checkly 7.12.1 → 7.13.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/ai-context/checkly.rules.md +11 -19
- package/dist/ai-context/skills-command/references/configure-agentic-checks.md +11 -19
- package/dist/constructs/agentic-check-codegen.d.ts +0 -12
- package/dist/constructs/agentic-check-codegen.js +7 -39
- package/dist/constructs/agentic-check-codegen.js.map +1 -1
- package/dist/constructs/agentic-check.d.ts +16 -72
- package/dist/constructs/agentic-check.js +11 -43
- package/dist/constructs/agentic-check.js.map +1 -1
- package/dist/tsconfig.tsbuildinfo +1 -1
- package/oclif.manifest.json +1 -1
- package/package.json +1 -1
|
@@ -152,15 +152,15 @@ Run `npx checkly skills manage plan` for the full reference.
|
|
|
152
152
|
- Agentic checks are AI-powered: instead of writing code, you describe what the check should do in natural language with the `prompt` property. The agent decides how to satisfy the prompt at runtime.
|
|
153
153
|
- Write prompts as concrete imperative steps, not vague goals. Tell the agent which URL to navigate to and what specific signals confirm success — for example, "Navigate to https://example.com/pricing and verify that at least three plan tiers are displayed", not "Check that pricing works".
|
|
154
154
|
- Keep prompts under 10000 characters. The construct will fail validation otherwise.
|
|
155
|
-
- **Frequency is
|
|
156
|
-
- **Locations are
|
|
155
|
+
- **Frequency is entitlement-gated by the backend.** You can configure the same `frequency` values as other checks, including `Frequency.EVERY_5M` / `5`. The backend enforces the fastest cadence available to the account.
|
|
156
|
+
- **Locations are entitlement-gated by the backend.** You can pass `locations` just like other public checks. The backend enforces the allowed number of regions for the account. `privateLocations` are not supported.
|
|
157
157
|
- **Several common check fields are intentionally omitted** from `AgenticCheckProps`: `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, `triggerIncident`, and `groupId`. The platform does not yet honor these for agentic checks. Setting them in the construct is a TypeScript error.
|
|
158
158
|
- **Important:** The target URL must be publicly accessible. Checks run on Checkly's cloud infrastructure, not locally. If the user is developing against localhost, suggest a tunneling tool (ngrok, cloudflare tunnel) or a preview/staging deployment.
|
|
159
159
|
- **Plan-gated:** Agentic checks require the `AGENTIC_CHECKS` entitlement on the account. Run `npx checkly skills manage` to check entitlements before using.
|
|
160
160
|
|
|
161
|
-
#### `agentRuntime`
|
|
161
|
+
#### `agentRuntime` skills
|
|
162
162
|
|
|
163
|
-
`agentRuntime` is
|
|
163
|
+
`agentRuntime` is used to add skills the agent may use at execution time.
|
|
164
164
|
|
|
165
165
|
```typescript
|
|
166
166
|
agentRuntime: {
|
|
@@ -173,23 +173,10 @@ agentRuntime: {
|
|
|
173
173
|
// - owner/repo form: 'addyosmani/web-quality-skills'
|
|
174
174
|
// - plain name: 'cost-optimization'
|
|
175
175
|
skills: ['addyosmani/web-quality-skills'],
|
|
176
|
-
|
|
177
|
-
// Environment variables the agent is allowed to read at runtime.
|
|
178
|
-
// Anything not listed here is hidden from the agent process — even
|
|
179
|
-
// if it's defined at the project or check level.
|
|
180
|
-
exposeEnvironmentVariables: [
|
|
181
|
-
// Bare string form: variable name only.
|
|
182
|
-
'ENVIRONMENT_URL',
|
|
183
|
-
// Object form: pair the variable with a description so the agent
|
|
184
|
-
// can decide when to read it. Descriptions are passed to the model
|
|
185
|
-
// and are truncated to 200 characters.
|
|
186
|
-
{ name: 'TEST_USER_EMAIL', description: 'Login email for the test account' },
|
|
187
|
-
],
|
|
188
176
|
},
|
|
189
177
|
```
|
|
190
178
|
|
|
191
|
-
-
|
|
192
|
-
- Descriptions are not just documentation — they steer the model's decisions. Use them to disambiguate variables that have non-obvious names.
|
|
179
|
+
- Reference Checkly environment variables directly in the prompt with double brackets, for example `{{ENVIRONMENT_URL}}`.
|
|
193
180
|
- The runner installs each skill via `npx skills add` at the start of every check run. The CLI does not validate the skill identifier at deploy time, so a typo will not surface until the first run.
|
|
194
181
|
- The `playwright-cli` skill is preloaded for every agentic check. Only declare additional skills here.
|
|
195
182
|
|
|
@@ -206,10 +193,15 @@ new AgenticCheck('example-agentic-check', {
|
|
|
206
193
|
name: 'Example Agentic Check',
|
|
207
194
|
prompt: 'Navigate to https://www.checklyhq.com/pricing and verify that at least three plan tiers are displayed on the page.',
|
|
208
195
|
activated: true,
|
|
196
|
+
locations: [
|
|
197
|
+
'us-east-1',
|
|
198
|
+
'eu-west-1',
|
|
199
|
+
'ap-southeast-1',
|
|
200
|
+
],
|
|
209
201
|
tags: [
|
|
210
202
|
'app:webshop',
|
|
211
203
|
],
|
|
212
|
-
frequency: Frequency.
|
|
204
|
+
frequency: Frequency.EVERY_5M,
|
|
213
205
|
alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(1, {
|
|
214
206
|
amount: 0,
|
|
215
207
|
interval: 5,
|
|
@@ -4,15 +4,15 @@
|
|
|
4
4
|
- Agentic checks are AI-powered: instead of writing code, you describe what the check should do in natural language with the `prompt` property. The agent decides how to satisfy the prompt at runtime.
|
|
5
5
|
- Write prompts as concrete imperative steps, not vague goals. Tell the agent which URL to navigate to and what specific signals confirm success — for example, "Navigate to https://example.com/pricing and verify that at least three plan tiers are displayed", not "Check that pricing works".
|
|
6
6
|
- Keep prompts under 10000 characters. The construct will fail validation otherwise.
|
|
7
|
-
- **Frequency is
|
|
8
|
-
- **Locations are
|
|
7
|
+
- **Frequency is entitlement-gated by the backend.** You can configure the same `frequency` values as other checks, including `Frequency.EVERY_5M` / `5`. The backend enforces the fastest cadence available to the account.
|
|
8
|
+
- **Locations are entitlement-gated by the backend.** You can pass `locations` just like other public checks. The backend enforces the allowed number of regions for the account. `privateLocations` are not supported.
|
|
9
9
|
- **Several common check fields are intentionally omitted** from `AgenticCheckProps`: `runParallel`, `retryStrategy`, `shouldFail`, `doubleCheck`, `triggerIncident`, and `groupId`. The platform does not yet honor these for agentic checks. Setting them in the construct is a TypeScript error.
|
|
10
10
|
- **Important:** The target URL must be publicly accessible. Checks run on Checkly's cloud infrastructure, not locally. If the user is developing against localhost, suggest a tunneling tool (ngrok, cloudflare tunnel) or a preview/staging deployment.
|
|
11
11
|
- **Plan-gated:** Agentic checks require the `AGENTIC_CHECKS` entitlement on the account. Run `npx checkly skills manage` to check entitlements before using.
|
|
12
12
|
|
|
13
|
-
## `agentRuntime`
|
|
13
|
+
## `agentRuntime` skills
|
|
14
14
|
|
|
15
|
-
`agentRuntime` is
|
|
15
|
+
`agentRuntime` is used to add skills the agent may use at execution time.
|
|
16
16
|
|
|
17
17
|
```typescript
|
|
18
18
|
agentRuntime: {
|
|
@@ -25,23 +25,10 @@ agentRuntime: {
|
|
|
25
25
|
// - owner/repo form: 'addyosmani/web-quality-skills'
|
|
26
26
|
// - plain name: 'cost-optimization'
|
|
27
27
|
skills: ['addyosmani/web-quality-skills'],
|
|
28
|
-
|
|
29
|
-
// Environment variables the agent is allowed to read at runtime.
|
|
30
|
-
// Anything not listed here is hidden from the agent process — even
|
|
31
|
-
// if it's defined at the project or check level.
|
|
32
|
-
exposeEnvironmentVariables: [
|
|
33
|
-
// Bare string form: variable name only.
|
|
34
|
-
'ENVIRONMENT_URL',
|
|
35
|
-
// Object form: pair the variable with a description so the agent
|
|
36
|
-
// can decide when to read it. Descriptions are passed to the model
|
|
37
|
-
// and are truncated to 200 characters.
|
|
38
|
-
{ name: 'TEST_USER_EMAIL', description: 'Login email for the test account' },
|
|
39
|
-
],
|
|
40
28
|
},
|
|
41
29
|
```
|
|
42
30
|
|
|
43
|
-
-
|
|
44
|
-
- Descriptions are not just documentation — they steer the model's decisions. Use them to disambiguate variables that have non-obvious names.
|
|
31
|
+
- Reference Checkly environment variables directly in the prompt with double brackets, for example `{{ENVIRONMENT_URL}}`.
|
|
45
32
|
- The runner installs each skill via `npx skills add` at the start of every check run. The CLI does not validate the skill identifier at deploy time, so a typo will not surface until the first run.
|
|
46
33
|
- The `playwright-cli` skill is preloaded for every agentic check. Only declare additional skills here.
|
|
47
34
|
|
|
@@ -58,10 +45,15 @@ new AgenticCheck('example-agentic-check', {
|
|
|
58
45
|
name: 'Example Agentic Check',
|
|
59
46
|
prompt: 'Navigate to https://www.checklyhq.com/pricing and verify that at least three plan tiers are displayed on the page.',
|
|
60
47
|
activated: true,
|
|
48
|
+
locations: [
|
|
49
|
+
'us-east-1',
|
|
50
|
+
'eu-west-1',
|
|
51
|
+
'ap-southeast-1',
|
|
52
|
+
],
|
|
61
53
|
tags: [
|
|
62
54
|
'app:webshop',
|
|
63
55
|
],
|
|
64
|
-
frequency: Frequency.
|
|
56
|
+
frequency: Frequency.EVERY_5M,
|
|
65
57
|
alertEscalationPolicy: AlertEscalationBuilder.runBasedEscalation(1, {
|
|
66
58
|
amount: 0,
|
|
67
59
|
interval: 5,
|
|
@@ -1,16 +1,5 @@
|
|
|
1
1
|
import { Codegen, Context } from './internal/codegen';
|
|
2
2
|
import { CheckResource } from './check-codegen';
|
|
3
|
-
/**
|
|
4
|
-
* Shape of a `selectedEnvironmentVariables` entry as stored on the backend
|
|
5
|
-
* inside `agenticCheckData`. The runner accepts two forms — a bare variable
|
|
6
|
-
* name, or an object with `key` and an optional `description`. The CLI
|
|
7
|
-
* construct uses `name` (not `key`), so the codegen translates object-form
|
|
8
|
-
* entries during emission.
|
|
9
|
-
*/
|
|
10
|
-
type StoredAgenticEnvironmentVariable = string | {
|
|
11
|
-
key: string;
|
|
12
|
-
description?: string;
|
|
13
|
-
};
|
|
14
3
|
/**
|
|
15
4
|
* Shape of `agenticCheckData` as stored on the backend and returned to the
|
|
16
5
|
* CLI during `checkly import`. Only fields the construct exposes are read.
|
|
@@ -20,7 +9,6 @@ type StoredAgenticEnvironmentVariable = string | {
|
|
|
20
9
|
*/
|
|
21
10
|
interface StoredAgenticCheckData {
|
|
22
11
|
skills?: string[] | null;
|
|
23
|
-
selectedEnvironmentVariables?: StoredAgenticEnvironmentVariable[] | null;
|
|
24
12
|
}
|
|
25
13
|
export interface AgenticCheckResource extends CheckResource {
|
|
26
14
|
checkType: 'AGENTIC';
|
|
@@ -16,31 +16,19 @@ class AgenticCheckCodegen extends codegen_1.Codegen {
|
|
|
16
16
|
});
|
|
17
17
|
const file = this.program.generatedConstructFile(filePath.fullPath);
|
|
18
18
|
file.namedImport(construct, 'checkly/constructs');
|
|
19
|
-
// `AgenticCheckProps` omits several fields that the platform does not
|
|
20
|
-
// yet honor (see `agentic-check.ts` for the full list and rationale).
|
|
21
|
-
// To keep the generated file type-checking against the construct, clear
|
|
22
|
-
// `locations` (which the construct hardcodes to a single value) and
|
|
23
|
-
// skip `retryStrategy` emission. The other omitted fields are already
|
|
24
|
-
// conditional in `buildCheckProps` and never populated for agentic
|
|
25
|
-
// checks today.
|
|
26
|
-
const sanitizedResource = {
|
|
27
|
-
...resource,
|
|
28
|
-
locations: undefined,
|
|
29
|
-
};
|
|
30
19
|
file.section((0, sourcegen_1.expr)((0, sourcegen_1.ident)(construct), builder => {
|
|
31
20
|
builder.new(builder => {
|
|
32
21
|
builder.string(logicalId);
|
|
33
22
|
builder.object(builder => {
|
|
34
23
|
builder.string('prompt', resource.prompt);
|
|
35
|
-
// Emit agentRuntime only when there's something meaningful to
|
|
36
|
-
//
|
|
37
|
-
//
|
|
38
|
-
// which is noise in the generated code.
|
|
24
|
+
// Emit agentRuntime only when there's something meaningful to carry.
|
|
25
|
+
// An imported check with no skills would otherwise produce an empty
|
|
26
|
+
// `agentRuntime: {}` block, which is noise in the generated code.
|
|
39
27
|
const agentRuntimeValue = buildAgentRuntimeObject(resource.agenticCheckData);
|
|
40
28
|
if (agentRuntimeValue !== undefined) {
|
|
41
29
|
builder.object('agentRuntime', agentRuntimeValue);
|
|
42
30
|
}
|
|
43
|
-
(0, check_codegen_1.buildCheckProps)(this.program, file, builder,
|
|
31
|
+
(0, check_codegen_1.buildCheckProps)(this.program, file, builder, resource, context, {
|
|
44
32
|
skipRetryStrategy: true,
|
|
45
33
|
});
|
|
46
34
|
});
|
|
@@ -51,17 +39,14 @@ class AgenticCheckCodegen extends codegen_1.Codegen {
|
|
|
51
39
|
exports.AgenticCheckCodegen = AgenticCheckCodegen;
|
|
52
40
|
/**
|
|
53
41
|
* Build an `agentRuntime: { ... }` object literal for the codegen output,
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
-
* `name`). Returns `undefined` when the input contains nothing worth
|
|
57
|
-
* emitting, so the caller can skip the property entirely.
|
|
42
|
+
* returning `undefined` when the input contains nothing worth emitting so the
|
|
43
|
+
* caller can skip the property entirely.
|
|
58
44
|
*/
|
|
59
45
|
function buildAgentRuntimeObject(data) {
|
|
60
46
|
if (!data)
|
|
61
47
|
return undefined;
|
|
62
48
|
const skills = (data.skills ?? []).filter(s => s.length > 0);
|
|
63
|
-
|
|
64
|
-
if (skills.length === 0 && storedEnvVars.length === 0) {
|
|
49
|
+
if (skills.length === 0) {
|
|
65
50
|
return undefined;
|
|
66
51
|
}
|
|
67
52
|
return builder => {
|
|
@@ -72,23 +57,6 @@ function buildAgentRuntimeObject(data) {
|
|
|
72
57
|
}
|
|
73
58
|
});
|
|
74
59
|
}
|
|
75
|
-
if (storedEnvVars.length > 0) {
|
|
76
|
-
builder.array('exposeEnvironmentVariables', arrayBuilder => {
|
|
77
|
-
for (const entry of storedEnvVars) {
|
|
78
|
-
if (typeof entry === 'string') {
|
|
79
|
-
arrayBuilder.string(entry);
|
|
80
|
-
}
|
|
81
|
-
else {
|
|
82
|
-
arrayBuilder.object(objectBuilder => {
|
|
83
|
-
objectBuilder.string('name', entry.key);
|
|
84
|
-
if (entry.description) {
|
|
85
|
-
objectBuilder.string('description', entry.description);
|
|
86
|
-
}
|
|
87
|
-
});
|
|
88
|
-
}
|
|
89
|
-
}
|
|
90
|
-
});
|
|
91
|
-
}
|
|
92
60
|
};
|
|
93
61
|
}
|
|
94
62
|
//# sourceMappingURL=agentic-check-codegen.js.map
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"agentic-check-codegen.js","sourceRoot":"","sources":["../../src/constructs/agentic-check-codegen.ts"],"names":[],"mappings":";;;AAAA,gDAAqD;AACrD,4CAA8D;AAC9D,mDAAgE;
|
|
1
|
+
{"version":3,"file":"agentic-check-codegen.js","sourceRoot":"","sources":["../../src/constructs/agentic-check-codegen.ts"],"names":[],"mappings":";;;AAAA,gDAAqD;AACrD,4CAA8D;AAC9D,mDAAgE;AAmBhE,MAAM,SAAS,GAAG,cAAc,CAAA;AAEhC,MAAa,mBAAoB,SAAQ,iBAA6B;IACpE,QAAQ,CAAE,QAA8B;QACtC,OAAO,kBAAkB,QAAQ,CAAC,IAAI,EAAE,CAAA;IAC1C,CAAC;IAED,OAAO,CAAE,SAAiB,EAAE,QAA8B,EAAE,OAAgB;QAC1E,MAAM,QAAQ,GAAG,OAAO,CAAC,QAAQ,CAAC,0BAA0B,EAAE,QAAQ,CAAC,IAAI,EAAE;YAC3E,IAAI,EAAE,QAAQ,CAAC,IAAI;YACnB,MAAM,EAAE,IAAI;SACb,CAAC,CAAA;QAEF,MAAM,IAAI,GAAG,IAAI,CAAC,OAAO,CAAC,sBAAsB,CAAC,QAAQ,CAAC,QAAQ,CAAC,CAAA;QAEnE,IAAI,CAAC,WAAW,CAAC,SAAS,EAAE,oBAAoB,CAAC,CAAA;QAEjD,IAAI,CAAC,OAAO,CAAC,IAAA,gBAAI,EAAC,IAAA,iBAAK,EAAC,SAAS,CAAC,EAAE,OAAO,CAAC,EAAE;YAC5C,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,EAAE;gBACpB,OAAO,CAAC,MAAM,CAAC,SAAS,CAAC,CAAA;gBACzB,OAAO,CAAC,MAAM,CAAC,OAAO,CAAC,EAAE;oBACvB,OAAO,CAAC,MAAM,CAAC,QAAQ,EAAE,QAAQ,CAAC,MAAM,CAAC,CAAA;oBAEzC,qEAAqE;oBACrE,oEAAoE;oBACpE,kEAAkE;oBAClE,MAAM,iBAAiB,GAAG,uBAAuB,CAAC,QAAQ,CAAC,gBAAgB,CAAC,CAAA;oBAC5E,IAAI,iBAAiB,KAAK,SAAS,EAAE,CAAC;wBACpC,OAAO,CAAC,MAAM,CAAC,cAAc,EAAE,iBAAiB,CAAC,CAAA;oBACnD,CAAC;oBAED,IAAA,+BAAe,EAAC,IAAI,CAAC,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,QAAQ,EAAE,OAAO,EAAE;wBAC9D,iBAAiB,EAAE,IAAI;qBACxB,CAAC,CAAA;gBACJ,CAAC,CAAC,CAAA;YACJ,CAAC,CAAC,CAAA;QACJ,CAAC,CAAC,CAAC,CAAA;IACL,CAAC;CACF;AApCD,kDAoCC;AAED;;;;GAIG;AACH,SAAS,uBAAuB,CAC9B,IAA+C;IAE/C,IAAI,CAAC,IAAI;QAAE,OAAO,SAAS,CAAA;IAE3B,MAAM,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,IAAI,EAAE,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,MAAM,GAAG,CAAC,CAAC,CAAA;IAE5D,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,SAAS,CAAA;IAClB,CAAC;IAED,OAAO,OAAO,CAAC,EAAE;QACf,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACtB,OAAO,CAAC,KAAK,CAAC,QAAQ,EAAE,YAAY,CAAC,EAAE;gBACrC,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;oBAC3B,YAAY,CAAC,MAAM,CAAC,KAAK,CAAC,CAAA;gBAC5B,CAAC;YACH,CAAC,CAAC,CAAA;QACJ,CAAC;IACH,CAAC,CAAA;AACH,CAAC"}
|
|
@@ -2,43 +2,15 @@ import { Check, CheckProps } from './check';
|
|
|
2
2
|
import { Frequency } from './frequency';
|
|
3
3
|
import { Diagnostics } from './diagnostics';
|
|
4
4
|
/**
|
|
5
|
-
* Frequencies
|
|
6
|
-
*
|
|
7
|
-
* (`EVERY_30M`, `EVERY_1H`, `EVERY_2H`, `EVERY_3H`, `EVERY_6H`, `EVERY_12H`,
|
|
8
|
-
* `EVERY_24H`) are also accepted.
|
|
5
|
+
* Frequencies for agentic checks are accepted locally and enforced by the
|
|
6
|
+
* backend according to the account's entitlements.
|
|
9
7
|
*/
|
|
10
|
-
export type AgenticCheckFrequency =
|
|
11
|
-
/**
|
|
12
|
-
* An environment variable the agent is permitted to read at runtime.
|
|
13
|
-
*
|
|
14
|
-
* Use the bare string form when the variable name is self-explanatory, or
|
|
15
|
-
* the object form to provide a `description` that helps the agent understand
|
|
16
|
-
* what the variable is for. Descriptions are passed to the model so it can
|
|
17
|
-
* make better decisions about when to read the variable.
|
|
18
|
-
*
|
|
19
|
-
* @example
|
|
20
|
-
* ```typescript
|
|
21
|
-
* 'API_KEY'
|
|
22
|
-
* { name: 'TOKEN_42', description: 'Feature flag service auth token' }
|
|
23
|
-
* ```
|
|
24
|
-
*/
|
|
25
|
-
export type AgentRuntimeEnvironmentVariable = string | {
|
|
26
|
-
/** The environment variable name. */
|
|
27
|
-
name: string;
|
|
28
|
-
/**
|
|
29
|
-
* Optional human-readable explanation of what the variable is for.
|
|
30
|
-
* Passed to the agent so it can decide when to read the variable.
|
|
31
|
-
* Truncated to {@link MAX_ENV_VAR_DESCRIPTION_LENGTH} characters.
|
|
32
|
-
*/
|
|
33
|
-
description?: string;
|
|
34
|
-
};
|
|
8
|
+
export type AgenticCheckFrequency = number | Frequency;
|
|
35
9
|
/**
|
|
36
10
|
* Configures the runtime context the agent has access to during a check.
|
|
37
11
|
*
|
|
38
|
-
* `agentRuntime`
|
|
39
|
-
*
|
|
40
|
-
* Treat it as a security boundary: the smaller the runtime surface, the
|
|
41
|
-
* smaller the blast radius of any prompt injection.
|
|
12
|
+
* `agentRuntime` lets you add extra skills on top of the defaults the runner
|
|
13
|
+
* provides automatically.
|
|
42
14
|
*/
|
|
43
15
|
export interface AgentRuntime {
|
|
44
16
|
/**
|
|
@@ -57,65 +29,38 @@ export interface AgentRuntime {
|
|
|
57
29
|
* @example ['addyosmani/web-quality-skills']
|
|
58
30
|
*/
|
|
59
31
|
skills?: string[];
|
|
60
|
-
/**
|
|
61
|
-
* Environment variables the agent is permitted to read at runtime.
|
|
62
|
-
*
|
|
63
|
-
* **Variables not listed here are not exposed to the agent**, even if
|
|
64
|
-
* they exist in the Checkly account. This is the primary defense against
|
|
65
|
-
* prompt injection: an attacker who controls content the agent reads
|
|
66
|
-
* cannot exfiltrate secrets the agent never had access to.
|
|
67
|
-
*
|
|
68
|
-
* Each entry is either a bare variable name, or an object with a
|
|
69
|
-
* `name` and an optional `description`. Descriptions help the agent
|
|
70
|
-
* understand what each variable is for.
|
|
71
|
-
*
|
|
72
|
-
* @example
|
|
73
|
-
* ```typescript
|
|
74
|
-
* exposeEnvironmentVariables: [
|
|
75
|
-
* 'API_KEY',
|
|
76
|
-
* { name: 'TEST_USER_PASSWORD', description: 'Login password for the test account' },
|
|
77
|
-
* ]
|
|
78
|
-
* ```
|
|
79
|
-
*/
|
|
80
|
-
exposeEnvironmentVariables?: AgentRuntimeEnvironmentVariable[];
|
|
81
32
|
}
|
|
82
33
|
/**
|
|
83
34
|
* Configuration properties for {@link AgenticCheck}.
|
|
84
35
|
*
|
|
85
36
|
* Agentic checks intentionally expose only the subset of options that the
|
|
86
37
|
* Checkly platform currently supports for them. Properties such as
|
|
87
|
-
* `
|
|
88
|
-
* `
|
|
89
|
-
*
|
|
90
|
-
*
|
|
38
|
+
* `privateLocations`, `runParallel`, `retryStrategy`, `shouldFail`,
|
|
39
|
+
* `doubleCheck`, `triggerIncident` and `groupId` are omitted because the
|
|
40
|
+
* platform does not yet honor them for agentic checks. They will be added back
|
|
41
|
+
* as additive, non-breaking changes once support lands.
|
|
91
42
|
*/
|
|
92
|
-
export interface AgenticCheckProps extends Omit<CheckProps, '
|
|
43
|
+
export interface AgenticCheckProps extends Omit<CheckProps, 'privateLocations' | 'runParallel' | 'retryStrategy' | 'shouldFail' | 'doubleCheck' | 'triggerIncident' | 'groupId' | 'frequency'> {
|
|
93
44
|
/**
|
|
94
45
|
* The prompt that defines what the agentic check should verify.
|
|
95
46
|
* Maximum 10,000 characters.
|
|
96
47
|
*/
|
|
97
48
|
prompt: string;
|
|
98
49
|
/**
|
|
99
|
-
* How often the check should run.
|
|
100
|
-
*
|
|
50
|
+
* How often the check should run. The backend enforces the fastest allowed
|
|
51
|
+
* cadence according to the account's entitlements. Defaults to
|
|
52
|
+
* {@link Frequency.EVERY_30M}.
|
|
101
53
|
*
|
|
102
54
|
* @example
|
|
103
55
|
* ```typescript
|
|
104
|
-
* frequency: Frequency.
|
|
56
|
+
* frequency: Frequency.EVERY_5M
|
|
105
57
|
* // or equivalently
|
|
106
|
-
* frequency:
|
|
58
|
+
* frequency: 5
|
|
107
59
|
* ```
|
|
108
60
|
*/
|
|
109
61
|
frequency?: AgenticCheckFrequency;
|
|
110
62
|
/**
|
|
111
|
-
* Configures
|
|
112
|
-
* which skills it can use, which environment variables it can read, and
|
|
113
|
-
* (in the future) other access surfaces such as network policies or tool
|
|
114
|
-
* allowlists.
|
|
115
|
-
*
|
|
116
|
-
* Treat `agentRuntime` as a security boundary. Anything not declared here
|
|
117
|
-
* is unavailable to the agent at runtime, which keeps the blast radius of
|
|
118
|
-
* any prompt injection as small as possible.
|
|
63
|
+
* Configures additional skills the agent can use during execution.
|
|
119
64
|
*/
|
|
120
65
|
agentRuntime?: AgentRuntime;
|
|
121
66
|
}
|
|
@@ -157,7 +102,6 @@ export declare class AgenticCheck extends Check {
|
|
|
157
102
|
prompt: string;
|
|
158
103
|
agentRuntime: {
|
|
159
104
|
skills: string[];
|
|
160
|
-
exposeEnvironmentVariables: AgentRuntimeEnvironmentVariable[];
|
|
161
105
|
};
|
|
162
106
|
activated: boolean | undefined;
|
|
163
107
|
muted: boolean | undefined;
|
|
@@ -6,21 +6,10 @@ const project_1 = require("./project");
|
|
|
6
6
|
const constants_1 = require("../constants");
|
|
7
7
|
const construct_diagnostics_1 = require("./construct-diagnostics");
|
|
8
8
|
/**
|
|
9
|
-
*
|
|
10
|
-
*
|
|
9
|
+
* Backwards-compatible default for checks that do not set a location and do
|
|
10
|
+
* not inherit one from the project config.
|
|
11
11
|
*/
|
|
12
|
-
const
|
|
13
|
-
/**
|
|
14
|
-
* The single location agentic checks currently run from. Until the platform
|
|
15
|
-
* supports running agentic checks from multiple locations, this value is
|
|
16
|
-
* forced server-side and the construct does not let users override it.
|
|
17
|
-
*/
|
|
18
|
-
const AGENTIC_CHECK_LOCATION = 'us-east-1';
|
|
19
|
-
/**
|
|
20
|
-
* Maximum length of an environment variable description, in characters.
|
|
21
|
-
* Matches the truncation length applied by the agentic runner.
|
|
22
|
-
*/
|
|
23
|
-
const MAX_ENV_VAR_DESCRIPTION_LENGTH = 200;
|
|
12
|
+
const DEFAULT_AGENTIC_CHECK_LOCATION = 'us-east-1';
|
|
24
13
|
/**
|
|
25
14
|
* Creates an Agentic Check that uses AI to monitor websites and applications.
|
|
26
15
|
*
|
|
@@ -54,12 +43,13 @@ class AgenticCheck extends check_1.Check {
|
|
|
54
43
|
super(logicalId, props);
|
|
55
44
|
this.prompt = props.prompt;
|
|
56
45
|
this.agentRuntime = props.agentRuntime;
|
|
46
|
+
// Preserve the old implicit single-region behavior for checks that do not
|
|
47
|
+
// set locations directly and do not inherit project-level locations.
|
|
48
|
+
this.locations ??= [DEFAULT_AGENTIC_CHECK_LOCATION];
|
|
57
49
|
// Defensive overrides: even though these props are omitted from the type,
|
|
58
50
|
// `Check.applyConfigDefaults()` may pull them in from the project-level
|
|
59
|
-
// `checks` config defaults. Force them to
|
|
60
|
-
//
|
|
61
|
-
// it doesn't.
|
|
62
|
-
this.locations = [AGENTIC_CHECK_LOCATION];
|
|
51
|
+
// `checks` config defaults. Force them to values the platform currently
|
|
52
|
+
// honors so the construct never claims to support something it doesn't.
|
|
63
53
|
this.privateLocations = undefined;
|
|
64
54
|
this.runParallel = false;
|
|
65
55
|
this.retryStrategy = undefined;
|
|
@@ -84,11 +74,6 @@ class AgenticCheck extends check_1.Check {
|
|
|
84
74
|
else if (this.prompt.length > 10000) {
|
|
85
75
|
diagnostics.add(new construct_diagnostics_1.InvalidPropertyValueDiagnostic('prompt', new Error(`"prompt" must be at most 10000 characters, got ${this.prompt.length}.`)));
|
|
86
76
|
}
|
|
87
|
-
if (this.frequency !== undefined
|
|
88
|
-
&& !ALLOWED_AGENTIC_FREQUENCIES.includes(this.frequency)) {
|
|
89
|
-
diagnostics.add(new construct_diagnostics_1.InvalidPropertyValueDiagnostic('frequency', new Error(`"frequency" must be one of ${ALLOWED_AGENTIC_FREQUENCIES.join(', ')} `
|
|
90
|
-
+ `for agentic checks, got ${this.frequency}.`)));
|
|
91
|
-
}
|
|
92
77
|
this.validateAgentRuntime(diagnostics);
|
|
93
78
|
}
|
|
94
79
|
// eslint-disable-next-line require-await
|
|
@@ -100,34 +85,17 @@ class AgenticCheck extends check_1.Check {
|
|
|
100
85
|
}
|
|
101
86
|
}
|
|
102
87
|
}
|
|
103
|
-
if (this.agentRuntime?.exposeEnvironmentVariables) {
|
|
104
|
-
for (const [index, entry] of this.agentRuntime.exposeEnvironmentVariables.entries()) {
|
|
105
|
-
const name = typeof entry === 'string' ? entry : entry?.name;
|
|
106
|
-
if (typeof name !== 'string' || name.trim().length === 0) {
|
|
107
|
-
diagnostics.add(new construct_diagnostics_1.InvalidPropertyValueDiagnostic('agentRuntime.exposeEnvironmentVariables', new Error(`"agentRuntime.exposeEnvironmentVariables[${index}]" must have a non-empty name.`)));
|
|
108
|
-
continue;
|
|
109
|
-
}
|
|
110
|
-
if (typeof entry !== 'string'
|
|
111
|
-
&& typeof entry.description === 'string'
|
|
112
|
-
&& entry.description.length > MAX_ENV_VAR_DESCRIPTION_LENGTH) {
|
|
113
|
-
diagnostics.add(new construct_diagnostics_1.InvalidPropertyValueDiagnostic('agentRuntime.exposeEnvironmentVariables', new Error(`"agentRuntime.exposeEnvironmentVariables[${index}].description" must be at most `
|
|
114
|
-
+ `${MAX_ENV_VAR_DESCRIPTION_LENGTH} characters, got ${entry.description.length}.`)));
|
|
115
|
-
}
|
|
116
|
-
}
|
|
117
|
-
}
|
|
118
88
|
}
|
|
119
89
|
synthesize() {
|
|
120
90
|
return {
|
|
121
91
|
...super.synthesize(),
|
|
122
92
|
checkType: constants_1.CheckTypes.AGENTIC,
|
|
123
93
|
prompt: this.prompt,
|
|
124
|
-
// Always emit `agentRuntime` so the backend has an explicit
|
|
125
|
-
//
|
|
126
|
-
//
|
|
127
|
-
// them", not "preserve whatever was there before".
|
|
94
|
+
// Always emit `agentRuntime` so the backend has an explicit picture of
|
|
95
|
+
// the skills the user wants. Omitted skills mean "the agent should not
|
|
96
|
+
// have them", not "preserve whatever was there before".
|
|
128
97
|
agentRuntime: {
|
|
129
98
|
skills: this.agentRuntime?.skills ?? [],
|
|
130
|
-
exposeEnvironmentVariables: this.agentRuntime?.exposeEnvironmentVariables ?? [],
|
|
131
99
|
},
|
|
132
100
|
};
|
|
133
101
|
}
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"agentic-check.js","sourceRoot":"","sources":["../../src/constructs/agentic-check.ts"],"names":[],"mappings":";;;AAAA,mCAA2C;AAE3C,uCAAmC;AACnC,4CAAyC;AAEzC,mEAAwE;
|
|
1
|
+
{"version":3,"file":"agentic-check.js","sourceRoot":"","sources":["../../src/constructs/agentic-check.ts"],"names":[],"mappings":";;;AAAA,mCAA2C;AAE3C,uCAAmC;AACnC,4CAAyC;AAEzC,mEAAwE;AAQxE;;;GAGG;AACH,MAAM,8BAA8B,GAAG,WAAW,CAAA;AAyElD;;;;;;;;;;;;;;;;;;;GAmBG;AACH,MAAa,YAAa,SAAQ,aAAK;IAC5B,MAAM,CAAQ;IACd,YAAY,CAAe;IAEpC;;;;;OAKG;IACH,YAAa,SAAiB,EAAE,KAAwB;QACtD,KAAK,CAAC,SAAS,EAAE,KAAK,CAAC,CAAA;QACvB,IAAI,CAAC,MAAM,GAAG,KAAK,CAAC,MAAM,CAAA;QAC1B,IAAI,CAAC,YAAY,GAAG,KAAK,CAAC,YAAY,CAAA;QAEtC,0EAA0E;QAC1E,qEAAqE;QACrE,IAAI,CAAC,SAAS,KAAK,CAAC,8BAA8B,CAAC,CAAA;QAEnD,0EAA0E;QAC1E,wEAAwE;QACxE,wEAAwE;QACxE,wEAAwE;QACxE,IAAI,CAAC,gBAAgB,GAAG,SAAS,CAAA;QACjC,IAAI,CAAC,WAAW,GAAG,KAAK,CAAA;QACxB,IAAI,CAAC,aAAa,GAAG,SAAS,CAAA;QAC9B,IAAI,CAAC,UAAU,GAAG,SAAS,CAAA;QAC3B,IAAI,CAAC,WAAW,GAAG,SAAS,CAAA;QAC5B,IAAI,CAAC,eAAe,GAAG,SAAS,CAAA;QAEhC,4DAA4D;QAC5D,IAAI,IAAI,CAAC,SAAS,KAAK,SAAS,EAAE,CAAC;YACjC,IAAI,CAAC,SAAS,GAAG,EAAE,CAAA;QACrB,CAAC;QAED,iBAAO,CAAC,iBAAiB,CAAC,IAAI,CAAC,CAAA;QAC/B,IAAI,CAAC,gBAAgB,EAAE,CAAA;IACzB,CAAC;IAED,QAAQ;QACN,OAAO,gBAAgB,IAAI,CAAC,SAAS,EAAE,CAAA;IACzC,CAAC;IAED,KAAK,CAAC,QAAQ,CAAE,WAAwB;QACtC,MAAM,KAAK,CAAC,QAAQ,CAAC,WAAW,CAAC,CAAA;QAEjC,IAAI,CAAC,IAAI,CAAC,MAAM,IAAI,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YACpD,WAAW,CAAC,GAAG,CAAC,IAAI,sDAA8B,CAChD,QAAQ,EACR,IAAI,KAAK,CAAC,6CAA6C,CAAC,CACzD,CAAC,CAAA;QACJ,CAAC;aAAM,IAAI,IAAI,CAAC,MAAM,CAAC,MAAM,GAAG,KAAM,EAAE,CAAC;YACvC,WAAW,CAAC,GAAG,CAAC,IAAI,sDAA8B,CAChD,QAAQ,EACR,IAAI,KAAK,CAAC,kDAAkD,IAAI,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,CACnF,CAAC,CAAA;QACJ,CAAC;QAED,IAAI,CAAC,oBAAoB,CAAC,WAAW,CAAC,CAAA;IACxC,CAAC;IAED,yCAAyC;IAC/B,KAAK,CAAC,oBAAoB,CAAE,WAAwB;QAC5D,IAAI,IAAI,CAAC,YAAY,EAAE,MAAM,EAAE,CAAC;YAC9B,KAAK,MAAM,CAAC,KAAK,EAAE,KAAK,CAAC,IAAI,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,OAAO,EAAE,EAAE,CAAC;gBAChE,IAAI,OAAO,KAAK,KAAK,QAAQ,IAAI,KAAK,CAAC,IAAI,EAAE,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;oBAC3D,WAAW,CAAC,GAAG,CAAC,IAAI,sDAA8B,CAChD,qBAAqB,EACrB,IAAI,KAAK,CACP,wBAAwB,KAAK,gCAAgC,CAC9D,CACF,CAAC,CAAA;gBACJ,CAAC;YACH,CAAC;QACH,CAAC;IACH,CAAC;IAED,UAAU;QACR,OAAO;YACL,GAAG,KAAK,CAAC,UAAU,EAAE;YACrB,SAAS,EAAE,sBAAU,CAAC,OAAO;YAC7B,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,uEAAuE;YACvE,uEAAuE;YACvE,wDAAwD;YACxD,YAAY,EAAE;gBACZ,MAAM,EAAE,IAAI,CAAC,YAAY,EAAE,MAAM,IAAI,EAAE;aACxC;SACF,CAAA;IACH,CAAC;CACF;AA1FD,oCA0FC"}
|