@fjall/components-infrastructure 19.0.0 → 21.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +50 -0
- package/dist/lib/patterns/aws/clickhouseDatabase.d.ts +23 -3
- package/dist/lib/patterns/aws/clickhouseDatabase.js +116 -26
- package/dist/lib/patterns/aws/computeEcs.d.ts +14 -16
- package/dist/lib/patterns/aws/computeEcs.js +30 -74
- package/dist/lib/patterns/aws/interfaces/database.d.ts +14 -13
- package/dist/lib/resources/aws/compute/ecs.js +2 -1
- package/dist/lib/resources/aws/compute/ecsTypes.d.ts +5 -3
- package/dist/lib/resources/aws/database/clickhouseBackupScript.d.ts +12 -0
- package/dist/lib/resources/aws/database/clickhouseBackupScript.js +34 -14
- package/dist/lib/resources/aws/database/clickhouseConstants.d.ts +30 -2
- package/dist/lib/resources/aws/database/clickhouseConstants.js +30 -2
- package/dist/lib/resources/aws/database/clickhouseMaintenanceScript.d.ts +69 -0
- package/dist/lib/resources/aws/database/clickhouseMaintenanceScript.js +131 -0
- package/dist/lib/resources/aws/database/rdsAurora.js +3 -0
- package/dist/lib/resources/aws/database/rdsInstance.js +4 -0
- package/dist/lib/resources/aws/monitoring/clickhouseAlarms.d.ts +12 -0
- package/dist/lib/resources/aws/monitoring/clickhouseAlarms.js +79 -3
- package/dist/lib/resources/aws/monitoring/ecsAlarms.d.ts +10 -1
- package/dist/lib/resources/aws/monitoring/ecsAlarms.js +14 -4
- package/dist/lib/resources/aws/networking/vpc.js +6 -2
- package/dist/lib/resources/aws/storage/s3.js +13 -3
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -2,6 +2,56 @@
|
|
|
2
2
|
|
|
3
3
|
Published fjall infrastructure components and utilities.
|
|
4
4
|
|
|
5
|
+
## Upgrade notes
|
|
6
|
+
|
|
7
|
+
### 21.0.0 — ClickHouse identities by workload class (breaking)
|
|
8
|
+
|
|
9
|
+
`ClickHouseDatabase` no longer shares one `schemaAdmin` principal across the
|
|
10
|
+
migration task, the schema gate and the maintenance sidecars. Each workload
|
|
11
|
+
class now authenticates as its own identity, so no two classes contend for
|
|
12
|
+
the same `max_concurrent_queries_for_user` cap:
|
|
13
|
+
|
|
14
|
+
| identity | holder |
|
|
15
|
+
| ------------------------- | ----------------------------- |
|
|
16
|
+
| `schemaAdmin.name` (XML) | migration task ONLY |
|
|
17
|
+
| `fjall_schema_gate` (SQL) | materialised gate container |
|
|
18
|
+
| `fjall_maintenance` (SQL) | OPTIMIZE FINAL + BACKUP tasks |
|
|
19
|
+
|
|
20
|
+
What changes in a synthesised stack:
|
|
21
|
+
|
|
22
|
+
- **App containers lose `SCHEMA_ADMIN_USER` / `SCHEMA_ADMIN_PASSWORD`.** They
|
|
23
|
+
keep `EXPECTED_CH_SCHEMA_VERSION`. A container image that runs its own
|
|
24
|
+
in-process ClickHouse schema check reading those two variables must drop
|
|
25
|
+
it — the materialised gate container performs the check.
|
|
26
|
+
- **The gate container authenticates as `fjall_schema_gate`** via
|
|
27
|
+
`FJALL_SCHEMA_GATE_CH_USER` / `FJALL_SCHEMA_GATE_CH_PASSWORD` (read-only,
|
|
28
|
+
`SELECT` on `<db>._schema_migrations` only).
|
|
29
|
+
- **The OPTIMIZE and BACKUP scheduled tasks authenticate as
|
|
30
|
+
`fjall_maintenance`.** Their execution roles read that identity's secret,
|
|
31
|
+
not the schema admin's. Both scripts open with a migrate ↔ maintenance
|
|
32
|
+
activity pre-flight and log `fjall:maintenance:status=<ok|deferred|failed>`
|
|
33
|
+
markers; `status=failed` pages via a new `ClickHouseMaintenanceFailedAlarm`.
|
|
34
|
+
- **Two new fixed-name Secrets Manager secrets** —
|
|
35
|
+
`fjall/clickhouse/fjall-schema-gate-password` and
|
|
36
|
+
`fjall/clickhouse/fjall-maintenance-password` — are minted by the construct.
|
|
37
|
+
They are NOT injected into the ClickHouse service container, so the
|
|
38
|
+
ClickHouse service task definition is unchanged by the upgrade.
|
|
39
|
+
- **The `fjall_` user-name prefix is reserved.** A `schemaAdmin.name` or
|
|
40
|
+
`managedPasswords` entry starting with `fjall_` fails synth.
|
|
41
|
+
|
|
42
|
+
Deploy order: the identities are SQL-provisioned by the migration runner
|
|
43
|
+
(`@fjall/clickhouse` 21 — bump the migration image before this construct),
|
|
44
|
+
so the migration task must run once under 21.x BEFORE anything authenticates
|
|
45
|
+
as the new identities. Deploy the Database stack, then the Compute stack that
|
|
46
|
+
carries the migration task, and only then let the schedules fire.
|
|
47
|
+
|
|
48
|
+
Rollback: rolling back to 20.x deletes the two framework secrets, which
|
|
49
|
+
Secrets Manager only schedules (30-day recovery window). Re-adopting 21.x
|
|
50
|
+
inside that window fails on the held names — first run
|
|
51
|
+
`aws secretsmanager restore-secret --secret-id fjall/clickhouse/<name>-password`
|
|
52
|
+
or `aws secretsmanager delete-secret --secret-id … --force-delete-without-recovery`
|
|
53
|
+
for each of the two secrets.
|
|
54
|
+
|
|
5
55
|
## Useful commands
|
|
6
56
|
|
|
7
57
|
- `npm run build` compile typescript to js
|
|
@@ -192,8 +192,12 @@ export interface ClickHouseDatabaseProps {
|
|
|
192
192
|
* `<access_management>1</access_management>` so it can manage GRANT /
|
|
193
193
|
* CREATE USER from SQL. `profile` MUST name a key in `profiles:` (or
|
|
194
194
|
* `ClickHouseDefaultProfiles`) and is bound directly via the XML config.
|
|
195
|
-
*
|
|
196
|
-
*
|
|
195
|
+
* The migration task is the ONLY holder: the scheduled tasks (OPTIMIZE
|
|
196
|
+
* FINAL, BACKUP DATABASE) authenticate as the framework's
|
|
197
|
+
* `fjall_maintenance` identity and the materialised schema gate as
|
|
198
|
+
* `fjall_schema_gate`, both SQL-provisioned by the migration runner from
|
|
199
|
+
* `getMigrationContributions()`. The name must not start with the
|
|
200
|
+
* reserved `fjall_` prefix.
|
|
197
201
|
*/
|
|
198
202
|
schemaAdmin: ClickHouseSchemaAdmin;
|
|
199
203
|
/**
|
|
@@ -241,7 +245,9 @@ export interface ClickHouseDatabaseProps {
|
|
|
241
245
|
/**
|
|
242
246
|
* Ops alarm SNS topic for the full ClickHouse alarm surface: host-posture
|
|
243
247
|
* alarms (CPU / disk warn+critical), the ECS-layer service alarms (cgroup
|
|
244
|
-
* memory, `RunningTaskCount` liveness
|
|
248
|
+
* memory, `RunningTaskCount` liveness — which enables Container Insights on
|
|
249
|
+
* the inner cluster, ~$0.30/metric-month — and the task-stop watchdog), the
|
|
250
|
+
* server-log
|
|
245
251
|
* signal alarms (errors, failed queries, backpressure, cold-tier S3), and —
|
|
246
252
|
* when `backupSchedule` is enabled — the backup-failure alarm plus the
|
|
247
253
|
* backup success heartbeat (absence + arming-gate alarms and the paging
|
|
@@ -376,8 +382,22 @@ export declare class ClickHouseDatabase extends Construct implements IClickHouse
|
|
|
376
382
|
* `process.env`. The helper reads the env vars directly — no runtime
|
|
377
383
|
* `secretsmanager:GetSecretValue` SDK call is needed; ECS executionRole
|
|
378
384
|
* carries the IAM grant via the standard `secretsImport` framework path.
|
|
385
|
+
*
|
|
386
|
+
* **Framework identities.** `FJALL_CLICKHOUSE_USERS` (the gate and
|
|
387
|
+
* maintenance identities) ship the same way under their own manifest,
|
|
388
|
+
* `CLICKHOUSE_FRAMEWORK_USERS`, each with a `USER_<NAME>_PASSWORD` import,
|
|
389
|
+
* plus `CLICKHOUSE_MAINTENANCE_PROFILE` naming the schema admin's settings
|
|
390
|
+
* profile so the runner can bind `fjall_maintenance` to the same
|
|
391
|
+
* instance-scaled caps without knowing any numbers.
|
|
379
392
|
*/
|
|
380
393
|
getMigrationContributions(): MigrationContributions;
|
|
394
|
+
/**
|
|
395
|
+
* The materialised gate's ClickHouse credentials: the gate authenticates as
|
|
396
|
+
* the read-only `fjall_schema_gate` identity, never as the schema admin.
|
|
397
|
+
* Env `FJALL_SCHEMA_GATE_CH_USER` carries the fixed name; secret
|
|
398
|
+
* `FJALL_SCHEMA_GATE_CH_PASSWORD` imports that identity's password.
|
|
399
|
+
* `undefined` when no `migrations:` config is declared — nothing to gate.
|
|
400
|
+
*/
|
|
381
401
|
getSchemaGateContribution(): {
|
|
382
402
|
readonly environment: Record<string, string>;
|
|
383
403
|
readonly secretsImport: Record<string, SecretImport>;
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { CLICKHOUSE_MANAGED_USERS_ENV, pickLatestClickHouseMigration, SCHEMA_ADMIN_PASSWORD_ENV, SCHEMA_ADMIN_USER_ENV } from "@fjall/util/migration";
|
|
1
|
+
import { CLICKHOUSE_FRAMEWORK_USERS_ENV, CLICKHOUSE_MAINTENANCE_PROFILE_ENV, CLICKHOUSE_MANAGED_USERS_ENV, FJALL_CLICKHOUSE_USER_PREFIX, FJALL_CLICKHOUSE_USERS, FJALL_MAINTENANCE_USER, FJALL_SCHEMA_GATE_USER, isReservedClickHouseUserName, pickLatestClickHouseMigration, SCHEMA_ADMIN_PASSWORD_ENV, SCHEMA_ADMIN_USER_ENV, SCHEMA_GATE_CH_PASSWORD_ENV, SCHEMA_GATE_CH_USER_ENV } from "@fjall/util/migration";
|
|
2
2
|
import { Annotations, CfnOutput, Duration, Fn, Stack } from "aws-cdk-lib";
|
|
3
3
|
import { Connections, Port, UserData } from "aws-cdk-lib/aws-ec2";
|
|
4
4
|
import { Monitoring } from "aws-cdk-lib/aws-autoscaling";
|
|
@@ -16,6 +16,7 @@ import { S3Bucket } from "../../resources/aws/storage/s3.js";
|
|
|
16
16
|
import { Secret } from "../../resources/aws/secrets/secret.js";
|
|
17
17
|
import { LogGroup } from "../../resources/aws/logging/logGroup.js";
|
|
18
18
|
import { createClickHouseSecurityGroup } from "../../resources/aws/database/clickhouseSecurityGroup.js";
|
|
19
|
+
import { buildMaintenancePreamble, maintenanceOkMarker, maintenanceRun } from "../../resources/aws/database/clickhouseMaintenanceScript.js";
|
|
19
20
|
import { buildClickHouseEntrypointWrapper, buildClickHouseUserData, generateUsersConfigXml } from "../../resources/aws/database/clickhouseUserData.js";
|
|
20
21
|
import { toPascalCase } from "../../utils/capitaliseString.js";
|
|
21
22
|
import { resolveAlertsTopic } from "../../utils/resolveAlertsTopic.js";
|
|
@@ -154,6 +155,21 @@ function expectDefined(value, invariant) {
|
|
|
154
155
|
}
|
|
155
156
|
return value;
|
|
156
157
|
}
|
|
158
|
+
/**
|
|
159
|
+
* Mints the password secret for one ClickHouse user — customer-declared
|
|
160
|
+
* (`schemaAdmin`, `managedPasswords`) and framework-owned
|
|
161
|
+
* (`FJALL_CLICKHOUSE_USERS`) alike, so every identity has the same removal
|
|
162
|
+
* policy and the same fixed `fjall/clickhouse/<name>-password` name.
|
|
163
|
+
*
|
|
164
|
+
* Fixed names carry a rollback cure. A rollback to a constructs version
|
|
165
|
+
* without the framework identities deletes their secrets, which Secrets
|
|
166
|
+
* Manager only SCHEDULES (30-day recovery window); re-adopting the identities
|
|
167
|
+
* inside that window fails with `ResourceExistsException` because the name
|
|
168
|
+
* is still held by the scheduled-for-deletion secret. Before re-deploying,
|
|
169
|
+
* either restore it (`aws secretsmanager restore-secret --secret-id
|
|
170
|
+
* fjall/clickhouse/<name>-password`, then let CloudFormation import) or
|
|
171
|
+
* remove it outright (`aws secretsmanager delete-secret --secret-id … --force-delete-without-recovery`).
|
|
172
|
+
*/
|
|
157
173
|
function createClickHouseUserSecret(scope, name) {
|
|
158
174
|
return new Secret(scope, `ClickHouseUser${toPascalCase(name)}`, {
|
|
159
175
|
secretName: clickHouseUserSecretName(name),
|
|
@@ -202,6 +218,7 @@ export class ClickHouseDatabase extends Construct {
|
|
|
202
218
|
#httpPort;
|
|
203
219
|
#nativePort;
|
|
204
220
|
#users;
|
|
221
|
+
#frameworkSecrets;
|
|
205
222
|
#schemaAdmin;
|
|
206
223
|
#managedPasswordNames;
|
|
207
224
|
#backupBucket;
|
|
@@ -259,6 +276,12 @@ export class ClickHouseDatabase extends Construct {
|
|
|
259
276
|
}
|
|
260
277
|
seenNames.add(name);
|
|
261
278
|
}
|
|
279
|
+
for (const name of seenNames) {
|
|
280
|
+
if (isReservedClickHouseUserName(name)) {
|
|
281
|
+
throw new Error(`ClickHouseDatabase: user name '${name}' uses the reserved '${FJALL_CLICKHOUSE_USER_PREFIX}' prefix — ` +
|
|
282
|
+
`framework identities (${FJALL_CLICKHOUSE_USERS.join(", ")}) are provisioned by the framework and cannot be declared in schemaAdmin or managedPasswords`);
|
|
283
|
+
}
|
|
284
|
+
}
|
|
262
285
|
const allUserNames = [
|
|
263
286
|
schemaAdmin.name,
|
|
264
287
|
...managedPasswords
|
|
@@ -328,7 +351,16 @@ export class ClickHouseDatabase extends Construct {
|
|
|
328
351
|
for (const name of allUserNames) {
|
|
329
352
|
userSecrets.set(name, createClickHouseUserSecret(this, name));
|
|
330
353
|
}
|
|
354
|
+
// Framework identities are minted OUTSIDE `userSecrets`: they are never
|
|
355
|
+
// injected into the ClickHouse service container (the migration runner
|
|
356
|
+
// provisions them over SQL), never exposed via `getUser`, and never
|
|
357
|
+
// declared as stack outputs — adopting them changes no service task def.
|
|
358
|
+
const frameworkSecrets = new Map();
|
|
359
|
+
for (const name of FJALL_CLICKHOUSE_USERS) {
|
|
360
|
+
frameworkSecrets.set(name, createClickHouseUserSecret(this, name));
|
|
361
|
+
}
|
|
331
362
|
this.#users = userSecrets;
|
|
363
|
+
this.#frameworkSecrets = frameworkSecrets;
|
|
332
364
|
this.#schemaAdmin = schemaAdmin;
|
|
333
365
|
this.#managedPasswordNames = managedPasswords;
|
|
334
366
|
const backupBucket = props.backupBucket ??
|
|
@@ -358,6 +390,13 @@ export class ClickHouseDatabase extends Construct {
|
|
|
358
390
|
retention: RetentionDays.TWO_WEEKS
|
|
359
391
|
})
|
|
360
392
|
: undefined;
|
|
393
|
+
// Explicit rather than `logRetention` so the maintenance-status metric
|
|
394
|
+
// filters can attach to it alongside the backup task's group.
|
|
395
|
+
const optimiseTaskLogGroup = optimiseEnabled
|
|
396
|
+
? new LogGroup(this, "ClickHouseOptimiseTaskLogGroup", {
|
|
397
|
+
retention: RetentionDays.ONE_WEEK
|
|
398
|
+
})
|
|
399
|
+
: undefined;
|
|
361
400
|
// Resolved BEFORE the user data is built: the declared volume size is
|
|
362
401
|
// embedded into the script so a storageGb change versions the launch
|
|
363
402
|
// template (see `dataVolumeSizeGb` on BuildClickHouseUserDataOptions).
|
|
@@ -416,10 +455,17 @@ export class ClickHouseDatabase extends Construct {
|
|
|
416
455
|
const clickHouseHost = this.getHostEndpoint();
|
|
417
456
|
const backupDestUrl = `https://${backupBucket.bucketName}.s3.${Stack.of(this).region}.amazonaws.com/backup/`;
|
|
418
457
|
const optimiseStatements = [
|
|
419
|
-
...REPLACING_MERGE_TREE_TABLES.map((table) =>
|
|
420
|
-
|
|
458
|
+
...REPLACING_MERGE_TREE_TABLES.map((table) => ({
|
|
459
|
+
table,
|
|
460
|
+
sql: `OPTIMIZE TABLE ${CLICKHOUSE_DATABASE_NAME}.${table} FINAL`
|
|
461
|
+
})),
|
|
462
|
+
...OPTIMISE_MV_TABLES.map((table) => ({
|
|
463
|
+
table,
|
|
464
|
+
sql: `OPTIMIZE TABLE ${CLICKHOUSE_DATABASE_NAME}.${table}`
|
|
465
|
+
}))
|
|
421
466
|
];
|
|
422
467
|
const adminSecret = expectDefined(userSecrets.get(schemaAdmin.name), `schemaAdmin '${schemaAdmin.name}' secret not minted.`);
|
|
468
|
+
const maintenanceSecret = expectDefined(frameworkSecrets.get(FJALL_MAINTENANCE_USER), `framework identity '${FJALL_MAINTENANCE_USER}' secret not minted.`);
|
|
423
469
|
const sidecarTlsPreamble = tlsActive
|
|
424
470
|
? `set -eu && printf '%s\\n' "$CLICKHOUSE_CA_CERT" > /tmp/ca.crt && printf '%s' '<?xml version="1.0"?><config><openSSL><client><caConfig>/tmp/ca.crt</caConfig><verificationMode>strict</verificationMode><loadDefaultCAFile>false</loadDefaultCAFile><invalidCertificateHandler><name>RejectCertificateHandler</name></invalidCertificateHandler></client></openSSL></config>' > /tmp/clickhouse-client.xml && `
|
|
425
471
|
: "";
|
|
@@ -432,33 +478,49 @@ export class ClickHouseDatabase extends Construct {
|
|
|
432
478
|
// One clickhouse-client invocation per statement: a single multi-statement
|
|
433
479
|
// --query aborts on the first failure, so one missing table (a migration
|
|
434
480
|
// deploy-order window, or a dropped table lingering in the constants)
|
|
435
|
-
// would permanently skip every optimise after it.
|
|
436
|
-
// the task still exits non-zero so they stay
|
|
437
|
-
// Both scheduled tasks authenticate
|
|
438
|
-
//
|
|
439
|
-
|
|
481
|
+
// would permanently skip every optimise after it. Each failure emits its
|
|
482
|
+
// own status marker and the task still exits non-zero so they stay
|
|
483
|
+
// visible in the task history. Both scheduled tasks authenticate as the
|
|
484
|
+
// maintenance identity; the password arrives via CLICKHOUSE_PASSWORD
|
|
485
|
+
// rather than argv.
|
|
486
|
+
const clickHouseClient = `clickhouse-client --host ${clickHouseHost} --port ${nativePort} --user ${FJALL_MAINTENANCE_USER}${sidecarTlsClientArgs}`;
|
|
440
487
|
const optimiseScript = [
|
|
441
488
|
"FAILED=0",
|
|
442
|
-
...optimiseStatements.map((
|
|
489
|
+
...optimiseStatements.map(({ table, sql }) => `${maintenanceRun(`optimise_${table}`, sql)} || FAILED=1`),
|
|
490
|
+
`if [ "$FAILED" = 0 ]; then ${maintenanceOkMarker("optimise")}; fi`,
|
|
443
491
|
'exit "$FAILED"'
|
|
444
492
|
].join("; ");
|
|
445
|
-
|
|
493
|
+
const optimisePreamble = buildMaintenancePreamble({
|
|
494
|
+
client: clickHouseClient,
|
|
495
|
+
job: "optimise",
|
|
496
|
+
schemaAdminName: schemaAdmin.name
|
|
497
|
+
});
|
|
498
|
+
const maintenanceSecrets = {
|
|
499
|
+
CLICKHOUSE_PASSWORD: EcsSecret.fromSecretsManager(maintenanceSecret.secret, "password"),
|
|
500
|
+
...sidecarTlsSecrets
|
|
501
|
+
};
|
|
446
502
|
const scheduledTasks = [];
|
|
447
|
-
|
|
503
|
+
const maintenanceLogGroups = [];
|
|
504
|
+
if (optimiseEnabled && optimiseTaskLogGroup !== undefined) {
|
|
448
505
|
scheduledTasks.push({
|
|
449
506
|
name: "ClickHouseOptimiseTask",
|
|
450
507
|
schedule: optimiseSchedule,
|
|
451
508
|
image: ContainerImage.fromRegistry(CLICKHOUSE_IMAGE),
|
|
452
509
|
cpu: OPTIMISE_TASK_CPU_UNITS,
|
|
453
510
|
memoryLimitMiB: OPTIMISE_TASK_MEMORY_MIB,
|
|
454
|
-
command: [
|
|
455
|
-
|
|
456
|
-
|
|
457
|
-
|
|
458
|
-
|
|
459
|
-
|
|
511
|
+
command: [
|
|
512
|
+
"sh",
|
|
513
|
+
"-c",
|
|
514
|
+
`${sidecarTlsPreamble}${optimisePreamble}\n${optimiseScript}`
|
|
515
|
+
],
|
|
516
|
+
secrets: maintenanceSecrets,
|
|
517
|
+
logGroup: optimiseTaskLogGroup,
|
|
460
518
|
securityGroups: [securityGroup]
|
|
461
519
|
});
|
|
520
|
+
maintenanceLogGroups.push({
|
|
521
|
+
job: "optimise",
|
|
522
|
+
logGroup: optimiseTaskLogGroup
|
|
523
|
+
});
|
|
462
524
|
}
|
|
463
525
|
if (backupEnabled && backupTaskLogGroup !== undefined) {
|
|
464
526
|
scheduledTasks.push({
|
|
@@ -473,6 +535,7 @@ export class ClickHouseDatabase extends Construct {
|
|
|
473
535
|
// Password via CLICKHOUSE_PASSWORD env, not --password on argv (argv → /proc/<pid>/cmdline).
|
|
474
536
|
`${sidecarTlsPreamble}${buildClickHouseBackupScript({
|
|
475
537
|
client: clickHouseClient,
|
|
538
|
+
schemaAdminName: schemaAdmin.name,
|
|
476
539
|
databaseName: CLICKHOUSE_DATABASE_NAME,
|
|
477
540
|
backupDestUrl,
|
|
478
541
|
diskFreeCriticalGib,
|
|
@@ -480,13 +543,14 @@ export class ClickHouseDatabase extends Construct {
|
|
|
480
543
|
verify: backupVerifyEnabled
|
|
481
544
|
})}`
|
|
482
545
|
],
|
|
483
|
-
secrets:
|
|
484
|
-
CLICKHOUSE_PASSWORD: EcsSecret.fromSecretsManager(adminSecret.secret, "password"),
|
|
485
|
-
...sidecarTlsSecrets
|
|
486
|
-
},
|
|
546
|
+
secrets: maintenanceSecrets,
|
|
487
547
|
logGroup: backupTaskLogGroup,
|
|
488
548
|
securityGroups: [securityGroup]
|
|
489
549
|
});
|
|
550
|
+
maintenanceLogGroups.push({
|
|
551
|
+
job: "backup",
|
|
552
|
+
logGroup: backupTaskLogGroup
|
|
553
|
+
});
|
|
490
554
|
}
|
|
491
555
|
const clickHouseContainerSecrets = {};
|
|
492
556
|
for (const name of allUserNames) {
|
|
@@ -505,7 +569,11 @@ export class ClickHouseDatabase extends Construct {
|
|
|
505
569
|
type: "ecs",
|
|
506
570
|
vpc,
|
|
507
571
|
...(resolvedAlertsTopic !== undefined && {
|
|
508
|
-
alertsTopic: resolvedAlertsTopic
|
|
572
|
+
alertsTopic: resolvedAlertsTopic,
|
|
573
|
+
// `RunningTaskCount` liveness exists only under Container Insights;
|
|
574
|
+
// enabled together with the alarm surface so the documented liveness
|
|
575
|
+
// alarm has a metric to watch (and costs nothing when alarms are off).
|
|
576
|
+
containerInsights: true
|
|
509
577
|
}),
|
|
510
578
|
...(applicationId !== undefined && { applicationId }),
|
|
511
579
|
cluster: {
|
|
@@ -697,6 +765,7 @@ export class ClickHouseDatabase extends Construct {
|
|
|
697
765
|
config: alarmThresholds,
|
|
698
766
|
diskFreeCriticalGib,
|
|
699
767
|
backupVerifyEnabled,
|
|
768
|
+
maintenanceLogGroups,
|
|
700
769
|
...(applicationId !== undefined && { applicationId }),
|
|
701
770
|
...(backupTaskLogGroup !== undefined && { backupTaskLogGroup })
|
|
702
771
|
});
|
|
@@ -900,17 +969,31 @@ export class ClickHouseDatabase extends Construct {
|
|
|
900
969
|
* `process.env`. The helper reads the env vars directly — no runtime
|
|
901
970
|
* `secretsmanager:GetSecretValue` SDK call is needed; ECS executionRole
|
|
902
971
|
* carries the IAM grant via the standard `secretsImport` framework path.
|
|
972
|
+
*
|
|
973
|
+
* **Framework identities.** `FJALL_CLICKHOUSE_USERS` (the gate and
|
|
974
|
+
* maintenance identities) ship the same way under their own manifest,
|
|
975
|
+
* `CLICKHOUSE_FRAMEWORK_USERS`, each with a `USER_<NAME>_PASSWORD` import,
|
|
976
|
+
* plus `CLICKHOUSE_MAINTENANCE_PROFILE` naming the schema admin's settings
|
|
977
|
+
* profile so the runner can bind `fjall_maintenance` to the same
|
|
978
|
+
* instance-scaled caps without knowing any numbers.
|
|
903
979
|
*/
|
|
904
980
|
getMigrationContributions() {
|
|
905
981
|
const environment = {
|
|
906
982
|
CLICKHOUSE_URL: this.getUrl(),
|
|
907
983
|
CLICKHOUSE_DATABASE: this.getDatabaseName(),
|
|
908
|
-
[SCHEMA_ADMIN_USER_ENV]: this.#schemaAdmin.name
|
|
984
|
+
[SCHEMA_ADMIN_USER_ENV]: this.#schemaAdmin.name,
|
|
985
|
+
[CLICKHOUSE_MAINTENANCE_PROFILE_ENV]: this.#schemaAdmin.profile
|
|
909
986
|
};
|
|
910
987
|
const secretsImport = {};
|
|
911
988
|
const adminSecret = this.getUser(this.#schemaAdmin.name);
|
|
912
989
|
secretsImport[SCHEMA_ADMIN_PASSWORD_ENV] =
|
|
913
990
|
adminSecret.getImport("password");
|
|
991
|
+
for (const [name, secret] of this.#frameworkSecrets) {
|
|
992
|
+
secretsImport[userPasswordEnvName(name)] = secret.getImport("password");
|
|
993
|
+
}
|
|
994
|
+
environment[CLICKHOUSE_FRAMEWORK_USERS_ENV] = JSON.stringify([
|
|
995
|
+
...FJALL_CLICKHOUSE_USERS
|
|
996
|
+
]);
|
|
914
997
|
const caCertImport = this.getTlsCaCertImport();
|
|
915
998
|
if (caCertImport !== undefined) {
|
|
916
999
|
secretsImport.CLICKHOUSE_CA_CERT = caCertImport;
|
|
@@ -942,16 +1025,23 @@ export class ClickHouseDatabase extends Construct {
|
|
|
942
1025
|
}
|
|
943
1026
|
return { environment, secretsImport, egress };
|
|
944
1027
|
}
|
|
1028
|
+
/**
|
|
1029
|
+
* The materialised gate's ClickHouse credentials: the gate authenticates as
|
|
1030
|
+
* the read-only `fjall_schema_gate` identity, never as the schema admin.
|
|
1031
|
+
* Env `FJALL_SCHEMA_GATE_CH_USER` carries the fixed name; secret
|
|
1032
|
+
* `FJALL_SCHEMA_GATE_CH_PASSWORD` imports that identity's password.
|
|
1033
|
+
* `undefined` when no `migrations:` config is declared — nothing to gate.
|
|
1034
|
+
*/
|
|
945
1035
|
getSchemaGateContribution() {
|
|
946
1036
|
if (this.#migrationsConfig === undefined)
|
|
947
1037
|
return undefined;
|
|
948
|
-
const
|
|
1038
|
+
const gateSecret = expectDefined(this.#frameworkSecrets.get(FJALL_SCHEMA_GATE_USER), `framework identity '${FJALL_SCHEMA_GATE_USER}' secret not minted.`);
|
|
949
1039
|
return {
|
|
950
1040
|
environment: {
|
|
951
|
-
[
|
|
1041
|
+
[SCHEMA_GATE_CH_USER_ENV]: FJALL_SCHEMA_GATE_USER
|
|
952
1042
|
},
|
|
953
1043
|
secretsImport: {
|
|
954
|
-
[
|
|
1044
|
+
[SCHEMA_GATE_CH_PASSWORD_ENV]: gateSecret.getImport("password")
|
|
955
1045
|
}
|
|
956
1046
|
};
|
|
957
1047
|
}
|
|
@@ -72,10 +72,7 @@ export declare function expandMigrationsSugar(service: EcsServiceConfig, userCon
|
|
|
72
72
|
* container gains a `dependsOn` SUCCESS edge on it.
|
|
73
73
|
* @internal Exported for testing only
|
|
74
74
|
*/
|
|
75
|
-
export declare function buildContainerConfigs(service: EcsServiceConfig, schemaVersionEnv?: Record<string, string>, annotationsScope?: Construct,
|
|
76
|
-
environment: Record<string, string>;
|
|
77
|
-
secretsImport: Record<string, SecretImport>;
|
|
78
|
-
}, schemaGate?: SchemaGateContainerSpec): EcsClusterProps["services"][number]["containers"];
|
|
75
|
+
export declare function buildContainerConfigs(service: EcsServiceConfig, schemaVersionEnv?: Record<string, string>, annotationsScope?: Construct, chSchemaVersionEnv?: Record<string, string>, schemaGate?: SchemaGateContainerSpec): EcsClusterProps["services"][number]["containers"];
|
|
79
76
|
/**
|
|
80
77
|
* Resolved scaling configuration for an ECS service.
|
|
81
78
|
* @internal Exported for testing only
|
|
@@ -131,17 +128,16 @@ export declare class EcsCompute extends Construct implements IEcsCompute {
|
|
|
131
128
|
private resolveSchemaVersionEnv;
|
|
132
129
|
/**
|
|
133
130
|
* ClickHouse mirror of `resolveSchemaVersionEnv`. Returns the
|
|
134
|
-
* `EXPECTED_CH_SCHEMA_VERSION` env entry
|
|
135
|
-
*
|
|
136
|
-
*
|
|
131
|
+
* `EXPECTED_CH_SCHEMA_VERSION` env entry, or `undefined` when the service
|
|
132
|
+
* is not gated. Env only: app containers receive no ClickHouse credentials
|
|
133
|
+
* from the gate — the materialised gate container authenticates as the
|
|
134
|
+
* read-only gate identity via `getSchemaGateContribution()`, composed in
|
|
135
|
+
* `resolveSchemaGateMaterialisation`.
|
|
137
136
|
*
|
|
138
137
|
* - `schemaGate: false` → returns `undefined` (auditable opt-out — same flag
|
|
139
138
|
* covers BOTH PG and CH gates)
|
|
140
139
|
* - No migrated CH in connections → returns `undefined`
|
|
141
|
-
* - Exactly one migrated CH → returns
|
|
142
|
-
* `SCHEMA_ADMIN_USER`, plus `secretsImport.SCHEMA_ADMIN_PASSWORD`. The
|
|
143
|
-
* creds piggyback on the same gate resolution so adding the gate to a
|
|
144
|
-
* service can't ship without the creds it needs to use it.
|
|
140
|
+
* - Exactly one migrated CH → returns `{ EXPECTED_CH_SCHEMA_VERSION }`
|
|
145
141
|
* - Two or more migrated CHs → throws via `resolveClickHouseDatabaseForService`
|
|
146
142
|
*/
|
|
147
143
|
private resolveClickHouseSchemaVersionEnv;
|
|
@@ -166,11 +162,13 @@ export declare class EcsCompute extends Construct implements IEcsCompute {
|
|
|
166
162
|
* - The default image URI cannot be derived because the constructs package
|
|
167
163
|
* version is unresolvable and no `schemaGate.image` override is set
|
|
168
164
|
*
|
|
169
|
-
*
|
|
170
|
-
* `getSchemaGateContribution()
|
|
171
|
-
*
|
|
172
|
-
*
|
|
173
|
-
*
|
|
165
|
+
* Both halves re-resolve the migrated DB for its
|
|
166
|
+
* `getSchemaGateContribution()`: the relational half for the
|
|
167
|
+
* credential-free URL base + user/password secret imports, the CH half for
|
|
168
|
+
* the read-only gate identity (`FJALL_SCHEMA_GATE_CH_USER` +
|
|
169
|
+
* `FJALL_SCHEMA_GATE_CH_PASSWORD`) composed with `CLICKHOUSE_URL` /
|
|
170
|
+
* database name / CA-cert import — `chSchemaVersionEnv` alone carries only
|
|
171
|
+
* the version entry the app containers get.
|
|
174
172
|
*/
|
|
175
173
|
private resolveSchemaGateMaterialisation;
|
|
176
174
|
private materialiseScheduledTasks;
|