@meyicloud/meyi-cost-server 1.4.1 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +11 -24
- package/README.md +48 -49
- package/index.js +0 -1
- package/package.json +4 -3
- package/src/controllers/cost-analysis.controller.js +39 -9
- package/src/lib/cost-analysis-schedule.js +45 -0
- package/src/plugin.js +16 -12
- package/src/repositories/aws-onboarding.repository.js +6 -4
- package/src/repositories/cost-analysis-report.repository.js +127 -0
- package/src/routes/index.js +6 -1
- package/src/schema/cost-analysis-report.schema.js +53 -0
- package/src/services/cost-analyser.service.js +158 -0
- package/src/services/cost-analysis-report.service.js +68 -0
- package/src/services/cost-report-artifact.service.js +22 -0
- package/src/workers/cost-analysis-report.worker.js +67 -0
- package/src/lib/cost-analysis.js +0 -116
- package/src/lib/llm-provider.js +0 -239
- package/src/repositories/cost-analysis.repository.js +0 -50
- package/src/schema/cost-analysis.schema.js +0 -7
- package/src/services/cost-analysis-data.service.js +0 -39
- package/src/services/cost-analysis.service.js +0 -190
package/AGENTS.md
CHANGED
|
@@ -35,23 +35,13 @@ Keep HTTP concerns in routes/controllers, AWS and database behavior in services,
|
|
|
35
35
|
schema creation in `schema`, and pure calculations in `lib`. `index.js` and
|
|
36
36
|
`cur.js` must remain thin compatibility entry points.
|
|
37
37
|
|
|
38
|
-
##
|
|
38
|
+
## External cost analyser
|
|
39
39
|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
"AWS Bedrock" resolve correctly.
|
|
46
|
-
|
|
47
|
-
`CostAnalysisService.resolveProvider(tenantId)` picks between a host-supplied
|
|
48
|
-
`providerResolver` and the `COST_AI_*` environment defaults. Two rules matter:
|
|
49
|
-
the analysis fingerprint is computed from the **effective** model so switching
|
|
50
|
-
provider cannot serve a cached answer from the previous one, and a resolver
|
|
51
|
-
failure degrades to the environment rather than failing the request.
|
|
52
|
-
|
|
53
|
-
When adding a provider, add it here and to `test/llm-provider.test.js`; no
|
|
54
|
-
other file should need to change.
|
|
40
|
+
The backend package never calls an LLM directly. `CostAnalysisReportWorker`
|
|
41
|
+
claims due tenant schedules and delegates execution to `CostAnalyserService`,
|
|
42
|
+
which starts one tenant-scoped `meyi-cost-ai-analyser` Fargate task. The task
|
|
43
|
+
writes canonical Markdown and derived PDF artifacts to private S3. Keep ECS
|
|
44
|
+
task overrides tenant-scoped and never log the customer role external ID.
|
|
55
45
|
|
|
56
46
|
## Request and data flow
|
|
57
47
|
|
|
@@ -62,20 +52,17 @@ flowchart LR
|
|
|
62
52
|
Enabled --> Router[/api/v1/cost router]
|
|
63
53
|
Router --> Controller[Cost controller]
|
|
64
54
|
Controller --> Customer[Customer AWS context service]
|
|
65
|
-
|
|
66
|
-
Controller --> Source{Configured source}
|
|
67
|
-
Source -->|CUR| SaaS[SaaS Athena context service]
|
|
55
|
+
Controller --> SaaS[SaaS Athena context service]
|
|
68
56
|
SaaS --> Athena[Central tenant CUR partition]
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
Athena --> Controller
|
|
73
|
-
CE --> Controller
|
|
57
|
+
Athena --> Scheduler[Scheduled report worker]
|
|
58
|
+
Scheduler --> ECS[Cost analyser Fargate task]
|
|
59
|
+
ECS --> Artifacts[Private Markdown and PDF S3 objects]
|
|
74
60
|
```
|
|
75
61
|
|
|
76
62
|
The plugin currently owns these route groups under `${apiBaseUri}/cost`:
|
|
77
63
|
|
|
78
64
|
- `GET /accounts`, `/data-status`, `/overview`, `/filter-options`, `/reports`, `/tags`
|
|
65
|
+
- `GET`/`PUT /analysis/schedule`, `GET /analysis/reports`, and tenant-scoped Markdown/PDF downloads
|
|
79
66
|
- `GET /budgets` and `POST /budgets`
|
|
80
67
|
- `DELETE /budgets/:id`
|
|
81
68
|
- `GET /budget-alert-dismissals` and `POST /budget-alert-dismissals`
|
package/README.md
CHANGED
|
@@ -30,6 +30,9 @@ and AWS onboarding. The plugin owns cost routes and its budget-related tables.
|
|
|
30
30
|
- Explicit CUR setup, pending-data, and unavailable states
|
|
31
31
|
- Tenant-scoped budget rules
|
|
32
32
|
- User-, tenant-, month-, and status-scoped budget-alert dismissals
|
|
33
|
+
- Tenant-configurable daily, weekly, and monthly AI report schedules
|
|
34
|
+
- PostgreSQL-backed background report jobs with failure history
|
|
35
|
+
- Server-generated PDF reports available through tenant-scoped downloads
|
|
33
36
|
|
|
34
37
|
Budgets are application rules stored in PostgreSQL; they are not AWS Budgets
|
|
35
38
|
resources. The consuming application evaluates them against current cost and
|
|
@@ -94,7 +97,7 @@ For a local package installation test:
|
|
|
94
97
|
|
|
95
98
|
```bash
|
|
96
99
|
npm pack
|
|
97
|
-
npm install /path/to/meyicloud-meyi-cost-server-1.
|
|
100
|
+
npm install /path/to/meyicloud-meyi-cost-server-1.5.0.tgz
|
|
98
101
|
```
|
|
99
102
|
|
|
100
103
|
## Publish to npm
|
|
@@ -255,58 +258,54 @@ SaaS Athena resources.
|
|
|
255
258
|
|
|
256
259
|
### AI analysis variables
|
|
257
260
|
|
|
258
|
-
AI
|
|
259
|
-
|
|
260
|
-
|
|
261
|
+
AI reports are generated by a separate `meyi-cost-ai-analyser` Fargate task.
|
|
262
|
+
The backend only schedules jobs, launches and monitors the task, and serves
|
|
263
|
+
tenant-scoped artifacts. It does not invoke Bedrock or generate PDFs itself.
|
|
264
|
+
Daily reports cover the previous complete day, weekly reports cover the
|
|
265
|
+
previous seven complete days, and monthly reports cover the previous calendar
|
|
266
|
+
month in the configured timezone.
|
|
261
267
|
|
|
262
|
-
|
|
263
|
-
| --- | --- | --- | --- |
|
|
264
|
-
| `COST_AI_ENABLED` | `false` | No | Enables the AI analysis endpoints and Bedrock client. |
|
|
265
|
-
| `COST_AI_REGION` | `AWS_REGION` or `us-east-1` | When enabled | Region used by the Bedrock Runtime client. |
|
|
266
|
-
| `COST_AI_MODEL_ID` | Global Claude Sonnet inference profile | When enabled | Bedrock model or inference-profile ID. |
|
|
267
|
-
| `COST_AI_MAX_TOKENS` | `1600` | No | Maximum response tokens, clamped from 600 to 3000. |
|
|
268
|
-
| `COST_AI_CACHE_TTL_MS` | `21600000` | No | Tenant analysis cache duration; defaults to six hours. |
|
|
269
|
-
| `COST_AI_HOURLY_LIMIT` | `6` | No | Maximum non-cached generations per tenant per hour. |
|
|
270
|
-
| `COST_AI_TIMEOUT_MS` | `120000` | No | Per-request timeout for key-based providers; Bedrock uses the SDK default. |
|
|
271
|
-
|
|
272
|
-
### Choosing a provider
|
|
268
|
+
The authenticated report API is:
|
|
273
269
|
|
|
274
|
-
|
|
275
|
-
|
|
276
|
-
`
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
provider: "anthropic", // bedrock | anthropic | openai
|
|
283
|
-
modelId: "claude-sonnet-4-5",
|
|
284
|
-
apiKey: "...", // required for anthropic and openai
|
|
285
|
-
region: "ap-south-2", // bedrock only
|
|
286
|
-
accessKeyId: "...", // bedrock only; omit to use ambient credentials
|
|
287
|
-
secretAccessKey: "...",
|
|
288
|
-
}),
|
|
289
|
-
})
|
|
290
|
-
```
|
|
291
|
-
|
|
292
|
-
Returning `null` falls back to the environment configuration, so a host that
|
|
293
|
-
does not pass a resolver behaves exactly as before. A resolver that throws is
|
|
294
|
-
logged and also falls back, so a credential-store outage degrades the feature
|
|
295
|
-
rather than failing the request.
|
|
296
|
-
|
|
297
|
-
Why this exists: Bedrock authenticates with SigV4 and is subject to
|
|
298
|
-
account-level Anthropic model access. The key-based providers are not, so a
|
|
299
|
-
deployment blocked on Bedrock model access can still run analysis. It also
|
|
300
|
-
lets a customer bring their own model account.
|
|
270
|
+
| Method | Route | Purpose |
|
|
271
|
+
| --- | --- | --- |
|
|
272
|
+
| `GET` | `/analysis/schedule` | Read this tenant's schedule. |
|
|
273
|
+
| `PUT` | `/analysis/schedule` | Update the schedule; requires `req.user.role=admin`. |
|
|
274
|
+
| `GET` | `/analysis/reports` | List tenant report history without PDF bytes. |
|
|
275
|
+
| `GET` | `/analysis/reports/:id` | Read one tenant report. |
|
|
276
|
+
| `GET` | `/analysis/reports/:id/pdf` | Download a completed report PDF. |
|
|
277
|
+
| `GET` | `/analysis/reports/:id/markdown` | Download the canonical Markdown report. |
|
|
301
278
|
|
|
302
|
-
|
|
303
|
-
|
|
304
|
-
|
|
279
|
+
Schedule and report rows live in `cost_ai_report_schedules` and
|
|
280
|
+
`cost_ai_reports`. The worker atomically claims due schedules with a lease, so
|
|
281
|
+
a restarted task can resume future work without relying on in-memory timers.
|
|
305
282
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
`
|
|
309
|
-
|
|
283
|
+
| Variable | Default | Required | Purpose |
|
|
284
|
+
| --- | --- | --- | --- |
|
|
285
|
+
| `COST_AI_ENABLED` | `false` | No | Enables the schedule worker and external analyser integration. |
|
|
286
|
+
| `COST_AI_ANALYSER_REGION` | `AWS_REGION` | When enabled | ECS and report-bucket region. |
|
|
287
|
+
| `COST_AI_ANALYSER_CLUSTER` | Empty | When enabled | ECS cluster ARN or name. |
|
|
288
|
+
| `COST_AI_ANALYSER_TASK_DEFINITION` | Empty | When enabled | Fargate task definition ARN. |
|
|
289
|
+
| `COST_AI_ANALYSER_CONTAINER_NAME` | `cost-ai-analyser` | No | Container override target. |
|
|
290
|
+
| `COST_AI_ANALYSER_SUBNETS` | Empty | When enabled | Comma-separated private subnet IDs. |
|
|
291
|
+
| `COST_AI_ANALYSER_SECURITY_GROUPS` | Empty | When enabled | Comma-separated task security groups. |
|
|
292
|
+
| `COST_AI_ANALYSER_REPORT_BUCKET` | Empty | When enabled | Private S3 Markdown/PDF artifact bucket. |
|
|
293
|
+
| `COST_AI_ANALYSER_BEDROCK_REGION` | Empty | When enabled | Bedrock region passed to the analyser task. |
|
|
294
|
+
| `COST_AI_ANALYSER_BEDROCK_MODEL_ID` | Empty | When enabled | Model passed only to the analyser task. |
|
|
295
|
+
| `COST_AI_ANALYSER_TARGET_REGIONS` | `AWS_REGION` | No | Customer regions inspected for resource detail. |
|
|
296
|
+
| `COST_AI_ANALYSER_TOP_N_SERVICES` | `10` | No | Maximum high-cost service agents run per report. |
|
|
297
|
+
| `COST_AI_ANALYSER_POLL_INTERVAL_MS` | `15000` | No | ECS task status polling interval. |
|
|
298
|
+
| `COST_AI_ANALYSER_TIMEOUT_MS` | `2700000` | No | Maximum external task duration. |
|
|
299
|
+
| `COST_AI_REPORT_POLL_INTERVAL_MS` | `60000` | No | How often the background worker claims due schedules; minimum 10 seconds. |
|
|
300
|
+
| `COST_AI_REPORT_LEASE_MS` | `3600000` | No | Claim lease used to recover from an interrupted worker. |
|
|
301
|
+
| `COST_AI_REPORT_BATCH_SIZE` | `5` | No | Maximum schedules claimed by one worker tick; clamped from 1 to 25. |
|
|
302
|
+
|
|
303
|
+
`GET /analysis/status` reports `meyi-cost-ai-analyser` with an `ecs-task`
|
|
304
|
+
source. The analyser uses the shared ECS task role for Athena, S3, Bedrock, and
|
|
305
|
+
cross-account role assumption. New reports store task and private artifact
|
|
306
|
+
references in `cost_ai_reports`; existing database-backed PDF rows remain
|
|
307
|
+
downloadable for compatibility. Model or task failures mark only that report as
|
|
308
|
+
failed and do not change CUR, standard reports, or onboarding.
|
|
310
309
|
|
|
311
310
|
### CUR/Athena variables
|
|
312
311
|
|
package/index.js
CHANGED
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@meyicloud/meyi-cost-server",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.6.0",
|
|
4
4
|
"description": "Tenant-aware AWS CUR and Athena cost plugin server for MeyiConnect",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "./index.js",
|
|
@@ -19,13 +19,14 @@
|
|
|
19
19
|
],
|
|
20
20
|
"dependencies": {
|
|
21
21
|
"@aws-sdk/client-athena": "^3.850.0",
|
|
22
|
-
"@aws-sdk/client-
|
|
22
|
+
"@aws-sdk/client-ecs": "^3.850.0",
|
|
23
23
|
"@aws-sdk/client-glue": "^3.850.0",
|
|
24
24
|
"@aws-sdk/client-s3": "^3.850.0",
|
|
25
25
|
"@aws-sdk/client-sts": "^3.850.0",
|
|
26
26
|
"@aws-sdk/credential-providers": "^3.850.0",
|
|
27
27
|
"drizzle-orm": "^0.44.7",
|
|
28
|
-
"express": "^4.21.1"
|
|
28
|
+
"express": "^4.21.1",
|
|
29
|
+
"luxon": "^3.7.2"
|
|
29
30
|
},
|
|
30
31
|
"peerDependencies": {
|
|
31
32
|
"pg": ">=8"
|
|
@@ -1,8 +1,7 @@
|
|
|
1
|
-
import { dateRange } from "../lib/cost-utils.js";
|
|
2
|
-
|
|
3
1
|
export class CostAnalysisController {
|
|
4
|
-
constructor({ service }) {
|
|
2
|
+
constructor({ service, reportService }) {
|
|
5
3
|
this.service = service;
|
|
4
|
+
this.reportService = reportService;
|
|
6
5
|
}
|
|
7
6
|
|
|
8
7
|
async status(req, res, next) {
|
|
@@ -13,16 +12,47 @@ export class CostAnalysisController {
|
|
|
13
12
|
|
|
14
13
|
async latest(req, res, next) {
|
|
15
14
|
try {
|
|
16
|
-
return res.json({ analysis: await this.
|
|
15
|
+
return res.json({ analysis: await this.reportService.latest(req) });
|
|
16
|
+
} catch (error) { return next(error); }
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
async schedule(req, res, next) {
|
|
20
|
+
try { return res.json(await this.reportService.getSchedule(req)); }
|
|
21
|
+
catch (error) { return next(error); }
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
async updateSchedule(req, res, next) {
|
|
25
|
+
try { return res.json(await this.reportService.saveSchedule(req, req.body || {})); }
|
|
26
|
+
catch (error) { return next(error); }
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
async reports(req, res, next) {
|
|
30
|
+
try { return res.json({ reports: await this.reportService.list(req, req.query.limit) }); }
|
|
31
|
+
catch (error) { return next(error); }
|
|
32
|
+
}
|
|
33
|
+
|
|
34
|
+
async report(req, res, next) {
|
|
35
|
+
try { return res.json(await this.reportService.get(req, req.params.id)); }
|
|
36
|
+
catch (error) { return next(error); }
|
|
37
|
+
}
|
|
38
|
+
|
|
39
|
+
async downloadPdf(req, res, next) {
|
|
40
|
+
try {
|
|
41
|
+
const report = await this.reportService.pdf(req, req.params.id);
|
|
42
|
+
res.setHeader("Content-Type", "application/pdf");
|
|
43
|
+
res.setHeader("Content-Disposition", `attachment; filename="${report.filename}"`);
|
|
44
|
+
res.setHeader("Content-Length", report.data.length);
|
|
45
|
+
return res.send(report.data);
|
|
17
46
|
} catch (error) { return next(error); }
|
|
18
47
|
}
|
|
19
48
|
|
|
20
|
-
async
|
|
49
|
+
async downloadMarkdown(req, res, next) {
|
|
21
50
|
try {
|
|
22
|
-
const
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
51
|
+
const report = await this.reportService.markdown(req, req.params.id);
|
|
52
|
+
res.setHeader("Content-Type", "text/markdown; charset=utf-8");
|
|
53
|
+
res.setHeader("Content-Disposition", `attachment; filename="${report.filename}"`);
|
|
54
|
+
res.setHeader("Content-Length", report.data.length);
|
|
55
|
+
return res.send(report.data);
|
|
26
56
|
} catch (error) { return next(error); }
|
|
27
57
|
}
|
|
28
58
|
}
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
import { DateTime, IANAZone } from "luxon";
|
|
2
|
+
|
|
3
|
+
const FREQUENCIES = new Set(["daily", "weekly", "monthly"]);
|
|
4
|
+
|
|
5
|
+
export function validateSchedule(input = {}) {
|
|
6
|
+
const frequency = String(input.frequency || "weekly").toLowerCase();
|
|
7
|
+
const timezone = String(input.timezone || "UTC").trim();
|
|
8
|
+
const time = String(input.time || "09:00").trim();
|
|
9
|
+
if (!FREQUENCIES.has(frequency)) throw Object.assign(new Error("Frequency must be daily, weekly, or monthly"), { statusCode: 400 });
|
|
10
|
+
if (!IANAZone.isValidZone(timezone)) throw Object.assign(new Error("A valid IANA timezone is required"), { statusCode: 400 });
|
|
11
|
+
if (!/^([01]\d|2[0-3]):[0-5]\d$/.test(time)) throw Object.assign(new Error("Time must use 24-hour HH:mm format"), { statusCode: 400 });
|
|
12
|
+
const dayOfWeek = frequency === "weekly" ? Number(input.dayOfWeek ?? 1) : null;
|
|
13
|
+
const dayOfMonth = frequency === "monthly" ? Number(input.dayOfMonth ?? 1) : null;
|
|
14
|
+
if (dayOfWeek !== null && (!Number.isInteger(dayOfWeek) || dayOfWeek < 0 || dayOfWeek > 6)) throw Object.assign(new Error("Weekly day must be between 0 and 6"), { statusCode: 400 });
|
|
15
|
+
if (dayOfMonth !== null && (!Number.isInteger(dayOfMonth) || dayOfMonth < 1 || dayOfMonth > 28)) throw Object.assign(new Error("Monthly day must be between 1 and 28"), { statusCode: 400 });
|
|
16
|
+
return { enabled: Boolean(input.enabled), frequency, timezone, time, dayOfWeek, dayOfMonth };
|
|
17
|
+
}
|
|
18
|
+
|
|
19
|
+
function atTime(date, time) {
|
|
20
|
+
const [hour, minute] = time.split(":").map(Number);
|
|
21
|
+
return date.set({ hour, minute, second: 0, millisecond: 0 });
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
export function nextScheduleRun(schedule, after = new Date()) {
|
|
25
|
+
const local = DateTime.fromJSDate(after, { zone: schedule.timezone });
|
|
26
|
+
let candidate = atTime(local.startOf("day"), schedule.time);
|
|
27
|
+
if (schedule.frequency === "daily") {
|
|
28
|
+
if (candidate <= local) candidate = candidate.plus({ days: 1 });
|
|
29
|
+
} else if (schedule.frequency === "weekly") {
|
|
30
|
+
const targetWeekday = schedule.dayOfWeek === 0 ? 7 : schedule.dayOfWeek;
|
|
31
|
+
candidate = atTime(local.startOf("week").plus({ days: targetWeekday - 1 }), schedule.time);
|
|
32
|
+
if (candidate <= local) candidate = candidate.plus({ weeks: 1 });
|
|
33
|
+
} else {
|
|
34
|
+
candidate = atTime(local.startOf("month").set({ day: schedule.dayOfMonth }), schedule.time);
|
|
35
|
+
if (candidate <= local) candidate = atTime(candidate.plus({ months: 1 }).startOf("month").set({ day: schedule.dayOfMonth }), schedule.time);
|
|
36
|
+
}
|
|
37
|
+
return candidate.toUTC().toJSDate();
|
|
38
|
+
}
|
|
39
|
+
|
|
40
|
+
export function reportRange(schedule, scheduledFor) {
|
|
41
|
+
const scheduledDay = DateTime.fromJSDate(new Date(scheduledFor), { zone: schedule.timezone }).startOf("day");
|
|
42
|
+
const end = schedule.frequency === "monthly" ? scheduledDay.startOf("month") : scheduledDay;
|
|
43
|
+
const start = schedule.frequency === "monthly" ? end.minus({ months: 1 }) : end.minus({ days: schedule.frequency === "weekly" ? 7 : 1 });
|
|
44
|
+
return { Start: start.toISODate(), End: end.toISODate() };
|
|
45
|
+
}
|
package/src/plugin.js
CHANGED
|
@@ -11,10 +11,12 @@ import { BudgetController } from "./controllers/budget.controller.js";
|
|
|
11
11
|
import { CostAnalysisController } from "./controllers/cost-analysis.controller.js";
|
|
12
12
|
import { createCostRouter } from "./routes/index.js";
|
|
13
13
|
import { installCostBudgetSchema } from "./schema/cost-budget.schema.js";
|
|
14
|
-
import {
|
|
15
|
-
import {
|
|
16
|
-
import {
|
|
17
|
-
import {
|
|
14
|
+
import { installCostAnalysisReportSchema } from "./schema/cost-analysis-report.schema.js";
|
|
15
|
+
import { CostAnalysisReportRepository } from "./repositories/cost-analysis-report.repository.js";
|
|
16
|
+
import { CostAnalyserService } from "./services/cost-analyser.service.js";
|
|
17
|
+
import { CostAnalysisReportService } from "./services/cost-analysis-report.service.js";
|
|
18
|
+
import { CostReportArtifactService } from "./services/cost-report-artifact.service.js";
|
|
19
|
+
import { CostAnalysisReportWorker } from "./workers/cost-analysis-report.worker.js";
|
|
18
20
|
import { installCurDiscoverySchema } from "./cur-discovery/schema.js";
|
|
19
21
|
import { CurDiscoveryRepository } from "./cur-discovery/cur-discovery.repository.js";
|
|
20
22
|
import { CurDiscoveryAws } from "./cur-discovery/cur-discovery.aws.js";
|
|
@@ -22,7 +24,7 @@ import { CurDiscoveryService } from "./cur-discovery/cur-discovery.service.js";
|
|
|
22
24
|
import { CurDiscoveryWorker } from "./cur-discovery/cur-discovery.worker.js";
|
|
23
25
|
import { safeSchema } from "./lib/cost-utils.js";
|
|
24
26
|
|
|
25
|
-
export function createInsightCost({ app, db, apiBaseUri = "/api/v1", logger = console
|
|
27
|
+
export function createInsightCost({ app, db, apiBaseUri = "/api/v1", logger = console } = {}) {
|
|
26
28
|
if (!db) throw new Error("db is required");
|
|
27
29
|
const schema = safeSchema(process.env.DB_SCHEMA || "meyiconnect");
|
|
28
30
|
const qSchema = `"${schema}"`;
|
|
@@ -31,7 +33,7 @@ export function createInsightCost({ app, db, apiBaseUri = "/api/v1", logger = co
|
|
|
31
33
|
const contextService = new CustomerAwsContextService({ repository: onboardingRepository, defaultTenant });
|
|
32
34
|
const budgetRepository = new BudgetRepository({ db, qSchema });
|
|
33
35
|
const budgetAlertRepository = new BudgetAlertRepository({ db, qSchema });
|
|
34
|
-
const
|
|
36
|
+
const costAnalysisReportRepository = new CostAnalysisReportRepository({ db, qSchema });
|
|
35
37
|
const budgetService = new BudgetService({ repository: budgetRepository });
|
|
36
38
|
const budgetAlertService = new BudgetAlertService({ repository: budgetAlertRepository });
|
|
37
39
|
const athenaContextService = new SaasAthenaContextService();
|
|
@@ -49,18 +51,18 @@ export function createInsightCost({ app, db, apiBaseUri = "/api/v1", logger = co
|
|
|
49
51
|
const discoveryWorker = new CurDiscoveryWorker({ repository: discoveryRepository, service: discoveryService, logger });
|
|
50
52
|
const costController = new CostController({ contextService, curProvider, discoveryRepository, logger });
|
|
51
53
|
const budgetController = new BudgetController({ contextService, budgetService, budgetAlertService });
|
|
52
|
-
const
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
const
|
|
56
|
-
const costAnalysisController = new CostAnalysisController({ service:
|
|
54
|
+
const costAnalyserService = new CostAnalyserService({ contextService, athenaContextService, logger });
|
|
55
|
+
const artifactService = new CostReportArtifactService();
|
|
56
|
+
const costAnalysisReportService = new CostAnalysisReportService({ contextService, repository: costAnalysisReportRepository, artifactService });
|
|
57
|
+
const costAnalysisReportWorker = new CostAnalysisReportWorker({ repository: costAnalysisReportRepository, analyserService: costAnalyserService, logger });
|
|
58
|
+
const costAnalysisController = new CostAnalysisController({ service: costAnalyserService, reportService: costAnalysisReportService });
|
|
57
59
|
const router = createCostRouter({ costController, budgetController, costAnalysisController }, logger);
|
|
58
60
|
let mounted = false;
|
|
59
61
|
|
|
60
62
|
return {
|
|
61
63
|
async install() {
|
|
62
64
|
await installCostBudgetSchema(db, qSchema);
|
|
63
|
-
await
|
|
65
|
+
await installCostAnalysisReportSchema(db, qSchema);
|
|
64
66
|
await installCurDiscoverySchema(db, qSchema);
|
|
65
67
|
logger.log?.("[Cost] Database migration verified");
|
|
66
68
|
},
|
|
@@ -70,10 +72,12 @@ export function createInsightCost({ app, db, apiBaseUri = "/api/v1", logger = co
|
|
|
70
72
|
mounted = true;
|
|
71
73
|
}
|
|
72
74
|
discoveryWorker.start();
|
|
75
|
+
costAnalysisReportWorker.start();
|
|
73
76
|
logger.log?.(`[Cost] Routes active at ${apiBaseUri}/cost/*`);
|
|
74
77
|
},
|
|
75
78
|
async stop() {
|
|
76
79
|
discoveryWorker.stop();
|
|
80
|
+
costAnalysisReportWorker.stop();
|
|
77
81
|
},
|
|
78
82
|
router,
|
|
79
83
|
};
|
|
@@ -24,9 +24,10 @@ export class AwsOnboardingRepository {
|
|
|
24
24
|
|
|
25
25
|
const connection = rows(await this.db.execute(sql`
|
|
26
26
|
SELECT
|
|
27
|
-
connection_id,
|
|
28
|
-
role_arn,
|
|
29
|
-
external_id
|
|
27
|
+
connection_id,
|
|
28
|
+
role_arn,
|
|
29
|
+
external_id,
|
|
30
|
+
management_account_id
|
|
30
31
|
FROM ${sql.raw(this.connectionsTable)}
|
|
31
32
|
WHERE tenant_id::text = ${tenant}
|
|
32
33
|
AND status = 'CONNECTED'
|
|
@@ -72,7 +73,8 @@ export class AwsOnboardingRepository {
|
|
|
72
73
|
status: item.status,
|
|
73
74
|
})),
|
|
74
75
|
meta: {
|
|
75
|
-
payerRoleArn: connection.role_arn,
|
|
76
|
+
payerRoleArn: connection.role_arn,
|
|
77
|
+
managementAccountId: connection.management_account_id,
|
|
76
78
|
externalId: connection.external_id,
|
|
77
79
|
connectionId: connection.connection_id,
|
|
78
80
|
curExportArn: curConfig.export_arn,
|
|
@@ -0,0 +1,127 @@
|
|
|
1
|
+
import { randomUUID } from "node:crypto";
|
|
2
|
+
import { sql } from "drizzle-orm";
|
|
3
|
+
import { rows } from "../lib/cost-utils.js";
|
|
4
|
+
|
|
5
|
+
export class CostAnalysisReportRepository {
|
|
6
|
+
constructor({ db, qSchema }) {
|
|
7
|
+
this.db = db;
|
|
8
|
+
this.schedules = `${qSchema}.cost_ai_report_schedules`;
|
|
9
|
+
this.reports = `${qSchema}.cost_ai_reports`;
|
|
10
|
+
}
|
|
11
|
+
|
|
12
|
+
async getSchedule(tenantId) {
|
|
13
|
+
return rows(await this.db.execute(sql`SELECT * FROM ${sql.raw(this.schedules)} WHERE tenant_id = ${tenantId}`))[0] || null;
|
|
14
|
+
}
|
|
15
|
+
|
|
16
|
+
async saveSchedule(tenantId, schedule, nextRunAt) {
|
|
17
|
+
return rows(await this.db.execute(sql`
|
|
18
|
+
INSERT INTO ${sql.raw(this.schedules)}
|
|
19
|
+
(tenant_id, enabled, frequency, time_of_day, timezone, day_of_week, day_of_month, next_run_at)
|
|
20
|
+
VALUES (${tenantId}, ${schedule.enabled}, ${schedule.frequency}, ${schedule.time}, ${schedule.timezone}, ${schedule.dayOfWeek}, ${schedule.dayOfMonth}, ${nextRunAt})
|
|
21
|
+
ON CONFLICT (tenant_id) DO UPDATE SET
|
|
22
|
+
enabled = EXCLUDED.enabled, frequency = EXCLUDED.frequency,
|
|
23
|
+
time_of_day = EXCLUDED.time_of_day, timezone = EXCLUDED.timezone,
|
|
24
|
+
day_of_week = EXCLUDED.day_of_week, day_of_month = EXCLUDED.day_of_month,
|
|
25
|
+
next_run_at = EXCLUDED.next_run_at, updated_at = now()
|
|
26
|
+
RETURNING *
|
|
27
|
+
`))[0];
|
|
28
|
+
}
|
|
29
|
+
|
|
30
|
+
async claimDue(limit, leaseMs) {
|
|
31
|
+
const leaseUntil = new Date(Date.now() + leaseMs);
|
|
32
|
+
const batchSize = Math.min(Math.max(Number(limit) || 10, 1), 50);
|
|
33
|
+
return rows(await this.db.execute(sql`
|
|
34
|
+
WITH due AS (
|
|
35
|
+
SELECT tenant_id, next_run_at AS scheduled_for
|
|
36
|
+
FROM ${sql.raw(this.schedules)}
|
|
37
|
+
WHERE enabled = true AND next_run_at IS NOT NULL AND next_run_at <= now()
|
|
38
|
+
ORDER BY next_run_at
|
|
39
|
+
FOR UPDATE SKIP LOCKED
|
|
40
|
+
LIMIT ${batchSize}
|
|
41
|
+
)
|
|
42
|
+
UPDATE ${sql.raw(this.schedules)} s
|
|
43
|
+
SET next_run_at = ${leaseUntil}, updated_at = now()
|
|
44
|
+
FROM due
|
|
45
|
+
WHERE s.tenant_id = due.tenant_id
|
|
46
|
+
RETURNING s.*, due.scheduled_for
|
|
47
|
+
`));
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
async setNextRun(tenantId, nextRunAt, completed = false, leaseUntil = null) {
|
|
51
|
+
await this.db.execute(sql`
|
|
52
|
+
UPDATE ${sql.raw(this.schedules)}
|
|
53
|
+
SET next_run_at = ${nextRunAt}, last_run_at = CASE WHEN ${completed} THEN now() ELSE last_run_at END, updated_at = now()
|
|
54
|
+
WHERE tenant_id = ${tenantId}
|
|
55
|
+
AND (${leaseUntil}::timestamptz IS NULL OR next_run_at = ${leaseUntil})
|
|
56
|
+
`);
|
|
57
|
+
}
|
|
58
|
+
|
|
59
|
+
async createReport(schedule, range) {
|
|
60
|
+
const inserted = rows(await this.db.execute(sql`
|
|
61
|
+
INSERT INTO ${sql.raw(this.reports)}
|
|
62
|
+
(id, tenant_id, schedule_frequency, scheduled_for, period_start, period_end)
|
|
63
|
+
VALUES (${randomUUID()}, ${schedule.tenant_id}, ${schedule.frequency}, ${schedule.scheduled_for}, ${range.Start}, ${range.End})
|
|
64
|
+
ON CONFLICT (tenant_id, scheduled_for) DO NOTHING
|
|
65
|
+
RETURNING *
|
|
66
|
+
`));
|
|
67
|
+
return inserted[0] || null;
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
async markRunning(id) {
|
|
71
|
+
return rows(await this.db.execute(sql`UPDATE ${sql.raw(this.reports)} SET status = 'RUNNING', attempt_count = attempt_count + 1, started_at = now(), error = NULL, updated_at = now() WHERE id = ${id} AND status = 'PENDING' RETURNING *`))[0] || null;
|
|
72
|
+
}
|
|
73
|
+
|
|
74
|
+
async setTaskArn(id, taskArn) {
|
|
75
|
+
await this.db.execute(sql`UPDATE ${sql.raw(this.reports)} SET ecs_task_arn = ${taskArn}, updated_at = now() WHERE id = ${id}`);
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
async completeExternal(id, result) {
|
|
79
|
+
await this.db.execute(sql`
|
|
80
|
+
UPDATE ${sql.raw(this.reports)} SET status = 'COMPLETED', ecs_task_arn = ${result.taskArn},
|
|
81
|
+
model_id = ${result.modelId}, data_source = ${result.dataSource}, facts = ${JSON.stringify(result.facts || {})}::jsonb,
|
|
82
|
+
result = ${JSON.stringify(result.summary || {})}::jsonb, artifact_bucket = ${result.artifactBucket},
|
|
83
|
+
markdown_key = ${result.markdownKey}, markdown_size = ${Number(result.markdownSize || 0)},
|
|
84
|
+
pdf_key = ${result.pdfKey}, pdf_size = ${Number(result.pdfSize || 0)}, pdf_data = NULL,
|
|
85
|
+
completed_at = now(), updated_at = now()
|
|
86
|
+
WHERE id = ${id}
|
|
87
|
+
`);
|
|
88
|
+
}
|
|
89
|
+
|
|
90
|
+
async complete(id, analysis, pdf) {
|
|
91
|
+
await this.db.execute(sql`
|
|
92
|
+
UPDATE ${sql.raw(this.reports)} SET status = 'COMPLETED', model_id = ${analysis.modelId}, data_source = ${analysis.dataSource || null},
|
|
93
|
+
facts = ${JSON.stringify(analysis.facts)}::jsonb,
|
|
94
|
+
result = ${JSON.stringify({ summary: analysis.summary, findings: analysis.findings, recommendations: analysis.recommendations, limitations: analysis.limitations })}::jsonb,
|
|
95
|
+
pdf_data = ${pdf}, pdf_size = ${pdf.length}, completed_at = now(), updated_at = now()
|
|
96
|
+
WHERE id = ${id}
|
|
97
|
+
`);
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
async fail(id, error) {
|
|
101
|
+
await this.db.execute(sql`UPDATE ${sql.raw(this.reports)} SET status = 'FAILED', ecs_task_arn = COALESCE(${error?.taskArn || null}, ecs_task_arn), error = ${String(error?.message || error).slice(0, 2000)}, completed_at = now(), updated_at = now() WHERE id = ${id}`);
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
async list(tenantId, limit = 100) {
|
|
105
|
+
return rows(await this.db.execute(sql`
|
|
106
|
+
SELECT id, schedule_frequency, scheduled_for, period_start, period_end, status,
|
|
107
|
+
attempt_count, model_id, data_source, facts, result, artifact_bucket,
|
|
108
|
+
markdown_key, markdown_size, pdf_key, pdf_size, error,
|
|
109
|
+
started_at, completed_at, created_at
|
|
110
|
+
FROM ${sql.raw(this.reports)} WHERE tenant_id = ${tenantId}
|
|
111
|
+
ORDER BY scheduled_for DESC LIMIT ${Math.min(Math.max(Number(limit) || 100, 1), 250)}
|
|
112
|
+
`));
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
async find(tenantId, id, includePdf = false) {
|
|
116
|
+
if (includePdf) {
|
|
117
|
+
return rows(await this.db.execute(sql`SELECT * FROM ${sql.raw(this.reports)} WHERE tenant_id = ${tenantId} AND id = ${id} LIMIT 1`))[0] || null;
|
|
118
|
+
}
|
|
119
|
+
return rows(await this.db.execute(sql`
|
|
120
|
+
SELECT id, schedule_frequency, scheduled_for, period_start, period_end, status,
|
|
121
|
+
attempt_count, model_id, data_source, facts, result, artifact_bucket,
|
|
122
|
+
markdown_key, markdown_size, pdf_key, pdf_size, error,
|
|
123
|
+
started_at, completed_at, created_at
|
|
124
|
+
FROM ${sql.raw(this.reports)} WHERE tenant_id = ${tenantId} AND id = ${id} LIMIT 1
|
|
125
|
+
`))[0] || null;
|
|
126
|
+
}
|
|
127
|
+
}
|
package/src/routes/index.js
CHANGED
|
@@ -11,7 +11,12 @@ export function createCostRouter({ costController, budgetController, costAnalysi
|
|
|
11
11
|
router.get("/cur-discovery", costController.curDiscovery.bind(costController));
|
|
12
12
|
router.get("/analysis/status", costAnalysisController.status.bind(costAnalysisController));
|
|
13
13
|
router.get("/analysis/latest", costAnalysisController.latest.bind(costAnalysisController));
|
|
14
|
-
router.
|
|
14
|
+
router.get("/analysis/schedule", costAnalysisController.schedule.bind(costAnalysisController));
|
|
15
|
+
router.put("/analysis/schedule", costAnalysisController.updateSchedule.bind(costAnalysisController));
|
|
16
|
+
router.get("/analysis/reports", costAnalysisController.reports.bind(costAnalysisController));
|
|
17
|
+
router.get("/analysis/reports/:id", costAnalysisController.report.bind(costAnalysisController));
|
|
18
|
+
router.get("/analysis/reports/:id/pdf", costAnalysisController.downloadPdf.bind(costAnalysisController));
|
|
19
|
+
router.get("/analysis/reports/:id/markdown", costAnalysisController.downloadMarkdown.bind(costAnalysisController));
|
|
15
20
|
router.get("/budgets", budgetController.list.bind(budgetController));
|
|
16
21
|
router.post("/budgets", budgetController.create.bind(budgetController));
|
|
17
22
|
router.delete("/budgets/:id", budgetController.delete.bind(budgetController));
|
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
import { sql } from "drizzle-orm";
|
|
2
|
+
|
|
3
|
+
export async function installCostAnalysisReportSchema(db, qSchema) {
|
|
4
|
+
await db.execute(sql.raw(`CREATE TABLE IF NOT EXISTS ${qSchema}.cost_ai_report_schedules (
|
|
5
|
+
tenant_id text PRIMARY KEY,
|
|
6
|
+
enabled boolean NOT NULL DEFAULT false,
|
|
7
|
+
frequency text NOT NULL DEFAULT 'weekly' CHECK (frequency IN ('daily', 'weekly', 'monthly')),
|
|
8
|
+
time_of_day time NOT NULL DEFAULT '09:00:00',
|
|
9
|
+
timezone text NOT NULL DEFAULT 'UTC',
|
|
10
|
+
day_of_week smallint CHECK (day_of_week BETWEEN 0 AND 6),
|
|
11
|
+
day_of_month smallint CHECK (day_of_month BETWEEN 1 AND 28),
|
|
12
|
+
next_run_at timestamptz,
|
|
13
|
+
last_run_at timestamptz,
|
|
14
|
+
created_at timestamptz NOT NULL DEFAULT now(),
|
|
15
|
+
updated_at timestamptz NOT NULL DEFAULT now()
|
|
16
|
+
)`));
|
|
17
|
+
await db.execute(sql.raw(`CREATE INDEX IF NOT EXISTS cost_ai_report_schedules_due_idx ON ${qSchema}.cost_ai_report_schedules (enabled, next_run_at)`));
|
|
18
|
+
|
|
19
|
+
await db.execute(sql.raw(`CREATE TABLE IF NOT EXISTS ${qSchema}.cost_ai_reports (
|
|
20
|
+
id text PRIMARY KEY,
|
|
21
|
+
tenant_id text NOT NULL,
|
|
22
|
+
schedule_frequency text NOT NULL CHECK (schedule_frequency IN ('daily', 'weekly', 'monthly')),
|
|
23
|
+
scheduled_for timestamptz NOT NULL,
|
|
24
|
+
period_start date NOT NULL,
|
|
25
|
+
period_end date NOT NULL,
|
|
26
|
+
status text NOT NULL DEFAULT 'PENDING' CHECK (status IN ('PENDING', 'RUNNING', 'COMPLETED', 'FAILED')),
|
|
27
|
+
attempt_count integer NOT NULL DEFAULT 0,
|
|
28
|
+
model_id text,
|
|
29
|
+
data_source text,
|
|
30
|
+
facts jsonb,
|
|
31
|
+
result jsonb,
|
|
32
|
+
pdf_data bytea,
|
|
33
|
+
pdf_size integer,
|
|
34
|
+
ecs_task_arn text,
|
|
35
|
+
artifact_bucket text,
|
|
36
|
+
markdown_key text,
|
|
37
|
+
markdown_size integer,
|
|
38
|
+
pdf_key text,
|
|
39
|
+
error text,
|
|
40
|
+
started_at timestamptz,
|
|
41
|
+
completed_at timestamptz,
|
|
42
|
+
created_at timestamptz NOT NULL DEFAULT now(),
|
|
43
|
+
updated_at timestamptz NOT NULL DEFAULT now(),
|
|
44
|
+
UNIQUE (tenant_id, scheduled_for)
|
|
45
|
+
)`));
|
|
46
|
+
await db.execute(sql.raw(`CREATE INDEX IF NOT EXISTS cost_ai_reports_tenant_created_idx ON ${qSchema}.cost_ai_reports (tenant_id, created_at DESC)`));
|
|
47
|
+
await db.execute(sql.raw(`CREATE INDEX IF NOT EXISTS cost_ai_reports_status_idx ON ${qSchema}.cost_ai_reports (status, created_at)`));
|
|
48
|
+
await db.execute(sql.raw(`ALTER TABLE ${qSchema}.cost_ai_reports ADD COLUMN IF NOT EXISTS ecs_task_arn text`));
|
|
49
|
+
await db.execute(sql.raw(`ALTER TABLE ${qSchema}.cost_ai_reports ADD COLUMN IF NOT EXISTS artifact_bucket text`));
|
|
50
|
+
await db.execute(sql.raw(`ALTER TABLE ${qSchema}.cost_ai_reports ADD COLUMN IF NOT EXISTS markdown_key text`));
|
|
51
|
+
await db.execute(sql.raw(`ALTER TABLE ${qSchema}.cost_ai_reports ADD COLUMN IF NOT EXISTS markdown_size integer`));
|
|
52
|
+
await db.execute(sql.raw(`ALTER TABLE ${qSchema}.cost_ai_reports ADD COLUMN IF NOT EXISTS pdf_key text`));
|
|
53
|
+
}
|