@toa.io/extensions.telemetry 1.0.0-alpha.303 → 1.0.0-alpha.305
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/grafana.md +34 -15
- package/package.json +6 -6
- package/readme.md +30 -3
- package/transpiled/extension.d.ts +1 -1
- package/transpiled/extension.js +29 -13
- package/transpiled/extension.js.map +1 -1
package/grafana.md
CHANGED
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
# Grafana stack setup
|
|
2
2
|
|
|
3
|
-
Toa-specific notes for wiring application traces and
|
|
4
|
-
Reference configuration: [`observability/`](../../observability) in the Toa
|
|
3
|
+
Toa-specific notes for wiring application traces, metrics and logs into Tempo, Prometheus, Loki
|
|
4
|
+
and Grafana. Reference configuration: [`observability/`](../../observability) in the Toa
|
|
5
|
+
repository.
|
|
5
6
|
|
|
6
7
|
## Local environment
|
|
7
8
|
|
|
@@ -11,8 +12,11 @@ The application environment needs:
|
|
|
11
12
|
(see [`tempo.yaml`](../../observability/tempo.yaml) for the metrics-generator wiring)
|
|
12
13
|
- Prometheus with `--web.enable-remote-write-receiver`, which is where Tempo's generator
|
|
13
14
|
writes, and `--web.enable-otlp-receiver`, which is where Toa posts its own metrics
|
|
14
|
-
-
|
|
15
|
-
|
|
15
|
+
- [Loki](https://grafana.com/oss/loki/) with the OTLP receiver (`/otlp/v1/logs`), on the `tsdb`
|
|
16
|
+
store and schema `v13` — structured metadata, which is where record attributes land, is legal
|
|
17
|
+
only there (see [`loki.yaml`](../../observability/loki.yaml))
|
|
18
|
+
- Grafana with a Prometheus datasource, a Tempo datasource with `serviceMap.datasourceUid`
|
|
19
|
+
pointing at it, and a Loki datasource
|
|
16
20
|
(see [`grafana-datasources.yaml`](../../observability/grafana-datasources.yaml))
|
|
17
21
|
|
|
18
22
|
Enable the export in the application context:
|
|
@@ -30,6 +34,10 @@ telemetry:
|
|
|
30
34
|
exporters:
|
|
31
35
|
otlp:
|
|
32
36
|
endpoint: http://localhost:9090/api/v1/otlp
|
|
37
|
+
logs:
|
|
38
|
+
exporters:
|
|
39
|
+
otlp:
|
|
40
|
+
endpoint: http://localhost:3100/otlp
|
|
33
41
|
```
|
|
34
42
|
|
|
35
43
|
[`dashboards/toa.json`](../../observability/dashboards/toa.json) reads those metrics: operations,
|
|
@@ -38,11 +46,15 @@ exposition, the outbox, the broker, storage, what is in flight and the event loo
|
|
|
38
46
|
## Production
|
|
39
47
|
|
|
40
48
|
- Point `traces.exporters.otlp.endpoint` at any OTLP/HTTP receiver (`/v1/traces` is appended),
|
|
41
|
-
|
|
49
|
+
`metrics.exporters.otlp.endpoint` at one that takes metrics (`/v1/metrics` is appended), and
|
|
50
|
+
`logs.exporters.otlp.endpoint` at one that takes logs (`/v1/logs` is appended).
|
|
42
51
|
Use `headers` for authentication (e.g. Grafana Cloud `Authorization: Basic ...`).
|
|
43
|
-
- Metrics carry the context as `service.namespace` and the environment as
|
|
52
|
+
- Metrics and logs carry the context as `service.namespace` and the environment as
|
|
44
53
|
`deployment.environment.name`, so several products and several environments reaching one
|
|
45
54
|
backend separate by `job` rather than by metric name.
|
|
55
|
+
- The console log exporter stays on unless `logs.exporters.console: false` says otherwise. Turn it
|
|
56
|
+
off where the node also scrapes the pod's stdout into the same backend, or every record arrives
|
|
57
|
+
twice.
|
|
46
58
|
- Skip the usual per-process `service.name` setup (`OTEL_SERVICE_NAME` and the like):
|
|
47
59
|
Toa attributes spans to component ids (`default.orders`) or `exposition` automatically,
|
|
48
60
|
even when multiple components run in a single process.
|
|
@@ -53,27 +65,34 @@ exposition, the outbox, the broker, storage, what is in flight and the event loo
|
|
|
53
65
|
- Sampling is head-based and decided by Toa (see [Sampling](readme.md#sampling)),
|
|
54
66
|
so no tail sampling is required on the collector side; `sample` and `rate`
|
|
55
67
|
are the knobs controlling the exported volume.
|
|
56
|
-
- Spans are sent in batches (512
|
|
57
|
-
abrupt termination (`SIGKILL`) may lose the last batch.
|
|
68
|
+
- Spans and log records are sent in batches (512 of them or every 5 seconds) and flushed on
|
|
69
|
+
process exit; abrupt termination (`SIGKILL`) may lose the last batch.
|
|
58
70
|
- Metrics are cumulative, so a lost export costs resolution and not totals: the next one carries
|
|
59
71
|
what the lost one would have.
|
|
60
72
|
|
|
61
73
|
## Trace to logs
|
|
62
74
|
|
|
63
75
|
Toa stamps `trace_id` and `span_id` into every log entry written within a span
|
|
64
|
-
(including unsampled traces), so
|
|
65
|
-
traces and logs link up with two datasource settings:
|
|
76
|
+
(including unsampled traces), so traces and logs link up with two datasource settings.
|
|
66
77
|
|
|
67
|
-
|
|
68
|
-
|
|
78
|
+
Exported over OTLP, a record's line is the message alone and everything else is a label or
|
|
79
|
+
structured metadata beside it — Loki maps `service.name` to the `service_name` label and the
|
|
80
|
+
record's trace and span ids to `trace_id` and `span_id`:
|
|
81
|
+
|
|
82
|
+
- Tempo datasource, _Trace to logs_: select the Loki datasource, set time shifts (e.g.
|
|
83
|
+
`-5m`/`+5m`), and a custom query, as the trace id is not in the line:
|
|
69
84
|
|
|
70
85
|
```logql
|
|
71
|
-
{
|
|
86
|
+
{service_name="$SERVICE"} | trace_id = `${__span.traceId}`
|
|
72
87
|
```
|
|
73
88
|
|
|
74
89
|
Append `` | span_id = `${__span.spanId}` `` to narrow down to a single span.
|
|
75
90
|
|
|
76
|
-
- Loki datasource, _Derived fields_: regex `
|
|
77
|
-
|
|
91
|
+
- Loki datasource, _Derived fields_: matcher type _Label_, regex `trace_id`, query
|
|
92
|
+
`${__value.raw}`, internal link to the Tempo datasource.
|
|
93
|
+
|
|
94
|
+
Where the records are scraped off the pod's stdout instead, the line is the whole JSON entry, and
|
|
95
|
+
the two settings read it as such: `| json | trace_id = ...` in the query, and a derived field
|
|
96
|
+
matching the regular expression `"trace_id":"([0-9a-f]+)"`.
|
|
78
97
|
|
|
79
98
|
Logs of unsampled traces carry a `trace_id` that does not exist in Tempo.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@toa.io/extensions.telemetry",
|
|
3
|
-
"version": "1.0.0-alpha.
|
|
3
|
+
"version": "1.0.0-alpha.305",
|
|
4
4
|
"type": "module",
|
|
5
5
|
"description": "Toa Telemetry extension",
|
|
6
6
|
"author": "temich <tema.gurtovoy@gmail.com>",
|
|
@@ -29,10 +29,10 @@
|
|
|
29
29
|
"transpile": "tsc"
|
|
30
30
|
},
|
|
31
31
|
"dependencies": {
|
|
32
|
-
"@toa.io/core": "1.0.0-alpha.
|
|
33
|
-
"@toa.io/definitions": "1.0.0-alpha.
|
|
34
|
-
"@toa.io/generic": "1.0.0-alpha.
|
|
35
|
-
"openspan": "1.0.0-alpha.
|
|
32
|
+
"@toa.io/core": "1.0.0-alpha.305",
|
|
33
|
+
"@toa.io/definitions": "1.0.0-alpha.305",
|
|
34
|
+
"@toa.io/generic": "1.0.0-alpha.304",
|
|
35
|
+
"openspan": "1.0.0-alpha.305"
|
|
36
36
|
},
|
|
37
|
-
"gitHead": "
|
|
37
|
+
"gitHead": "6a28476303826e2484cc2ea6306efee6f19fb594"
|
|
38
38
|
}
|
package/readme.md
CHANGED
|
@@ -60,6 +60,31 @@ telemetry:
|
|
|
60
60
|
level: debug
|
|
61
61
|
```
|
|
62
62
|
|
|
63
|
+
### Logs exporting
|
|
64
|
+
|
|
65
|
+
An entry goes to every configured log exporter. The console writes the JSON line and is on unless
|
|
66
|
+
it is turned off; `otlp` posts entries to an OTLP/HTTP endpoint and is off until one is
|
|
67
|
+
configured. The two are independent: configuring one says nothing about the other.
|
|
68
|
+
|
|
69
|
+
```yaml
|
|
70
|
+
# context.toa.yaml
|
|
71
|
+
|
|
72
|
+
telemetry:
|
|
73
|
+
logs:
|
|
74
|
+
level: info
|
|
75
|
+
exporters:
|
|
76
|
+
console: false # stops the JSON line being written; on when omitted
|
|
77
|
+
otlp:
|
|
78
|
+
endpoint: http://loki:3100/otlp # POSTs to {endpoint}/v1/logs
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
The severity threshold is the one above, per component: what a process writes is what it exports.
|
|
82
|
+
A missing or unavailable endpoint costs a single warning and dropped entries, and never the
|
|
83
|
+
throughput or the shutdown.
|
|
84
|
+
|
|
85
|
+
See [Logs](/documentation/logs.md) for what a record carries and the resource it is exported
|
|
86
|
+
under.
|
|
87
|
+
|
|
63
88
|
## Tracing
|
|
64
89
|
|
|
65
90
|
Each operation invocation runs within a _span_.
|
|
@@ -266,7 +291,9 @@ Logs are not comments or documentation, nor are they a replacement for them.
|
|
|
266
291
|
:-1: Don't:
|
|
267
292
|
|
|
268
293
|
```javascript
|
|
269
|
-
context.logs.error(
|
|
294
|
+
context.logs.error(
|
|
295
|
+
'Failed to send the email, please check the email server configuration'
|
|
296
|
+
)
|
|
270
297
|
```
|
|
271
298
|
|
|
272
299
|
:+1: Do:
|
|
@@ -274,7 +301,7 @@ context.logs.error('Failed to send the email, please check the email server conf
|
|
|
274
301
|
```javascript
|
|
275
302
|
context.logs.error('Failed to send the email', {
|
|
276
303
|
reason: 'SMTP error',
|
|
277
|
-
status: response.statusCode
|
|
304
|
+
status: response.statusCode
|
|
278
305
|
})
|
|
279
306
|
```
|
|
280
307
|
|
|
@@ -315,7 +342,7 @@ context.logs.debug('Configuration', context.configuration)
|
|
|
315
342
|
```javascript
|
|
316
343
|
context.logs.debug('Limits', {
|
|
317
344
|
max: context.configuration.limits.max,
|
|
318
|
-
min: context.configuration.limits.min
|
|
345
|
+
min: context.configuration.limits.min
|
|
319
346
|
})
|
|
320
347
|
```
|
|
321
348
|
|
|
@@ -3,7 +3,7 @@ import type { Locator } from '@toa.io/core';
|
|
|
3
3
|
import type { extensions } from '@toa.io/core/types';
|
|
4
4
|
type Resident = extensions.Resident;
|
|
5
5
|
export declare class Factory implements extensions.Factory {
|
|
6
|
-
private readonly
|
|
6
|
+
private readonly logsConfig;
|
|
7
7
|
private readonly ready;
|
|
8
8
|
constructor();
|
|
9
9
|
aspect(locator: Locator, declaration: Declaration): extensions.Aspect[];
|
package/transpiled/extension.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { console, metrics, traces } from 'openspan';
|
|
1
|
+
import { console, logs, metrics, traces } from 'openspan';
|
|
2
2
|
import { environment } from '@toa.io/generic';
|
|
3
3
|
import { LOGS_PREFIX, METRICS_ENV, TRACES_ENV } from '@toa.io/definitions/extensions.telemetry';
|
|
4
4
|
import { Logs } from './Logs.js';
|
|
@@ -6,14 +6,16 @@ import { Span } from './Span.js';
|
|
|
6
6
|
import { Metrics } from './Metrics.js';
|
|
7
7
|
import { Ready } from './Ready.js';
|
|
8
8
|
export class Factory {
|
|
9
|
-
|
|
9
|
+
logsConfig;
|
|
10
10
|
ready;
|
|
11
11
|
constructor() {
|
|
12
12
|
const globEnv = environment.get(LOGS_PREFIX);
|
|
13
13
|
const level = environment.get('TOA_DEV') === '1' ? 'trace' : 'info';
|
|
14
|
-
this.
|
|
15
|
-
this.
|
|
16
|
-
console.configure({ level: this.
|
|
14
|
+
this.logsConfig = globEnv === undefined ? { level } : JSON.parse(globEnv);
|
|
15
|
+
this.logsConfig.level ??= level;
|
|
16
|
+
console.configure({ level: this.logsConfig.level });
|
|
17
|
+
// after the level, which is what decides whether an entry is written at all
|
|
18
|
+
logs({ exporters: records(this.logsConfig.exporters) });
|
|
17
19
|
const tracesEnv = environment.get(TRACES_ENV);
|
|
18
20
|
const options = tracesEnv === undefined ? development() : JSON.parse(tracesEnv);
|
|
19
21
|
// `openspan` is not Toa's and reads the environment, which no longer holds the context by
|
|
@@ -45,19 +47,24 @@ export class Factory {
|
|
|
45
47
|
createLogs(locator) {
|
|
46
48
|
const overEnv = environment.get(`${LOGS_PREFIX}_${locator.uppercase}`);
|
|
47
49
|
const override = overEnv !== undefined ? JSON.parse(overEnv) : undefined;
|
|
48
|
-
const { level } = Object.assign({}, this.
|
|
50
|
+
const { level } = Object.assign({}, this.logsConfig, override);
|
|
49
51
|
return new Logs(locator, { level });
|
|
50
52
|
}
|
|
51
53
|
}
|
|
52
54
|
/**
|
|
53
|
-
*
|
|
54
|
-
*
|
|
55
|
-
*
|
|
56
|
-
* traces annotates `telemetry.traces.exporters`.
|
|
57
|
-
*
|
|
58
|
-
* `extensions/exposition/source/Factory.ts` says the same thing for the gateway process,
|
|
59
|
-
* which boots without this extension.
|
|
55
|
+
* What the records say they came from. The same resource the series carry, so that one backend
|
|
56
|
+
* holding several products and several environments separates them by the same attributes
|
|
57
|
+
* whichever signal is being read.
|
|
60
58
|
*/
|
|
59
|
+
function records(exporters) {
|
|
60
|
+
if (exporters?.otlp !== undefined) {
|
|
61
|
+
const resource = (exporters.otlp.resource ??= {});
|
|
62
|
+
resource['service.name'] ??= environment.get('TOA_CONTEXT');
|
|
63
|
+
resource['service.namespace'] ??= environment.get('TOA_CONTEXT');
|
|
64
|
+
resource['deployment.environment.name'] ??= environment.get('TOA_ENV');
|
|
65
|
+
}
|
|
66
|
+
return exporters;
|
|
67
|
+
}
|
|
61
68
|
/**
|
|
62
69
|
* What the series say they came from. The metric names are the same in every deployment, so what
|
|
63
70
|
* tells two products apart in one backend is the resource: the context is the namespace, and the
|
|
@@ -79,6 +86,15 @@ function measurements() {
|
|
|
79
86
|
}
|
|
80
87
|
return options;
|
|
81
88
|
}
|
|
89
|
+
/**
|
|
90
|
+
* Tracing is off unless it is configured. The console exporter is a local development
|
|
91
|
+
* mechanism, so it is turned on for `toa dev` and for a boot trace the CLI has already
|
|
92
|
+
* asked for (`runtime/boot/src/span.js`), and nowhere else — a deployment that wants
|
|
93
|
+
* traces annotates `telemetry.traces.exporters`.
|
|
94
|
+
*
|
|
95
|
+
* `extensions/exposition/source/Factory.ts` says the same thing for the gateway process,
|
|
96
|
+
* which boots without this extension.
|
|
97
|
+
*/
|
|
82
98
|
function development() {
|
|
83
99
|
const local = environment.get('TOA_DEV') === '1' || environment.get('TOA_BOOT_TRACE') === '1';
|
|
84
100
|
return local ? { exporters: { console: {} } } : {};
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"extension.js","sourceRoot":"","sources":["../source/extension.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,UAAU,CAAA;
|
|
1
|
+
{"version":3,"file":"extension.js","sourceRoot":"","sources":["../source/extension.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,UAAU,CAAA;AACzD,OAAO,EAAE,WAAW,EAAE,MAAM,iBAAiB,CAAA;AAC7C,OAAO,EACL,WAAW,EACX,WAAW,EACX,UAAU,EACX,MAAM,0CAA0C,CAAA;AACjD,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAChC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAA;AAChC,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAA;AACtC,OAAO,EAAE,KAAK,EAAE,MAAM,YAAY,CAAA;AASlC,MAAM,OAAO,OAAO;IACD,UAAU,CAAY;IACtB,KAAK,CAAc;IAEpC;QACE,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,CAAC,WAAW,CAAC,CAAA;QAC5C,MAAM,KAAK,GAAG,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,GAAG,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAA;QAEnE,IAAI,CAAC,UAAU,GAAG,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAA;QACzE,IAAI,CAAC,UAAU,CAAC,KAAK,KAAK,KAAK,CAAA;QAE/B,OAAO,CAAC,SAAS,CAAC,EAAE,KAAK,EAAE,IAAI,CAAC,UAAU,CAAC,KAAK,EAAE,CAAC,CAAA;QAEnD,4EAA4E;QAC5E,IAAI,CAAC,EAAE,SAAS,EAAE,OAAO,CAAC,IAAI,CAAC,UAAU,CAAC,SAAS,CAAC,EAAE,CAAC,CAAA;QAEvD,MAAM,SAAS,GAAG,WAAW,CAAC,GAAG,CAAC,UAAU,CAAC,CAAA;QAC7C,MAAM,OAAO,GACX,SAAS,KAAK,SAAS,CAAC,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAE,IAAI,CAAC,KAAK,CAAC,SAAS,CAAmB,CAAA;QAEpF,0FAA0F;QAC1F,+DAA+D;QAC/D,IAAI,OAAO,CAAC,SAAS,EAAE,IAAI,KAAK,SAAS;YACvC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC,OAAO,KAAK,WAAW,CAAC,GAAG,CAAC,aAAa,CAAC,CAAA;QAEnE,MAAM,CAAC,OAAO,CAAC,CAAA;QACf,OAAO,CAAC,YAAY,EAAE,CAAC,CAAA;QAEvB,IAAI,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,EAAE,CAAA;IAC7B,CAAC;IAEM,MAAM,CAAC,OAAgB,EAAE,WAAwB;QACtD,MAAM,IAAI,GAAG,IAAI,CAAC,UAAU,CAAC,OAAO,CAAC,CAAA;QACrC,MAAM,IAAI,GAAG,IAAI,IAAI,CAAC,OAAO,CAAC,CAAA;QAC9B,MAAM,OAAO,GAAG,IAAI,OAAO,CAAC,OAAO,EAAE,WAAW,IAAI,EAAE,CAAC,CAAA;QAEvD,OAAO,CAAC,IAAI,EAAE,IAAI,EAAE,OAAO,CAAC,CAAA;IAC9B,CAAC;IAED;;;;;;;;OAQG;IACI,QAAQ;QACb,OAAO,IAAI,CAAC,KAAK,CAAA;IACnB,CAAC;IAEO,UAAU,CAAC,OAAgB;QACjC,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,CAAC,GAAG,WAAW,IAAI,OAAO,CAAC,SAAS,EAAE,CAAC,CAAA;QACtE,MAAM,QAAQ,GAAG,OAAO,KAAK,SAAS,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,SAAS,CAAA;QAExE,MAAM,EAAE,KAAK,EAAE,GAAG,MAAM,CAAC,MAAM,CAAC,EAAE,EAAE,IAAI,CAAC,UAAU,EAAE,QAAQ,CAAC,CAAA;QAE9D,OAAO,IAAI,IAAI,CAAC,OAAO,EAAE,EAAE,KAAK,EAAE,CAAC,CAAA;IACrC,CAAC;CACF;AAED;;;;GAIG;AACH,SAAS,OAAO,CAAC,SAA8B;IAC7C,IAAI,SAAS,EAAE,IAAI,KAAK,SAAS,EAAE,CAAC;QAClC,MAAM,QAAQ,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,QAAQ,KAAK,EAAE,CAAC,CAAA;QAEjD,QAAQ,CAAC,cAAc,CAAC,KAAK,WAAW,CAAC,GAAG,CAAC,aAAa,CAAC,CAAA;QAC3D,QAAQ,CAAC,mBAAmB,CAAC,KAAK,WAAW,CAAC,GAAG,CAAC,aAAa,CAAC,CAAA;QAChE,QAAQ,CAAC,6BAA6B,CAAC,KAAK,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,CAAA;IACxE,CAAC;IAED,OAAO,SAAS,CAAA;AAClB,CAAC;AAED;;;;;;;GAOG;AACH,SAAS,YAAY;IACnB,MAAM,GAAG,GAAG,WAAW,CAAC,GAAG,CAAC,WAAW,CAAC,CAAA;IAExC,IAAI,GAAG,KAAK,SAAS;QAAE,OAAO,SAAS,CAAA;IAEvC,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,CAAmB,CAAA;IAEjD,OAAO,CAAC,MAAM,KAAK,KAAK,CAAA;IAExB,IAAI,OAAO,CAAC,SAAS,EAAE,IAAI,KAAK,SAAS,EAAE,CAAC;QAC1C,MAAM,QAAQ,GAAG,CAAC,OAAO,CAAC,SAAS,CAAC,IAAI,CAAC,QAAQ,KAAK,EAAE,CAAC,CAAA;QAEzD,QAAQ,CAAC,mBAAmB,CAAC,KAAK,WAAW,CAAC,GAAG,CAAC,aAAa,CAAC,CAAA;QAChE,QAAQ,CAAC,6BAA6B,CAAC,KAAK,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,CAAA;IACxE,CAAC;IAED,OAAO,OAAO,CAAA;AAChB,CAAC;AAMD;;;;;;;;GAQG;AACH,SAAS,WAAW;IAClB,MAAM,KAAK,GACT,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,KAAK,GAAG,IAAI,WAAW,CAAC,GAAG,CAAC,gBAAgB,CAAC,KAAK,GAAG,CAAA;IAEjF,OAAO,KAAK,CAAC,CAAC,CAAC,EAAE,SAAS,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,EAAE,CAAC,CAAC,CAAC,EAAE,CAAA;AACpD,CAAC"}
|