@forestadmin/agent-bff 1.32.0 → 1.33.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +107 -1
- package/dist/tracing-handle.d.ts +26 -0
- package/dist/tracing-handle.js +40 -0
- package/dist/tracing-preload.d.ts +2 -0
- package/dist/tracing-preload.js +27 -0
- package/dist/tracing.d.ts +79 -0
- package/dist/tracing.js +168 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -36,6 +36,112 @@ dropped collection reachable.
|
|
|
36
36
|
Two ways to run it: embedded in a Forest agent (`agent.addBff()`, see
|
|
37
37
|
[Embedded in an agent](#embedded-in-an-agent)) or standalone, described here.
|
|
38
38
|
|
|
39
|
+
### Docker (recommended)
|
|
40
|
+
|
|
41
|
+
```bash
|
|
42
|
+
cp .env.example .env # then fill in the secrets
|
|
43
|
+
docker compose up
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
The template targets a local, non-containerised run, so one value has to change for Docker:
|
|
47
|
+
set `AGENT_URL=http://host.docker.internal:3351`. Left at `localhost`, it resolves to the BFF
|
|
48
|
+
container itself and every agent call fails (see the note below).
|
|
49
|
+
|
|
50
|
+
The `docker-compose.yml` at the root of this package starts a single BFF instance. See
|
|
51
|
+
`.env.example` for the full list of environment variables and their descriptions.
|
|
52
|
+
|
|
53
|
+
Or run the image directly:
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
docker run -d \
|
|
57
|
+
-p 3450:3450 \
|
|
58
|
+
--stop-timeout 15 \
|
|
59
|
+
--add-host host.docker.internal:host-gateway \
|
|
60
|
+
-e FOREST_AUTH_SECRET="..." \
|
|
61
|
+
-e FOREST_ENV_SECRET="..." \
|
|
62
|
+
-e FOREST_SERVER_URL="https://api.forestadmin.com" \
|
|
63
|
+
-e FOREST_APP_URL="https://app.forestadmin.com" \
|
|
64
|
+
-e AGENT_URL="http://host.docker.internal:3351" \
|
|
65
|
+
-e BFF_TOKEN_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
|
|
66
|
+
ghcr.io/forestadmin/agent-bff:latest
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
> **Note:** When the BFF runs in Docker and your agent runs on the host machine, use
|
|
70
|
+
> `host.docker.internal` instead of `localhost` in `AGENT_URL`. Docker Desktop resolves that
|
|
71
|
+
> name natively; on Docker Engine for Linux it does not exist unless you map it, hence the
|
|
72
|
+
> `--add-host` above (the Compose setup does the same through `extra_hosts`).
|
|
73
|
+
|
|
74
|
+
The image's entry point is the CLI, so the subcommands below work the same way:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
docker run --rm ghcr.io/forestadmin/agent-bff:latest openapi > openapi.json
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Tags follow the npm package: `:latest`, `:1`, `:1.20` and the immutable `:1.20.2`.
|
|
81
|
+
|
|
82
|
+
The package is public, so none of the commands above need a login. That visibility is set once,
|
|
83
|
+
by hand, on the GHCR package: a package GHCR creates on its first push is private, and the
|
|
84
|
+
workflow's `GITHUB_TOKEN` can push to it but not change what it is. Until someone flips it (the
|
|
85
|
+
same step `workflow-executor` went through), pulls need
|
|
86
|
+
`docker login ghcr.io -u <user> -p <token-with-read:packages>`.
|
|
87
|
+
|
|
88
|
+
On `SIGTERM` or `SIGINT` the BFF stops accepting connections and gives the requests already in
|
|
89
|
+
flight 10 seconds to finish before cutting their sockets, then exits 0. A second signal gives up on
|
|
90
|
+
the wait and exits 1.
|
|
91
|
+
|
|
92
|
+
Allow for that in your orchestrator's grace period. The whole budget is up to 11 seconds — the 10
|
|
93
|
+
second deadline plus a 1 second fallback for the exit itself — and `docker stop` defaults to 10,
|
|
94
|
+
so under load it would SIGKILL exactly when the shutdown is doing its job. Hence `--stop-timeout 15`
|
|
95
|
+
above and `stop_grace_period: 15s` in the Compose file; on Kubernetes the default
|
|
96
|
+
`terminationGracePeriodSeconds` of 30 already covers it.
|
|
97
|
+
|
|
98
|
+
### Observability (OpenTelemetry)
|
|
99
|
+
|
|
100
|
+
The Docker image ships with [OpenTelemetry](https://opentelemetry.io/) APM built in, and works with
|
|
101
|
+
any OTLP-compatible backend (Datadog, Grafana Tempo, Jaeger, Honeycomb, etc.). It is **off by
|
|
102
|
+
default** and turns on as soon as you point it at an OTLP receiver — no code changes or extra
|
|
103
|
+
installs required. A setup that cannot start logs a warning and runs untraced rather than taking
|
|
104
|
+
the process down with it. Tracing is set up before the app starts (auto-instrumentation for HTTP and the
|
|
105
|
+
outbound calls to the agent and the Forest SaaS). The graceful shutdown described above waits for
|
|
106
|
+
the buffered spans to be exported before it exits, but gives that its own 2 second deadline rather
|
|
107
|
+
than the 10 seconds in-flight requests get: an unreachable collector costs you the last spans, never
|
|
108
|
+
the ability to stop. Worst case it adds ~3 seconds to a shutdown.
|
|
109
|
+
|
|
110
|
+
Configure it entirely through the standard OTel environment variables:
|
|
111
|
+
|
|
112
|
+
| Variable | Description |
|
|
113
|
+
| --- | --- |
|
|
114
|
+
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP receiver URL (e.g. `http://collector:4318`). **Tracing stays off until this or `OTEL_TRACES_EXPORTER` is set.** |
|
|
115
|
+
| `OTEL_SERVICE_NAME` | Service name reported in traces. Falls back to `service.name` in `OTEL_RESOURCE_ATTRIBUTES`, then to `forestadmin-agent-bff`. |
|
|
116
|
+
| `OTEL_RESOURCE_ATTRIBUTES` | Extra resource attributes, e.g. `deployment.environment=production`. A `service.name` here is honoured when `OTEL_SERVICE_NAME` is unset. |
|
|
117
|
+
| `OTEL_SDK_DISABLED` | Set to `true` (case-insensitive) to force-disable tracing whatever else is configured. |
|
|
118
|
+
| `OTEL_TRACES_EXPORTER` | Which exporter the SDK builds: `otlp` (the default), `console`, `zipkin`, `none`, or a list. Setting it alone turns tracing on without an OTLP endpoint, which is what makes `console` usable for debugging. `none` keeps instrumentation running with nothing exported, so trace context still propagates to the agent and the Forest SaaS. |
|
|
119
|
+
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | Per-signal endpoint, taking precedence over the generic one above. Setting either turns tracing on. |
|
|
120
|
+
| `OTEL_EXPORTER_OTLP_PROTOCOL` | `http/protobuf` (the default), `http/json` or `grpc`, with `OTEL_EXPORTER_OTLP_TRACES_PROTOCOL` for traces alone. |
|
|
121
|
+
| `OTEL_METRICS_EXPORTER` / `OTEL_LOGS_EXPORTER` | Default to `none` here, against the SDK's own `otlp`: this image arms tracing, and leaving them unset would otherwise export metrics and logs to `http://localhost:4318` on the side. Set either one (`otlp`, `console`, …) to opt that signal back in. |
|
|
122
|
+
|
|
123
|
+
```bash
|
|
124
|
+
docker run -d \
|
|
125
|
+
-p 3450:3450 \
|
|
126
|
+
--stop-timeout 15 \
|
|
127
|
+
--add-host host.docker.internal:host-gateway \
|
|
128
|
+
-e FOREST_AUTH_SECRET="..." \
|
|
129
|
+
-e FOREST_ENV_SECRET="..." \
|
|
130
|
+
-e FOREST_SERVER_URL="https://api.forestadmin.com" \
|
|
131
|
+
-e FOREST_APP_URL="https://app.forestadmin.com" \
|
|
132
|
+
-e AGENT_URL="http://host.docker.internal:3351" \
|
|
133
|
+
-e BFF_TOKEN_ENCRYPTION_KEY="$(openssl rand -base64 32)" \
|
|
134
|
+
-e OTEL_EXPORTER_OTLP_ENDPOINT="http://collector:4318" \
|
|
135
|
+
ghcr.io/forestadmin/agent-bff:latest
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
> **Note:** These variables only do anything in the Docker image. The image's entry point loads
|
|
139
|
+
> the tracing preload before the CLI; the npm `forest-bff` bin runs the CLI on its own, and the
|
|
140
|
+
> OpenTelemetry packages are not npm dependencies — so outside Docker an `OTEL_*` variable is
|
|
141
|
+
> read by nothing and the process starts untraced, silently.
|
|
142
|
+
|
|
143
|
+
### Without Docker
|
|
144
|
+
|
|
39
145
|
Packaged / production — run the bin:
|
|
40
146
|
|
|
41
147
|
```bash
|
|
@@ -99,7 +205,7 @@ yarn start:dev # node --env-file=.env dist/cli.js
|
|
|
99
205
|
| `FOREST_APP_URL` | yes | Forest front base URL, used to build the OAuth front-channel redirect (`src/oauth/oauth-routes.ts`). |
|
|
100
206
|
| `AGENT_URL` | yes | The customer agent base URL the BFF calls via agent-client. |
|
|
101
207
|
| `BFF_TOKEN_ENCRYPTION_KEY` | for OAuth | Base64-encoded 32-byte AES-256 key encrypting stored refresh tokens. Until it is set, the `/oauth/*` token-issuance routes are disabled and `/health` reports `configured.oauth: false` — but it stays `ok`, since the key gates OAuth and not boot; already-issued `bff_access` tokens still authenticate on `/agent/*` whenever `FOREST_AUTH_SECRET` is present. |
|
|
102
|
-
| `HTTP_PORT` | no | Server port, integer 0–65535. Defaults to `3450`. `0` binds an OS-assigned ephemeral port. |
|
|
208
|
+
| `HTTP_PORT` | no | Server port, integer 0–65535. Defaults to `3450`. `0` binds an OS-assigned ephemeral port — useful for a local run, unusable in the Docker image, where nothing outside the process learns which port it got: it can be neither published nor probed, and the image's healthcheck would report the container unhealthy forever. |
|
|
103
209
|
| `BFF_ALLOWED_ORIGINS` | no | Comma-separated CORS allow-list of origins (scheme + host + port). An entry may carry a single `*` as the leading host label — `https://*.apps.zdusercontent.com` — which matches exactly one DNS label there, and nothing else: not two labels, not the apex, and never the scheme or the port. The host left after `*.` must be at least two non-empty labels, so `https://*.com` is refused; a two-label public suffix such as `https://*.co.uk` is not, and would allow every site under it. Any other `*` in the host is refused and warned about at boot; a `*` outside the host — in userinfo, a path or a query — is stripped along with the rest of the URL, so `https://*@example.com` is simply the exact origin `https://example.com`. Empty ⇒ no cross-origin browser access. |
|
|
104
210
|
| `BFF_DEFAULT_TIMEZONE` | no | Fallback IANA timezone used when a request carries neither an `X-Forest-Timezone` header nor a body `timezone`. |
|
|
105
211
|
| `BFF_PUBLIC_URL` | no | The BFF's own external base URL, published as `servers[0].url` in the OpenAPI document so a generated client resolves endpoints without being configured by hand. Absent, `servers[0].url` stays `/`, which a consumer that fetched the document over HTTP resolves against that URL — but which leaves a client generated from an offline `forest-bff openapi` export with no base URL at all. Trailing slashes are stripped. A malformed value fails the boot, and so does one carrying credentials, a query string or a fragment: credentials would be published to every reader of the document, and anything behind a `?` or `#` swallows the path a generated client appends. |
|
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The one piece of state the preload and the CLI have to share.
|
|
3
|
+
*
|
|
4
|
+
* `--require` runs the preload in its own module, before `cli.js` is even loaded, so the SDK it
|
|
5
|
+
* builds has nowhere to go but a module both sides resolve to the same instance. Keeping it here —
|
|
6
|
+
* rather than having the preload arm a signal handler of its own — is what lets the shutdown path
|
|
7
|
+
* WAIT for the flush instead of racing it, and keeps the preload from consuming a signal the CLI
|
|
8
|
+
* has not armed a handler for yet.
|
|
9
|
+
*
|
|
10
|
+
* Nothing OpenTelemetry-specific is imported here, so the npm bin pays only for an empty variable.
|
|
11
|
+
*/
|
|
12
|
+
export interface TracingHandle {
|
|
13
|
+
shutdown(): Promise<void>;
|
|
14
|
+
}
|
|
15
|
+
export declare function setTracingHandle(sdk: TracingHandle | undefined): void;
|
|
16
|
+
export declare function getTracingHandle(): TracingHandle | undefined;
|
|
17
|
+
/**
|
|
18
|
+
* Flushes buffered spans, or resolves immediately when tracing was never armed — the shutdown path
|
|
19
|
+
* should not have to know which. A failing export is swallowed: a dead collector must not turn a
|
|
20
|
+
* clean shutdown into a failed one.
|
|
21
|
+
*
|
|
22
|
+
* No deadline here on purpose: `armShutdown` bounds whatever flush it is given, and it is the only
|
|
23
|
+
* caller. Bounding it twice would say the deadline lives in two places when it does not.
|
|
24
|
+
*/
|
|
25
|
+
export declare function flushTracing(): Promise<void>;
|
|
26
|
+
//# sourceMappingURL=tracing-handle.d.ts.map
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* The one piece of state the preload and the CLI have to share.
|
|
4
|
+
*
|
|
5
|
+
* `--require` runs the preload in its own module, before `cli.js` is even loaded, so the SDK it
|
|
6
|
+
* builds has nowhere to go but a module both sides resolve to the same instance. Keeping it here —
|
|
7
|
+
* rather than having the preload arm a signal handler of its own — is what lets the shutdown path
|
|
8
|
+
* WAIT for the flush instead of racing it, and keeps the preload from consuming a signal the CLI
|
|
9
|
+
* has not armed a handler for yet.
|
|
10
|
+
*
|
|
11
|
+
* Nothing OpenTelemetry-specific is imported here, so the npm bin pays only for an empty variable.
|
|
12
|
+
*/
|
|
13
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
14
|
+
exports.setTracingHandle = setTracingHandle;
|
|
15
|
+
exports.getTracingHandle = getTracingHandle;
|
|
16
|
+
exports.flushTracing = flushTracing;
|
|
17
|
+
let handle;
|
|
18
|
+
function setTracingHandle(sdk) {
|
|
19
|
+
handle = sdk;
|
|
20
|
+
}
|
|
21
|
+
function getTracingHandle() {
|
|
22
|
+
return handle;
|
|
23
|
+
}
|
|
24
|
+
/**
|
|
25
|
+
* Flushes buffered spans, or resolves immediately when tracing was never armed — the shutdown path
|
|
26
|
+
* should not have to know which. A failing export is swallowed: a dead collector must not turn a
|
|
27
|
+
* clean shutdown into a failed one.
|
|
28
|
+
*
|
|
29
|
+
* No deadline here on purpose: `armShutdown` bounds whatever flush it is given, and it is the only
|
|
30
|
+
* caller. Bounding it twice would say the deadline lives in two places when it does not.
|
|
31
|
+
*/
|
|
32
|
+
async function flushTracing() {
|
|
33
|
+
try {
|
|
34
|
+
await handle?.shutdown();
|
|
35
|
+
}
|
|
36
|
+
catch {
|
|
37
|
+
/* istanbul ignore next — nothing to do about it, and it must not change the exit code. */
|
|
38
|
+
}
|
|
39
|
+
}
|
|
40
|
+
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoidHJhY2luZy1oYW5kbGUuanMiLCJzb3VyY2VSb290IjoiIiwic291cmNlcyI6WyIuLi9zcmMvdHJhY2luZy1oYW5kbGUudHMiXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6IjtBQUFBOzs7Ozs7Ozs7O0dBVUc7O0FBUUgsNENBRUM7QUFFRCw0Q0FFQztBQVVELG9DQU1DO0FBeEJELElBQUksTUFBaUMsQ0FBQztBQUV0QyxTQUFnQixnQkFBZ0IsQ0FBQyxHQUE4QjtJQUM3RCxNQUFNLEdBQUcsR0FBRyxDQUFDO0FBQ2YsQ0FBQztBQUVELFNBQWdCLGdCQUFnQjtJQUM5QixPQUFPLE1BQU0sQ0FBQztBQUNoQixDQUFDO0FBRUQ7Ozs7Ozs7R0FPRztBQUNJLEtBQUssVUFBVSxZQUFZO0lBQ2hDLElBQUksQ0FBQztRQUNILE1BQU0sTUFBTSxFQUFFLFFBQVEsRUFBRSxDQUFDO0lBQzNCLENBQUM7SUFBQyxNQUFNLENBQUM7UUFDUCwwRkFBMEY7SUFDNUYsQ0FBQztBQUNILENBQUMifQ==
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
3
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
4
|
+
};
|
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
+
// The `--require` entry point of the Docker image (see the Dockerfile's ENTRYPOINT). Separate from
|
|
7
|
+
// `tracing.ts` on purpose: importing the setup must never arm an SDK, only calling it should.
|
|
8
|
+
//
|
|
9
|
+
// The SDK is handed to `tracing-handle` rather than wired to a signal here. The CLI arms the only
|
|
10
|
+
// termination handler, and it flushes through that handle — so a signal arriving before the CLI is
|
|
11
|
+
// up cannot be swallowed by a listener that does not terminate anything.
|
|
12
|
+
const console_logger_1 = __importDefault(require("./adapters/console-logger"));
|
|
13
|
+
const tracing_1 = __importDefault(require("./tracing"));
|
|
14
|
+
const tracing_handle_1 = require("./tracing-handle");
|
|
15
|
+
// Nothing here may throw. A `--require` runs before the entry point, so an exception aborts the
|
|
16
|
+
// process before the BFF exists at all — a dead container in exchange for optional, best-effort
|
|
17
|
+
// telemetry. `initTracing` degrades on its own for the cases it knows about; this catches the rest,
|
|
18
|
+
// including whatever a future SDK version decides to throw from `start()`.
|
|
19
|
+
try {
|
|
20
|
+
(0, tracing_handle_1.setTracingHandle)((0, tracing_1.default)());
|
|
21
|
+
}
|
|
22
|
+
catch (error) {
|
|
23
|
+
(0, console_logger_1.default)()('Warn', 'OpenTelemetry failed to initialise, starting untraced', {
|
|
24
|
+
reason: error instanceof Error ? error.message : String(error),
|
|
25
|
+
});
|
|
26
|
+
}
|
|
27
|
+
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoidHJhY2luZy1wcmVsb2FkLmpzIiwic291cmNlUm9vdCI6IiIsInNvdXJjZXMiOlsiLi4vc3JjL3RyYWNpbmctcHJlbG9hZC50cyJdLCJuYW1lcyI6W10sIm1hcHBpbmdzIjoiOzs7OztBQUFBLG1HQUFtRztBQUNuRyw4RkFBOEY7QUFDOUYsRUFBRTtBQUNGLGtHQUFrRztBQUNsRyxtR0FBbUc7QUFDbkcseUVBQXlFO0FBQ3pFLCtFQUE0RDtBQUM1RCx3REFBb0M7QUFDcEMscURBQW9EO0FBRXBELGdHQUFnRztBQUNoRyxnR0FBZ0c7QUFDaEcsb0dBQW9HO0FBQ3BHLDJFQUEyRTtBQUMzRSxJQUFJLENBQUM7SUFDSCxJQUFBLGlDQUFnQixFQUFDLElBQUEsaUJBQVcsR0FBRSxDQUFDLENBQUM7QUFDbEMsQ0FBQztBQUFDLE9BQU8sS0FBSyxFQUFFLENBQUM7SUFDZixJQUFBLHdCQUFtQixHQUFFLENBQUMsTUFBTSxFQUFFLHVEQUF1RCxFQUFFO1FBQ3JGLE1BQU0sRUFBRSxLQUFLLFlBQVksS0FBSyxDQUFDLENBQUMsQ0FBQyxLQUFLLENBQUMsT0FBTyxDQUFDLENBQUMsQ0FBQyxNQUFNLENBQUMsS0FBSyxDQUFDO0tBQy9ELENBQUMsQ0FBQztBQUNMLENBQUMifQ==
|
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
import type { Logger } from './ports/logger-port';
|
|
2
|
+
/**
|
|
3
|
+
* The OpenTelemetry setup. `tracing-preload.ts` is what actually runs it, via `--require` before
|
|
4
|
+
* cli.js so the auto-instrumentation patches land before anything the app imports; keeping the two
|
|
5
|
+
* apart is what lets this module be imported by a test without arming an SDK.
|
|
6
|
+
*
|
|
7
|
+
* The SDK is initialised only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set, so an install that never
|
|
8
|
+
* opted into APM pays nothing. Neither ending the process nor flushing on the way out is this
|
|
9
|
+
* module's business: it hands the SDK back, the preload parks it in `tracing-handle`, and the
|
|
10
|
+
* shutdown path flushes through that. Arming a signal handler here would consume a signal the CLI
|
|
11
|
+
* has not armed its own handler for yet, and nothing would then terminate the process.
|
|
12
|
+
* All configuration goes through the standard OTel environment variables:
|
|
13
|
+
*
|
|
14
|
+
* OTEL_EXPORTER_OTLP_ENDPOINT OTLP receiver (e.g. http://localhost:4318)
|
|
15
|
+
* OTEL_SERVICE_NAME default: forestadmin-agent-bff
|
|
16
|
+
* OTEL_SDK_DISABLED set to "true" to force-disable
|
|
17
|
+
* OTEL_RESOURCE_ATTRIBUTES e.g. deployment.environment=production
|
|
18
|
+
* OTEL_METRICS_EXPORTER default: none (the SDK's own default is otlp)
|
|
19
|
+
* OTEL_LOGS_EXPORTER default: none (the SDK's own default is otlp)
|
|
20
|
+
*
|
|
21
|
+
* The OTel packages are installed only into the Docker image's isolated deps (see
|
|
22
|
+
* `packages/agent-bff/docker/`), never shipped to npm consumers of the CLI — hence the dynamic
|
|
23
|
+
* require behind a seam, and the warning rather than a crash when they are absent.
|
|
24
|
+
*/
|
|
25
|
+
export declare const DEFAULT_SERVICE_NAME = "forestadmin-agent-bff";
|
|
26
|
+
export interface OtelSdk {
|
|
27
|
+
start(): void;
|
|
28
|
+
shutdown(): Promise<void>;
|
|
29
|
+
}
|
|
30
|
+
export interface OtelModules {
|
|
31
|
+
NodeSDK: new (options: Record<string, unknown>) => OtelSdk;
|
|
32
|
+
getNodeAutoInstrumentations: () => unknown;
|
|
33
|
+
}
|
|
34
|
+
export interface TracingOptions {
|
|
35
|
+
/**
|
|
36
|
+
* The environment OUR decisions read: whether to arm at all, and what to name the service when
|
|
37
|
+
* nothing else does. It stops there — the SDK reads the real `process.env` for everything about
|
|
38
|
+
* the export itself, so a test injecting `env` cannot assert where spans go. Identical objects in
|
|
39
|
+
* production, so this is a limit on what the seam can test, not a behaviour.
|
|
40
|
+
*/
|
|
41
|
+
env?: NodeJS.ProcessEnv;
|
|
42
|
+
logger?: Logger;
|
|
43
|
+
/** The dynamic require, as a seam: the packages exist only in the Docker image. */
|
|
44
|
+
load?: () => OtelModules | undefined;
|
|
45
|
+
}
|
|
46
|
+
/** The packages this image installs for APM, and the export it takes from each. */
|
|
47
|
+
export declare const OTEL_MODULE_IDS: {
|
|
48
|
+
readonly sdk: "@opentelemetry/sdk-node";
|
|
49
|
+
readonly instrumentations: "@opentelemetry/auto-instrumentations-node";
|
|
50
|
+
};
|
|
51
|
+
/**
|
|
52
|
+
* The module resolution, as a seam. These packages are absent everywhere except the Docker image,
|
|
53
|
+
* so the real `require` can only ever take the failure branch here — passing a loader is what lets
|
|
54
|
+
* the success branch, and the package ids it asks for, be exercised at all. A typo in one of them
|
|
55
|
+
* would otherwise surface as an untraced image and nothing else.
|
|
56
|
+
*/
|
|
57
|
+
export type ModuleLoader = (id: string) => Record<string, unknown>;
|
|
58
|
+
export declare function loadOtelModules(load?: ModuleLoader): OtelModules | undefined;
|
|
59
|
+
/**
|
|
60
|
+
* The service name the environment already carries, by the spec's precedence: `OTEL_SERVICE_NAME`
|
|
61
|
+
* wins, then a `service.name` entry in the comma-separated `OTEL_RESOURCE_ATTRIBUTES` list.
|
|
62
|
+
* Undefined when neither names one, which is the only case where our own default should apply.
|
|
63
|
+
*/
|
|
64
|
+
export declare function serviceNameFromEnv(env: NodeJS.ProcessEnv): string | undefined;
|
|
65
|
+
/**
|
|
66
|
+
* Strips the credentials out of the endpoint before it reaches the logs. `https://user:token@host`
|
|
67
|
+
* is a legitimate way to reach a collector behind basic auth, and container logs are the last place
|
|
68
|
+
* that token should end up — this package does not echo a secret anywhere else either.
|
|
69
|
+
*
|
|
70
|
+
* The query string goes with them, unread: collectors that take their key as `?api_key=` exist, and
|
|
71
|
+
* telling them apart from a harmless parameter means knowing every vendor's spelling. Nothing about
|
|
72
|
+
* the destination is lost — the host and path are what the line is for.
|
|
73
|
+
*
|
|
74
|
+
* An endpoint that does not parse is dropped from the log entirely rather than passed through: it
|
|
75
|
+
* cannot be redacted, so it cannot be shown. The SDK will fail on it soon enough on its own.
|
|
76
|
+
*/
|
|
77
|
+
export declare function redactEndpoint(endpoint: string): string | undefined;
|
|
78
|
+
export default function initTracing(options?: TracingOptions): OtelSdk | undefined;
|
|
79
|
+
//# sourceMappingURL=tracing.d.ts.map
|
package/dist/tracing.js
ADDED
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
3
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
4
|
+
};
|
|
5
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
6
|
+
exports.OTEL_MODULE_IDS = exports.DEFAULT_SERVICE_NAME = void 0;
|
|
7
|
+
exports.loadOtelModules = loadOtelModules;
|
|
8
|
+
exports.serviceNameFromEnv = serviceNameFromEnv;
|
|
9
|
+
exports.redactEndpoint = redactEndpoint;
|
|
10
|
+
exports.default = initTracing;
|
|
11
|
+
const console_logger_1 = __importDefault(require("./adapters/console-logger"));
|
|
12
|
+
/**
|
|
13
|
+
* The OpenTelemetry setup. `tracing-preload.ts` is what actually runs it, via `--require` before
|
|
14
|
+
* cli.js so the auto-instrumentation patches land before anything the app imports; keeping the two
|
|
15
|
+
* apart is what lets this module be imported by a test without arming an SDK.
|
|
16
|
+
*
|
|
17
|
+
* The SDK is initialised only when `OTEL_EXPORTER_OTLP_ENDPOINT` is set, so an install that never
|
|
18
|
+
* opted into APM pays nothing. Neither ending the process nor flushing on the way out is this
|
|
19
|
+
* module's business: it hands the SDK back, the preload parks it in `tracing-handle`, and the
|
|
20
|
+
* shutdown path flushes through that. Arming a signal handler here would consume a signal the CLI
|
|
21
|
+
* has not armed its own handler for yet, and nothing would then terminate the process.
|
|
22
|
+
* All configuration goes through the standard OTel environment variables:
|
|
23
|
+
*
|
|
24
|
+
* OTEL_EXPORTER_OTLP_ENDPOINT OTLP receiver (e.g. http://localhost:4318)
|
|
25
|
+
* OTEL_SERVICE_NAME default: forestadmin-agent-bff
|
|
26
|
+
* OTEL_SDK_DISABLED set to "true" to force-disable
|
|
27
|
+
* OTEL_RESOURCE_ATTRIBUTES e.g. deployment.environment=production
|
|
28
|
+
* OTEL_METRICS_EXPORTER default: none (the SDK's own default is otlp)
|
|
29
|
+
* OTEL_LOGS_EXPORTER default: none (the SDK's own default is otlp)
|
|
30
|
+
*
|
|
31
|
+
* The OTel packages are installed only into the Docker image's isolated deps (see
|
|
32
|
+
* `packages/agent-bff/docker/`), never shipped to npm consumers of the CLI — hence the dynamic
|
|
33
|
+
* require behind a seam, and the warning rather than a crash when they are absent.
|
|
34
|
+
*/
|
|
35
|
+
exports.DEFAULT_SERVICE_NAME = 'forestadmin-agent-bff';
|
|
36
|
+
/** The packages this image installs for APM, and the export it takes from each. */
|
|
37
|
+
exports.OTEL_MODULE_IDS = {
|
|
38
|
+
sdk: '@opentelemetry/sdk-node',
|
|
39
|
+
instrumentations: '@opentelemetry/auto-instrumentations-node',
|
|
40
|
+
};
|
|
41
|
+
function loadOtelModules(load = require) {
|
|
42
|
+
try {
|
|
43
|
+
const modules = {
|
|
44
|
+
NodeSDK: load(exports.OTEL_MODULE_IDS.sdk).NodeSDK,
|
|
45
|
+
getNodeAutoInstrumentations: load(exports.OTEL_MODULE_IDS.instrumentations)
|
|
46
|
+
.getNodeAutoInstrumentations,
|
|
47
|
+
};
|
|
48
|
+
// Resolving is not the same as finding what we came for. A package that loads but no longer
|
|
49
|
+
// exports the name we read — a rename on a version bump — would otherwise hand back a truthy
|
|
50
|
+
// object of undefined members, skip the "packages not available" branch, and throw a
|
|
51
|
+
// TypeError inside a `--require`, before the entry point runs at all. That is a dead container
|
|
52
|
+
// for an APM layer that is supposed to degrade to a warning.
|
|
53
|
+
if (Object.values(modules).some(exported => typeof exported !== 'function'))
|
|
54
|
+
return undefined;
|
|
55
|
+
return modules;
|
|
56
|
+
}
|
|
57
|
+
catch {
|
|
58
|
+
return undefined;
|
|
59
|
+
}
|
|
60
|
+
}
|
|
61
|
+
/**
|
|
62
|
+
* The OTel specification defines its boolean environment variables as case-insensitive, and this
|
|
63
|
+
* one is the kill switch: reading `TRUE` as "not disabled" would leave tracing running for someone
|
|
64
|
+
* who just asked for it to stop.
|
|
65
|
+
*/
|
|
66
|
+
function isDisabled(raw) {
|
|
67
|
+
return raw?.trim().toLowerCase() === 'true';
|
|
68
|
+
}
|
|
69
|
+
/**
|
|
70
|
+
* The service name the environment already carries, by the spec's precedence: `OTEL_SERVICE_NAME`
|
|
71
|
+
* wins, then a `service.name` entry in the comma-separated `OTEL_RESOURCE_ATTRIBUTES` list.
|
|
72
|
+
* Undefined when neither names one, which is the only case where our own default should apply.
|
|
73
|
+
*/
|
|
74
|
+
function serviceNameFromEnv(env) {
|
|
75
|
+
const explicit = env.OTEL_SERVICE_NAME?.trim();
|
|
76
|
+
if (explicit)
|
|
77
|
+
return explicit;
|
|
78
|
+
return env.OTEL_RESOURCE_ATTRIBUTES?.split(',')
|
|
79
|
+
.map(entry => entry.split('='))
|
|
80
|
+
.filter(([key, value]) => key?.trim() === 'service.name' && value?.trim())
|
|
81
|
+
.map(([, value]) => value.trim())
|
|
82
|
+
.pop();
|
|
83
|
+
}
|
|
84
|
+
/**
|
|
85
|
+
* Strips the credentials out of the endpoint before it reaches the logs. `https://user:token@host`
|
|
86
|
+
* is a legitimate way to reach a collector behind basic auth, and container logs are the last place
|
|
87
|
+
* that token should end up — this package does not echo a secret anywhere else either.
|
|
88
|
+
*
|
|
89
|
+
* The query string goes with them, unread: collectors that take their key as `?api_key=` exist, and
|
|
90
|
+
* telling them apart from a harmless parameter means knowing every vendor's spelling. Nothing about
|
|
91
|
+
* the destination is lost — the host and path are what the line is for.
|
|
92
|
+
*
|
|
93
|
+
* An endpoint that does not parse is dropped from the log entirely rather than passed through: it
|
|
94
|
+
* cannot be redacted, so it cannot be shown. The SDK will fail on it soon enough on its own.
|
|
95
|
+
*/
|
|
96
|
+
function redactEndpoint(endpoint) {
|
|
97
|
+
try {
|
|
98
|
+
const url = new URL(endpoint);
|
|
99
|
+
if (!url.username && !url.password && !url.search)
|
|
100
|
+
return endpoint;
|
|
101
|
+
url.username = '';
|
|
102
|
+
url.password = '';
|
|
103
|
+
url.search = '';
|
|
104
|
+
return url.toString();
|
|
105
|
+
}
|
|
106
|
+
catch {
|
|
107
|
+
return undefined;
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
function initTracing(options = {}) {
|
|
111
|
+
const { env = process.env, logger = (0, console_logger_1.default)(), load = loadOtelModules } = options;
|
|
112
|
+
// Any of the three standard ways to say "export traces somewhere" turns tracing on. The per-signal
|
|
113
|
+
// endpoint counts as much as the generic one, and OTEL_TRACES_EXPORTER counts on its own —
|
|
114
|
+
// requiring an OTLP collector before `console` does anything would be nonsense for an exporter
|
|
115
|
+
// that writes to stdout, and it is the obvious first thing to reach for when debugging.
|
|
116
|
+
//
|
|
117
|
+
// A blank value counts as unset throughout, not as "export to the OTel default": `env_file` hands
|
|
118
|
+
// an empty string through for a variable left blank in a `.env`, and arming the SDK against
|
|
119
|
+
// localhost:4318 is not what leaving the line empty asks for.
|
|
120
|
+
const endpoint = env.OTEL_EXPORTER_OTLP_TRACES_ENDPOINT?.trim() || env.OTEL_EXPORTER_OTLP_ENDPOINT?.trim();
|
|
121
|
+
const requestedExporter = env.OTEL_TRACES_EXPORTER?.trim().toLowerCase();
|
|
122
|
+
if ((!endpoint && !requestedExporter) || isDisabled(env.OTEL_SDK_DISABLED))
|
|
123
|
+
return undefined;
|
|
124
|
+
const modules = load();
|
|
125
|
+
if (!modules) {
|
|
126
|
+
logger('Warn', 'OpenTelemetry packages not available, skipping APM initialisation');
|
|
127
|
+
return undefined;
|
|
128
|
+
}
|
|
129
|
+
const { NodeSDK, getNodeAutoInstrumentations } = modules;
|
|
130
|
+
// Passing `serviceName` unconditionally would override OTEL_RESOURCE_ATTRIBUTES=service.name=...,
|
|
131
|
+
// which is a documented knob — someone setting only that would silently get our default. So we
|
|
132
|
+
// fill it in only when the environment names no service at all, and otherwise leave the SDK to
|
|
133
|
+
// apply the spec's own precedence (OTEL_SERVICE_NAME first, then the resource attribute).
|
|
134
|
+
const named = serviceNameFromEnv(env);
|
|
135
|
+
// NodeSDK reads OTEL_METRICS_EXPORTER and OTEL_LOGS_EXPORTER itself, and an unset one does not
|
|
136
|
+
// mean "off" — it means otlp. So arming traces alone also starts a metric reader and a log
|
|
137
|
+
// processor aimed at the OTLP default, http://localhost:4318. Under OTEL_TRACES_EXPORTER=console,
|
|
138
|
+
// which asks for no collector at all, that is a POST every minute to a port nobody named; where
|
|
139
|
+
// something else on the host does listen there, it is telemetry that install never opted into.
|
|
140
|
+
// This module arms tracing. The other two signals stay opt-in, through their own variable, which
|
|
141
|
+
// still works: we only fill in what the environment left blank.
|
|
142
|
+
env.OTEL_METRICS_EXPORTER = env.OTEL_METRICS_EXPORTER?.trim() || 'none';
|
|
143
|
+
env.OTEL_LOGS_EXPORTER = env.OTEL_LOGS_EXPORTER?.trim() || 'none';
|
|
144
|
+
// No `traceExporter`, deliberately. Passing one puts NodeSDK on its manual-configuration path,
|
|
145
|
+
// where it stops reading the environment — which is how OTEL_TRACES_EXPORTER, then
|
|
146
|
+
// OTEL_EXPORTER_OTLP_PROTOCOL, then the per-signal endpoint each turned out to be silently
|
|
147
|
+
// ignored, one review round after another. They were three symptoms of doing the SDK's job.
|
|
148
|
+
//
|
|
149
|
+
// sdk-node depends on every OTLP exporter and on the Zipkin one, so all of them are in the image
|
|
150
|
+
// already; leaving the choice to the SDK costs nothing and makes the whole standard surface work,
|
|
151
|
+
// protocol and per-signal overrides included. Our only decisions are whether to arm at all, and
|
|
152
|
+
// the service name when nothing else supplies one.
|
|
153
|
+
const sdk = new NodeSDK({
|
|
154
|
+
...(named ? {} : { serviceName: exports.DEFAULT_SERVICE_NAME }),
|
|
155
|
+
instrumentations: [getNodeAutoInstrumentations()],
|
|
156
|
+
});
|
|
157
|
+
sdk.start();
|
|
158
|
+
// What was ASKED for, not what the SDK settled on — we no longer choose, so claiming a
|
|
159
|
+
// destination we did not pick would be inventing one. The endpoint is dropped when an exporter
|
|
160
|
+
// was named instead, since it would read as a promise this line cannot keep.
|
|
161
|
+
logger('Info', 'OpenTelemetry tracing enabled', {
|
|
162
|
+
serviceName: named ?? exports.DEFAULT_SERVICE_NAME,
|
|
163
|
+
...(requestedExporter ? { exporter: requestedExporter } : {}),
|
|
164
|
+
...(endpoint && !requestedExporter ? { endpoint: redactEndpoint(endpoint) } : {}),
|
|
165
|
+
});
|
|
166
|
+
return sdk;
|
|
167
|
+
}
|
|
168
|
+
//# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJmaWxlIjoidHJhY2luZy5qcyIsInNvdXJjZVJvb3QiOiIiLCJzb3VyY2VzIjpbIi4uL3NyYy90cmFjaW5nLnRzIl0sIm5hbWVzIjpbXSwibWFwcGluZ3MiOiI7Ozs7OztBQW1FQSwwQ0FtQkM7QUFnQkQsZ0RBVUM7QUFjRCx3Q0FjQztBQUVELDhCQXFFQztBQWpORCwrRUFBNEQ7QUFFNUQ7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7Ozs7R0FzQkc7QUFFVSxRQUFBLG9CQUFvQixHQUFHLHVCQUF1QixDQUFDO0FBeUI1RCxtRkFBbUY7QUFDdEUsUUFBQSxlQUFlLEdBQUc7SUFDN0IsR0FBRyxFQUFFLHlCQUF5QjtJQUM5QixnQkFBZ0IsRUFBRSwyQ0FBMkM7Q0FDckQsQ0FBQztBQVVYLFNBQWdCLGVBQWUsQ0FBQyxPQUFxQixPQUFPO0lBQzFELElBQUksQ0FBQztRQUNILE1BQU0sT0FBTyxHQUFHO1lBQ2QsT0FBTyxFQUFFLElBQUksQ0FBQyx1QkFBZSxDQUFDLEdBQUcsQ0FBQyxDQUFDLE9BQU87WUFDMUMsMkJBQTJCLEVBQUUsSUFBSSxDQUFDLHVCQUFlLENBQUMsZ0JBQWdCLENBQUM7aUJBQ2hFLDJCQUEyQjtTQUMvQixDQUFDO1FBRUYsNEZBQTRGO1FBQzVGLDZGQUE2RjtRQUM3RixxRkFBcUY7UUFDckYsK0ZBQStGO1FBQy9GLDZEQUE2RDtRQUM3RCxJQUFJLE1BQU0sQ0FBQyxNQUFNLENBQUMsT0FBTyxDQUFDLENBQUMsSUFBSSxDQUFDLFFBQVEsQ0FBQyxFQUFFLENBQUMsT0FBTyxRQUFRLEtBQUssVUFBVSxDQUFDO1lBQUUsT0FBTyxTQUFTLENBQUM7UUFFOUYsT0FBTyxPQUFzQixDQUFDO0lBQ2hDLENBQUM7SUFBQyxNQUFNLENBQUM7UUFDUCxPQUFPLFNBQVMsQ0FBQztJQUNuQixDQUFDO0FBQ0gsQ0FBQztBQUVEOzs7O0dBSUc7QUFDSCxTQUFTLFVBQVUsQ0FBQyxHQUF1QjtJQUN6QyxPQUFPLEdBQUcsRUFBRSxJQUFJLEVBQUUsQ0FBQyxXQUFXLEVBQUUsS0FBSyxNQUFNLENBQUM7QUFDOUMsQ0FBQztBQUVEOzs7O0dBSUc7QUFDSCxTQUFnQixrQkFBa0IsQ0FBQyxHQUFzQjtJQUN2RCxNQUFNLFFBQVEsR0FBRyxHQUFHLENBQUMsaUJBQWlCLEVBQUUsSUFBSSxFQUFFLENBQUM7SUFFL0MsSUFBSSxRQUFRO1FBQUUsT0FBTyxRQUFRLENBQUM7SUFFOUIsT0FBTyxHQUFHLENBQUMsd0JBQXdCLEVBQUUsS0FBSyxDQUFDLEdBQUcsQ0FBQztTQUM1QyxHQUFHLENBQUMsS0FBSyxDQUFDLEVBQUUsQ0FBQyxLQUFLLENBQUMsS0FBSyxDQUFDLEdBQUcsQ0FBQyxDQUFDO1NBQzlCLE1BQU0sQ0FBQyxDQUFDLENBQUMsR0FBRyxFQUFFLEtBQUssQ0FBQyxFQUFFLEVBQUUsQ0FBQyxHQUFHLEVBQUUsSUFBSSxFQUFFLEtBQUssY0FBYyxJQUFJLEtBQUssRUFBRSxJQUFJLEVBQUUsQ0FBQztTQUN6RSxHQUFHLENBQUMsQ0FBQyxDQUFDLEVBQUUsS0FBSyxDQUFDLEVBQUUsRUFBRSxDQUFDLEtBQUssQ0FBQyxJQUFJLEVBQUUsQ0FBQztTQUNoQyxHQUFHLEVBQUUsQ0FBQztBQUNYLENBQUM7QUFFRDs7Ozs7Ozs7Ozs7R0FXRztBQUNILFNBQWdCLGNBQWMsQ0FBQyxRQUFnQjtJQUM3QyxJQUFJLENBQUM7UUFDSCxNQUFNLEdBQUcsR0FBRyxJQUFJLEdBQUcsQ0FBQyxRQUFRLENBQUMsQ0FBQztRQUU5QixJQUFJLENBQUMsR0FBRyxDQUFDLFFBQVEsSUFBSSxDQUFDLEdBQUcsQ0FBQyxRQUFRLElBQUksQ0FBQyxHQUFHLENBQUMsTUFBTTtZQUFFLE9BQU8sUUFBUSxDQUFDO1FBRW5FLEdBQUcsQ0FBQyxRQUFRLEdBQUcsRUFBRSxDQUFDO1FBQ2xCLEdBQUcsQ0FBQyxRQUFRLEdBQUcsRUFBRSxDQUFDO1FBQ2xCLEdBQUcsQ0FBQyxNQUFNLEdBQUcsRUFBRSxDQUFDO1FBRWhCLE9BQU8sR0FBRyxDQUFDLFFBQVEsRUFBRSxDQUFDO0lBQ3hCLENBQUM7SUFBQyxNQUFNLENBQUM7UUFDUCxPQUFPLFNBQVMsQ0FBQztJQUNuQixDQUFDO0FBQ0gsQ0FBQztBQUVELFNBQXdCLFdBQVcsQ0FBQyxVQUEwQixFQUFFO0lBQzlELE1BQU0sRUFBRSxHQUFHLEdBQUcsT0FBTyxDQUFDLEdBQUcsRUFBRSxNQUFNLEdBQUcsSUFBQSx3QkFBbUIsR0FBRSxFQUFFLElBQUksR0FBRyxlQUFlLEVBQUUsR0FBRyxPQUFPLENBQUM7SUFFOUYsbUdBQW1HO0lBQ25HLDJGQUEyRjtJQUMzRiwrRkFBK0Y7SUFDL0Ysd0ZBQXdGO0lBQ3hGLEVBQUU7SUFDRixrR0FBa0c7SUFDbEcsNEZBQTRGO0lBQzVGLDhEQUE4RDtJQUM5RCxNQUFNLFFBQVEsR0FDWixHQUFHLENBQUMsa0NBQWtDLEVBQUUsSUFBSSxFQUFFLElBQUksR0FBRyxDQUFDLDJCQUEyQixFQUFFLElBQUksRUFBRSxDQUFDO0lBQzVGLE1BQU0saUJBQWlCLEdBQUcsR0FBRyxDQUFDLG9CQUFvQixFQUFFLElBQUksRUFBRSxDQUFDLFdBQVcsRUFBRSxDQUFDO0lBRXpFLElBQUksQ0FBQyxDQUFDLFFBQVEsSUFBSSxDQUFDLGlCQUFpQixDQUFDLElBQUksVUFBVSxDQUFDLEdBQUcsQ0FBQyxpQkFBaUIsQ0FBQztRQUFFLE9BQU8sU0FBUyxDQUFDO0lBRTdGLE1BQU0sT0FBTyxHQUFHLElBQUksRUFBRSxDQUFDO0lBRXZCLElBQUksQ0FBQyxPQUFPLEVBQUUsQ0FBQztRQUNiLE1BQU0sQ0FBQyxNQUFNLEVBQUUsbUVBQW1FLENBQUMsQ0FBQztRQUVwRixPQUFPLFNBQVMsQ0FBQztJQUNuQixDQUFDO0lBRUQsTUFBTSxFQUFFLE9BQU8sRUFBRSwyQkFBMkIsRUFBRSxHQUFHLE9BQU8sQ0FBQztJQUV6RCxrR0FBa0c7SUFDbEcsK0ZBQStGO0lBQy9GLCtGQUErRjtJQUMvRiwwRkFBMEY7SUFDMUYsTUFBTSxLQUFLLEdBQUcsa0JBQWtCLENBQUMsR0FBRyxDQUFDLENBQUM7SUFFdEMsK0ZBQStGO0lBQy9GLDJGQUEyRjtJQUMzRixrR0FBa0c7SUFDbEcsZ0dBQWdHO0lBQ2hHLCtGQUErRjtJQUMvRixpR0FBaUc7SUFDakcsZ0VBQWdFO0lBQ2hFLEdBQUcsQ0FBQyxxQkFBcUIsR0FBRyxHQUFHLENBQUMscUJBQXFCLEVBQUUsSUFBSSxFQUFFLElBQUksTUFBTSxDQUFDO0lBQ3hFLEdBQUcsQ0FBQyxrQkFBa0IsR0FBRyxHQUFHLENBQUMsa0JBQWtCLEVBQUUsSUFBSSxFQUFFLElBQUksTUFBTSxDQUFDO0lBRWxFLCtGQUErRjtJQUMvRixtRkFBbUY7SUFDbkYsMkZBQTJGO0lBQzNGLDRGQUE0RjtJQUM1RixFQUFFO0lBQ0YsaUdBQWlHO0lBQ2pHLGtHQUFrRztJQUNsRyxnR0FBZ0c7SUFDaEcsbURBQW1EO0lBQ25ELE1BQU0sR0FBRyxHQUFHLElBQUksT0FBTyxDQUFDO1FBQ3RCLEdBQUcsQ0FBQyxLQUFLLENBQUMsQ0FBQyxDQUFDLEVBQUUsQ0FBQyxDQUFDLENBQUMsRUFBRSxXQUFXLEVBQUUsNEJBQW9CLEVBQUUsQ0FBQztRQUN2RCxnQkFBZ0IsRUFBRSxDQUFDLDJCQUEyQixFQUFFLENBQUM7S0FDbEQsQ0FBQyxDQUFDO0lBRUgsR0FBRyxDQUFDLEtBQUssRUFBRSxDQUFDO0lBRVosdUZBQXVGO0lBQ3ZGLCtGQUErRjtJQUMvRiw2RUFBNkU7SUFDN0UsTUFBTSxDQUFDLE1BQU0sRUFBRSwrQkFBK0IsRUFBRTtRQUM5QyxXQUFXLEVBQUUsS0FBSyxJQUFJLDRCQUFvQjtRQUMxQyxHQUFHLENBQUMsaUJBQWlCLENBQUMsQ0FBQyxDQUFDLEVBQUUsUUFBUSxFQUFFLGlCQUFpQixFQUFFLENBQUMsQ0FBQyxDQUFDLEVBQUUsQ0FBQztRQUM3RCxHQUFHLENBQUMsUUFBUSxJQUFJLENBQUMsaUJBQWlCLENBQUMsQ0FBQyxDQUFDLEVBQUUsUUFBUSxFQUFFLGNBQWMsQ0FBQyxRQUFRLENBQUMsRUFBRSxDQUFDLENBQUMsQ0FBQyxFQUFFLENBQUM7S0FDbEYsQ0FBQyxDQUFDO0lBRUgsT0FBTyxHQUFHLENBQUM7QUFDYixDQUFDIn0=
|