@cocreate/server-telemetry 1.16.1 → 1.16.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +136 -37
- package/package.json +13 -10
- package/src/index.js +49 -14
package/README.md
CHANGED
|
@@ -1,65 +1,164 @@
|
|
|
1
|
-
# CoCreate
|
|
1
|
+
# CoCreate Server Telemetry
|
|
2
2
|
|
|
3
|
-
A
|
|
3
|
+
A high-frequency, non-blocking telemetry engine designed to monitor multi-tenant resource saturation, loop latency, and connection bandwidth consumption across clustered environments. This module streams real-time system performance insights directly into the CoCreate self-healing autoscaler ecosystem.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
5
|
+
---
|
|
9
6
|
|
|
10
|
-
|
|
7
|
+
## Documentation
|
|
11
8
|
|
|
12
|
-
[CoCreate
|
|
9
|
+
For full API references, custom performance metrics, and dashboard ingestion guides, refer to the [CoCreate Server Telemetry Documentation](https://cocreatejs.com/docs/server-telemetry).
|
|
13
10
|
|
|
14
|
-
|
|
11
|
+
---
|
|
15
12
|
|
|
16
|
-
|
|
13
|
+
## Table of Contents
|
|
17
14
|
|
|
18
|
-
|
|
15
|
+
* [Low-Overhead High-Frequency Telemetry Engine](#low-overhead-high-frequency-telemetry-engine)
|
|
16
|
+
* [Why CoCreate Server Telemetry?](#why-cocreate-server-telemetry)
|
|
17
|
+
* [The Metric Profile Plane](#the-metric-profile-plane)
|
|
18
|
+
* [The Saturation Index Calculation](#the-saturation-index-calculation)
|
|
19
|
+
* [Non-Blocking Multi-Threaded Streaming](#non-blocking-multi-threaded-streaming)
|
|
20
|
+
* [Configuration](#configuration)
|
|
21
|
+
* [Installation](#installation)
|
|
22
|
+
* [Announcements](#announcements)
|
|
23
|
+
* [Roadmap](#roadmap)
|
|
24
|
+
* [How to Contribute](#how-to-contribute)
|
|
25
|
+
* [About](#about)
|
|
19
26
|
|
|
20
|
-
|
|
21
|
-
- [Announcements](#announcements)
|
|
22
|
-
- [Roadmap](#roadmap)
|
|
23
|
-
- [How to Contribute](#how-to-contribute)
|
|
24
|
-
- [About](#about)
|
|
25
|
-
- [License](#license)
|
|
27
|
+
---
|
|
26
28
|
|
|
27
|
-
|
|
29
|
+
## Low-Overhead High-Frequency Telemetry Engine
|
|
28
30
|
|
|
29
|
-
|
|
31
|
+
Monitoring real-time systems often introduces an instrumentation tax—where the observer itself degrades performance. `@cocreate/server-telemetry` is engineered using low-level, asynchronous polling loops to monitor internal runtime state with near-zero overhead.
|
|
30
32
|
|
|
31
|
-
|
|
33
|
+
### Event Loop Lag Detection
|
|
32
34
|
|
|
33
|
-
|
|
35
|
+
Instead of tracking abstract hypervisor data, this module monitors actual V8 thread starvation by measuring scheduling delay intervals. A precision timer schedules microtask check-ins; deviations reveal when application code is blocking the single thread.
|
|
34
36
|
|
|
35
|
-
|
|
37
|
+
### Zero-Allocation Ring Buffers
|
|
36
38
|
|
|
37
|
-
|
|
39
|
+
To prevent triggering V8 Garbage Collection (GC) pauses during intense data collection, all raw telemetry samples are held in pre-allocated, fixed-size typed array ring buffers in memory, ensuring consistent application execution.
|
|
38
40
|
|
|
39
|
-
|
|
41
|
+
```text
|
|
42
|
+
[Metrics Gatherer] ──► [Pre-Allocated Ring Buffer] ──► [Asynchronous Flush] ──► [Autoscaler Engine]
|
|
43
|
+
(Zero GC Overhead)
|
|
44
|
+
```
|
|
40
45
|
|
|
41
|
-
|
|
46
|
+
---
|
|
42
47
|
|
|
43
|
-
|
|
48
|
+
## Why CoCreate Server Telemetry?
|
|
44
49
|
|
|
45
|
-
|
|
50
|
+
Standard infrastructure monitoring agents (like Datadog or Prometheus Node Exporter) sample system states at coarse 10 to 60-second intervals and look at machines globally. They completely miss the short, critical spikes that degrade real-time WebSocket experiences—like a single tenant executing a complex query that triggers a 500ms event loop freeze.
|
|
46
51
|
|
|
47
|
-
|
|
52
|
+
`@cocreate/server-telemetry` isolates metrics down to the individual **Organization ID** and **Socket Connection Instance**. It calculates instant health profiles at sub-second intervals, giving the `@cocreate/server-autoscaler` the granular insight required to gracefully shed or migrate specific tenant workloads before users experience lag.
|
|
48
53
|
|
|
49
|
-
|
|
54
|
+
---
|
|
50
55
|
|
|
51
|
-
|
|
56
|
+
## The Metric Profile Plane
|
|
52
57
|
|
|
53
|
-
|
|
58
|
+
The telemetry engine continually samples four primary dimensions across the cluster:
|
|
54
59
|
|
|
55
|
-
|
|
60
|
+
| Metric Group | Collection Method | Key Use Case |
|
|
61
|
+
| --- | --- | --- |
|
|
62
|
+
| **V8 Event Loop Lag** | Asynchronous Microtask Polling | Detects thread starvation caused by heavy computation or synchronous execution blockages. |
|
|
63
|
+
| **Memory Profiles** | `process.memoryUsage.rss()` | Tracks memory growth and leaks within individual worker instances. |
|
|
64
|
+
| **Socket Density** | Active Session Tracking | Measures connection volumes to protect memory limits. |
|
|
65
|
+
| **Bandwidth Deltas** | Network Interface Stream Counters | Monitors tenant-isolated data rates to identify runaway processes or DDoS vectors. |
|
|
56
66
|
|
|
57
|
-
|
|
67
|
+
---
|
|
58
68
|
|
|
59
|
-
|
|
69
|
+
## The Saturation Index Calculation
|
|
60
70
|
|
|
61
|
-
|
|
71
|
+
The absolute core of the CoCreate self-healing cluster is the **Saturation Index ($S_i$)**. The telemetry module calculates this value at regular 500ms intervals using a dynamic weighted linear combination:
|
|
62
72
|
|
|
63
|
-
|
|
73
|
+
$$S_i = \alpha \cdot \left(\frac{\text{Loop Lag (ms)}}{\text{Max Lag Limit}}\right) + \beta \cdot \left(\frac{\text{RAM Used}}{\text{RAM Allocated}}\right) + \gamma \cdot \left(\frac{\text{Active Sockets}}{\text{Max Socket Capacity}}\right)$$
|
|
64
74
|
|
|
65
|
-
|
|
75
|
+
Where the weight constants are tuned by default to prioritize thread fluidity:
|
|
76
|
+
|
|
77
|
+
* $\alpha = 0.50$ (Event Loop Fluctuations)
|
|
78
|
+
* $\beta = 0.30$ (Memory Saturation)
|
|
79
|
+
* $\gamma = 0.20$ (Connection Density)
|
|
80
|
+
|
|
81
|
+
When $S_i > 0.85$ for a consecutive duration exceeding 15 seconds, the telemetry plane fires a critical notification to the server orchestration layer to prompt a defensive horizontal scaling action.
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Non-Blocking Multi-Threaded Streaming
|
|
86
|
+
|
|
87
|
+
Telemetry data shouldn't compete with application throughput. When configured to stream metrics to a central database or external timeseries sink, `@cocreate/server-telemetry` offloads data serialization work entirely.
|
|
88
|
+
|
|
89
|
+
Using Node.js `worker_threads`, the main application thread offloads raw data arrays to a background thread using zero-copy `ArrayBuffer` transfers. The background worker serializes and ships the metric payload asynchronously, leaving the primary thread free to process WebSocket packets uninterrupted.
|
|
90
|
+
|
|
91
|
+
---
|
|
92
|
+
|
|
93
|
+
## Configuration
|
|
94
|
+
|
|
95
|
+
Drop a `telemetry.config.json` into your server root directory to manage monitoring frequencies and aggregation targets.
|
|
96
|
+
|
|
97
|
+
```json
|
|
98
|
+
{
|
|
99
|
+
"samplingIntervalMs": 500,
|
|
100
|
+
"flushIntervalMs": 5000,
|
|
101
|
+
"retentionPeriodHours": 24,
|
|
102
|
+
"limits": {
|
|
103
|
+
"maxAllowedV8LagMs": 100,
|
|
104
|
+
"highMemoryThresholdPercent": 85
|
|
105
|
+
},
|
|
106
|
+
"streaming": {
|
|
107
|
+
"enabled": true,
|
|
108
|
+
"useWorkerThread": true,
|
|
109
|
+
"target": "cluster-registry"
|
|
110
|
+
}
|
|
111
|
+
}
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
---
|
|
115
|
+
|
|
116
|
+
## Installation
|
|
117
|
+
|
|
118
|
+
### NPM Installation
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
npm i @cocreate/server-telemetry
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
### Yarn Installation
|
|
125
|
+
|
|
126
|
+
```bash
|
|
127
|
+
yarn add @cocreate/server-telemetry
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
## Announcements
|
|
133
|
+
|
|
134
|
+
Performance improvements, kernel tracing updates, and profiling modifications are documented on the [CoCreate Server Telemetry GitHub Releases](https://github.com/CoCreate-app/CoCreate-server-telemetry/releases) page. Subscribe to stay informed about optimization releases and new features.
|
|
135
|
+
|
|
136
|
+
---
|
|
137
|
+
|
|
138
|
+
## Roadmap
|
|
139
|
+
|
|
140
|
+
Upcoming capabilities for our monitoring plane include:
|
|
141
|
+
|
|
142
|
+
* **eBPF Network Inspections:** Low-overhead, kernel-level packet inspection to isolate network throughput anomalies down to precise system sockets.
|
|
143
|
+
* **GC Execution Visualizer:** Deep tracing hooks that flag exactly which tenant code block caused V8 mark-and-sweep garbage collection freezes.
|
|
144
|
+
|
|
145
|
+
---
|
|
146
|
+
|
|
147
|
+
## How to Contribute
|
|
148
|
+
|
|
149
|
+
We welcome pull requests targeting performance profiling, adapter additions, and bug fixes. Please read our [CONTRIBUTING.md](https://github.com/CoCreate-app/CoCreate-server-telemetry/blob/master/CONTRIBUTING.md) guide for local development workflows.
|
|
150
|
+
|
|
151
|
+
If you encounter any bugs or wish to request new features, please submit an issue on our [GitHub Issues](https://github.com/CoCreate-app/CoCreate-server-telemetry/issues) tracker. We want this library to be community-driven, and CoCreate-led. We need your help to shape its future.
|
|
152
|
+
|
|
153
|
+
For broader system configurations and API guides, please visit our [CoCreate Server Telemetry Documentation](https://cocreatejs.com/docs/server-telemetry).
|
|
154
|
+
|
|
155
|
+
---
|
|
156
|
+
|
|
157
|
+
## About
|
|
158
|
+
|
|
159
|
+
`@cocreate/server-telemetry` is designed, built, and supported by the CoCreate Developer Experience Team.
|
|
160
|
+
|
|
161
|
+
> [!NOTE]
|
|
162
|
+
> For questions regarding telemetry pipelines, custom metric adapters, or deployment profiling, please visit our [GitHub Discussions](https://github.com/CoCreate-app/CoCreate-server-telemetry/discussions) or join our Discord community.
|
|
163
|
+
|
|
164
|
+
`@cocreate/server-telemetry` is maintained and funded by CoCreate. The names and logos for CoCreate are trademarks of CoCreate, LLC.
|
package/package.json
CHANGED
|
@@ -1,16 +1,19 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@cocreate/server-telemetry",
|
|
3
|
-
"version": "1.16.
|
|
4
|
-
"description": "CoCreate-
|
|
3
|
+
"version": "1.16.2",
|
|
4
|
+
"description": "High-throughput, non-blocking dual-buffered telemetry orchestrator for CoCreate Server. Measures real-time multi-tenant bandwidth, API operation tracing, and consumption analytics without event-loop lag.",
|
|
5
5
|
"keywords": [
|
|
6
|
-
"cocreate
|
|
7
|
-
"
|
|
8
|
-
"
|
|
9
|
-
"
|
|
10
|
-
"
|
|
11
|
-
"
|
|
12
|
-
"
|
|
13
|
-
"
|
|
6
|
+
"cocreate",
|
|
7
|
+
"cocreate-server",
|
|
8
|
+
"telemetry",
|
|
9
|
+
"multi-tenant-analytics",
|
|
10
|
+
"event-driven-telemetry",
|
|
11
|
+
"bandwidth-monitoring",
|
|
12
|
+
"api-tracing",
|
|
13
|
+
"time-series-metrics",
|
|
14
|
+
"byodb-metrics",
|
|
15
|
+
"performance-monitoring",
|
|
16
|
+
"non-blocking"
|
|
14
17
|
],
|
|
15
18
|
"type": "module",
|
|
16
19
|
"main": "./src/index.js",
|
package/src/index.js
CHANGED
|
@@ -26,7 +26,7 @@ import os from 'node:os';
|
|
|
26
26
|
import Config from '@cocreate/config';
|
|
27
27
|
import { getValueFromObject } from '@cocreate/utils';
|
|
28
28
|
|
|
29
|
-
let server = null;
|
|
29
|
+
let server = null; // Reference to the main CoCreateServer instance
|
|
30
30
|
let activeIntervalId = null; // Storage for the core execution loop interval
|
|
31
31
|
let lastCpuInfo = null; // Cache for the previous CPU telemetry sample
|
|
32
32
|
let telemetryConfig = null; // In-memory reference to loaded configuration map
|
|
@@ -311,27 +311,40 @@ export function read() {
|
|
|
311
311
|
getValueFromObject(telemetryConfig, 'server-telemetry.safetyOptions.cordonCpuThreshold') ||
|
|
312
312
|
getValueFromObject(telemetryConfig, 'server-telemetry.cordonCpuThreshold'), 10
|
|
313
313
|
)
|
|
314
|
-
:
|
|
314
|
+
: 95;
|
|
315
315
|
|
|
316
316
|
const actualMemoryThreshold = isNaN(cordonMemoryThreshold) ? 90 : cordonMemoryThreshold;
|
|
317
|
-
const actualCpuThreshold = isNaN(cordonCpuThreshold) ?
|
|
317
|
+
const actualCpuThreshold = isNaN(cordonCpuThreshold) ? 95 : cordonCpuThreshold;
|
|
318
318
|
|
|
319
|
-
|
|
320
|
-
|
|
319
|
+
// Memory limits are evaluated instantaneously on the 1-second loop
|
|
320
|
+
const isMemoryLimitBreached = currentTelemetry.memory.percentage >= actualMemoryThreshold;
|
|
321
321
|
|
|
322
322
|
const recoveryBuffer = 5;
|
|
323
|
+
// Recovering is still active on the 1-second tick: both current CPU and Memory must be safely down
|
|
323
324
|
const isResourceSafeToRecover = currentTelemetry.memory.percentage < (actualMemoryThreshold - recoveryBuffer) &&
|
|
324
325
|
currentTelemetry.cpu.totalUsage < (actualCpuThreshold - recoveryBuffer);
|
|
325
326
|
|
|
326
327
|
// Decoupled state execution: Telemetry acts as a sensor sending updates to the Server core.
|
|
327
|
-
|
|
328
|
-
|
|
328
|
+
if (isMemoryLimitBreached && !server.isCordon) {
|
|
329
|
+
// Detailed log displaying active memory and threshold limits when Cordon activates instantly
|
|
330
|
+
console.warn(
|
|
331
|
+
`[@cocreate/server-telemetry] [CORDON TRIGGER] Server reached instantaneous memory limit! Firing cordon to sever new connections.\n` +
|
|
332
|
+
` » Active Memory: ${currentTelemetry.memory.percentage}% (Threshold: ${actualMemoryThreshold}%)`
|
|
333
|
+
);
|
|
334
|
+
|
|
329
335
|
server.send({
|
|
330
336
|
method: 'cordon',
|
|
331
337
|
broadcast: true,
|
|
332
338
|
active: true
|
|
333
339
|
});
|
|
334
|
-
} else if (!
|
|
340
|
+
} else if (!isMemoryLimitBreached && server.isCordon && isResourceSafeToRecover) {
|
|
341
|
+
// Detailed log showing that the worker has settled back under safe thresholds, letting traffic back in
|
|
342
|
+
console.log(
|
|
343
|
+
`[@cocreate/server-telemetry] [CORDON RECOVERY] Server has recovered to safe operational margins! Removing cordon.\n` +
|
|
344
|
+
` » Active Memory: ${currentTelemetry.memory.percentage}% (Safe limit must be < ${actualMemoryThreshold - recoveryBuffer}%)\n` +
|
|
345
|
+
` » Active CPU: ${currentTelemetry.cpu.totalUsage}% (Safe limit must be < ${actualCpuThreshold - recoveryBuffer}%)`
|
|
346
|
+
);
|
|
347
|
+
|
|
335
348
|
server.send({
|
|
336
349
|
method: 'cordon',
|
|
337
350
|
broadcast: true,
|
|
@@ -377,6 +390,30 @@ export async function flushTelemetryToDatabase() {
|
|
|
377
390
|
const cpuMax = Math.max(...cpuUsages);
|
|
378
391
|
const cpuAvg = cpuUsages.reduce((sum, val) => sum + val, 0) / totalSamples;
|
|
379
392
|
|
|
393
|
+
// --- STAGGERED AVERAGE CPU CORDON TRACKING ---
|
|
394
|
+
const cordonCpuThreshold = telemetryConfig
|
|
395
|
+
? parseInt(
|
|
396
|
+
getValueFromObject(telemetryConfig, 'server-telemetry.safetyLimits.cordonCpuThreshold') ||
|
|
397
|
+
getValueFromObject(telemetryConfig, 'server-telemetry.safetyOptions.cordonCpuThreshold') ||
|
|
398
|
+
getValueFromObject(telemetryConfig, 'server-telemetry.cordonCpuThreshold'), 10
|
|
399
|
+
)
|
|
400
|
+
: 95;
|
|
401
|
+
const actualCpuThreshold = isNaN(cordonCpuThreshold) ? 95 : cordonCpuThreshold;
|
|
402
|
+
|
|
403
|
+
// If average CPU utilization over the sample window breaches the threshold, fire cordon
|
|
404
|
+
if (cpuAvg >= actualCpuThreshold && !server.isCordon) {
|
|
405
|
+
console.warn(
|
|
406
|
+
`[@cocreate/server-telemetry] [CORDON TRIGGER - CPU AVERAGE] Server average CPU utilization over sample period breached threshold! Firing cordon to protect host.\n` +
|
|
407
|
+
` » Average CPU (over ${totalSamples} samples): ${cpuAvg.toFixed(2)}% (Threshold: ${actualCpuThreshold}%)`
|
|
408
|
+
);
|
|
409
|
+
|
|
410
|
+
server.send({
|
|
411
|
+
method: 'cordon',
|
|
412
|
+
broadcast: true,
|
|
413
|
+
active: true
|
|
414
|
+
});
|
|
415
|
+
}
|
|
416
|
+
|
|
380
417
|
const memMin = Math.min(...memUsages);
|
|
381
418
|
const memMax = Math.max(...memUsages);
|
|
382
419
|
const memAvg = memUsages.reduce((sum, val) => sum + val, 0) / totalSamples;
|
|
@@ -496,8 +533,8 @@ export function get() {
|
|
|
496
533
|
/**
|
|
497
534
|
* Integrates server-telemetry capabilities, loads configuration, and schedules staggered execution threads.
|
|
498
535
|
*/
|
|
499
|
-
export async function init(
|
|
500
|
-
server =
|
|
536
|
+
export async function init(serverInstance) {
|
|
537
|
+
server = serverInstance;
|
|
501
538
|
|
|
502
539
|
const telemetryAPI = {
|
|
503
540
|
init,
|
|
@@ -511,9 +548,7 @@ export async function init(serverCtx) {
|
|
|
511
548
|
flushTelemetryToDatabase
|
|
512
549
|
};
|
|
513
550
|
|
|
514
|
-
telemetryConfig = await Config(
|
|
515
|
-
'server-telemetry': ''
|
|
516
|
-
});
|
|
551
|
+
telemetryConfig = await Config.get('telemetry');
|
|
517
552
|
|
|
518
553
|
const telemetryEnabled = getValueFromObject(telemetryConfig, 'server-telemetry.enabled') !== false && getValueFromObject(telemetryConfig, 'server-telemetry.enabled') !== 'false';
|
|
519
554
|
|
|
@@ -537,7 +572,7 @@ export async function init(serverCtx) {
|
|
|
537
572
|
|
|
538
573
|
dbBuffer = [];
|
|
539
574
|
|
|
540
|
-
console.log(`[@cocreate/server-telemetry]
|
|
575
|
+
console.log(`[@cocreate/server-telemetry] [WORKER ${workerId}] Monitoring telemetry.`);
|
|
541
576
|
activeIntervalId = setInterval(read, collectionInterval);
|
|
542
577
|
|
|
543
578
|
return telemetryAPI;
|