whalewatch-agent 0.0.0-stage → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +67 -2
- package/bin/whalewatch-agent.js +45 -0
- package/docker-compose.yml +22 -0
- package/install.sh +467 -0
- package/package.json +30 -4
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Hemanth Reddy
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,3 +1,68 @@
|
|
|
1
|
-
#
|
|
1
|
+
# whale-agent
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Server monitoring for Whale: a pinned OpenTelemetry Collector (contrib 0.161.0,
|
|
4
|
+
sha256-checked) installed as a systemd service. Per host it reports:
|
|
5
|
+
|
|
6
|
+
- CPU, memory, load, disk I/O, filesystems (used / size) and network
|
|
7
|
+
- the busiest processes — CPU and memory once a minute; idle small ones are dropped,
|
|
8
|
+
except common databases, caches, queues and proxies (Redis, Postgres, RabbitMQ, nginx …)
|
|
9
|
+
- Docker containers (CPU, memory, network) when `/var/run/docker.sock` exists
|
|
10
|
+
- any log files you name, each under its own service in Logs
|
|
11
|
+
- the OS, and on AWS the instance type, region, zone and instance id
|
|
12
|
+
|
|
13
|
+
## Install
|
|
14
|
+
|
|
15
|
+
curl -fsSL https://app.whale.balajitech.xyz/agent/install.sh | sudo bash -s -- --key <ingest key>
|
|
16
|
+
|
|
17
|
+
Or from npm, on a server with Node 16 or later:
|
|
18
|
+
|
|
19
|
+
npx whalewatch-agent --key <ingest key>
|
|
20
|
+
|
|
21
|
+
It runs the same installer with the same options. Unless you're root it re-runs
|
|
22
|
+
itself with sudo, so expect a password prompt.
|
|
23
|
+
|
|
24
|
+
Linux (x86-64, ARM64) with systemd. `--help` lists the options: `--url` (self-hosted
|
|
25
|
+
ingest), `--project`, `--host`, `--logs NAME=GLOB`, `--interval`, `--no-processes`,
|
|
26
|
+
`--no-docker`, `--uninstall`. Re-running upgrades or reconfigures; a new config is
|
|
27
|
+
validated before it replaces the running one.
|
|
28
|
+
|
|
29
|
+
Files: `/opt/whale-agent` (collector), `/etc/whale-agent/config.yaml`,
|
|
30
|
+
`/etc/whale-agent/agent.env` (the key, mode 600), `/var/lib/whale-agent` (log
|
|
31
|
+
offsets). Its own logs: `journalctl -u whale-agent`.
|
|
32
|
+
|
|
33
|
+
## Docker instead of systemd
|
|
34
|
+
|
|
35
|
+
bash install.sh --print-config --root-path /hostfs > config.yaml
|
|
36
|
+
WHALE_API_KEY=<ingest key> docker compose up -d
|
|
37
|
+
|
|
38
|
+
## Notes
|
|
39
|
+
|
|
40
|
+
- A host that also runs whalewatch is one row in Hosts (both report the OS hostname).
|
|
41
|
+
The agent already covers host metrics, so `WHALE_HOST_METRICS=0` in those services
|
|
42
|
+
avoids sending them twice.
|
|
43
|
+
- Volume: about 45 datapoints per 30 s scrape plus a few per busy process per minute —
|
|
44
|
+
roughly 8M datapoints a month for a typical server.
|
|
45
|
+
- Exports are gzipped OTLP/HTTP protobuf with the `x-api-key` header. Ingest must
|
|
46
|
+
decompress gzip (`whale-api/src/ingest/decompress.ts`); deploy that before agents.
|
|
47
|
+
- Command lines never leave the machine (they can carry secrets); processes report
|
|
48
|
+
pid, executable name and owner.
|
|
49
|
+
- Upgrading the collector: bump `OTELCOL_VERSION` and both SHA256s (from the release's
|
|
50
|
+
`.tar.gz.sha256` assets), check the config with `otelcol-contrib validate`, then
|
|
51
|
+
`pnpm sync:agent` to refresh the copy the web app serves
|
|
52
|
+
(`whale-webapp/public/agent/install.sh`).
|
|
53
|
+
|
|
54
|
+
## AWS (CloudWatch + resource map)
|
|
55
|
+
|
|
56
|
+
`aws/whale-aws.yaml` is a CloudFormation stack, deployed once per account and region:
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
curl -fsSLO https://app.whale.balajitech.xyz/agent/aws/whale-aws.yaml
|
|
60
|
+
aws cloudformation deploy --stack-name whale-aws --template-file whale-aws.yaml \
|
|
61
|
+
--capabilities CAPABILITY_IAM --region ap-south-1 \
|
|
62
|
+
--parameter-overrides WhaleIngestKey=<ingest-key>
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
- **Metrics:** a CloudWatch metric stream (OpenTelemetry 1.0 format) → Kinesis Data Firehose → `POST /v1/aws/firehose` on the ingest, authenticated with the ingest key. Curated EC2, ALB/NLB, RDS, ElastiCache, Lambda and SQS metrics; edit `IncludeFilters` to change the set. Firehose needs an HTTPS endpoint — self-hosters set `WhaleIngestUrl`.
|
|
66
|
+
- **Map:** a read-only Lambda lists load balancers, target groups (with target health), EC2, RDS, ElastiCache, Lambda (with event-source mappings) and SQS every `InventoryMinutes` (10), infers compute → database/cache links from security groups, and sends the snapshot to `/v1/metrics` as OTLP (`aws.resource`, `aws.relation`). Invoke the function once after deploying to see the map immediately.
|
|
67
|
+
- **Cost:** AWS bills metric streams per metric update and Firehose per GB — the curated metric list keeps both small.
|
|
68
|
+
- **Removal:** `aws cloudformation delete-stack --stack-name whale-aws`. The failed-delivery S3 bucket is kept (objects expire after 7 days); delete it by hand if you want it gone.
|
|
@@ -0,0 +1,45 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
'use strict'
|
|
3
|
+
// npx whalewatch-agent --key <ingest key> [options]
|
|
4
|
+
//
|
|
5
|
+
// Runs the bundled install.sh (the same script Whale serves at
|
|
6
|
+
// /agent/install.sh) with the arguments given. Installing and uninstalling
|
|
7
|
+
// need root, so a non-root run goes through sudo; --help and --print-config
|
|
8
|
+
// run as the caller.
|
|
9
|
+
|
|
10
|
+
const { spawnSync } = require('node:child_process')
|
|
11
|
+
const path = require('node:path')
|
|
12
|
+
|
|
13
|
+
const script = path.join(__dirname, '..', 'install.sh')
|
|
14
|
+
const args = process.argv.slice(2)
|
|
15
|
+
const asCaller = args.some((a) => a === '-h' || a === '--help' || a === '--print-config')
|
|
16
|
+
|
|
17
|
+
// install.sh checks for root before it checks the OS, so say the real reason here.
|
|
18
|
+
if (!asCaller && process.platform !== 'linux') {
|
|
19
|
+
console.error('whalewatch-agent: the installer is Linux-only; elsewhere run the collector in Docker (see --print-config)')
|
|
20
|
+
process.exit(1)
|
|
21
|
+
}
|
|
22
|
+
|
|
23
|
+
const isRoot = typeof process.getuid === 'function' && process.getuid() === 0
|
|
24
|
+
let cmd = 'bash'
|
|
25
|
+
let argv = [script, ...args]
|
|
26
|
+
if (!asCaller && !isRoot) {
|
|
27
|
+
// sudo starts from a clean environment; carry the installer's own settings across.
|
|
28
|
+
const env = ['WHALE_API_KEY', 'WHALE_URL', 'WHALE_PROJECT', 'WHALE_HOST']
|
|
29
|
+
.filter((k) => process.env[k])
|
|
30
|
+
.map((k) => `${k}=${process.env[k]}`)
|
|
31
|
+
cmd = 'sudo'
|
|
32
|
+
argv = env.length > 0 ? ['env', ...env, 'bash', script, ...args] : ['bash', script, ...args]
|
|
33
|
+
}
|
|
34
|
+
|
|
35
|
+
const r = spawnSync(cmd, argv, { stdio: 'inherit' })
|
|
36
|
+
if (r.error) {
|
|
37
|
+
const missing = r.error.code === 'ENOENT'
|
|
38
|
+
console.error(
|
|
39
|
+
missing && cmd === 'sudo'
|
|
40
|
+
? 'whalewatch-agent: sudo not found; run this as root'
|
|
41
|
+
: `whalewatch-agent: ${missing ? `${cmd} not found` : r.error.message}`,
|
|
42
|
+
)
|
|
43
|
+
process.exit(1)
|
|
44
|
+
}
|
|
45
|
+
process.exit(r.status === null ? 1 : r.status)
|
|
@@ -0,0 +1,22 @@
|
|
|
1
|
+
# whale-agent in a container instead of a systemd service:
|
|
2
|
+
# bash install.sh --print-config --root-path /hostfs > config.yaml
|
|
3
|
+
# WHALE_API_KEY=<ingest key> docker compose up -d
|
|
4
|
+
services:
|
|
5
|
+
whale-agent:
|
|
6
|
+
image: otel/opentelemetry-collector-contrib:0.161.0
|
|
7
|
+
command: ["--config=/etc/whale-agent/config.yaml"]
|
|
8
|
+
restart: unless-stopped
|
|
9
|
+
network_mode: host # host interfaces and hostname
|
|
10
|
+
pid: host # host processes
|
|
11
|
+
user: "0:0" # other users' /proc entries and the Docker socket
|
|
12
|
+
environment:
|
|
13
|
+
WHALE_API_KEY: ${WHALE_API_KEY:?set WHALE_API_KEY}
|
|
14
|
+
WHALE_URL: ${WHALE_URL:-https://ingest.whale.balajitech.xyz}
|
|
15
|
+
OTEL_RESOURCE_ATTRIBUTES: service.name=whale-agent,whale.project=${WHALE_PROJECT:-default}
|
|
16
|
+
volumes:
|
|
17
|
+
- ./config.yaml:/etc/whale-agent/config.yaml:ro
|
|
18
|
+
- /:/hostfs:ro
|
|
19
|
+
- /var/run/docker.sock:/var/run/docker.sock:ro
|
|
20
|
+
- whale-agent-state:/var/lib/whale-agent
|
|
21
|
+
volumes:
|
|
22
|
+
whale-agent-state:
|
package/install.sh
ADDED
|
@@ -0,0 +1,467 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# whale-agent — server monitoring for Whale.
|
|
3
|
+
#
|
|
4
|
+
# Installs a pinned OpenTelemetry Collector (contrib build) as a systemd service
|
|
5
|
+
# that sends this machine's CPU, memory, load, disks, filesystems and network,
|
|
6
|
+
# its busiest processes, Docker containers and any log files you name to
|
|
7
|
+
# Whale's OTLP ingest. On EC2 the instance type, region and AZ come along.
|
|
8
|
+
#
|
|
9
|
+
# curl -fsSL https://<your-whale>/agent/install.sh | sudo bash -s -- --key <ingest key>
|
|
10
|
+
#
|
|
11
|
+
# Re-run it to upgrade or change settings; --uninstall removes everything.
|
|
12
|
+
# --print-config prints the collector config for running it in Docker instead.
|
|
13
|
+
#
|
|
14
|
+
# Source of truth: whale-agent/install.sh. The copy served at /agent/install.sh
|
|
15
|
+
# (whale-webapp/public/agent/) is refreshed with `pnpm sync:agent`.
|
|
16
|
+
#
|
|
17
|
+
# Everything runs from main() on the last line, so a truncated download
|
|
18
|
+
# executes nothing.
|
|
19
|
+
|
|
20
|
+
set -euo pipefail
|
|
21
|
+
|
|
22
|
+
OTELCOL_VERSION="0.161.0"
|
|
23
|
+
SHA256_AMD64="778c689efa681ff6e4722ce9f66b9b7f57c3ba009ab2e2b43dc2e0315862c731"
|
|
24
|
+
SHA256_ARM64="cd5de93213a0dbb90e4998b3b9e4e15ed691ec635cf7cf4147f95799fb16b676"
|
|
25
|
+
DEFAULT_URL="https://ingest.whale.balajitech.xyz" # = whale-node DEFAULT_INGEST_URL
|
|
26
|
+
|
|
27
|
+
PREFIX="/opt/whale-agent"
|
|
28
|
+
CONF_DIR="/etc/whale-agent"
|
|
29
|
+
STATE_DIR="/var/lib/whale-agent"
|
|
30
|
+
UNIT_FILE="/etc/systemd/system/whale-agent.service"
|
|
31
|
+
|
|
32
|
+
# Characters allowed in a log glob. No quotes, spaces or commas: commas separate
|
|
33
|
+
# globs, and everything lands inside single-quoted YAML.
|
|
34
|
+
GLOB_RE='/[A-Za-z0-9._/*?@+~-]+'
|
|
35
|
+
|
|
36
|
+
die() { echo "whale-agent: $*" >&2; exit 1; }
|
|
37
|
+
say() { echo "whale-agent: $*" >&2; }
|
|
38
|
+
|
|
39
|
+
usage() {
|
|
40
|
+
cat <<'EOF'
|
|
41
|
+
whale-agent — ships this server's metrics and logs to Whale.
|
|
42
|
+
|
|
43
|
+
curl -fsSL https://<your-whale>/agent/install.sh | sudo bash -s -- --key <ingest key> [options]
|
|
44
|
+
|
|
45
|
+
--key KEY Ingest key (Settings → API keys). Or set WHALE_API_KEY.
|
|
46
|
+
--url URL Ingest URL. Default: Whale's hosted ingest. Self-hosters must set it.
|
|
47
|
+
--project SLUG Project to file this host under (a project-scoped key wins).
|
|
48
|
+
--host NAME Report this name instead of the OS hostname.
|
|
49
|
+
--logs SPEC Tail log files as a service: NAME=GLOB[,GLOB...]. Repeatable.
|
|
50
|
+
e.g. --logs nginx=/var/log/nginx/*.log --logs api=/srv/api/*.log
|
|
51
|
+
--interval SECS Host metrics interval, 10-3600 (default 30).
|
|
52
|
+
--no-processes Don't report per-process CPU and memory.
|
|
53
|
+
--docker Report containers even if /var/run/docker.sock isn't here yet.
|
|
54
|
+
--no-docker Never report containers.
|
|
55
|
+
--root-path PATH Where the host's / is mounted when running in a container.
|
|
56
|
+
--print-config Print the collector config and exit (Docker / Kubernetes).
|
|
57
|
+
--uninstall Stop the agent and remove everything it installed.
|
|
58
|
+
EOF
|
|
59
|
+
}
|
|
60
|
+
|
|
61
|
+
parse_args() {
|
|
62
|
+
while [ $# -gt 0 ]; do
|
|
63
|
+
case "$1" in
|
|
64
|
+
--key | --url | --project | --host | --logs | --interval | --root-path)
|
|
65
|
+
[ $# -ge 2 ] || die "$1 needs a value"
|
|
66
|
+
case "$1" in
|
|
67
|
+
--key) KEY="$2" ;;
|
|
68
|
+
--url) URL="$2" ;;
|
|
69
|
+
--project) PROJECT="$2" ;;
|
|
70
|
+
--host) HOST_NAME="$2" ;;
|
|
71
|
+
--logs) LOGS="${LOGS}${2}"$'\n' ;;
|
|
72
|
+
--interval) INTERVAL="$2" ;;
|
|
73
|
+
--root-path) ROOT_PATH="$2" ;;
|
|
74
|
+
esac
|
|
75
|
+
shift 2
|
|
76
|
+
;;
|
|
77
|
+
--no-processes) PROCESSES=0; shift ;;
|
|
78
|
+
--docker) DOCKER=1; shift ;;
|
|
79
|
+
--no-docker) DOCKER=0; shift ;;
|
|
80
|
+
--print-config) MODE=print; shift ;;
|
|
81
|
+
--uninstall) MODE=uninstall; shift ;;
|
|
82
|
+
-h | --help) usage; exit 0 ;;
|
|
83
|
+
*) die "unknown option: $1 (try --help)" ;;
|
|
84
|
+
esac
|
|
85
|
+
done
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
matches() { printf '%s' "$1" | grep -Eq "$2"; }
|
|
89
|
+
|
|
90
|
+
validate_common() {
|
|
91
|
+
URL="${URL%/}"
|
|
92
|
+
matches "$URL" '^https?://[A-Za-z0-9.:_/-]+$' || die "--url must look like https://ingest.example.com"
|
|
93
|
+
matches "$PROJECT" '^[A-Za-z0-9._-]{1,64}$' || die "--project may only use letters, digits, . _ -"
|
|
94
|
+
if [ -n "$HOST_NAME" ]; then
|
|
95
|
+
matches "$HOST_NAME" '^[A-Za-z0-9._-]{1,253}$' || die "--host may only use letters, digits, . _ -"
|
|
96
|
+
fi
|
|
97
|
+
if ! matches "$INTERVAL" '^[0-9]+$' || [ "$INTERVAL" -lt 10 ] || [ "$INTERVAL" -gt 3600 ]; then
|
|
98
|
+
die "--interval must be 10-3600 seconds"
|
|
99
|
+
fi
|
|
100
|
+
if [ -n "$ROOT_PATH" ]; then
|
|
101
|
+
matches "$ROOT_PATH" '^/[A-Za-z0-9._/-]*$' || die "--root-path must be an absolute path"
|
|
102
|
+
ROOT_PATH="${ROOT_PATH%/}"
|
|
103
|
+
fi
|
|
104
|
+
local spec name
|
|
105
|
+
while IFS= read -r spec; do
|
|
106
|
+
[ -n "$spec" ] || continue
|
|
107
|
+
name="${spec%%=*}"
|
|
108
|
+
[ "$name" != "$spec" ] || die "--logs expects NAME=GLOB, got: $spec"
|
|
109
|
+
matches "$name" '^[a-z0-9][a-z0-9_-]{0,62}$' || die "log name must be lowercase letters, digits, _ -: $name"
|
|
110
|
+
matches "${spec#*=}" "^${GLOB_RE}(,${GLOB_RE})*\$" || die "log globs must be absolute paths without spaces or quotes: ${spec#*=}"
|
|
111
|
+
done <<EOF
|
|
112
|
+
$LOGS
|
|
113
|
+
EOF
|
|
114
|
+
}
|
|
115
|
+
|
|
116
|
+
validate_key() {
|
|
117
|
+
[ -n "$KEY" ] || die "an ingest key is required: --key <key> (create one under Settings → API keys)"
|
|
118
|
+
matches "$KEY" '^[A-Za-z0-9._:+/=-]{8,256}$' || die "that doesn't look like an ingest key"
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
docker_on() {
|
|
122
|
+
[ "$DOCKER" = 1 ] && return 0
|
|
123
|
+
[ "$DOCKER" = auto ] && [ -S /var/run/docker.sock ]
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
# One file_log receiver per --logs NAME, so each source files under its own
|
|
127
|
+
# service name in Whale's Logs tab.
|
|
128
|
+
log_receivers() {
|
|
129
|
+
local spec name globs out="" g sep
|
|
130
|
+
set -f
|
|
131
|
+
while IFS= read -r spec; do
|
|
132
|
+
[ -n "$spec" ] || continue
|
|
133
|
+
name="${spec%%=*}"
|
|
134
|
+
globs="${spec#*=}"
|
|
135
|
+
out="${out} file_log/${name}:"$'\n'" include: ["
|
|
136
|
+
sep=""
|
|
137
|
+
# set -f above: the globs must reach the collector unexpanded.
|
|
138
|
+
for g in $(printf '%s' "$globs" | tr ',' ' '); do
|
|
139
|
+
out="${out}${sep}'${ROOT_PATH}${g}'"
|
|
140
|
+
sep=", "
|
|
141
|
+
done
|
|
142
|
+
out="${out}]"$'\n'" start_at: end"$'\n'" storage: file_storage"$'\n'
|
|
143
|
+
out="${out} resource:"$'\n'" service.name: ${name}"$'\n'
|
|
144
|
+
done <<EOF
|
|
145
|
+
$LOGS
|
|
146
|
+
EOF
|
|
147
|
+
set +f
|
|
148
|
+
printf '%s' "$out"
|
|
149
|
+
}
|
|
150
|
+
|
|
151
|
+
log_receiver_names() {
|
|
152
|
+
local spec names=""
|
|
153
|
+
while IFS= read -r spec; do
|
|
154
|
+
[ -n "$spec" ] || continue
|
|
155
|
+
names="${names:+$names, }file_log/${spec%%=*}"
|
|
156
|
+
done <<EOF
|
|
157
|
+
$LOGS
|
|
158
|
+
EOF
|
|
159
|
+
printf '%s' "$names"
|
|
160
|
+
}
|
|
161
|
+
|
|
162
|
+
# Processes reported however idle (see filter/small_processes). An RE2 regex
|
|
163
|
+
# over the executable name; [.] keeps backslashes out of the OTTL string.
|
|
164
|
+
KEEP_PROCESSES='^(redis-server|valkey-server|keydb-server|memcached|beam[.]smp|nats-server|postgres|postmaster|mysqld|mariadbd|mongod|mongos|clickhouse-server|nginx|haproxy|caddy|traefik|envoy|etcd|consul|minio)$'
|
|
165
|
+
|
|
166
|
+
gen_config() {
|
|
167
|
+
local root_line="" metrics_receivers="host_metrics" logs_names process_block="" docker_block="" logs_pipeline=""
|
|
168
|
+
if [ -n "$ROOT_PATH" ]; then root_line=" root_path: ${ROOT_PATH}"; fi
|
|
169
|
+
|
|
170
|
+
if [ "$PROCESSES" = 1 ]; then
|
|
171
|
+
metrics_receivers="${metrics_receivers}, host_metrics/processes"
|
|
172
|
+
process_block="
|
|
173
|
+
# Per-process CPU and memory, once a minute. Only the two metrics the Hosts
|
|
174
|
+
# page shows, and filter/small_processes below drops idle, small processes —
|
|
175
|
+
# a busy server has hundreds and they would dominate the bill otherwise.
|
|
176
|
+
# Command lines stay on the machine: they can carry secrets.
|
|
177
|
+
host_metrics/processes:
|
|
178
|
+
collection_interval: 60s
|
|
179
|
+
${root_line}
|
|
180
|
+
scrapers:
|
|
181
|
+
process:
|
|
182
|
+
mute_process_all_errors: true
|
|
183
|
+
resource_attributes:
|
|
184
|
+
process.command_line: { enabled: false }
|
|
185
|
+
process.executable.path: { enabled: false }
|
|
186
|
+
metrics:
|
|
187
|
+
process.cpu.time: { enabled: false }
|
|
188
|
+
process.cpu.utilization: { enabled: true }
|
|
189
|
+
process.disk.io: { enabled: false }
|
|
190
|
+
process.memory.virtual: { enabled: false }
|
|
191
|
+
"
|
|
192
|
+
fi
|
|
193
|
+
|
|
194
|
+
if docker_on; then
|
|
195
|
+
metrics_receivers="${metrics_receivers}, docker_stats"
|
|
196
|
+
docker_block="
|
|
197
|
+
# Containers: CPU (docker-stats style — 100% is one core), memory, network.
|
|
198
|
+
docker_stats:
|
|
199
|
+
endpoint: unix:///var/run/docker.sock
|
|
200
|
+
collection_interval: ${INTERVAL}s
|
|
201
|
+
metrics:
|
|
202
|
+
container.blockio.io_service_bytes_recursive: { enabled: false }
|
|
203
|
+
container.cpu.usage.kernelmode: { enabled: false }
|
|
204
|
+
container.cpu.usage.total: { enabled: false }
|
|
205
|
+
container.cpu.usage.usermode: { enabled: false }
|
|
206
|
+
container.memory.file: { enabled: false }
|
|
207
|
+
container.memory.total_cache: { enabled: false }
|
|
208
|
+
container.memory.usage.limit: { enabled: false }
|
|
209
|
+
container.network.io.usage.rx_dropped: { enabled: false }
|
|
210
|
+
container.network.io.usage.tx_dropped: { enabled: false }
|
|
211
|
+
"
|
|
212
|
+
fi
|
|
213
|
+
|
|
214
|
+
logs_names="$(log_receiver_names)"
|
|
215
|
+
if [ -n "$logs_names" ]; then
|
|
216
|
+
logs_pipeline=" logs:
|
|
217
|
+
receivers: [${logs_names}]
|
|
218
|
+
processors: [memory_limiter, resource_detection, batch]
|
|
219
|
+
exporters: [otlp_http]"
|
|
220
|
+
fi
|
|
221
|
+
|
|
222
|
+
cat <<EOF
|
|
223
|
+
# whale-agent config — generated by install.sh for otelcol-contrib ${OTELCOL_VERSION}.
|
|
224
|
+
# Re-run the installer to change it; hand edits are overwritten.
|
|
225
|
+
|
|
226
|
+
extensions:
|
|
227
|
+
file_storage:
|
|
228
|
+
directory: ${STATE_DIR}
|
|
229
|
+
create_directory: true
|
|
230
|
+
|
|
231
|
+
receivers:
|
|
232
|
+
host_metrics:
|
|
233
|
+
collection_interval: ${INTERVAL}s
|
|
234
|
+
${root_line}
|
|
235
|
+
scrapers:
|
|
236
|
+
cpu:
|
|
237
|
+
metrics:
|
|
238
|
+
system.cpu.time: { enabled: false }
|
|
239
|
+
system.cpu.utilization: { enabled: true }
|
|
240
|
+
system.cpu.logical.count: { enabled: true }
|
|
241
|
+
load: {}
|
|
242
|
+
memory:
|
|
243
|
+
metrics:
|
|
244
|
+
system.memory.utilization: { enabled: true }
|
|
245
|
+
disk:
|
|
246
|
+
exclude:
|
|
247
|
+
devices: ['^(loop|ram|sr|fd|zram|nbd|dm-)[0-9]*$']
|
|
248
|
+
match_type: regexp
|
|
249
|
+
metrics:
|
|
250
|
+
system.disk.io_time: { enabled: false }
|
|
251
|
+
system.disk.merged: { enabled: false }
|
|
252
|
+
system.disk.operation_time: { enabled: false }
|
|
253
|
+
system.disk.operations: { enabled: false }
|
|
254
|
+
system.disk.pending_operations: { enabled: false }
|
|
255
|
+
system.disk.weighted_io_time: { enabled: false }
|
|
256
|
+
filesystem:
|
|
257
|
+
exclude_fs_types:
|
|
258
|
+
fs_types: [autofs, binfmt_misc, bpf, cgroup, cgroup2, configfs, debugfs, devpts, devtmpfs, efivarfs, fuse.lxcfs, fusectl, hugetlbfs, mqueue, nsfs, overlay, proc, pstore, rpc_pipefs, securityfs, selinuxfs, squashfs, sysfs, tmpfs, tracefs]
|
|
259
|
+
match_type: strict
|
|
260
|
+
exclude_mount_points:
|
|
261
|
+
mount_points: ['^/(dev|proc|sys|run|snap|var/lib/docker|var/lib/containerd|var/lib/kubelet)(/|$)']
|
|
262
|
+
match_type: regexp
|
|
263
|
+
metrics:
|
|
264
|
+
system.filesystem.inodes.usage: { enabled: false }
|
|
265
|
+
system.filesystem.utilization: { enabled: true }
|
|
266
|
+
network:
|
|
267
|
+
exclude:
|
|
268
|
+
interfaces: ['^(lo|docker[0-9]*|veth.*|br-.*|cni.*|flannel.*|cali.*|virbr.*)$']
|
|
269
|
+
match_type: regexp
|
|
270
|
+
metrics:
|
|
271
|
+
system.network.connections: { enabled: false }
|
|
272
|
+
system.network.dropped: { enabled: false }
|
|
273
|
+
system.network.errors: { enabled: false }
|
|
274
|
+
system.network.packets: { enabled: false }
|
|
275
|
+
# system.uptime also carries the host's OS / cloud details into Whale.
|
|
276
|
+
system:
|
|
277
|
+
metrics:
|
|
278
|
+
system.uptime: { enabled: true }
|
|
279
|
+
${process_block}${docker_block}$(log_receivers)
|
|
280
|
+
processors:
|
|
281
|
+
memory_limiter:
|
|
282
|
+
check_interval: 1s
|
|
283
|
+
limit_mib: 256
|
|
284
|
+
spike_limit_mib: 64
|
|
285
|
+
# host.name comes from the OS hostname — the same value whalewatch reports —
|
|
286
|
+
# so a host running both lines up as one row. ec2 adds instance type,
|
|
287
|
+
# region and AZ on AWS and does nothing elsewhere.
|
|
288
|
+
resource_detection:
|
|
289
|
+
detectors: [env, system, ec2]
|
|
290
|
+
timeout: 5s
|
|
291
|
+
override: false
|
|
292
|
+
system:
|
|
293
|
+
hostname_sources: [os]
|
|
294
|
+
resource_attributes:
|
|
295
|
+
host.arch: { enabled: true }
|
|
296
|
+
os.description: { enabled: true }
|
|
297
|
+
# Databases, caches, queues and proxies always pass, however idle: a quiet
|
|
298
|
+
# Redis is still one someone watches (Infrastructure matches services by
|
|
299
|
+
# process name, so a dropped idle daemon would read as stopped).
|
|
300
|
+
filter/small_processes:
|
|
301
|
+
error_mode: ignore
|
|
302
|
+
metrics:
|
|
303
|
+
datapoint:
|
|
304
|
+
- 'metric.name == "process.cpu.utilization" and value_double < 0.005 and not IsMatch(resource.attributes["process.executable.name"], "${KEEP_PROCESSES}")'
|
|
305
|
+
- 'metric.name == "process.memory.usage" and value_int < 52428800 and not IsMatch(resource.attributes["process.executable.name"], "${KEEP_PROCESSES}")'
|
|
306
|
+
batch:
|
|
307
|
+
timeout: 10s
|
|
308
|
+
|
|
309
|
+
exporters:
|
|
310
|
+
otlp_http:
|
|
311
|
+
endpoint: \${env:WHALE_URL}
|
|
312
|
+
headers:
|
|
313
|
+
x-api-key: \${env:WHALE_API_KEY}
|
|
314
|
+
compression: gzip
|
|
315
|
+
|
|
316
|
+
service:
|
|
317
|
+
extensions: [file_storage]
|
|
318
|
+
telemetry:
|
|
319
|
+
logs:
|
|
320
|
+
level: warn
|
|
321
|
+
metrics:
|
|
322
|
+
level: none
|
|
323
|
+
pipelines:
|
|
324
|
+
metrics:
|
|
325
|
+
receivers: [${metrics_receivers}]
|
|
326
|
+
processors: [memory_limiter, resource_detection, filter/small_processes, batch]
|
|
327
|
+
exporters: [otlp_http]
|
|
328
|
+
${logs_pipeline}
|
|
329
|
+
EOF
|
|
330
|
+
}
|
|
331
|
+
|
|
332
|
+
write_env() {
|
|
333
|
+
local attrs="service.name=whale-agent,whale.project=${PROJECT}"
|
|
334
|
+
if [ -n "$HOST_NAME" ]; then attrs="${attrs},host.name=${HOST_NAME}"; fi
|
|
335
|
+
printf 'WHALE_API_KEY="%s"\nWHALE_URL="%s"\nOTEL_RESOURCE_ATTRIBUTES="%s"\n' "$KEY" "$URL" "$attrs"
|
|
336
|
+
}
|
|
337
|
+
|
|
338
|
+
write_unit() {
|
|
339
|
+
cat <<EOF
|
|
340
|
+
[Unit]
|
|
341
|
+
Description=Whale agent (OpenTelemetry Collector ${OTELCOL_VERSION})
|
|
342
|
+
After=network-online.target
|
|
343
|
+
Wants=network-online.target
|
|
344
|
+
|
|
345
|
+
[Service]
|
|
346
|
+
# root: other users' /proc entries and the Docker socket need it.
|
|
347
|
+
EnvironmentFile=${CONF_DIR}/agent.env
|
|
348
|
+
ExecStart=${PREFIX}/otelcol-contrib --config=${CONF_DIR}/config.yaml
|
|
349
|
+
Restart=always
|
|
350
|
+
RestartSec=10
|
|
351
|
+
MemoryMax=512M
|
|
352
|
+
Nice=10
|
|
353
|
+
|
|
354
|
+
[Install]
|
|
355
|
+
WantedBy=multi-user.target
|
|
356
|
+
EOF
|
|
357
|
+
}
|
|
358
|
+
|
|
359
|
+
# A 401 here beats a silently rejected agent.
|
|
360
|
+
preflight() {
|
|
361
|
+
local code
|
|
362
|
+
code="$(curl -s -o /dev/null -w '%{http_code}' -m 15 -X POST "${URL}/v1/metrics" \
|
|
363
|
+
-H "x-api-key: ${KEY}" -H 'content-type: application/json' \
|
|
364
|
+
--data '{"resourceMetrics":[]}' || true)"
|
|
365
|
+
case "$code" in
|
|
366
|
+
2??) ;;
|
|
367
|
+
401 | 403) die "${URL} rejected the key (HTTP ${code}) — check --key, and --url if you self-host" ;;
|
|
368
|
+
*) say "warning: couldn't confirm ${URL} (HTTP ${code:-no response}); installing anyway" ;;
|
|
369
|
+
esac
|
|
370
|
+
}
|
|
371
|
+
|
|
372
|
+
install_agent() {
|
|
373
|
+
local arch sum tgz tmp
|
|
374
|
+
[ "$(id -u)" -eq 0 ] || die "run as root: curl ... | sudo bash -s -- --key <key>"
|
|
375
|
+
[ "$(uname -s)" = Linux ] || die "the installer is Linux-only; elsewhere run the collector in Docker (see --print-config)"
|
|
376
|
+
command -v systemctl >/dev/null 2>&1 || die "systemd not found; use --print-config and run the collector in Docker"
|
|
377
|
+
for c in curl tar sha256sum; do command -v "$c" >/dev/null 2>&1 || die "$c is required"; done
|
|
378
|
+
case "$(uname -m)" in
|
|
379
|
+
x86_64 | amd64) arch=amd64; sum="$SHA256_AMD64" ;;
|
|
380
|
+
aarch64 | arm64) arch=arm64; sum="$SHA256_ARM64" ;;
|
|
381
|
+
*) die "unsupported CPU: $(uname -m)" ;;
|
|
382
|
+
esac
|
|
383
|
+
|
|
384
|
+
preflight
|
|
385
|
+
mkdir -p "$PREFIX" "$CONF_DIR" "$STATE_DIR"
|
|
386
|
+
|
|
387
|
+
if [ "$(cat "$PREFIX/VERSION" 2>/dev/null || true)" != "$OTELCOL_VERSION" ]; then
|
|
388
|
+
tmp="$(mktemp -d)"
|
|
389
|
+
WORK="$tmp"
|
|
390
|
+
tgz="otelcol-contrib_${OTELCOL_VERSION}_linux_${arch}.tar.gz"
|
|
391
|
+
say "downloading OpenTelemetry Collector ${OTELCOL_VERSION} (${arch}, ~110 MB)"
|
|
392
|
+
curl -fsSL --retry 3 -o "$tmp/$tgz" \
|
|
393
|
+
"https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v${OTELCOL_VERSION}/${tgz}"
|
|
394
|
+
echo "${sum} $tmp/$tgz" | sha256sum -c - >/dev/null 2>&1 || die "checksum mismatch on ${tgz} — not installing it"
|
|
395
|
+
tar -xzf "$tmp/$tgz" -C "$tmp" otelcol-contrib
|
|
396
|
+
install -m 0755 "$tmp/otelcol-contrib" "$PREFIX/otelcol-contrib.new"
|
|
397
|
+
mv -f "$PREFIX/otelcol-contrib.new" "$PREFIX/otelcol-contrib"
|
|
398
|
+
echo "$OTELCOL_VERSION" >"$PREFIX/VERSION"
|
|
399
|
+
fi
|
|
400
|
+
|
|
401
|
+
# Write next to the live files and validate before swapping them in, so a
|
|
402
|
+
# bad flag never takes down a running agent.
|
|
403
|
+
(umask 077 && write_env >"$CONF_DIR/agent.env.new")
|
|
404
|
+
gen_config >"$CONF_DIR/config.yaml.new"
|
|
405
|
+
if ! (set -a && . "$CONF_DIR/agent.env.new" && set +a &&
|
|
406
|
+
"$PREFIX/otelcol-contrib" validate --config="$CONF_DIR/config.yaml.new"); then
|
|
407
|
+
rm -f "$CONF_DIR/agent.env.new" "$CONF_DIR/config.yaml.new"
|
|
408
|
+
die "the generated config didn't validate — nothing changed"
|
|
409
|
+
fi
|
|
410
|
+
mv -f "$CONF_DIR/agent.env.new" "$CONF_DIR/agent.env"
|
|
411
|
+
mv -f "$CONF_DIR/config.yaml.new" "$CONF_DIR/config.yaml"
|
|
412
|
+
chmod 600 "$CONF_DIR/agent.env"
|
|
413
|
+
chmod 644 "$CONF_DIR/config.yaml"
|
|
414
|
+
|
|
415
|
+
write_unit >"$UNIT_FILE"
|
|
416
|
+
systemctl daemon-reload
|
|
417
|
+
systemctl enable whale-agent >/dev/null 2>&1
|
|
418
|
+
systemctl restart whale-agent
|
|
419
|
+
sleep 5
|
|
420
|
+
if ! systemctl is-active --quiet whale-agent; then
|
|
421
|
+
journalctl -u whale-agent -n 30 --no-pager >&2 || true
|
|
422
|
+
die "the service didn't stay up — log above"
|
|
423
|
+
fi
|
|
424
|
+
say "running. $(hostname) shows up under Hosts within a minute."
|
|
425
|
+
if [ -n "$(log_receiver_names)" ]; then say "tailing new lines of: $(log_receiver_names)"; fi
|
|
426
|
+
say "config: ${CONF_DIR}/config.yaml · logs: journalctl -u whale-agent"
|
|
427
|
+
}
|
|
428
|
+
|
|
429
|
+
uninstall_agent() {
|
|
430
|
+
[ "$(id -u)" -eq 0 ] || die "run as root"
|
|
431
|
+
systemctl disable --now whale-agent >/dev/null 2>&1 || true
|
|
432
|
+
rm -f "$UNIT_FILE"
|
|
433
|
+
systemctl daemon-reload >/dev/null 2>&1 || true
|
|
434
|
+
rm -rf "$PREFIX" "$CONF_DIR" "$STATE_DIR"
|
|
435
|
+
say "removed."
|
|
436
|
+
}
|
|
437
|
+
|
|
438
|
+
main() {
|
|
439
|
+
WORK=""
|
|
440
|
+
trap '[ -z "$WORK" ] || rm -rf "$WORK"' EXIT
|
|
441
|
+
KEY="${WHALE_API_KEY:-}"
|
|
442
|
+
URL="${WHALE_URL:-$DEFAULT_URL}"
|
|
443
|
+
PROJECT="${WHALE_PROJECT:-default}"
|
|
444
|
+
HOST_NAME="${WHALE_HOST:-}"
|
|
445
|
+
INTERVAL=30
|
|
446
|
+
PROCESSES=1
|
|
447
|
+
DOCKER=auto
|
|
448
|
+
ROOT_PATH=""
|
|
449
|
+
LOGS=""
|
|
450
|
+
MODE=install
|
|
451
|
+
parse_args "$@"
|
|
452
|
+
case "$MODE" in
|
|
453
|
+
uninstall) uninstall_agent ;;
|
|
454
|
+
print)
|
|
455
|
+
validate_common
|
|
456
|
+
gen_config
|
|
457
|
+
say "run it with WHALE_API_KEY, WHALE_URL and OTEL_RESOURCE_ATTRIBUTES=service.name=whale-agent,whale.project=${PROJECT} set"
|
|
458
|
+
;;
|
|
459
|
+
install)
|
|
460
|
+
validate_common
|
|
461
|
+
validate_key
|
|
462
|
+
install_agent
|
|
463
|
+
;;
|
|
464
|
+
esac
|
|
465
|
+
}
|
|
466
|
+
|
|
467
|
+
main "$@"
|
package/package.json
CHANGED
|
@@ -1,6 +1,32 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "whalewatch-agent",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Server monitoring for Whale. Installs a pinned OpenTelemetry Collector as a systemd service that ships this server's metrics, processes, containers and logs to Whale.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"homepage": "https://whale.balajitech.xyz/docs/",
|
|
7
|
+
"bin": {
|
|
8
|
+
"whalewatch-agent": "bin/whalewatch-agent.js"
|
|
9
|
+
},
|
|
10
|
+
"files": [
|
|
11
|
+
"bin",
|
|
12
|
+
"install.sh",
|
|
13
|
+
"docker-compose.yml",
|
|
14
|
+
"README.md",
|
|
15
|
+
"LICENSE"
|
|
16
|
+
],
|
|
17
|
+
"engines": {
|
|
18
|
+
"node": ">=16"
|
|
19
|
+
},
|
|
20
|
+
"keywords": [
|
|
21
|
+
"observability",
|
|
22
|
+
"monitoring",
|
|
23
|
+
"opentelemetry",
|
|
24
|
+
"otel",
|
|
25
|
+
"collector",
|
|
26
|
+
"agent",
|
|
27
|
+
"metrics",
|
|
28
|
+
"logs",
|
|
29
|
+
"systemd",
|
|
30
|
+
"whale"
|
|
31
|
+
]
|
|
32
|
+
}
|