mdbkit 0.5.0__tar.gz → 0.5.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {mdbkit-0.5.0 → mdbkit-0.5.2}/PKG-INFO +1120 -1062
- {mdbkit-0.5.0 → mdbkit-0.5.2}/README.md +59 -4
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/__init__.py +1 -1
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/PKG-INFO +1120 -1062
- {mdbkit-0.5.0 → mdbkit-0.5.2}/pyproject.toml +13 -4
- {mdbkit-0.5.0 → mdbkit-0.5.2}/setup.cfg +4 -4
- {mdbkit-0.5.0 → mdbkit-0.5.2}/LICENSE +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/advisor.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/analysis.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/cli.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/compare.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/demo.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/explain.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/filtering.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/ftdc.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/lab.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/oslog.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/parser.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/rebuild.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/render.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/report.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/scripts.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/serverstatus.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit/triage.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/SOURCES.txt +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/dependency_links.txt +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/entry_points.txt +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/requires.txt +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/mdbkit.egg-info/top_level.txt +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_demo_lab.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_explain.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_ftdc.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_mdbkit.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_oslog_serverstatus.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_rebuild_report.py +0 -0
- {mdbkit-0.5.0 → mdbkit-0.5.2}/tests/test_triage.py +0 -0
|
@@ -1,1062 +1,1120 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: mdbkit
|
|
3
|
-
Version: 0.5.
|
|
4
|
-
Summary: Offline toolkit for MongoDB 4.4+ structured logs: log analysis, slow-query shapes, connection churn, and deterministic index advice. A spiritual successor to mtools' log tools.
|
|
5
|
-
Author: Saqib Ameen Subhan
|
|
6
|
-
License: MIT
|
|
7
|
-
Project-URL: Homepage, https://github.com/
|
|
8
|
-
Project-URL:
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
Classifier:
|
|
14
|
-
Classifier:
|
|
15
|
-
Classifier:
|
|
16
|
-
Classifier:
|
|
17
|
-
Classifier:
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
[](https://github.com/saqibameen86/mdbkit/actions)
|
|
31
|
+
[](https://pypi.org/project/mdbkit/)
|
|
32
|
+
[](https://pypi.org/project/mdbkit/)
|
|
33
|
+
[](LICENSE)
|
|
34
|
+
|
|
35
|
+
**An offline toolkit for MongoDB structured logs** — slow-query analysis,
|
|
36
|
+
deterministic index advice, incident triage, and diagnostic-data decoding.
|
|
37
|
+
For MongoDB 4.4 – 8.0, from the terminal, without connecting to anything.
|
|
38
|
+
|
|
39
|
+
A spiritual successor to mtools' log tools, which never learned to read the
|
|
40
|
+
JSON log format introduced in 4.4.
|
|
41
|
+
|
|
42
|
+
```
|
|
43
|
+
namespace op count cumMs docsEx scan plan shape
|
|
44
|
+
shop.events aggregate 29 3.2m 25,810,000 98889:1 COLLSCAN+SORT {tenantId:eq, ts:gte} sort:{ts:-1}
|
|
45
|
+
shop.orders find 48 1.4m 6,000,000 2976:1 COLLSCAN+SORT {status:eq, createdAt:gt}
|
|
46
|
+
shop.users find 31 3.7s 31 1:1 IXSCAN{email} {email:eq}
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Two of those need an index. One is already fine. That distinction is the
|
|
50
|
+
whole point.
|
|
51
|
+
|
|
52
|
+
---
|
|
53
|
+
|
|
54
|
+
## Try it right now — no MongoDB required
|
|
55
|
+
|
|
56
|
+
`mdbkit demo` writes a realistic log containing a real incident, so you can
|
|
57
|
+
evaluate the tool in about a minute without touching a cluster:
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
pip install mdbkit # macOS has no pip by default — see Install below
|
|
61
|
+
|
|
62
|
+
mdbkit demo --with-extras -o demo.log # a log + indexes.json, schema.json, explain.json
|
|
63
|
+
|
|
64
|
+
mdbkit loginfo demo.log # what is in this log?
|
|
65
|
+
mdbkit queries demo.log # which query shapes cost the most?
|
|
66
|
+
mdbkit triage demo.log --window 0 # what went wrong, and when?
|
|
67
|
+
mdbkit connections demo.log # who connected, and did anyone fail to?
|
|
68
|
+
mdbkit advise demo.log --indexes indexes.json --schema schema.json
|
|
69
|
+
mdbkit explain explain.json # read a saved explain plan
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
The generated log contains a connection storm from one client, a replica set
|
|
73
|
+
election, an index build, five failed logins from a service account, and an
|
|
74
|
+
aggregation burning 25 million document reads to return 261 documents — then
|
|
75
|
+
`advise` tells you which index fixes it.
|
|
76
|
+
|
|
77
|
+
Output is deterministic: the same `--seed` always produces the same log, so a
|
|
78
|
+
demo behaves identically every time. Scenarios are `incident`, `healthy` (the
|
|
79
|
+
control case — useful for seeing what "nothing wrong" looks like) and `mixed`.
|
|
80
|
+
|
|
81
|
+
Ready for a real server? Jump to [the workflows](#the-four-questions-it-answers).
|
|
82
|
+
|
|
83
|
+
---
|
|
84
|
+
|
|
85
|
+
## Is it safe to run on a production server?
|
|
86
|
+
|
|
87
|
+
This is the right question to ask of any tool someone hands you. The honest
|
|
88
|
+
answer, and how to check it yourself.
|
|
89
|
+
|
|
90
|
+
**What mdbkit never does:**
|
|
91
|
+
|
|
92
|
+
| | |
|
|
93
|
+
|---|---|
|
|
94
|
+
| Connect to your database | Analysis commands read **files**. There is no driver, no URI, no connection. |
|
|
95
|
+
| Send anything anywhere | There is no network code at all. No telemetry, no update check, no crash reporting. |
|
|
96
|
+
| Change anything | It is strictly read-only. Where an action would help, it **prints the command** for you to review and run. |
|
|
97
|
+
| Execute what it reads | Log lines and explain files are parsed as data with `json.loads`. Nothing is ever evaluated. |
|
|
98
|
+
| Pull in dependencies | Zero runtime dependencies. Nothing in the supply chain but the Python standard library. |
|
|
99
|
+
|
|
100
|
+
**Verify it yourself in 60 seconds** — this is a small, dependency-free
|
|
101
|
+
codebase specifically so that you can:
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
# 1. No network, no shell-outs, no eval anywhere in the analysis code
|
|
105
|
+
pip show -f mdbkit | head -3
|
|
106
|
+
grep -rn "socket\|urllib\|requests\|http\|eval(\|exec(" $(python -c "import mdbkit,os;print(os.path.dirname(mdbkit.__file__))")
|
|
107
|
+
|
|
108
|
+
# 2. Confirm it has no dependencies
|
|
109
|
+
pip show mdbkit | grep Requires
|
|
110
|
+
|
|
111
|
+
# 3. Watch it make no connections while it runs (Linux)
|
|
112
|
+
strace -f -e trace=network mdbkit queries mongod.log 2>&1 | grep -c socket
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
The grep returns nothing for every analysis module. The only file that starts
|
|
116
|
+
a process is `lab.py`, which exists to create a *throwaway test cluster* and
|
|
117
|
+
is documented as an explicit exception below.
|
|
118
|
+
|
|
119
|
+
**What it does read:** the log file you point it at; optionally
|
|
120
|
+
`diagnostic.data` (metrics only, never documents); optionally `indexes.json`
|
|
121
|
+
and `schema.json` that **you** generate with scripts mdbkit prints for you to
|
|
122
|
+
inspect first. On the database host it also reads `/proc` and calls `statvfs`
|
|
123
|
+
for disk and memory figures — nothing that leaves the machine.
|
|
124
|
+
|
|
125
|
+
**What leaves your machine: nothing.** There is no server to send it to.
|
|
126
|
+
|
|
127
|
+
**Still cautious?** That is reasonable. Run `mdbkit demo` first and see what
|
|
128
|
+
the output looks like on synthetic data, or `mdbkit lab` to try it against a
|
|
129
|
+
disposable local cluster before you point it at anything real. Both exist for
|
|
130
|
+
exactly this reason.
|
|
131
|
+
|
|
132
|
+
Full detail: [SECURITY.md](SECURITY.md).
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
|
|
136
|
+
## Install
|
|
137
|
+
|
|
138
|
+
### macOS
|
|
139
|
+
|
|
140
|
+
macOS ships no `pip` and no `pipx`, so `pip install` fails out of the box.
|
|
141
|
+
Pick whichever of these matches your setup.
|
|
142
|
+
|
|
143
|
+
**With Homebrew (recommended — keeps mdbkit in its own environment):**
|
|
144
|
+
```bash
|
|
145
|
+
brew install pipx
|
|
146
|
+
pipx ensurepath # then open a new terminal window
|
|
147
|
+
pipx install mdbkit
|
|
148
|
+
```
|
|
149
|
+
|
|
150
|
+
**Without Homebrew, using the Python that comes with macOS:**
|
|
151
|
+
```bash
|
|
152
|
+
python3 --version # accept the Command Line Tools prompt if it appears
|
|
153
|
+
python3 -m pip install --user mdbkit
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Then put the install location on your `PATH` — macOS does not do this for
|
|
157
|
+
you:
|
|
158
|
+
```bash
|
|
159
|
+
echo 'export PATH="$HOME/Library/Python/'"$(python3 -c 'import sys;print(f"{sys.version_info.major}.{sys.version_info.minor}")')"'/bin:$PATH"' >> ~/.zshrc
|
|
160
|
+
source ~/.zshrc
|
|
161
|
+
mdbkit --version
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
**If you use uv:**
|
|
165
|
+
```bash
|
|
166
|
+
uv tool install mdbkit
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
**If pip reports `externally-managed-environment`**, add
|
|
170
|
+
`--break-system-packages`, or use the pipx route above, which avoids the
|
|
171
|
+
problem entirely.
|
|
172
|
+
|
|
173
|
+
Do not install with `sudo`. mdbkit is a user-level CLI and needs no
|
|
174
|
+
elevated privileges — not to install, and not to run.
|
|
175
|
+
|
|
176
|
+
### Linux
|
|
177
|
+
|
|
178
|
+
**Most distributions:**
|
|
179
|
+
```bash
|
|
180
|
+
pip install mdbkit
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
**Ubuntu 20.04 / Debian / Amazon Linux 2 (Python 3.8 hosts):**
|
|
184
|
+
```bash
|
|
185
|
+
sudo apt install pipx # or: sudo dnf install pipx
|
|
186
|
+
pipx install mdbkit
|
|
187
|
+
pipx ensurepath && source ~/.bashrc
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
**Modern Ubuntu/Debian complaining about "externally-managed-environment":**
|
|
191
|
+
```bash
|
|
192
|
+
pip install mdbkit --break-system-packages
|
|
193
|
+
```
|
|
194
|
+
|
|
195
|
+
### Windows
|
|
196
|
+
|
|
197
|
+
```powershell
|
|
198
|
+
py -m pip install mdbkit
|
|
199
|
+
py -m mdbkit --version
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
If `mdbkit` is not recognised as a command afterwards, the Scripts directory
|
|
203
|
+
is not on your `PATH`; `py -m mdbkit` works regardless.
|
|
204
|
+
|
|
205
|
+
### Air-gapped database hosts
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
pip download mdbkit -d ./wheels # on a connected machine
|
|
209
|
+
# copy ./wheels across, then:
|
|
210
|
+
pip install --no-index --find-links ./wheels mdbkit
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Zero runtime dependencies means this is a single wheel — nothing else to
|
|
214
|
+
resolve.
|
|
215
|
+
|
|
216
|
+
### Upgrading
|
|
217
|
+
|
|
218
|
+
```bash
|
|
219
|
+
pip install --upgrade mdbkit # or: pipx upgrade mdbkit
|
|
220
|
+
mdbkit --version
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Requires Python 3.8+. mdbkit never updates itself and never checks for
|
|
224
|
+
updates — upgrades are always explicit.
|
|
225
|
+
|
|
226
|
+
> mdbkit is a Python package on PyPI. It is **not** in `apt`/`dnf`/`yum`.
|
|
227
|
+
|
|
228
|
+
---
|
|
229
|
+
|
|
230
|
+
## The four questions it answers
|
|
231
|
+
|
|
232
|
+
Every command reads files or stdin and accepts several files or a glob, so
|
|
233
|
+
rotated logs work as one stream: `mdbkit queries "mongod.log*"`.
|
|
234
|
+
|
|
235
|
+
### 1. Why is my database slow?
|
|
236
|
+
|
|
237
|
+
```bash
|
|
238
|
+
mdbkit queries mongod.log # shapes ranked by total time
|
|
239
|
+
mdbkit queries mongod.log --sort scanRatio # worst examined:returned first
|
|
240
|
+
mdbkit queries mongod.log --shape 1 # full detail on one shape
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
`cumMs` is time summed across **all** occurrences of a shape, not one query.
|
|
244
|
+
`scan` is documents examined per document returned — `1:1` is healthy,
|
|
245
|
+
`98889:1` is a missing index. `plan` shows what MongoDB actually chose.
|
|
246
|
+
|
|
247
|
+
### 2. What index would fix it?
|
|
248
|
+
|
|
249
|
+
```bash
|
|
250
|
+
# Optional but much sharper: export what already exists.
|
|
251
|
+
mdbkit export-script indexes > export_indexes.js
|
|
252
|
+
mdbkit export-script schema > export_schema.js
|
|
253
|
+
|
|
254
|
+
mongosh --quiet --host your_db_host --port 27017 \
|
|
255
|
+
--username your_username --password your_password \
|
|
256
|
+
--authenticationDatabase admin \
|
|
257
|
+
--eval "$(cat export_indexes.js)" > indexes.json # repeat for schema
|
|
258
|
+
|
|
259
|
+
mdbkit advise mongod.log --indexes indexes.json --schema schema.json --ns shop.orders
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
Every recommendation states the evidence it reasoned from, a confidence
|
|
263
|
+
level, the caveats, and how to validate it. It says *candidate*, not
|
|
264
|
+
*command*, and it never tells you to drop an index.
|
|
265
|
+
|
|
266
|
+
### 3. What happened at 3am?
|
|
267
|
+
|
|
268
|
+
```bash
|
|
269
|
+
mdbkit triage /var/log/mongodb/mongod.log # last 60 minutes by default
|
|
270
|
+
mdbkit triage mongod.log --window 0 # the whole file
|
|
271
|
+
mdbkit triage mongod.log --report incident.html # something to attach to a ticket
|
|
272
|
+
```
|
|
273
|
+
|
|
274
|
+
Cluster health, elections, connection storms, hot collections, index builds,
|
|
275
|
+
error clusters, slow-query peaks — plus disk, memory, CPU and FTDC metrics
|
|
276
|
+
when run on the database host. Every finding ends with the next command to
|
|
277
|
+
run.
|
|
278
|
+
|
|
279
|
+
### 4. Did my change actually help?
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
mdbkit compare before.log --after after.log
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
```
|
|
286
|
+
slow-query time DOWN 32% (6.0m -> 4.1m across compared shapes)
|
|
287
|
+
shapes: 1 improved, 0 regressed, 0 new, 0 gone, 4 unchanged
|
|
288
|
+
|
|
289
|
+
IMPROVED
|
|
290
|
+
shop.orders {createdAt:gt, status:eq} sort:{createdAt:-1}
|
|
291
|
+
mean 1.7s -> 33ms (-98%) scan 2976:1 -> 1:1 [COLLSCAN -> index, in-memory sort gone]
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
The natural follow-up to `advise`: you created the index, a day passed, and
|
|
295
|
+
this tells you whether it worked.
|
|
296
|
+
|
|
297
|
+
### 5. Why did it die? (and what does serverStatus say?)
|
|
298
|
+
|
|
299
|
+
```bash
|
|
300
|
+
mdbkit oslog /var/log/syslog # OOM kills, fd limits, I/O errors
|
|
301
|
+
mdbkit triage mongod.log --oslog /var/log/syslog
|
|
302
|
+
```
|
|
303
|
+
|
|
304
|
+
A mongod log cannot record its own OOM kill — the process is gone before it
|
|
305
|
+
can write anything. The system log has the answer in one line, and `triage`
|
|
306
|
+
will correlate it with the unexplained restart.
|
|
307
|
+
|
|
308
|
+
```bash
|
|
309
|
+
mdbkit export-script serverstatus > export_serverstatus.js
|
|
310
|
+
mongosh --quiet --host HOST --eval "$(cat export_serverstatus.js)" > status.json
|
|
311
|
+
mdbkit serverstatus status.json
|
|
312
|
+
```
|
|
313
|
+
|
|
314
|
+
```
|
|
315
|
+
[CRIT] Concurrency tickets: Exhausted tickets queue every new operation,
|
|
316
|
+
which looks like slowness with no slow query to blame.
|
|
317
|
+
- read: 1 of 128 free (1%)
|
|
318
|
+
[CRIT] WiredTiger cache: Above 80% WiredTiger evicts in the background;
|
|
319
|
+
above 95% application threads are made to evict.
|
|
320
|
+
- 31.0 GiB of 32.0 GiB used (96.9%)
|
|
321
|
+
```
|
|
322
|
+
|
|
323
|
+
**Bonus — who connected?**
|
|
324
|
+
|
|
325
|
+
```bash
|
|
326
|
+
mdbkit connections mongod.log
|
|
327
|
+
```
|
|
328
|
+
|
|
329
|
+
Per-IP churn with first/last seen, plus an authenticated-users table showing
|
|
330
|
+
successful and failed logins per account and when each last authenticated —
|
|
331
|
+
the question that starts most access incidents.
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## Running it on a schedule
|
|
336
|
+
|
|
337
|
+
mdbkit is deterministic, offline and read-only, which makes it well suited to
|
|
338
|
+
a cron job. It deliberately **cannot send anything anywhere** — there is no
|
|
339
|
+
network code and never will be. So mdbkit produces the verdict and your own
|
|
340
|
+
script does the talking.
|
|
341
|
+
|
|
342
|
+
The primitive that makes this work is `--exit-code`:
|
|
343
|
+
|
|
344
|
+
| Exit code | Meaning |
|
|
345
|
+
|---|---|
|
|
346
|
+
| `0` | Nothing above INFO |
|
|
347
|
+
| `1` | At least one WARN |
|
|
348
|
+
| `2` | At least one CRIT |
|
|
349
|
+
|
|
350
|
+
```bash
|
|
351
|
+
#!/usr/bin/env bash
|
|
352
|
+
# /usr/local/bin/mdbkit-watch.sh — hourly health check
|
|
353
|
+
set -uo pipefail
|
|
354
|
+
|
|
355
|
+
LOG=/var/log/mongodb/mongod.log
|
|
356
|
+
OUT=$(mktemp)
|
|
357
|
+
|
|
358
|
+
mdbkit triage "$LOG" --window 60 --oslog /var/log/syslog \
|
|
359
|
+
--only CRIT,WARN --exit-code > "$OUT" 2>&1
|
|
360
|
+
STATUS=$?
|
|
361
|
+
|
|
362
|
+
if [ "$STATUS" -ge 2 ]; then
|
|
363
|
+
# CRIT: wake someone up. Your channel, your call — mdbkit stays offline.
|
|
364
|
+
curl -sf -X POST -H 'Content-type: application/json' \
|
|
365
|
+
--data "{\"text\": \"MongoDB CRIT on $(hostname)\n\`\`\`$(cat "$OUT")\`\`\`\"}" \
|
|
366
|
+
"$SLACK_WEBHOOK_URL"
|
|
367
|
+
elif [ "$STATUS" -eq 1 ]; then
|
|
368
|
+
mail -s "MongoDB warnings on $(hostname)" dba@example.com < "$OUT"
|
|
369
|
+
fi
|
|
370
|
+
|
|
371
|
+
rm -f "$OUT"
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
```cron
|
|
375
|
+
# hourly triage; only speaks up when something is wrong
|
|
376
|
+
0 * * * * /usr/local/bin/mdbkit-watch.sh
|
|
377
|
+
|
|
378
|
+
# daily slow-query digest, kept for trend comparison
|
|
379
|
+
30 6 * * * mdbkit queries /var/log/mongodb/mongod.log \
|
|
380
|
+
--report /var/log/mdbkit/$(date +\%F).html
|
|
381
|
+
```
|
|
382
|
+
|
|
383
|
+
Because the reports are dated, `compare` turns them into a trend:
|
|
384
|
+
|
|
385
|
+
```bash
|
|
386
|
+
mdbkit compare /var/log/mongodb/mongod.log.1 --after /var/log/mongodb/mongod.log
|
|
387
|
+
```
|
|
388
|
+
|
|
389
|
+
Two things worth knowing. `--only CRIT,WARN` keeps the mail short, and a run
|
|
390
|
+
that finds nothing prints almost nothing — so a silent cron job means a
|
|
391
|
+
healthy database rather than a broken script. And because every command is
|
|
392
|
+
read-only and never connects to the database, running this hourly on a
|
|
393
|
+
production host costs one log read and no risk.
|
|
394
|
+
|
|
395
|
+
---
|
|
396
|
+
|
|
397
|
+
## Trying it against a real cluster
|
|
398
|
+
|
|
399
|
+
`mdbkit lab` starts a **disposable local MongoDB** so you can test against a
|
|
400
|
+
real server without touching anything that matters.
|
|
401
|
+
|
|
402
|
+
```bash
|
|
403
|
+
mdbkit lab start # 3-node replica set on 127.0.0.1:28110-28112
|
|
404
|
+
mdbkit lab seed # 50k documents + a deliberately mixed workload
|
|
405
|
+
mdbkit queries $(mdbkit lab logs | head -1)
|
|
406
|
+
mdbkit lab destroy --yes # remove it entirely
|
|
407
|
+
```
|
|
408
|
+
|
|
409
|
+
It binds to localhost only, uses ports far from 27017 so it can never be
|
|
410
|
+
confused with a real deployment, and refuses to touch any directory it did
|
|
411
|
+
not create. It is the one command that starts external processes — see
|
|
412
|
+
[SECURITY.md](SECURITY.md).
|
|
413
|
+
|
|
414
|
+
Full options and more examples: [`mdbkit lab`](#mdbkit-lab) in the reference.
|
|
415
|
+
|
|
416
|
+
---
|
|
417
|
+
|
|
418
|
+
## Command reference
|
|
419
|
+
|
|
420
|
+
Every command reads files or stdin and writes to stdout. `--help` works on any
|
|
421
|
+
command (`mdbkit queries --help`). Global: `mdbkit --version`.
|
|
422
|
+
|
|
423
|
+
All commands that read a log accept one or more paths, a shell glob, a
|
|
424
|
+
rotated `.gz` file, or `-` for stdin. Several files are read as a single
|
|
425
|
+
stream in filename order, which matches MongoDB's rotation naming:
|
|
426
|
+
|
|
427
|
+
```bash
|
|
428
|
+
mdbkit queries mongod.log # one file
|
|
429
|
+
mdbkit queries mongod.log.1 mongod.log # explicit list
|
|
430
|
+
mdbkit queries "mongod.log*" # glob (quote it)
|
|
431
|
+
mdbkit queries /var/log/mongodb/mongod.log.*.gz # compressed archives
|
|
432
|
+
cat mongod.log | mdbkit queries - # stdin
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
---
|
|
436
|
+
|
|
437
|
+
### `mdbkit loginfo <log>`
|
|
438
|
+
|
|
439
|
+
Overall log summary: server version, host, restarts, connections accepted,
|
|
440
|
+
slow-query count, warning/error counts, and a per-component line breakdown.
|
|
441
|
+
|
|
442
|
+
| Option | Description |
|
|
443
|
+
|---|---|
|
|
444
|
+
| `--json` | Machine-readable output |
|
|
445
|
+
|
|
446
|
+
```bash
|
|
447
|
+
mdbkit loginfo /var/log/mongodb/mongod.log
|
|
448
|
+
mdbkit loginfo mongod.log.2.gz --json
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
---
|
|
452
|
+
|
|
453
|
+
### `mdbkit queries <log>`
|
|
454
|
+
|
|
455
|
+
Slow queries grouped by **query shape** — literal values stripped, so the same
|
|
456
|
+
query with different parameters is counted once.
|
|
457
|
+
|
|
458
|
+
| Option | Default | Description |
|
|
459
|
+
|---|---|---|
|
|
460
|
+
| `--sort FIELD` | `totalMs` | Order by `totalMs`, `count`, `mean`, `max`, `docsExamined`, or `scanRatio` |
|
|
461
|
+
| `--limit N` | all | Show only the top N shapes |
|
|
462
|
+
| `--min-ms N` | 0 | Ignore operations faster than N milliseconds |
|
|
463
|
+
| `--include-system` | off | Include internal `admin`/`config`/`local` namespaces (hidden by default — they are server housekeeping, not your workload) |
|
|
464
|
+
| `--report FILE` | | Write a shareable `.md` or `.html` report instead (see [Shareable reports](#shareable-reports----report-file)) |
|
|
465
|
+
| `--json` | | Machine-readable output |
|
|
466
|
+
|
|
467
|
+
**Reading the columns:**
|
|
468
|
+
|
|
469
|
+
| Column | Meaning |
|
|
470
|
+
|---|---|
|
|
471
|
+
| `cumMs` | Time summed across **all** occurrences of that shape — not one query |
|
|
472
|
+
| `mean` / `max` | Per-occurrence average and worst case |
|
|
473
|
+
| `docsEx` | Documents examined, summed across all occurrences |
|
|
474
|
+
| `scan` | Documents examined per document returned. `1:1` is ideal; `3444:1` means a missing or weak index |
|
|
475
|
+
| `plan` | The plan MongoDB chose: `COLLSCAN` (no index), `IXSCAN{fields}` (index used), `IDHACK` (`_id` lookup), `+SORT` (in-memory sort). `?` = the plan was not recorded on that line |
|
|
476
|
+
| `shape` | Fields and operators queried, with the sort |
|
|
477
|
+
|
|
478
|
+
```bash
|
|
479
|
+
mdbkit queries mongod.log
|
|
480
|
+
mdbkit queries mongod.log --sort scanRatio --limit 10
|
|
481
|
+
mdbkit queries mongod.log --min-ms 500 --json
|
|
482
|
+
mdbkit queries "mongod.log*" # rotated logs as one stream
|
|
483
|
+
mdbkit queries mongod.log --shape 1 # drill into the worst offender
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
`--shape N` expands one row of the table:
|
|
487
|
+
|
|
488
|
+
```
|
|
489
|
+
namespace : shop.events
|
|
490
|
+
shape : {tenantId:eq, ts:gte} sort:{ts:-1}
|
|
491
|
+
|
|
492
|
+
occurrences : 29
|
|
493
|
+
total time : 3.2m
|
|
494
|
+
mean / max : 6.6s / 8.9s
|
|
495
|
+
docs examined : 25,810,000
|
|
496
|
+
docs returned : 261
|
|
497
|
+
scan ratio : 98889 examined per document returned
|
|
498
|
+
|
|
499
|
+
plans observed
|
|
500
|
+
COLLSCAN 29x
|
|
501
|
+
|
|
502
|
+
flags
|
|
503
|
+
COLLSCAN — no index used for at least one execution
|
|
504
|
+
in-memory SORT — results sorted after retrieval
|
|
505
|
+
|
|
506
|
+
client applications
|
|
507
|
+
ReportWorker 15x
|
|
508
|
+
OrderService 7x
|
|
509
|
+
```
|
|
510
|
+
|
|
511
|
+
---
|
|
512
|
+
|
|
513
|
+
### `mdbkit connections <log>`
|
|
514
|
+
|
|
515
|
+
Connection churn and **who authenticated**: totals, peak concurrent count,
|
|
516
|
+
per-source-IP breakdown with first/last seen, the client applications and
|
|
517
|
+
drivers, and a per-user table.
|
|
518
|
+
|
|
519
|
+
| Option | Description |
|
|
520
|
+
|---|---|
|
|
521
|
+
| `--json` | Machine-readable output |
|
|
522
|
+
|
|
523
|
+
```bash
|
|
524
|
+
mdbkit connections mongod.log
|
|
525
|
+
```
|
|
526
|
+
|
|
527
|
+
```
|
|
528
|
+
source ip accepted ended first seen last seen appName
|
|
529
|
+
---------- -------- ----- ------------------- ------------------- ------------
|
|
530
|
+
10.20.9.77 220 0 2026-07-01 08:49:30 2026-07-01 08:49:30 checkout-api
|
|
531
|
+
10.20.4.11 4 1 2026-07-01 08:00:15 2026-07-01 09:29:30 OrderService
|
|
532
|
+
|
|
533
|
+
authenticated users
|
|
534
|
+
user auth db ok failed last authenticated from
|
|
535
|
+
------------ ------- --- ------ ------------------- -----------
|
|
536
|
+
svc_checkout admin 221 0 2026-07-01 08:49:30 10.20.9.77
|
|
537
|
+
etl_batch admin 0 5 2026-07-01 08:51:54 10.20.11.40
|
|
538
|
+
|
|
539
|
+
etl_batch: 5 failed authentication(s) — last error: AuthenticationFailed
|
|
540
|
+
```
|
|
541
|
+
|
|
542
|
+
This answers the question that starts most access incidents: *did that
|
|
543
|
+
account connect, from where, and when last?* If the log shows no
|
|
544
|
+
authentication events at all, mdbkit says so — either auth is disabled, or
|
|
545
|
+
the window contains no new logins because clients are reusing connections.
|
|
546
|
+
|
|
547
|
+
---
|
|
548
|
+
|
|
549
|
+
### `mdbkit filter <log>`
|
|
550
|
+
|
|
551
|
+
Streams **matching raw log lines** to stdout. Output stays valid logv2 JSON, so
|
|
552
|
+
it chains with other tools (including mdbkit itself).
|
|
553
|
+
|
|
554
|
+
| Option | Description |
|
|
555
|
+
|---|---|
|
|
556
|
+
| `--component NAME` | `COMMAND`, `NETWORK`, `REPL`, `STORAGE`, `INDEX`, `WRITE`, `QUERY`, `CONTROL`, … |
|
|
557
|
+
| `--severity S` | `I` info, `W` warning, `E` error, `F` fatal |
|
|
558
|
+
| `--ns NAMESPACE` | Exact namespace, e.g. `shop.orders` |
|
|
559
|
+
| `--slow N` | Only operations with `durationMillis` >= N |
|
|
560
|
+
| `--from TIMESTAMP` | Lower time bound (inclusive) |
|
|
561
|
+
| `--to TIMESTAMP` | Upper time bound (inclusive) |
|
|
562
|
+
| `--msg TEXT` | Substring match on the message field |
|
|
563
|
+
| `--limit N` | Print only the **first** N matches |
|
|
564
|
+
| `--last N` | Print only the **last** N matches — usually what you want during an incident |
|
|
565
|
+
| `--as-explain` | Rebuild each matching slow query as a runnable `mongosh` `.explain()` command instead of printing the raw log line |
|
|
566
|
+
| `--explain-script` | With `--as-explain`, wrap in `EJSON.stringify()` plus usage comments so it can be saved as a `.js` file |
|
|
567
|
+
|
|
568
|
+
**Timestamp formats accepted** by `--from` / `--to`:
|
|
569
|
+
|
|
570
|
+
```
|
|
571
|
+
2026-07-01T08:00:00+04:00 with an explicit offset (production logs)
|
|
572
|
+
2026-07-01T08:00:00Z UTC
|
|
573
|
+
2026-07-01T08:00:00 no offset — read as the log's own timezone
|
|
574
|
+
2026-07-01 08:00:00 space instead of T
|
|
575
|
+
2026-07-01T08:00 minute precision
|
|
576
|
+
2026-07-01 whole day
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
```bash
|
|
580
|
+
mdbkit filter mongod.log --severity E --last 20 # errors (most recent 20)
|
|
581
|
+
mdbkit filter mongod.log --severity F # fatal — always investigate
|
|
582
|
+
mdbkit filter mongod.log --severity W --last 50 # warnings
|
|
583
|
+
mdbkit filter mongod.log --component REPL --msg election
|
|
584
|
+
mdbkit filter mongod.log --slow 500 --ns shop.orders --limit 50
|
|
585
|
+
mdbkit filter mongod.log --from 2026-07-01T14:30:00+04:00 --to 2026-07-01T15:00:00+04:00
|
|
586
|
+
mdbkit filter mongod.log --slow 100 | mdbkit queries -
|
|
587
|
+
```
|
|
588
|
+
|
|
589
|
+
**From a slow query in the log to an explain plan**, without hand-writing the
|
|
590
|
+
query — `--as-explain` rebuilds the command that ran:
|
|
591
|
+
|
|
592
|
+
```bash
|
|
593
|
+
# See the actual commands behind your slowest operations
|
|
594
|
+
mdbkit filter mongod.log --ns shop.orders --slow 500 --last 3 --as-explain
|
|
595
|
+
|
|
596
|
+
# Or produce a runnable script, get the plan, and analyze it
|
|
597
|
+
mdbkit filter mongod.log --slow 500 --last 1 --as-explain --explain-script > q.js
|
|
598
|
+
mongosh --quiet --host your_db_host --username your_username \\
|
|
599
|
+
--password your_password --authenticationDatabase admin \\
|
|
600
|
+
--eval "$(cat q.js)" > explain.json
|
|
601
|
+
mdbkit explain explain.json
|
|
602
|
+
```
|
|
603
|
+
|
|
604
|
+
> Rebuilt commands contain the **real values** from your log (not redacted
|
|
605
|
+
> shapes) — treat them as sensitive.
|
|
606
|
+
|
|
607
|
+
---
|
|
608
|
+
|
|
609
|
+
### `mdbkit advise <log>`
|
|
610
|
+
|
|
611
|
+
Deterministic **candidate** index recommendations from observed slow-query
|
|
612
|
+
shapes, using the ESR guideline (Equality → Sort → Range). Rules, not AI: the
|
|
613
|
+
same log always produces the same advice.
|
|
614
|
+
|
|
615
|
+
| Option | Default | Description |
|
|
616
|
+
|---|---|---|
|
|
617
|
+
| `--indexes FILE` | | `indexes.json` from `mdbkit export-script indexes` — enables overlap checks against existing indexes |
|
|
618
|
+
| `--schema FILE` | | `schema.json` from `mdbkit export-script schema` — enables field-type caveats and confidence adjustment |
|
|
619
|
+
| `--ns NAMESPACE` | all | Only advise on one namespace (recommended on large logs) |
|
|
620
|
+
| `--limit N` | 10 | Show only the top N recommendations (`0` = all) |
|
|
621
|
+
| `--min-ms N` | 0 | Ignore operations faster than N milliseconds |
|
|
622
|
+
| `--min-count N` | 1 | Only advise on shapes seen at least N times |
|
|
623
|
+
| `--include-system` | off | Include internal `admin`/`config`/`local` namespaces |
|
|
624
|
+
| `--json` | | Machine-readable output |
|
|
625
|
+
|
|
626
|
+
Each recommendation carries a candidate key pattern, the evidence behind it, a
|
|
627
|
+
confidence level, caveats, and a validation step. mdbkit never advises dropping
|
|
628
|
+
an index — at most it flags an overlap to investigate.
|
|
629
|
+
|
|
630
|
+
```bash
|
|
631
|
+
mdbkit advise mongod.log
|
|
632
|
+
mdbkit advise mongod.log --indexes indexes.json --schema schema.json
|
|
633
|
+
mdbkit advise mongod.log --ns shop.orders --limit 3
|
|
634
|
+
```
|
|
635
|
+
|
|
636
|
+
---
|
|
637
|
+
|
|
638
|
+
### `mdbkit explain <file>`
|
|
639
|
+
|
|
640
|
+
Analyzes a saved `explain("executionStats")` document: the plan chain, the
|
|
641
|
+
examined-vs-returned math, plain-English verdicts, and — when the plan needs
|
|
642
|
+
help — a candidate index from the same advisor engine.
|
|
643
|
+
|
|
644
|
+
| Option | Description |
|
|
645
|
+
|---|---|
|
|
646
|
+
| `--indexes FILE` | Overlap check against existing indexes |
|
|
647
|
+
| `--schema FILE` | Field-type caveats |
|
|
648
|
+
| `--json` | Machine-readable output |
|
|
649
|
+
|
|
650
|
+
**Full example.** Get a plan for a query and analyze it:
|
|
651
|
+
|
|
652
|
+
```bash
|
|
653
|
+
# 1. Capture the plan (adjust host/credentials for your deployment)
|
|
654
|
+
mongosh --quiet \\
|
|
655
|
+
--host your_db_host \\
|
|
656
|
+
--port 27017 \\
|
|
657
|
+
--username your_username \\
|
|
658
|
+
--password your_password \\
|
|
659
|
+
--authenticationDatabase admin \\
|
|
660
|
+
--eval 'EJSON.stringify(db.getSiblingDB("shop").orders.find({status:"open"}).sort({ts:-1}).explain("executionStats"))' \\
|
|
661
|
+
> explain.json
|
|
662
|
+
|
|
663
|
+
# 2. Analyze it
|
|
664
|
+
mdbkit explain explain.json
|
|
665
|
+
|
|
666
|
+
# 3. Sharper, with your existing indexes and sampled schema
|
|
667
|
+
mdbkit explain explain.json --indexes indexes.json --schema schema.json
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
Don't want to write the query by hand? `mdbkit filter ... --as-explain`
|
|
671
|
+
rebuilds it from the log for you (see the `filter` section above).
|
|
672
|
+
|
|
673
|
+
Legacy `mongo` shell and Compass output containing `NumberLong(...)`,
|
|
674
|
+
`ISODate(...)` or `ObjectId(...)` is accepted — mdbkit unwraps those
|
|
675
|
+
automatically, so you do not have to re-export.
|
|
676
|
+
|
|
677
|
+
---
|
|
678
|
+
|
|
679
|
+
### `mdbkit triage <log>`
|
|
680
|
+
|
|
681
|
+
**"Triage" means: quickly work out what is wrong and what to look at first.**
|
|
682
|
+
Run this when something has gone wrong — or has just gone wrong — and you need
|
|
683
|
+
one screen that says what happened, how bad it is, and where to look next.
|
|
684
|
+
**Defaults to the last 60 minutes of log time.**
|
|
685
|
+
|
|
686
|
+
| Option | Default | Description |
|
|
687
|
+
|---|---|---|
|
|
688
|
+
| `--window N` | 60 | Analyze the last N minutes of log time; `0` = the whole file |
|
|
689
|
+
| `--dbpath PATH` | auto | Override the data directory used for the disk check |
|
|
690
|
+
| `--no-sysprobe` | off | Skip local disk/memory/CPU probes — use when analyzing a log copied off the host |
|
|
691
|
+
| `--ftdc PATH` | | `diagnostic.data` directory — adds CPU, memory, cache, queue and connection history from MongoDB's own recorder |
|
|
692
|
+
| `--report FILE` | | Write a shareable `.md` or `.html` report instead of terminal output |
|
|
693
|
+
| `--json` | | Machine-readable output |
|
|
694
|
+
|
|
695
|
+
```bash
|
|
696
|
+
mdbkit triage /var/log/mongodb/mongod.log
|
|
697
|
+
mdbkit triage mongod.log --window 30
|
|
698
|
+
mdbkit triage mongod.log --ftdc /var/lib/mongodb/diagnostic.data
|
|
699
|
+
mdbkit triage mongod.log --report incident.html
|
|
700
|
+
mdbkit triage mongod.log --window 0 --no-sysprobe
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
---
|
|
704
|
+
|
|
705
|
+
### `mdbkit ftdc {summary|timeline|export} <path>`
|
|
706
|
+
|
|
707
|
+
Decodes `diagnostic.data` — **FTDC (Full-Time Diagnostic Data Capture)**, the
|
|
708
|
+
metrics recorder every mongod already runs. It holds CPU, memory, WiredTiger
|
|
709
|
+
cache, connection, queue and operation history for every node, with no
|
|
710
|
+
monitoring agent installed and no database connection. It is compressed BSON,
|
|
711
|
+
not encrypted; mdbkit decodes it offline.
|
|
712
|
+
|
|
713
|
+
| Action | Description |
|
|
714
|
+
|---|---|
|
|
715
|
+
| `summary` | min / avg / max / last per metric, plus per-second rates for counters |
|
|
716
|
+
| `timeline` | Values bucketed over time — shows *when* something spiked |
|
|
717
|
+
| `export` | CSV to stdout, for a spreadsheet or your own tooling |
|
|
718
|
+
|
|
719
|
+
| Option | Default | Description |
|
|
720
|
+
|---|---|---|
|
|
721
|
+
| `--last DURATION` | `4h` | Analyze only the most recent window — `90m`, `4h`, `2d` |
|
|
722
|
+
| `--all` | off | Analyze the entire history (see the performance note below) |
|
|
723
|
+
| `--metric LABEL` | all | Restrict to one metric (repeatable), e.g. `--metric conns.current` |
|
|
724
|
+
| `--step SECONDS` | 60 | Timeline bucket size |
|
|
725
|
+
| `--from` / `--to` | | Explicit time bounds (same formats as `filter`) |
|
|
726
|
+
| `--json` | | Machine-readable output |
|
|
727
|
+
|
|
728
|
+
**Performance note.** `diagnostic.data` can hold weeks of per-second samples —
|
|
729
|
+
a few hundred megabytes covering thousands of chunks and several thousand
|
|
730
|
+
metrics each. Decoding all of it is CPU-bound and takes minutes, so these
|
|
731
|
+
commands **default to the last 4 hours** and skip older chunks before
|
|
732
|
+
decompressing them. On a 250 MB directory that is the difference between about
|
|
733
|
+
a second and about a minute. Use `--last`/`--from`/`--to` to move the window,
|
|
734
|
+
and `--all` when you really do want the whole history.
|
|
735
|
+
|
|
736
|
+
```bash
|
|
737
|
+
mdbkit ftdc summary /var/lib/mongodb/diagnostic.data
|
|
738
|
+
mdbkit ftdc timeline diagnostic.data --metric conns.current --step 300
|
|
739
|
+
mdbkit ftdc export diagnostic.data > metrics.csv
|
|
740
|
+
```
|
|
741
|
+
|
|
742
|
+
Metric labels include `ops.*` (insert/query/update/delete/getmore/command),
|
|
743
|
+
`conns.current`, `conns.available`, `queue.readers`, `queue.writers`,
|
|
744
|
+
`cache.usedBytes`, `cache.maxBytes`, `cache.dirtyBytes`, `tickets.*`,
|
|
745
|
+
`mem.residentMB`, and on Linux `sys.cpu.*` and `sys.mem.availableKB`.
|
|
746
|
+
|
|
747
|
+
The data directory can be copied off the host and analyzed elsewhere — it
|
|
748
|
+
contains metrics only, never document contents.
|
|
749
|
+
|
|
750
|
+
---
|
|
751
|
+
|
|
752
|
+
### Shareable reports — `--report FILE`
|
|
753
|
+
|
|
754
|
+
`triage` and `queries` can write a self-contained report instead of printing to
|
|
755
|
+
the terminal — for a ticket, a handover, or a post-incident review.
|
|
756
|
+
|
|
757
|
+
```bash
|
|
758
|
+
mdbkit triage mongod.log --report incident.html # styled, self-contained
|
|
759
|
+
mdbkit triage mongod.log --report incident.md # for tickets and PRs
|
|
760
|
+
mdbkit queries mongod.log --limit 20 --report slow-queries.md
|
|
761
|
+
```
|
|
762
|
+
|
|
763
|
+
The format follows the file extension: `.html` or `.md`.
|
|
764
|
+
|
|
765
|
+
Markdown output looks like this:
|
|
766
|
+
|
|
767
|
+
```markdown
|
|
768
|
+
# MongoDB incident triage
|
|
769
|
+
|
|
770
|
+
*window 2026-07-01 08:10 -> 09:10 · generated 2026-07-01 09:12*
|
|
771
|
+
|
|
772
|
+
## Findings
|
|
773
|
+
|
|
774
|
+
- **[CRIT] Replica set instability** — 3 election/stepdown event(s) at 08:41:02, 08:58:14
|
|
775
|
+
- Starting an election, since we've seen no PRIMARY in election timeout period
|
|
776
|
+
- *next:* `Correlate with connection storms and slow checkpoints below`
|
|
777
|
+
- **[WARN] Connection storm** — 2 minute(s) at >= 60 new connections/min; peak 480 at 08:41
|
|
778
|
+
- 10.2.1.7: 312 in the peak minute
|
|
779
|
+
- *next:* `mdbkit connections <log>`
|
|
780
|
+
- **[OK] Errors** — No error/fatal severity lines in window.
|
|
781
|
+
```
|
|
782
|
+
|
|
783
|
+
The HTML version carries the same content with a dark, print-friendly
|
|
784
|
+
stylesheet. It is **fully self-contained**: inline CSS, no JavaScript, no
|
|
785
|
+
external assets or CDN references, so it opens on an air-gapped machine and
|
|
786
|
+
sends nothing anywhere.
|
|
787
|
+
|
|
788
|
+
Reports contain the same information as the terminal output — query **shapes**
|
|
789
|
+
and metrics, never literal values from your documents.
|
|
790
|
+
|
|
791
|
+
---
|
|
792
|
+
|
|
793
|
+
### `mdbkit demo`
|
|
794
|
+
|
|
795
|
+
Generates a realistic MongoDB structured log so you can evaluate mdbkit — or
|
|
796
|
+
run a live demo — without a cluster. Output is deterministic for a given
|
|
797
|
+
seed, so a demo behaves identically every time, including on a projector.
|
|
798
|
+
|
|
799
|
+
| Option | Default | Description |
|
|
800
|
+
|---|---|---|
|
|
801
|
+
| `--scenario` | `mixed` | `healthy`, `incident`, or `mixed` |
|
|
802
|
+
| `--minutes N` | 90 | How much log time to generate |
|
|
803
|
+
| `--seed N` | 7 | Same seed produces byte-identical output |
|
|
804
|
+
| `-o, --out FILE` | stdout | Write to a file |
|
|
805
|
+
| `--with-extras` | off | Also write `indexes.json`, `schema.json` and `explain.json` beside the log |
|
|
806
|
+
|
|
807
|
+
```bash
|
|
808
|
+
mdbkit demo -o demo.log # 90 minutes, mixed
|
|
809
|
+
mdbkit demo --scenario incident --minutes 30 -o incident.log
|
|
810
|
+
mdbkit demo --scenario healthy -o quiet.log # nothing wrong: the control case
|
|
811
|
+
mdbkit demo | mdbkit queries - # straight down a pipe
|
|
812
|
+
```
|
|
813
|
+
|
|
814
|
+
The `incident` scenario contains an index build, a connection storm from a
|
|
815
|
+
single client, a replica set election, plan-executor errors, a slow
|
|
816
|
+
WiredTiger checkpoint, and a burst of unindexed queries afterwards — the
|
|
817
|
+
shape of a real bad afternoon.
|
|
818
|
+
|
|
819
|
+
---
|
|
820
|
+
|
|
821
|
+
### `mdbkit lab`
|
|
822
|
+
|
|
823
|
+
Starts a **disposable local MongoDB** for testing, reproducing a slow query,
|
|
824
|
+
or rehearsing a demo. This is the only command that starts external
|
|
825
|
+
processes; see [SECURITY.md](SECURITY.md) for exactly how it is bounded.
|
|
826
|
+
|
|
827
|
+
Requires `mongod` on your `PATH` (and `mongosh` to initiate the replica set
|
|
828
|
+
and seed data). Linux and macOS.
|
|
829
|
+
|
|
830
|
+
| Action | What it does |
|
|
831
|
+
|---|---|
|
|
832
|
+
| `start` | Create and start a replica set, print the connection string and log paths |
|
|
833
|
+
| `seed` | Insert sample data and run a workload with deliberately interesting queries |
|
|
834
|
+
| `status` | Show ports, pids and whether each node is running |
|
|
835
|
+
| `logs` | Print the log file paths, ready to pipe into other commands |
|
|
836
|
+
| `stop` | Stop the nodes, keep the data |
|
|
837
|
+
| `destroy` | Stop and delete the lab (requires `--yes`) |
|
|
838
|
+
|
|
839
|
+
| Option | Default | Description |
|
|
840
|
+
|---|---|---|
|
|
841
|
+
| `--dir PATH` | `~/.mdbkit-lab` | Where the lab lives |
|
|
842
|
+
| `--nodes N` | 3 | Replica set size |
|
|
843
|
+
| `--port N` | 28110 | Base port — deliberately far from 27017 |
|
|
844
|
+
| `--slowms N` | 0 | Log every operation, which is what makes the log worth reading |
|
|
845
|
+
| `--standalone` | off | Single node, no replica set |
|
|
846
|
+
| `--docs N` | 50000 | Documents inserted by `seed` |
|
|
847
|
+
| `--yes` | | Confirm `destroy` |
|
|
848
|
+
|
|
849
|
+
**The full loop:**
|
|
850
|
+
|
|
851
|
+
```bash
|
|
852
|
+
mdbkit lab start # 3-node replica set on 28110-28112
|
|
853
|
+
mdbkit lab seed # sample data + a mixed workload
|
|
854
|
+
|
|
855
|
+
mdbkit queries $(mdbkit lab logs | head -1)
|
|
856
|
+
mdbkit advise $(mdbkit lab logs | head -1)
|
|
857
|
+
|
|
858
|
+
mdbkit lab destroy --yes # remove everything
|
|
859
|
+
```
|
|
860
|
+
|
|
861
|
+
**`mdbkit lab logs`** prints the log file path of every node, one per line,
|
|
862
|
+
so it composes with the other commands instead of you hunting for paths:
|
|
863
|
+
|
|
864
|
+
```bash
|
|
865
|
+
mdbkit lab logs
|
|
866
|
+
# /home/you/.mdbkit-lab/node0/mongod.log
|
|
867
|
+
# /home/you/.mdbkit-lab/node1/mongod.log
|
|
868
|
+
# /home/you/.mdbkit-lab/node2/mongod.log
|
|
869
|
+
|
|
870
|
+
mdbkit queries $(mdbkit lab logs | head -1) # just the primary
|
|
871
|
+
mdbkit triage $(mdbkit lab logs) # all three as one stream
|
|
872
|
+
mdbkit loginfo $(mdbkit lab logs | sed -n 2p) # a specific secondary
|
|
873
|
+
```
|
|
874
|
+
|
|
875
|
+
**A single node**, when you do not need replication — faster to start and it
|
|
876
|
+
does not require `mongosh`:
|
|
877
|
+
|
|
878
|
+
```bash
|
|
879
|
+
mdbkit lab start --standalone
|
|
880
|
+
mdbkit lab seed --docs 5000
|
|
881
|
+
mdbkit queries $(mdbkit lab logs)
|
|
882
|
+
mdbkit lab destroy --yes
|
|
883
|
+
```
|
|
884
|
+
|
|
885
|
+
**Several labs side by side**, for example to compare two MongoDB versions or
|
|
886
|
+
keep one running while you break another:
|
|
887
|
+
|
|
888
|
+
```bash
|
|
889
|
+
mdbkit lab start --dir ~/lab-a --port 28110
|
|
890
|
+
mdbkit lab start --dir ~/lab-b --port 28210 --standalone
|
|
891
|
+
|
|
892
|
+
mdbkit lab status --dir ~/lab-a
|
|
893
|
+
mdbkit lab destroy --dir ~/lab-b --yes
|
|
894
|
+
```
|
|
895
|
+
|
|
896
|
+
**Pause without losing data** — `stop` leaves the data directory intact so
|
|
897
|
+
you can start again later; only `destroy` deletes anything:
|
|
898
|
+
|
|
899
|
+
```bash
|
|
900
|
+
mdbkit lab stop # nodes down, data kept
|
|
901
|
+
mdbkit lab start # back up with the same data
|
|
902
|
+
mdbkit lab status # ports, pids, running or not
|
|
903
|
+
```
|
|
904
|
+
|
|
905
|
+
**A complete before/after experiment**, which is what `lab` is really for:
|
|
906
|
+
|
|
907
|
+
```bash
|
|
908
|
+
mdbkit lab start && mdbkit lab seed
|
|
909
|
+
cp $(mdbkit lab logs | head -1) before.log
|
|
910
|
+
|
|
911
|
+
mongosh --port 28110 --eval \
|
|
912
|
+
'db.getSiblingDB("shop").orders.createIndex({status:1, createdAt:-1})'
|
|
913
|
+
|
|
914
|
+
mdbkit lab seed # run the workload again with the index
|
|
915
|
+
cp $(mdbkit lab logs | head -1) after.log
|
|
916
|
+
|
|
917
|
+
mdbkit compare before.log --after after.log
|
|
918
|
+
mdbkit lab destroy --yes
|
|
919
|
+
```
|
|
920
|
+
|
|
921
|
+
`seed` runs indexed point lookups alongside deliberately unindexed queries —
|
|
922
|
+
an equality-plus-range-plus-sort with no supporting index, an aggregation
|
|
923
|
+
that scans the collection, and updates whose predicate has no index — so the
|
|
924
|
+
log immediately contains something worth analysing.
|
|
925
|
+
|
|
926
|
+
**Safety.** The lab binds to `127.0.0.1` only, refuses to use or delete any
|
|
927
|
+
directory it did not create, and never touches a MongoDB it did not start.
|
|
928
|
+
It is a laptop and scratch-VM tool, not a deployment tool.
|
|
929
|
+
|
|
930
|
+
---
|
|
931
|
+
|
|
932
|
+
### `mdbkit oslog [FILE...]`
|
|
933
|
+
|
|
934
|
+
Scans a system log for the things that affect a database process: OOM kills,
|
|
935
|
+
file-descriptor limits, segmentation faults, filesystem and I/O errors,
|
|
936
|
+
read-only remounts, conntrack exhaustion, and systemd service exits.
|
|
937
|
+
|
|
938
|
+
With no argument it reads `/var/log/syslog` or `/var/log/messages` if they are
|
|
939
|
+
readable.
|
|
940
|
+
|
|
941
|
+
| Option | Description |
|
|
942
|
+
|---|---|
|
|
943
|
+
| `--exit-code` | Exit 2 on CRIT, 1 on WARN, else 0 |
|
|
944
|
+
| `--json` | Machine-readable output |
|
|
945
|
+
|
|
946
|
+
```bash
|
|
947
|
+
mdbkit oslog # whichever system log exists
|
|
948
|
+
mdbkit oslog /var/log/messages
|
|
949
|
+
mdbkit oslog /var/log/syslog.1 /var/log/syslog
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
**On journald systems** there is no text log to read, and mdbkit does not run
|
|
953
|
+
commands on your behalf. It tells you what to capture instead:
|
|
954
|
+
|
|
955
|
+
```bash
|
|
956
|
+
journalctl -k --since '4 hours ago' > kern.log
|
|
957
|
+
journalctl -u mongod --since '4 hours ago' >> kern.log
|
|
958
|
+
mdbkit oslog kern.log
|
|
959
|
+
```
|
|
960
|
+
|
|
961
|
+
The same file can be handed to `triage --oslog`, which correlates it with the
|
|
962
|
+
mongod log — so an unexplained restart at 09:14 lines up with the OOM kill
|
|
963
|
+
that caused it.
|
|
964
|
+
|
|
965
|
+
---
|
|
966
|
+
|
|
967
|
+
### `mdbkit serverstatus FILE [--after FILE]`
|
|
968
|
+
|
|
969
|
+
Digests a saved `db.adminCommand({serverStatus: 1})` dump. That command
|
|
970
|
+
returns several hundred fields; this reports the handful that explain a
|
|
971
|
+
struggling server.
|
|
972
|
+
|
|
973
|
+
| Option | Description |
|
|
974
|
+
|---|---|
|
|
975
|
+
| `--after FILE` | A second dump taken later — turns cumulative counters into true rates |
|
|
976
|
+
| `--report FILE` | Write a shareable `.md` or `.html` report |
|
|
977
|
+
| `--exit-code` | Exit 2 on CRIT, 1 on WARN, else 0 |
|
|
978
|
+
| `--json` | Machine-readable output |
|
|
979
|
+
|
|
980
|
+
```bash
|
|
981
|
+
mdbkit export-script serverstatus > export_serverstatus.js
|
|
982
|
+
|
|
983
|
+
mongosh --quiet --host HOST --port PORT \
|
|
984
|
+
--username USER --password PASS --authenticationDatabase admin \
|
|
985
|
+
--eval "$(cat export_serverstatus.js)" > status.json
|
|
986
|
+
|
|
987
|
+
mdbkit serverstatus status.json
|
|
988
|
+
```
|
|
989
|
+
|
|
990
|
+
What it checks: **concurrency tickets** (exhaustion queues every operation and
|
|
991
|
+
looks like slowness with no slow query to blame), **WiredTiger cache** against
|
|
992
|
+
the 80% background-eviction and 95% application-thread-eviction thresholds,
|
|
993
|
+
**dirty cache**, **application-thread eviction**, **connection headroom**,
|
|
994
|
+
**queued readers and writers**, **assertions**, **flow control**, replication
|
|
995
|
+
role and process memory. Ticket layout is read from either
|
|
996
|
+
`wiredTiger.concurrentTransactions` (pre-7.0) or `queues.execution` (7.0+).
|
|
997
|
+
|
|
998
|
+
**Two dumps give true rates.** Almost everything in serverStatus is cumulative
|
|
999
|
+
since process start, so a single dump only yields lifetime averages:
|
|
1000
|
+
|
|
1001
|
+
```bash
|
|
1002
|
+
mongosh ... > before.json ; sleep 60 ; mongosh ... > after.json
|
|
1003
|
+
mdbkit serverstatus before.json --after after.json
|
|
1004
|
+
```
|
|
1005
|
+
|
|
1006
|
+
```
|
|
1007
|
+
[INFO] Operation counters: Measured over 60 seconds between the two dumps.
|
|
1008
|
+
- query 742,000 in 60s (12366.7/sec)
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
That is real current load. The same counter read from one dump would have
|
|
1012
|
+
reported 2,127/sec — the average since the server started ten days ago.
|
|
1013
|
+
|
|
1014
|
+
---
|
|
1015
|
+
|
|
1016
|
+
### `mdbkit compare BEFORE --after AFTER`
|
|
1017
|
+
|
|
1018
|
+
Diffs query shapes between two logs and reports what improved, what
|
|
1019
|
+
regressed, and what is new. The natural follow-up to `advise`: you created an
|
|
1020
|
+
index, a day passed, and this answers whether it worked.
|
|
1021
|
+
|
|
1022
|
+
| Option | Default | Description |
|
|
1023
|
+
|---|---|---|
|
|
1024
|
+
| `--after FILE...` | required | The log(s) from after the change |
|
|
1025
|
+
| `--ns NAMESPACE` | all | Compare only one namespace |
|
|
1026
|
+
| `--min-count N` | 3 | Ignore shapes seen fewer than N times, so noise in a quiet log does not read as a regression |
|
|
1027
|
+
| `--min-ms N` | 0 | Ignore operations faster than this |
|
|
1028
|
+
| `--limit N` | 15 | Shapes to print (`0` = all) |
|
|
1029
|
+
| `--include-system` | off | Include internal `admin`/`config`/`local` namespaces |
|
|
1030
|
+
| `--report FILE` | | Write a shareable `.md` or `.html` report |
|
|
1031
|
+
| `--json` | | Machine-readable output |
|
|
1032
|
+
|
|
1033
|
+
```bash
|
|
1034
|
+
mdbkit compare before.log --after after.log
|
|
1035
|
+
mdbkit compare before.log --after after.log --ns shop.orders
|
|
1036
|
+
mdbkit compare "old/mongod.log*" --after "new/mongod.log*" --report change.html
|
|
1037
|
+
```
|
|
1038
|
+
|
|
1039
|
+
```
|
|
1040
|
+
slow-query time DOWN 32% (6.0m -> 4.1m across compared shapes)
|
|
1041
|
+
shapes: 1 improved, 0 regressed, 0 new, 0 gone, 4 unchanged
|
|
1042
|
+
|
|
1043
|
+
IMPROVED
|
|
1044
|
+
shop.orders {createdAt:gt, status:eq} sort:{createdAt:-1}
|
|
1045
|
+
mean 1.7s -> 33ms (-98%) scan 2976:1 -> 1:1 [COLLSCAN -> index, in-memory sort gone]
|
|
1046
|
+
```
|
|
1047
|
+
|
|
1048
|
+
A shape counts as improved or regressed on a plan change (COLLSCAN becoming
|
|
1049
|
+
an index scan, or the reverse), on an in-memory sort disappearing, or on mean
|
|
1050
|
+
duration moving by more than 20%.
|
|
1051
|
+
|
|
1052
|
+
---
|
|
1053
|
+
|
|
1054
|
+
### `mdbkit export-script {schema|indexes}`
|
|
1055
|
+
|
|
1056
|
+
Prints a small `mongosh` script to stdout. **mdbkit never connects to your
|
|
1057
|
+
database** — you run these yourself, so you can read exactly what they do
|
|
1058
|
+
first. Both are read-only and export **field names and types only, never
|
|
1059
|
+
document values**.
|
|
1060
|
+
|
|
1061
|
+
```bash
|
|
1062
|
+
mdbkit export-script indexes > export_indexes.js
|
|
1063
|
+
mdbkit export-script schema > export_schema.js
|
|
1064
|
+
```
|
|
1065
|
+
|
|
1066
|
+
---
|
|
1067
|
+
|
|
1068
|
+
## Roadmap
|
|
1069
|
+
|
|
1070
|
+
Terminal output is and will remain first-class — this tool is built for the
|
|
1071
|
+
Linux box the database actually runs on.
|
|
1072
|
+
|
|
1073
|
+
**Shipped in v0.4:** `compare`, rotated-log globbing, per-shape drill-down —
|
|
1074
|
+
on top of v0.3's `demo` and `lab`, and v0.2's FTDC decoding, incident triage,
|
|
1075
|
+
query reconstruction and shareable reports.
|
|
1076
|
+
|
|
1077
|
+
Next up, roughly in order:
|
|
1078
|
+
|
|
1079
|
+
* **Sharded clusters.** `mongos` logs are a different shape, and the classic
|
|
1080
|
+
sharded failure — a query with no shard key fanning out to every shard — is
|
|
1081
|
+
visible in the log. Also chunk migrations, balancer windows and jumbo
|
|
1082
|
+
chunks. Would come with `mdbkit lab --sharded` so it can be tested.
|
|
1083
|
+
* **Startup configuration audit.** mongod logs warnings at startup about
|
|
1084
|
+
transparent huge pages, readahead, ulimits, NUMA and filesystem choice.
|
|
1085
|
+
These are classic production misconfigurations and they are already in
|
|
1086
|
+
your log — nothing new needs collecting.
|
|
1087
|
+
* **Index usage candidates.** Prefix-redundant indexes (an index on `{a: 1}`
|
|
1088
|
+
when `{a: 1, b: 1}` exists) are worth *examining*, but static analysis is
|
|
1089
|
+
not sufficient grounds to drop one — the planner may still be choosing it.
|
|
1090
|
+
So mdbkit will flag candidates and print an `$indexStats` script to confirm
|
|
1091
|
+
real usage first, never a drop recommendation.
|
|
1092
|
+
* **Confirming the FTDC-based checkpoint, eviction and flow-control
|
|
1093
|
+
detectors** against real `diagnostic.data` — see
|
|
1094
|
+
`docs/TESTING-PLAYBOOK.md`. Real logs and metrics very welcome.
|
|
1095
|
+
|
|
1096
|
+
mdbkit is validated against real-world structured logs (tens of thousands of
|
|
1097
|
+
lines) in addition to its synthetic test fixtures.
|
|
1098
|
+
|
|
1099
|
+
## Bugs, feature requests, questions
|
|
1100
|
+
|
|
1101
|
+
Please use [GitHub Issues](../../issues) — it keeps problems and fixes public
|
|
1102
|
+
so the next person can find them. Real-world log lines that parse wrongly are
|
|
1103
|
+
the most valuable bug reports of all (redact literals first!).
|
|
1104
|
+
|
|
1105
|
+
## Security
|
|
1106
|
+
|
|
1107
|
+
mdbkit is offline by design: the codebase contains no network code, never
|
|
1108
|
+
executes or evaluates input, and treats every log line as untrusted data
|
|
1109
|
+
(strict JSON parsing only — shell constructors are never evaluated). See
|
|
1110
|
+
[SECURITY.md](SECURITY.md) for the reporting process.
|
|
1111
|
+
|
|
1112
|
+
## Non-affiliation
|
|
1113
|
+
|
|
1114
|
+
mdbkit is an independent community project. It is **not affiliated with,
|
|
1115
|
+
endorsed by, or sponsored by MongoDB, Inc.** "MongoDB" is a registered
|
|
1116
|
+
trademark of MongoDB, Inc., used here only to describe compatibility.
|
|
1117
|
+
|
|
1118
|
+
## License
|
|
1119
|
+
|
|
1120
|
+
MIT — see [LICENSE](LICENSE).
|