mdbkit 0.2.0__tar.gz → 0.3.0__tar.gz

{mdbkit-0.2.0/mdbkit.egg-info → mdbkit-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdbkit
-Version: 0.2.0
+Version: 0.3.0
 Summary: Offline toolkit for MongoDB 4.4+ structured logs: log analysis, slow-query shapes, connection churn, and deterministic index advice. A spiritual successor to mtools' log tools.
 Author: Saqib Ameen Subhan
 License: MIT
@@ -36,6 +36,25 @@ MongoDB 4.4 switched to structured JSON logging, and the beloved mtools log comm
 
 mdbkit fills that gap: a single, dependency-free CLI that turns structured logs into answers.
 
+## Try it in 30 seconds
+
+No MongoDB required — `mdbkit demo` writes a realistic log with a real
+incident in it:
+
+```bash
+pip install mdbkit
+mdbkit demo --with-extras -o demo.log
+
+mdbkit loginfo demo.log
+mdbkit queries demo.log
+mdbkit triage demo.log --window 0 --no-sysprobe
+mdbkit advise demo.log --indexes indexes.json --schema schema.json
+```
+
+You will see a connection storm, a replica set election, an index build, and
+a collection scan burning 47 million document reads — then the candidate
+index that fixes it.
+
 ## Install
 
 **Recommended on any Linux/Mac/Windows:**
@@ -199,6 +218,8 @@ mdbkit triage mongod.log --no-sysprobe        # analyzing a log copied off-host
 **Defaults to the last 60 minutes of log time**, because triage is for
 incidents happening now or just finished. In one command:
 
+- **cluster health** — this node's replica set role, every peer's last known
+  state, heartbeat failures, and whether the node is still serving
 - restarts, error clusters, election/stepdown events
 - connection storms — with the peak minute and the top source IPs
 - slow-query volume and the peak minute, so you know *when* it hurt
@@ -211,8 +232,15 @@ incidents happening now or just finished. In one command:
 When run on the database host, mdbkit finds `dbPath` automatically — from the
 log's startup line, or the running `mongod` process, or `/etc/mongod.conf`,
 or common defaults — so the disk check works even when the current log has no
-startup event. All probing is stdlib-only (`/proc`, `statvfs`); no shell-outs,
-nothing leaves the machine.
+startup event. **`diagnostic.data` lives inside the dbPath, so it is picked up
+automatically too**: you only need `--ftdc` to point somewhere else, such as a
+directory copied off another host. All probing is stdlib-only (`/proc`,
+`statvfs`); no shell-outs, nothing leaves the machine.
+
+Cluster health is derived entirely from the log — no connection to the
+database. It reports what the node last said about itself and its peers, which
+is the honest limit of an offline tool, and enough to answer "is this node
+serving, and what does it think of the others?"
 
 Read-only: it never connects to the database, and every finding ends with a
 next step for a human to review. Detectors marked beta are pattern-matched and
@@ -295,7 +323,7 @@ query with different parameters is counted once.
 | `--limit N` | all | Show only the top N shapes |
 | `--min-ms N` | 0 | Ignore operations faster than N milliseconds |
 | `--include-system` | off | Include internal `admin`/`config`/`local` namespaces (hidden by default — they are server housekeeping, not your workload) |
-| `--report FILE` | | Write a shareable `.md` or `.html` report instead |
+| `--report FILE` | | Write a shareable `.md` or `.html` report instead (see [Shareable reports](#shareable-reports----report-file)) |
 | `--json` | | Machine-readable output |
 
 **Reading the columns:**
@@ -319,13 +347,38 @@ mdbkit queries mongod.log --min-ms 500 --json
 
 ### `mdbkit connections <log>`
 
-Connection churn: totals, peak concurrent count, per-source-IP breakdown, and
-the client applications and drivers seen.
+Connection churn and **who authenticated**: totals, peak concurrent count,
+per-source-IP breakdown with first/last seen, the client applications and
+drivers, and a per-user table.
 
 | Option | Description |
 |---|---|
 | `--json` | Machine-readable output |
 
+```bash
+mdbkit connections mongod.log
+```
+
+```
+source ip   accepted  ended  first seen           last seen            appName
+----------  --------  -----  -------------------  -------------------  ------------
+10.20.9.77  220       0      2026-07-01 08:49:30  2026-07-01 08:49:30  checkout-api
+10.20.4.11  4         1      2026-07-01 08:00:15  2026-07-01 09:29:30  OrderService
+
+authenticated users
+user          auth db  ok   failed  last authenticated   from
+------------  -------  ---  ------  -------------------  -----------
+svc_checkout  admin    221  0       2026-07-01 08:49:30  10.20.9.77
+etl_batch     admin    0    5       2026-07-01 08:51:54  10.20.11.40
+
+  etl_batch: 5 failed authentication(s) — last error: AuthenticationFailed
+```
+
+This answers the question that starts most access incidents: *did that
+account connect, from where, and when last?* If the log shows no
+authentication events at all, mdbkit says so — either auth is disabled, or
+the window contains no new logins because clients are reusing connections.
+
 ---
 
 ### `mdbkit filter <log>`
@@ -500,11 +553,21 @@ not encrypted; mdbkit decodes it offline.
 
 | Option | Default | Description |
 |---|---|---|
+| `--last DURATION` | `4h` | Analyze only the most recent window — `90m`, `4h`, `2d` |
+| `--all` | off | Analyze the entire history (see the performance note below) |
 | `--metric LABEL` | all | Restrict to one metric (repeatable), e.g. `--metric conns.current` |
 | `--step SECONDS` | 60 | Timeline bucket size |
-| `--from` / `--to` | | Time bounds (same formats as `filter`) |
+| `--from` / `--to` | | Explicit time bounds (same formats as `filter`) |
 | `--json` | | Machine-readable output |
 
+**Performance note.** `diagnostic.data` can hold weeks of per-second samples —
+a few hundred megabytes covering thousands of chunks and several thousand
+metrics each. Decoding all of it is CPU-bound and takes minutes, so these
+commands **default to the last 4 hours** and skip older chunks before
+decompressing them. On a 250 MB directory that is the difference between about
+a second and about a minute. Use `--last`/`--from`/`--to` to move the window,
+and `--all` when you really do want the whole history.
+
 ```bash
 mdbkit ftdc summary /var/lib/mongodb/diagnostic.data
 mdbkit ftdc timeline diagnostic.data --metric conns.current --step 300
@@ -521,6 +584,126 @@ contains metrics only, never document contents.
 
 ---
 
+### Shareable reports — `--report FILE`
+
+`triage` and `queries` can write a self-contained report instead of printing to
+the terminal — for a ticket, a handover, or a post-incident review.
+
+```bash
+mdbkit triage mongod.log --report incident.html     # styled, self-contained
+mdbkit triage mongod.log --report incident.md       # for tickets and PRs
+mdbkit queries mongod.log --limit 20 --report slow-queries.md
+```
+
+The format follows the file extension: `.html` or `.md`.
+
+Markdown output looks like this:
+
+```markdown
+# MongoDB incident triage
+
+*window 2026-07-01 08:10 -> 09:10  ·  generated 2026-07-01 09:12*
+
+## Findings
+
+- **[CRIT] Replica set instability** — 3 election/stepdown event(s) at 08:41:02, 08:58:14
+    - Starting an election, since we've seen no PRIMARY in election timeout period
+    - *next:* `Correlate with connection storms and slow checkpoints below`
+- **[WARN] Connection storm** — 2 minute(s) at >= 60 new connections/min; peak 480 at 08:41
+    - 10.2.1.7: 312 in the peak minute
+    - *next:* `mdbkit connections <log>`
+- **[OK] Errors** — No error/fatal severity lines in window.
+```
+
+The HTML version carries the same content with a dark, print-friendly
+stylesheet. It is **fully self-contained**: inline CSS, no JavaScript, no
+external assets or CDN references, so it opens on an air-gapped machine and
+sends nothing anywhere.
+
+Reports contain the same information as the terminal output — query **shapes**
+and metrics, never literal values from your documents.
+
+---
+
+### `mdbkit demo`
+
+Generates a realistic MongoDB structured log so you can evaluate mdbkit — or
+run a live demo — without a cluster. Output is deterministic for a given
+seed, so a demo behaves identically every time, including on a projector.
+
+| Option | Default | Description |
+|---|---|---|
+| `--scenario` | `mixed` | `healthy`, `incident`, or `mixed` |
+| `--minutes N` | 90 | How much log time to generate |
+| `--seed N` | 7 | Same seed produces byte-identical output |
+| `-o, --out FILE` | stdout | Write to a file |
+| `--with-extras` | off | Also write `indexes.json`, `schema.json` and `explain.json` beside the log |
+
+```bash
+mdbkit demo -o demo.log                          # 90 minutes, mixed
+mdbkit demo --scenario incident --minutes 30 -o incident.log
+mdbkit demo --scenario healthy -o quiet.log      # nothing wrong: the control case
+mdbkit demo | mdbkit queries -                   # straight down a pipe
+```
+
+The `incident` scenario contains an index build, a connection storm from a
+single client, a replica set election, plan-executor errors, a slow
+WiredTiger checkpoint, and a burst of unindexed queries afterwards — the
+shape of a real bad afternoon.
+
+---
+
+### `mdbkit lab`
+
+Starts a **disposable local MongoDB** for testing, reproducing a slow query,
+or rehearsing a demo. This is the only command that starts external
+processes; see [SECURITY.md](SECURITY.md) for exactly how it is bounded.
+
+Requires `mongod` on your `PATH` (and `mongosh` to initiate the replica set
+and seed data). Linux and macOS.
+
+| Action | What it does |
+|---|---|
+| `start` | Create and start a replica set, print the connection string and log paths |
+| `seed` | Insert sample data and run a workload with deliberately interesting queries |
+| `status` | Show ports, pids and whether each node is running |
+| `logs` | Print the log file paths, ready to pipe into other commands |
+| `stop` | Stop the nodes, keep the data |
+| `destroy` | Stop and delete the lab (requires `--yes`) |
+
+| Option | Default | Description |
+|---|---|---|
+| `--dir PATH` | `~/.mdbkit-lab` | Where the lab lives |
+| `--nodes N` | 3 | Replica set size |
+| `--port N` | 28110 | Base port — deliberately far from 27017 |
+| `--slowms N` | 0 | Log every operation, which is what makes the log worth reading |
+| `--standalone` | off | Single node, no replica set |
+| `--docs N` | 50000 | Documents inserted by `seed` |
+| `--yes` | | Confirm `destroy` |
+
+**The full loop:**
+
+```bash
+mdbkit lab start                    # 3-node replica set on 28110-28112
+mdbkit lab seed                     # sample data + a mixed workload
+
+mdbkit queries $(mdbkit lab logs | head -1)
+mdbkit advise  $(mdbkit lab logs | head -1)
+
+mdbkit lab destroy --yes            # remove everything
+```
+
+`seed` runs indexed point lookups alongside deliberately unindexed queries —
+an equality-plus-range-plus-sort with no supporting index, an aggregation
+that scans the collection, and updates whose predicate has no index — so the
+log immediately contains something worth analysing.
+
+**Safety.** The lab binds to `127.0.0.1` only, refuses to use or delete any
+directory it did not create, and never touches a MongoDB it did not start.
+It is a laptop and scratch-VM tool, not a deployment tool.
+
+---
+
 ### `mdbkit export-script {schema|indexes}`
 
 Prints a small `mongosh` script to stdout. **mdbkit never connects to your
@@ -540,12 +723,14 @@ mdbkit export-script schema  > export_schema.js
 Terminal output is and will remain first-class — this tool is built for the
 Linux box the database actually runs on.
 
-**Shipped in v0.2:** FTDC decoding, incident triage with system metrics,
-query reconstruction (`--as-explain`), and shareable Markdown/HTML reports.
+**Shipped in v0.3:** `demo` log generation and `lab` disposable clusters, on
+top of v0.2's FTDC decoding, incident triage, query reconstruction and
+shareable reports.
 
 Next up:
+* `mdbkit compare before.log after.log` — did the index actually help?
+* Multiple log files and globs in one command, for rotated logs.
 * Per-shape drill-down (`mdbkit queries --shape N` with full detail).
-* `serverStatus` snapshot digest for triage (two snapshots for true rates).
 * Graduating the remaining beta detectors (checkpoints, eviction, flow
   control) once validated against real incident logs — see
   `docs/TESTING-PLAYBOOK.md`. Real logs very welcome.