mdbkit 0.1.0__tar.gz

mdbkit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Saqib Ameen Subhan
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

mdbkit-0.1.0/PKG-INFO

@@ -0,0 +1,207 @@
+Metadata-Version: 2.4
+Name: mdbkit
+Version: 0.1.0
+Summary: Offline toolkit for MongoDB 4.4+ structured logs: log analysis, slow-query shapes, connection churn, and deterministic index advice. A spiritual successor to mtools' log tools.
+Author: Saqib Ameen Subhan
+License: MIT
+Project-URL: Homepage, https://github.com/saqibameen86/mdbkit
+Project-URL: Issues, https://github.com/saqibameen86/mdbkit/issues
+Keywords: mongodb,logs,logv2,mtools,index,dba,slow-query
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: System Administrators
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Classifier: Topic :: System :: Systems Administration
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# mdbkit
+
+**An offline toolkit for MongoDB structured logs — log analysis, slow-query shapes, connection churn, and deterministic index advice.**
+
+A spiritual successor to [mtools](https://github.com/rueckstiess/mtools)' log tools (`mloginfo`, `mlogfilter`) for the structured JSON log format MongoDB has used since 4.4 — the format mtools never supported. Built for DBAs and ops engineers running self-managed MongoDB 4.4 / 5.0 / 6.0 / 7.0 / 8.0.
+
+> **Privacy by design: mdbkit never makes a network call.** It reads log files (or stdin) and writes to stdout. No telemetry, no phoning home, no cloud. Your logs never leave your machine. It is safe to run on air-gapped database hosts.
+
+## Why
+
+MongoDB 4.4 switched to structured JSON logging, and the beloved mtools log commands stopped working — the issue has been open since 2020. Meanwhile, index recommendations from MongoDB's Performance Advisor require a paid Atlas tier or Cloud/Ops Manager. If you run Community Edition on your own infrastructure, you're back to reading raw JSON logs with `grep` and `jq`.
+
+mdbkit fills that gap: a single, dependency-free CLI that turns structured logs into answers.
+
+## Install
+
+```bash
+pipx install mdbkit        # recommended
+# or
+pip install mdbkit
+```
+
+Zero runtime dependencies (pure Python stdlib), so it also installs cleanly on air-gapped hosts from a single wheel:
+
+```bash
+pip download mdbkit -d ./wheels     # on a connected machine
+pip install --no-index --find-links ./wheels mdbkit   # on the DB host
+```
+
+Requires Python 3.9+.
+
+## Quick start
+
+```bash
+# Overall log summary: versions, restarts, connection counts, error/warning totals
+mdbkit loginfo /var/log/mongodb/mongod.log
+
+# Slow queries grouped by query shape (literals stripped), ranked by total time
+mdbkit queries mongod.log
+mdbkit queries mongod.log --sort scanRatio --limit 10 --json
+
+# Connection churn by source IP, appName, and driver
+mdbkit connections mongod.log
+
+# Filter raw log lines (output stays valid logv2 JSON — chainable)
+mdbkit filter mongod.log --slow 500 --component COMMAND --ns shop.orders
+mdbkit filter mongod.log --from 2026-07-01T08:00:00Z --to 2026-07-01T09:00:00Z | mdbkit queries -
+
+# Rotated/compressed logs work directly
+mdbkit queries mongod.log.2.gz
+```
+
+### Index advice
+
+```bash
+mdbkit advise mongod.log
+```
+
+Produces **candidate** indexes from observed slow-query shapes using the ESR
+(Equality → Sort → Range) guideline, with evidence, confidence, caveats, and a
+validation step for every recommendation:
+
+```
+[1] shop.orders  —  confidence: HIGH
+    query shape : {createdAt:gt, status:eq} sort:{createdAt:-1}
+    candidate   : { status: 1, createdAt: -1 }
+    evidence    : COLLSCAN observed in planSummary
+    evidence    : in-memory sort (hasSortStage) observed
+    evidence    : examined 251,400 docs to return 73 (3444:1)
+    caveat      : Every index adds write and storage overhead ...
+    validate    : Re-run the query with .explain('executionStats') ...
+```
+
+The advice gets sharper if you export your existing indexes and a sampled
+schema. mdbkit never connects to your database — instead it prints small
+`mongosh` scripts you run yourself, so you can read exactly what they do:
+
+```bash
+mdbkit export-script indexes > export_indexes.js
+mdbkit export-script schema  > export_schema.js
+mongosh --quiet "mongodb://localhost/shop" export_indexes.js > indexes.json
+mongosh --quiet "mongodb://localhost/shop" export_schema.js  > schema.json
+
+mdbkit advise mongod.log --indexes indexes.json --schema schema.json
+```
+
+With `--indexes`, mdbkit checks each candidate against your existing indexes
+(flagging when an existing index should already cover the query, or when a
+candidate would make an existing index redundant — it flags, never suggests
+dropping). With `--schema`, it warns about multikey (array) fields,
+low-cardinality fields, and field-name typos. The schema export records field
+**names and types only — no values.**
+
+### Incident triage (beta)
+
+```bash
+mdbkit triage /var/log/mongodb/mongod.log        # or a copied log + --no-sysprobe
+```
+
+One command during an incident: election/stepdown timeline, connection
+storms, hot collections, error clusters, slow checkpoints — plus local
+disk/memory/load probes when run on the DB host. Read-only: it never
+connects to the database, and every finding ends with a next step for a
+human to review. Detectors marked beta are validated against real logs
+where available and clearly labeled where broader validation is pending.
+
+### Explain-plan analysis
+
+Got a slow query in hand rather than a log? Save its explain output and ask
+mdbkit what's wrong:
+
+```bash
+# in mongosh:  EJSON.stringify(db.orders.find({...}).sort({...}).explain("executionStats"))
+mdbkit explain explain.json
+```
+
+You get the plan chain (`SORT -> COLLSCAN`), the examined/returned math, plain-
+English verdicts (full collection scan, blocking in-memory sort, weakly
+selective index, covered query), and — when the plan needs help — the same
+evidence-backed candidate index the advisor would produce. Works with find and
+aggregate explains, classic and SBE (6.0+) plans, and sharded winning plans.
+
+### Design principles
+
+* **Deterministic.** Same log in, same advice out. Rules, not AI. Every
+  recommendation shows its evidence and its rule of reasoning.
+* **Candidates, not commands.** mdbkit never tells you to blindly run
+  `createIndex`, and never advises dropping an index.
+* **Honest about uncertainty.** Shapes seen once are labeled low-confidence;
+  `$or`, `$regex`, `$in` and low-selectivity operators carry explicit caveats.
+* **Offline, always.** No network code exists in this codebase.
+
+## What it reads
+
+Any MongoDB 4.4+ structured log: `mongod.log`, `mongos` logs, rotated `.gz`
+files, or stdin (`-`). Slow-query lines (`"msg":"Slow query"`) are logged by
+default for operations over `slowms` (100 ms); lower `slowms` or enable
+profiling level 1 to capture more:
+
+```js
+db.setProfilingLevel(1, { slowms: 50 })
+```
+
+Pre-4.4 plain-text logs are detected and politely refused — for those, the
+original mtools still works.
+
+## Roadmap
+
+Terminal output is and will remain first-class — this tool is built for the
+Linux box the database actually runs on.
+
+* **v0.2** — FTDC (`diagnostic.data`) decoding: offline summaries of the
+  metrics MongoDB already records on every node; election/failover timeline
+  from REPL events; per-shape drill-down.
+* **v0.3** — shareable Markdown/HTML report export (for tickets and
+  post-incident reviews — a convenience layer, never a replacement for the
+  terminal).
+
+mdbkit is validated against real-world structured logs (tens of thousands of
+lines) in addition to its synthetic test fixtures.
+
+## Bugs, feature requests, questions
+
+Please use [GitHub Issues](../../issues) — it keeps problems and fixes public
+so the next person can find them. Real-world log lines that parse wrongly are
+the most valuable bug reports of all (redact literals first!).
+
+## Security
+
+mdbkit is offline by design: the codebase contains no network code, never
+executes or evaluates input, and treats every log line as untrusted data
+(strict JSON parsing only — shell constructors are never evaluated). See
+[SECURITY.md](SECURITY.md) for the reporting process.
+
+## Non-affiliation
+
+mdbkit is an independent community project. It is **not affiliated with,
+endorsed by, or sponsored by MongoDB, Inc.** "MongoDB" is a registered
+trademark of MongoDB, Inc., used here only to describe compatibility.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

mdbkit-0.1.0/README.md

@@ -0,0 +1,183 @@
+# mdbkit
+
+**An offline toolkit for MongoDB structured logs — log analysis, slow-query shapes, connection churn, and deterministic index advice.**
+
+A spiritual successor to [mtools](https://github.com/rueckstiess/mtools)' log tools (`mloginfo`, `mlogfilter`) for the structured JSON log format MongoDB has used since 4.4 — the format mtools never supported. Built for DBAs and ops engineers running self-managed MongoDB 4.4 / 5.0 / 6.0 / 7.0 / 8.0.
+
+> **Privacy by design: mdbkit never makes a network call.** It reads log files (or stdin) and writes to stdout. No telemetry, no phoning home, no cloud. Your logs never leave your machine. It is safe to run on air-gapped database hosts.
+
+## Why
+
+MongoDB 4.4 switched to structured JSON logging, and the beloved mtools log commands stopped working — the issue has been open since 2020. Meanwhile, index recommendations from MongoDB's Performance Advisor require a paid Atlas tier or Cloud/Ops Manager. If you run Community Edition on your own infrastructure, you're back to reading raw JSON logs with `grep` and `jq`.
+
+mdbkit fills that gap: a single, dependency-free CLI that turns structured logs into answers.
+
+## Install
+
+```bash
+pipx install mdbkit        # recommended
+# or
+pip install mdbkit
+```
+
+Zero runtime dependencies (pure Python stdlib), so it also installs cleanly on air-gapped hosts from a single wheel:
+
+```bash
+pip download mdbkit -d ./wheels     # on a connected machine
+pip install --no-index --find-links ./wheels mdbkit   # on the DB host
+```
+
+Requires Python 3.9+.
+
+## Quick start
+
+```bash
+# Overall log summary: versions, restarts, connection counts, error/warning totals
+mdbkit loginfo /var/log/mongodb/mongod.log
+
+# Slow queries grouped by query shape (literals stripped), ranked by total time
+mdbkit queries mongod.log
+mdbkit queries mongod.log --sort scanRatio --limit 10 --json
+
+# Connection churn by source IP, appName, and driver
+mdbkit connections mongod.log
+
+# Filter raw log lines (output stays valid logv2 JSON — chainable)
+mdbkit filter mongod.log --slow 500 --component COMMAND --ns shop.orders
+mdbkit filter mongod.log --from 2026-07-01T08:00:00Z --to 2026-07-01T09:00:00Z | mdbkit queries -
+
+# Rotated/compressed logs work directly
+mdbkit queries mongod.log.2.gz
+```
+
+### Index advice
+
+```bash
+mdbkit advise mongod.log
+```
+
+Produces **candidate** indexes from observed slow-query shapes using the ESR
+(Equality → Sort → Range) guideline, with evidence, confidence, caveats, and a
+validation step for every recommendation:
+
+```
+[1] shop.orders  —  confidence: HIGH
+    query shape : {createdAt:gt, status:eq} sort:{createdAt:-1}
+    candidate   : { status: 1, createdAt: -1 }
+    evidence    : COLLSCAN observed in planSummary
+    evidence    : in-memory sort (hasSortStage) observed
+    evidence    : examined 251,400 docs to return 73 (3444:1)
+    caveat      : Every index adds write and storage overhead ...
+    validate    : Re-run the query with .explain('executionStats') ...
+```
+
+The advice gets sharper if you export your existing indexes and a sampled
+schema. mdbkit never connects to your database — instead it prints small
+`mongosh` scripts you run yourself, so you can read exactly what they do:
+
+```bash
+mdbkit export-script indexes > export_indexes.js
+mdbkit export-script schema  > export_schema.js
+mongosh --quiet "mongodb://localhost/shop" export_indexes.js > indexes.json
+mongosh --quiet "mongodb://localhost/shop" export_schema.js  > schema.json
+
+mdbkit advise mongod.log --indexes indexes.json --schema schema.json
+```
+
+With `--indexes`, mdbkit checks each candidate against your existing indexes
+(flagging when an existing index should already cover the query, or when a
+candidate would make an existing index redundant — it flags, never suggests
+dropping). With `--schema`, it warns about multikey (array) fields,
+low-cardinality fields, and field-name typos. The schema export records field
+**names and types only — no values.**
+
+### Incident triage (beta)
+
+```bash
+mdbkit triage /var/log/mongodb/mongod.log        # or a copied log + --no-sysprobe
+```
+
+One command during an incident: election/stepdown timeline, connection
+storms, hot collections, error clusters, slow checkpoints — plus local
+disk/memory/load probes when run on the DB host. Read-only: it never
+connects to the database, and every finding ends with a next step for a
+human to review. Detectors marked beta are validated against real logs
+where available and clearly labeled where broader validation is pending.
+
+### Explain-plan analysis
+
+Got a slow query in hand rather than a log? Save its explain output and ask
+mdbkit what's wrong:
+
+```bash
+# in mongosh:  EJSON.stringify(db.orders.find({...}).sort({...}).explain("executionStats"))
+mdbkit explain explain.json
+```
+
+You get the plan chain (`SORT -> COLLSCAN`), the examined/returned math, plain-
+English verdicts (full collection scan, blocking in-memory sort, weakly
+selective index, covered query), and — when the plan needs help — the same
+evidence-backed candidate index the advisor would produce. Works with find and
+aggregate explains, classic and SBE (6.0+) plans, and sharded winning plans.
+
+### Design principles
+
+* **Deterministic.** Same log in, same advice out. Rules, not AI. Every
+  recommendation shows its evidence and its rule of reasoning.
+* **Candidates, not commands.** mdbkit never tells you to blindly run
+  `createIndex`, and never advises dropping an index.
+* **Honest about uncertainty.** Shapes seen once are labeled low-confidence;
+  `$or`, `$regex`, `$in` and low-selectivity operators carry explicit caveats.
+* **Offline, always.** No network code exists in this codebase.
+
+## What it reads
+
+Any MongoDB 4.4+ structured log: `mongod.log`, `mongos` logs, rotated `.gz`
+files, or stdin (`-`). Slow-query lines (`"msg":"Slow query"`) are logged by
+default for operations over `slowms` (100 ms); lower `slowms` or enable
+profiling level 1 to capture more:
+
+```js
+db.setProfilingLevel(1, { slowms: 50 })
+```
+
+Pre-4.4 plain-text logs are detected and politely refused — for those, the
+original mtools still works.
+
+## Roadmap
+
+Terminal output is and will remain first-class — this tool is built for the
+Linux box the database actually runs on.
+
+* **v0.2** — FTDC (`diagnostic.data`) decoding: offline summaries of the
+  metrics MongoDB already records on every node; election/failover timeline
+  from REPL events; per-shape drill-down.
+* **v0.3** — shareable Markdown/HTML report export (for tickets and
+  post-incident reviews — a convenience layer, never a replacement for the
+  terminal).
+
+mdbkit is validated against real-world structured logs (tens of thousands of
+lines) in addition to its synthetic test fixtures.
+
+## Bugs, feature requests, questions
+
+Please use [GitHub Issues](../../issues) — it keeps problems and fixes public
+so the next person can find them. Real-world log lines that parse wrongly are
+the most valuable bug reports of all (redact literals first!).
+
+## Security
+
+mdbkit is offline by design: the codebase contains no network code, never
+executes or evaluates input, and treats every log line as untrusted data
+(strict JSON parsing only — shell constructors are never evaluated). See
+[SECURITY.md](SECURITY.md) for the reporting process.
+
+## Non-affiliation
+
+mdbkit is an independent community project. It is **not affiliated with,
+endorsed by, or sponsored by MongoDB, Inc.** "MongoDB" is a registered
+trademark of MongoDB, Inc., used here only to describe compatibility.
+
+## License
+
+MIT — see [LICENSE](LICENSE).

mdbkit-0.1.0/mdbkit/__init__.py

@@ -0,0 +1,3 @@
+"""mdbkit — an offline toolkit for MongoDB structured logs and index advice."""
+
+__version__ = "0.1.0"