dbdock 1.1.24 → 1.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (139) hide show
  1. package/LICENSE +1 -1
  2. package/README.md +87 -728
  3. package/dist/alerts/alert-templates.d.ts +1 -1
  4. package/dist/alerts/alert-templates.js +6 -2
  5. package/dist/alerts/alert-templates.js.map +1 -1
  6. package/dist/alerts/alert.service.js +24 -11
  7. package/dist/alerts/alert.service.js.map +1 -1
  8. package/dist/alerts/alert.types.d.ts +1 -1
  9. package/dist/backup/backup.service.js.map +1 -1
  10. package/dist/cli/commands/analyze.js.map +1 -1
  11. package/dist/cli/commands/backup.js +10 -4
  12. package/dist/cli/commands/backup.js.map +1 -1
  13. package/dist/cli/commands/cleanup.js +12 -6
  14. package/dist/cli/commands/cleanup.js.map +1 -1
  15. package/dist/cli/commands/copydb.js +78 -37
  16. package/dist/cli/commands/copydb.js.map +1 -1
  17. package/dist/cli/commands/cross-migrate.js +9 -10
  18. package/dist/cli/commands/cross-migrate.js.map +1 -1
  19. package/dist/cli/commands/delete.js +46 -37
  20. package/dist/cli/commands/delete.js.map +1 -1
  21. package/dist/cli/commands/driver-copy.js +150 -40
  22. package/dist/cli/commands/driver-copy.js.map +1 -1
  23. package/dist/cli/commands/init.js +35 -13
  24. package/dist/cli/commands/init.js.map +1 -1
  25. package/dist/cli/commands/migrate-config.js +11 -6
  26. package/dist/cli/commands/migrate-config.js.map +1 -1
  27. package/dist/cli/commands/restore.js +14 -148
  28. package/dist/cli/commands/restore.js.map +1 -1
  29. package/dist/cli/commands/schedule.js +1 -1
  30. package/dist/cli/commands/schedule.js.map +1 -1
  31. package/dist/cli/commands/start.js +19 -14
  32. package/dist/cli/commands/start.js.map +1 -1
  33. package/dist/cli/commands/status.js +5 -2
  34. package/dist/cli/commands/status.js.map +1 -1
  35. package/dist/cli/commands/test.js +3 -38
  36. package/dist/cli/commands/test.js.map +1 -1
  37. package/dist/cli/index.js.map +1 -1
  38. package/dist/cli/utils/config.d.ts +7 -0
  39. package/dist/cli/utils/config.js.map +1 -1
  40. package/dist/cli/utils/progress.js.map +1 -1
  41. package/dist/cli/utils/retention.js +6 -1
  42. package/dist/cli/utils/retention.js.map +1 -1
  43. package/dist/config/config.schema.d.ts +8 -0
  44. package/dist/config/config.schema.js +42 -0
  45. package/dist/config/config.schema.js.map +1 -1
  46. package/dist/config/config.service.js +99 -63
  47. package/dist/config/config.service.js.map +1 -1
  48. package/dist/config/env-url.helper.js +5 -2
  49. package/dist/config/env-url.helper.js.map +1 -1
  50. package/dist/config/permissions.checker.js +5 -7
  51. package/dist/config/permissions.checker.js.map +1 -1
  52. package/dist/config/pgpass.helper.js.map +1 -1
  53. package/dist/config/secrets.validator.js +3 -1
  54. package/dist/config/secrets.validator.js.map +1 -1
  55. package/dist/crypto/crypto.service.js +1 -1
  56. package/dist/crypto/crypto.service.js.map +1 -1
  57. package/dist/dbdock.js.map +1 -1
  58. package/dist/engines/engine.types.d.ts +32 -0
  59. package/dist/engines/engine.types.js +3 -0
  60. package/dist/engines/engine.types.js.map +1 -0
  61. package/dist/engines/index.d.ts +4 -0
  62. package/dist/engines/index.js +30 -0
  63. package/dist/engines/index.js.map +1 -0
  64. package/dist/engines/mssql.engine.d.ts +2 -0
  65. package/dist/engines/mssql.engine.js +198 -0
  66. package/dist/engines/mssql.engine.js.map +1 -0
  67. package/dist/engines/mysql.engine.d.ts +2 -0
  68. package/dist/engines/mysql.engine.js +184 -0
  69. package/dist/engines/mysql.engine.js.map +1 -0
  70. package/dist/engines/postgres.engine.d.ts +2 -0
  71. package/dist/engines/postgres.engine.js +257 -0
  72. package/dist/engines/postgres.engine.js.map +1 -0
  73. package/dist/engines/redis-worker.d.ts +1 -0
  74. package/dist/engines/redis-worker.js +80 -0
  75. package/dist/engines/redis-worker.js.map +1 -0
  76. package/dist/engines/redis.engine.d.ts +2 -0
  77. package/dist/engines/redis.engine.js +124 -0
  78. package/dist/engines/redis.engine.js.map +1 -0
  79. package/dist/index.d.ts +7 -0
  80. package/dist/index.js +8 -1
  81. package/dist/index.js.map +1 -1
  82. package/dist/main.js +1 -1
  83. package/dist/main.js.map +1 -1
  84. package/dist/migration/analyzers/mongodb.analyzer.js +15 -7
  85. package/dist/migration/analyzers/mongodb.analyzer.js.map +1 -1
  86. package/dist/migration/analyzers/postgres.analyzer.js +1 -1
  87. package/dist/migration/analyzers/postgres.analyzer.js.map +1 -1
  88. package/dist/migration/config.manager.js.map +1 -1
  89. package/dist/migration/engines/migration.engine.js +1 -1
  90. package/dist/migration/engines/migration.engine.js.map +1 -1
  91. package/dist/migration/engines/mongo-to-postgres.engine.js +34 -16
  92. package/dist/migration/engines/mongo-to-postgres.engine.js.map +1 -1
  93. package/dist/migration/engines/postgres-to-mongo.engine.js +8 -10
  94. package/dist/migration/engines/postgres-to-mongo.engine.js.map +1 -1
  95. package/dist/migration/mappers/mongo-to-postgres.mapper.js +3 -6
  96. package/dist/migration/mappers/mongo-to-postgres.mapper.js.map +1 -1
  97. package/dist/migration/mappers/postgres-to-mongo.mapper.js +5 -3
  98. package/dist/migration/mappers/postgres-to-mongo.mapper.js.map +1 -1
  99. package/dist/migration/reference.detector.js.map +1 -1
  100. package/dist/migration/row.types.d.ts +12 -0
  101. package/dist/migration/row.types.js +3 -0
  102. package/dist/migration/row.types.js.map +1 -0
  103. package/dist/migration/type.mapper.d.ts +3 -3
  104. package/dist/migration/type.mapper.js +29 -14
  105. package/dist/migration/type.mapper.js.map +1 -1
  106. package/dist/migration/types.d.ts +1 -1
  107. package/dist/scheduler/schedule-manager.js +4 -1
  108. package/dist/scheduler/schedule-manager.js.map +1 -1
  109. package/dist/scheduler/scheduler.service.js +11 -6
  110. package/dist/scheduler/scheduler.service.js.map +1 -1
  111. package/dist/standalone/backup-standalone.js +17 -118
  112. package/dist/standalone/backup-standalone.js.map +1 -1
  113. package/dist/standalone/restore-standalone.d.ts +10 -0
  114. package/dist/standalone/restore-standalone.js +167 -0
  115. package/dist/standalone/restore-standalone.js.map +1 -0
  116. package/dist/storage/adapters/cloudinary.adapter.js +12 -7
  117. package/dist/storage/adapters/cloudinary.adapter.js.map +1 -1
  118. package/dist/storage/adapters/local.adapter.js +15 -12
  119. package/dist/storage/adapters/local.adapter.js.map +1 -1
  120. package/dist/storage/adapters/s3.adapter.d.ts +2 -1
  121. package/dist/storage/adapters/s3.adapter.js +23 -2
  122. package/dist/storage/adapters/s3.adapter.js.map +1 -1
  123. package/dist/storage/deletion-guard.d.ts +37 -0
  124. package/dist/storage/deletion-guard.js +102 -0
  125. package/dist/storage/deletion-guard.js.map +1 -0
  126. package/dist/storage/safe-storage.adapter.d.ts +35 -0
  127. package/dist/storage/safe-storage.adapter.js +161 -0
  128. package/dist/storage/safe-storage.adapter.js.map +1 -0
  129. package/dist/storage/storage.interface.d.ts +6 -0
  130. package/dist/storage/storage.service.d.ts +3 -0
  131. package/dist/storage/storage.service.js +18 -3
  132. package/dist/storage/storage.service.js.map +1 -1
  133. package/dist/utils/logger.js +32 -8
  134. package/dist/utils/logger.js.map +1 -1
  135. package/dist/wal/retention.service.js +23 -8
  136. package/dist/wal/retention.service.js.map +1 -1
  137. package/dist/wal/wal-archiver.service.js +7 -4
  138. package/dist/wal/wal-archiver.service.js.map +1 -1
  139. package/package.json +6 -4
package/README.md CHANGED
@@ -1,45 +1,35 @@
1
- # DBDock
1
+ # DBdock
2
2
 
3
- Stop writing backup scripts. Stop losing sleep over database migrations. DBDock handles PostgreSQL backups, restores, database copies, and cross-database migrations between MongoDB and PostgreSQL in one command.
3
+ The open-source database CLI for PostgreSQL backups, restores, database copies, and cross-database migrations between MongoDB and PostgreSQL with encryption, compression, and multi-cloud storage built in.
4
4
 
5
5
  [![npm version](https://img.shields.io/npm/v/dbdock.svg)](https://www.npmjs.com/package/dbdock)
6
6
  [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](LICENSE)
7
7
  [![Node.js](https://img.shields.io/badge/node-%3E%3D18-brightgreen)](https://nodejs.org)
8
- [![Documentation](https://img.shields.io/badge/docs-dbdock.mintlify.app-blue)](https://dbdock.mintlify.app)
8
+ [![Documentation](https://img.shields.io/badge/docs-docs.dbdock.xyz-blue)](https://docs.dbdock.xyz)
9
9
 
10
- [Full Docs](https://dbdock.mintlify.app) ・ [Discussions](https://github.com/naheemolaide/dbdock-support/discussions) ・ [Report a Bug](https://github.com/naheemolaide/dbdock-support/issues)
10
+ [Documentation](https://docs.dbdock.xyz) ・ [Discussions](https://github.com/dbdock/dbdock/discussions) ・ [Report a Bug](https://github.com/dbdock/dbdock/issues)
11
11
 
12
12
  ---
13
13
 
14
- ## The Problem
15
-
16
- Every time you need to backup a database, copy it to staging, or restore before a migration — it's the same boring steps. Connect, dump, upload, move files around, remember the right flags. Sure, you could ask AI to write you a script. But then you're maintaining that script, handling errors, adding encryption, switching storage providers, doing it again next week.
17
-
18
- It's not hard. It's just repetitive. And repetitive stuff should be one command.
19
-
20
- ## The Fix
14
+ Stop writing backup scripts. Stop maintaining migration code. One CLI, one command.
21
15
 
22
16
  ```bash
23
- npx dbdock init # One-time setup (takes 30 seconds)
24
- npx dbdock backup # Backup with encryption + compression
25
- npx dbdock restore # Restore from any backup
26
- npx dbdock copydb "db_url_1" "db_url_2" # Copy entire database, zero config
17
+ npx dbdock init # One-time setup
18
+ npx dbdock backup # Backup with encryption + compression
19
+ npx dbdock restore # Interactive restore
20
+ npx dbdock copydb "src_url" "dst_url" # Copy a database, zero config
27
21
  npx dbdock migrate "mongo_url" "postgres_url" # Cross-database migration
28
22
  ```
29
23
 
30
- That's it. No shell scripts. No manual uploads. No throwaway migration code.
31
-
32
- ---
33
-
34
24
  ## Install
35
25
 
36
- **Use directly with npx (no install needed):**
26
+ **With npx (no install):**
37
27
 
38
28
  ```bash
39
- npx dbdock backup
29
+ npx dbdock --help
40
30
  ```
41
31
 
42
- **Or install globally:**
32
+ **Global install:**
43
33
 
44
34
  ```bash
45
35
  npm install -g dbdock
@@ -47,759 +37,128 @@ npm install -g dbdock
47
37
 
48
38
  **Prerequisites:** Node.js 18+ and PostgreSQL client tools (`pg_dump`, `pg_restore`, `psql`).
49
39
 
40
+ <details>
41
+ <summary>Install PostgreSQL client tools</summary>
42
+
50
43
  ```bash
51
44
  # macOS
52
45
  brew install postgresql
53
46
 
54
47
  # Ubuntu/Debian
55
48
  sudo apt-get install postgresql-client
56
- ```
57
-
58
- ---
59
-
60
- ## Commands
61
-
62
- ### `dbdock init` — Set up in 30 seconds
63
-
64
- Run once. It walks you through everything interactively:
65
-
66
- ```bash
67
- npx dbdock init
68
- ```
69
-
70
- It asks for your database connection, picks your storage (Local, S3, R2, Cloudinary), sets up encryption if you want it, and optionally configures Slack/Email alerts.
71
-
72
- **What happens under the hood:**
73
- - Config (safe stuff) goes to `dbdock.config.json` — commit this
74
- - Secrets go to `.env` — never committed, `.gitignore` updated automatically
75
-
76
- You can also run **without a config file**: set `DBDOCK_DB_URL` (or `DATABASE_URL`) and other env vars and DBDock will use env-only configuration.
77
-
78
- ---
79
-
80
- ### `dbdock backup` — One command, full backup
81
-
82
- ```bash
83
- npx dbdock backup
84
- ```
85
-
86
- ```
87
- ████████████████████ | 100% | 45.23 MB | Speed: 12.50 MB/s | Uploading to S3
88
- ✔ Backup completed successfully
89
- ```
90
-
91
- Real-time progress. Streams directly to your storage provider. Done.
92
-
93
- **Options:**
94
-
95
- | Flag | What it does |
96
- |------|------|
97
- | `--encrypt` / `--no-encrypt` | Toggle AES-256 encryption |
98
- | `--compress` / `--no-compress` | Toggle Brotli compression |
99
- | `--encryption-key <key>` | Custom 64-char hex key |
100
- | `--compression-level <1-11>` | Compression intensity (default: 6) |
101
-
102
- **Need an encryption key?**
103
-
104
- ```bash
105
- node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
106
- ```
107
-
108
- **Backup formats:** `custom` (default binary), `plain` (SQL text), `directory`, `tar`
109
-
110
- ---
111
-
112
- ### `dbdock restore` — Interactive restore with smart filtering
113
-
114
- ```bash
115
- npx dbdock restore
116
- ```
117
-
118
- ```
119
- Progress:
120
- ────────────────────────────────────────────────────────
121
- ✔ Downloading backup
122
- ✔ Decrypting data
123
- ✔ Decompressing data
124
- ⟳ Restoring to database...
125
- ────────────────────────────────────────────────────────
126
- ✔ All steps completed in 8.42s
127
- ```
128
-
129
- Got 200+ backups? It auto-enables smart filtering — search by date, keyword, or just grab the most recent ones. No scrolling through walls of text.
130
-
131
- You can also restore to a completely different database. Pick "New Database Instance (Migrate)" when prompted and enter the target connection details.
132
-
133
- ---
134
-
135
- ### `dbdock copydb` — Copy a database with just two URLs
136
-
137
- This is the one people love. No config files. No setup. Just paste two PostgreSQL URLs:
138
-
139
- ```bash
140
- npx dbdock copydb "postgresql://user:pass@source:5432/mydb" "postgresql://user:pass@target:5432/mydb"
141
- ```
142
-
143
- It tests both connections, shows you the source DB size and table count, warns you if the target has existing data, and asks for confirmation before doing anything. Then it streams `pg_dump` directly into `pg_restore` — no temp files, no waiting.
144
-
145
- **Options:**
146
-
147
- | Flag | What it does |
148
- |------|------|
149
- | `--schema-only` | Copy tables, indexes, constraints — no data |
150
- | `--data-only` | Copy data only (schema must exist on target) |
151
- | `--verbose` | Show detailed pg_dump/pg_restore output |
152
-
153
- ```bash
154
- npx dbdock copydb --schema-only "source_url" "target_url"
155
- npx dbdock copydb --data-only "source_url" "target_url"
156
- ```
157
-
158
- **Perfect for:**
159
- - Moving between Neon, Supabase, Railway, RDS, or any Postgres host
160
- - Migrating cloud providers without the headache
161
-
162
- **Environment consolidation:**
163
-
164
- ```bash
165
- # Refresh staging with production data
166
- npx dbdock copydb "prod_url" "staging_url"
167
-
168
- # Promote staging to production
169
- npx dbdock copydb "staging_url" "prod_url"
170
-
171
- # Pull production to local for debugging
172
- npx dbdock copydb "prod_url" "postgresql://postgres:pass@localhost:5432/myapp"
173
-
174
- # Align schema across environments without touching data
175
- npx dbdock copydb --schema-only "prod_url" "staging_url"
176
- ```
177
-
178
- ---
179
-
180
- ### `dbdock list` — See all your backups
181
-
182
- ```bash
183
- npx dbdock list # Everything
184
- npx dbdock list --recent 10 # Last 10
185
- npx dbdock list --search keyword # Find specific backup
186
- npx dbdock list --days 7 # Last 7 days
187
- ```
188
-
189
- Auto-filters when you have 50+ backups so the output stays clean.
190
-
191
- ---
192
-
193
- ### `dbdock delete` — Remove backups
194
-
195
- ```bash
196
- npx dbdock delete # Interactive picker
197
- npx dbdock delete --key <id> # Delete specific backup
198
- npx dbdock delete --all # Nuke everything (with confirmation)
199
- ```
200
-
201
- ---
202
-
203
- ### `dbdock cleanup` — Auto-clean old backups
204
-
205
- ```bash
206
- npx dbdock cleanup # Interactive with preview
207
- npx dbdock cleanup --dry-run # See what would be deleted
208
- npx dbdock cleanup --force # Skip confirmation
209
- ```
210
-
211
- Shows you exactly what gets deleted and how much space you reclaim before doing anything.
212
-
213
- ---
214
-
215
- ### `dbdock status` — Check schedules and service health
216
-
217
- ```bash
218
- npx dbdock status
219
- ```
220
-
221
- ```
222
- ┌─────┬──────────────┬─────────────────┬──────────┐
223
- │ # │ Name │ Cron Expression │ Status │
224
- ├─────┼──────────────┼─────────────────┼──────────┤
225
- │ 1 │ daily │ 0 * * * * │ ✓ Active │
226
- │ 2 │ weekly │ 0 0 * * 0 │ ✗ Paused │
227
- └─────┴──────────────┴─────────────────┴──────────┘
228
- ```
229
-
230
- ---
231
-
232
- ### `dbdock schedule` — Manage cron schedules
233
-
234
- ```bash
235
- npx dbdock schedule
236
- ```
237
-
238
- Add, remove, or toggle backup schedules. Comes with presets (hourly, daily at midnight, daily at 2 AM, weekly, monthly) or use a custom cron expression.
239
-
240
- **Heads up:** Schedules only run when DBDock is integrated into your Node.js app (see [Programmatic Usage](#programmatic-usage) below). The CLI just manages the config.
241
-
242
- ---
243
-
244
- ### `dbdock test` — Verify everything works
245
-
246
- ```bash
247
- npx dbdock test
248
- ```
249
-
250
- Tests your database connection, storage provider, and alert config. Run this first if something feels off.
251
-
252
- ---
253
-
254
- ### `dbdock migrate-config` — Fix legacy configs
255
-
256
- ```bash
257
- npx dbdock migrate-config
258
- ```
259
-
260
- Got secrets sitting in `dbdock.config.json` from an older version? This extracts them to `.env`, cleans up your config, and updates `.gitignore`. One command, done.
261
-
262
- ---
263
-
264
- ## Cross-Database Migration (MongoDB ↔ PostgreSQL)
265
-
266
- Move your data between MongoDB and PostgreSQL without writing throwaway scripts. DBDock analyzes the source, proposes a schema mapping, lets you review it, and handles the transfer.
267
-
268
- ### `dbdock analyze` — Understand your database first
269
-
270
- ```bash
271
- npx dbdock analyze "mongodb://localhost:27017/myapp"
272
- npx dbdock analyze "postgresql://user:pass@localhost:5432/myapp"
273
- ```
274
-
275
- Scans the source database and shows you everything — collections/tables, field types, nested structures, inconsistencies (like a `price` field that's a string in 200 docs and a number in 15,000), and missing fields.
276
-
277
- ```
278
- DBDock - Database Analysis
279
- ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
280
-
281
- ℹ Database: MongoDB — myapp
282
- ℹ Host: localhost:27017
283
-
284
- ✓ Analysis complete
285
-
286
- Found 4 collections, 57.2K total documents
287
-
288
- users (45K docs)
289
- ├─ _id (objectId)
290
- ├─ name (string)
291
- ├─ email (string)
292
- ├─ address (object)
293
- │ ├─ street (string)
294
- │ ├─ city (string)
295
- │ └─ zip (string)
296
- ├─ phone (string) [62% present]
297
- └─ orders (array: object)
298
-
299
- products (12K docs)
300
- ├─ _id (objectId)
301
- ├─ title (string)
302
- ├─ price (string(203), number(11797)) ⚠ mixed types
303
- ├─ tags (array: string)
304
- └─ meta (object)
305
- ```
306
-
307
- ---
308
-
309
- ### `dbdock migrate` — Cross-database migration
310
-
311
- ```bash
312
- npx dbdock migrate "mongodb://localhost:27017/myapp" "postgresql://user:pass@localhost:5432/myapp"
313
- ```
314
-
315
- DBDock analyzes the source, generates a schema mapping proposal, and presents it for review before touching anything:
316
-
317
- ```
318
- DBDock - Cross-Database Migration
319
- ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
320
-
321
- ℹ Source: MongoDB — myapp
322
- ℹ Target: PostgreSQL — myapp
323
-
324
- ✓ Source analyzed: 4 collections, 57.2K documents
325
-
326
- Proposed Schema Mapping:
327
-
328
- users → users
329
- ├─ _id → (uuid_from_objectid) users.id (uuid) PK
330
- ├─ name → users.name (text)
331
- ├─ email → users.email (text)
332
- ├─ address {} → user_addresses (1:1 relation)
333
- │ ├─ street → street (text)
334
- │ ├─ city → city (text)
335
- │ └─ zip → zip (text)
336
- ├─ orders [] → user_orders child table
337
-
338
- products → products
339
- ├─ _id → (uuid_from_objectid) products.id (uuid) PK
340
- ├─ title → products.title (text)
341
- ├─ price → products.price (numeric) nullable
342
- ├─ tags [] → product_tags text[]
343
- ├─ meta {} → products.meta (jsonb)
344
-
345
- ⚠ Conflicts Found:
346
49
 
347
- products.price: string in 203 docs, number in 11797 docs
348
- Suggestion: cast to numeric, log failures
349
-
350
- • users.phone: missing in 38.0% of documents
351
- → Suggestion: nullable column
352
-
353
- ? Accept mapping? (Y / export / cancel)
50
+ # Windows
51
+ # Download from https://www.postgresql.org/download/windows/
354
52
  ```
53
+ </details>
355
54
 
356
- You review the mapping, then choose: execute it, export it as a config file to tweak, or cancel.
357
-
358
- ---
359
-
360
- ### What it handles
361
-
362
- | Scenario | How DBDock handles it |
363
- |---|---|
364
- | **Nested objects** | Consistent shape → flattened to a related table. Varying/messy → kept as `jsonb` column |
365
- | **Arrays of primitives** | `tags: ["a", "b"]` → PostgreSQL array column or junction table |
366
- | **Arrays of objects** | `orders: [{item, qty}]` → child table with foreign key back to parent |
367
- | **Missing fields** | Detects frequency across all documents, makes sparse fields nullable |
368
- | **Type mismatches** | Same field with different types → casts to majority type, logs failures to `_migration_errors` |
369
- | **ObjectId references** | Auto-detects `userId: ObjectId(...)` patterns → creates proper foreign keys |
370
- | **Deep nesting** | Flattens up to configurable depth (default: 2), stores deeper levels as `jsonb` |
371
- | **Postgres → Mongo** | 1:1 joins → embedded objects. Small 1:many → embedded arrays. Large 1:many → separate collections with refs |
372
- | **Many-to-many** | Junction tables → arrays of values in the document |
373
-
374
- ---
375
-
376
- ### The reverse — PostgreSQL to MongoDB
55
+ ## Features
377
56
 
378
- ```bash
379
- npx dbdock migrate "postgresql://user:pass@localhost:5432/myapp" "mongodb://localhost:27017/myapp"
380
- ```
381
-
382
- DBDock detects table relationships and proposes embedding vs referencing strategies:
383
-
384
- ```
385
- Proposed Document Mapping:
386
-
387
- users → users collection
388
- ├─ id → _id
389
- ├─ name → name
390
- ├─ email → email
391
- ├─ addresses → embed object as address
392
- ├─ orders → embed array as orders (< 1000 rows)
393
-
394
- products → products collection
395
- ├─ id → _id
396
- ├─ title → title
397
- ├─ price → price
398
- ├─ product_tags → embed as tags array
399
- ```
400
-
401
- Small 1:1 and 1:many relations get embedded. Large 1:many relations stay as separate collections with reference fields. You can override per table.
402
-
403
- ---
57
+ - **Beautiful CLI** — real-time progress bars, speed tracking, smart filtering
58
+ - **Multiple storage** local disk, AWS S3, Cloudflare R2, Cloudinary
59
+ - **Security-first** — AES-256-GCM encryption, env-var secrets, credential masking, `.pgpass` support
60
+ - **Retention policies** — auto-cleanup by count/age with a safety net
61
+ - **Alerts** email (SMTP) and Slack for backup success and failure
62
+ - **Cron schedules** — automated backups
63
+ - **Cross-database** — MongoDB ↔ PostgreSQL with schema mapping and dry runs
64
+ - **TypeScript SDK** — use DBdock programmatically in any Node.js app
404
65
 
405
- ### Dry run & validation
66
+ ## Quick start
406
67
 
407
68
  ```bash
408
- npx dbdock migrate --dry-run "mongodb://..." "postgresql://..."
69
+ npx dbdock init # Interactive setup (30 seconds)
70
+ npx dbdock test # Validate connections
71
+ npx dbdock backup # Create your first backup
72
+ npx dbdock restore # Interactive restore
409
73
  ```
410
74
 
411
- Runs the full migration into a temporary schema, validates counts and referential integrity, then cleans up. Nothing touches the real target:
412
-
413
- ```
414
- Dry Run Results:
415
- ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
416
- ✔ users: 45,000 → 45,000
417
- ✔ addresses: 45,000 → 44,998 (2 failed)
418
- ✔ orders: 312,400 → 312,400
419
- ✔ products: 12,000 → 12,000
420
- ✔ product_tags: 38,200 → 38,200
421
-
422
- ⚠ 2 rows failed (see _migration_errors)
423
- ✓ All foreign keys valid — dry run passed
424
- ```
425
-
426
- ---
427
-
428
- ### Incremental / delta migration
429
-
430
- Already migrated once but your app is still running on the old database? Sync only new or changed data:
431
-
432
- ```bash
433
- npx dbdock migrate --incremental --since "2026-03-10" "mongodb://..." "postgresql://..."
434
- ```
435
-
436
- Uses `createdAt`/`updatedAt` timestamps to only transfer new and changed documents. Existing rows in the target are upserted.
437
-
438
- ---
439
-
440
- ### Save & reuse config
441
-
442
- Export the mapping for your team to use reproducibly:
443
-
444
- ```bash
445
- npx dbdock migrate --export-config ./migration.yaml "mongodb://..." "postgresql://..."
446
- ```
75
+ See the [Quickstart guide](https://docs.dbdock.xyz/get-started/quickstart) for a full walkthrough.
447
76
 
448
- Then anyone runs the same migration:
77
+ ## Commands at a glance
449
78
 
450
- ```bash
451
- npx dbdock migrate --config ./migration.yaml "mongodb://..." "postgresql://..."
452
- ```
453
-
454
- Tweak the YAML/JSON config to adjust type mappings, embedding strategies, or rename target tables — then re-run.
455
-
456
- ---
457
-
458
- ### Migration options
459
-
460
- | Flag | What it does |
79
+ | Command | Purpose |
461
80
  |---|---|
462
- | `--dry-run` | Migrate into temp schema, validate, clean up |
463
- | `--incremental` | Only sync new/changed data |
464
- | `--since <date>` | Cutoff date for incremental mode (ISO format) |
465
- | `--config <path>` | Load a saved migration config (YAML or JSON) |
466
- | `--export-config <path>` | Save migration plan to file without executing |
467
- | `--batch-size <n>` | Documents per batch (default: 1000) |
468
- | `--max-depth <n>` | Max nesting depth before storing as jsonb (default: 2) |
469
-
470
- ### Core principles
471
-
472
- - **Never lose data** failed rows go to `_migration_errors`, never silently dropped
473
- - **Never surprise the user** — full mapping shown before execution, conflicts flagged
474
- - **Always let you review** nothing executes without explicit confirmation
475
- - **Reproducible** — export/import configs for team-wide consistency
476
-
477
- ---
478
-
479
- ## Storage Providers
480
-
481
- Pick your storage during `dbdock init`, or set it in `dbdock.config.json`:
482
-
483
- ### Local
484
-
485
- ```json
486
- { "storage": { "provider": "local", "local": { "path": "./backups" } } }
487
- ```
488
-
489
- ### AWS S3
490
-
491
- ```json
492
- {
493
- "storage": {
494
- "provider": "s3",
495
- "s3": { "bucket": "my-backups", "region": "us-east-1" }
496
- }
497
- }
498
- ```
499
-
500
- ```bash
501
- DBDOCK_STORAGE_ACCESS_KEY=your-access-key
502
- DBDOCK_STORAGE_SECRET_KEY=your-secret-key
503
- ```
504
-
505
- Needs IAM permissions: `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject`
506
-
507
- ### Cloudflare R2
508
-
509
- ```json
510
- {
511
- "storage": {
512
- "provider": "r2",
513
- "s3": {
514
- "bucket": "my-backups",
515
- "region": "auto",
516
- "endpoint": "https://ACCOUNT_ID.r2.cloudflarestorage.com"
517
- }
518
- }
519
- }
520
- ```
521
-
522
- Same env vars as S3 above.
523
-
524
- ### Cloudinary
525
-
526
- ```json
527
- {
528
- "storage": {
529
- "provider": "cloudinary",
530
- "cloudinary": { "cloudName": "your-cloud" }
531
- }
532
- }
533
- ```
534
-
535
- ```bash
536
- DBDOCK_CLOUDINARY_API_KEY=your-api-key
537
- DBDOCK_CLOUDINARY_API_SECRET=your-api-secret
538
- ```
539
-
540
- ---
541
-
542
- ## Security
543
-
544
- DBDock splits your config into two parts by design:
81
+ | [`init`](https://docs.dbdock.xyz/cli/init) | Interactive setup wizard |
82
+ | [`backup`](https://docs.dbdock.xyz/cli/backup) | Create a backup |
83
+ | [`restore`](https://docs.dbdock.xyz/cli/restore) | Restore a backup (with filtering) |
84
+ | [`copydb`](https://docs.dbdock.xyz/cli/copydb) | Copy a database between two URLs |
85
+ | [`list`](https://docs.dbdock.xyz/cli/list) | List backups |
86
+ | [`delete`](https://docs.dbdock.xyz/cli/delete) | Delete specific or all backups |
87
+ | [`cleanup`](https://docs.dbdock.xyz/cli/cleanup) | Apply retention policy |
88
+ | [`schedule`](https://docs.dbdock.xyz/cli/schedule) | Manage cron schedules |
89
+ | [`status`](https://docs.dbdock.xyz/cli/status) | View schedules and service health |
90
+ | [`test`](https://docs.dbdock.xyz/cli/test) | Validate configuration |
91
+ | [`analyze`](https://docs.dbdock.xyz/migration/analyze) | Inspect a database structure |
92
+ | [`migrate`](https://docs.dbdock.xyz/migration/migrate) | Cross-database migration |
93
+ | [`migrate-config`](https://docs.dbdock.xyz/cli/migrate-config) | Move legacy secrets to env vars |
545
94
 
546
- | What | Where | Git safe? |
547
- |------|-------|-----------|
548
- | Host, port, bucket names, settings | `dbdock.config.json` or env | Yes (if in config) |
549
- | Passwords, API keys, secrets | `.env` or `DBDOCK_DB_URL` | No (auto-gitignored) |
95
+ Full reference at [docs.dbdock.xyz/cli/overview](https://docs.dbdock.xyz/cli/overview).
550
96
 
551
- ### Environment Variables
97
+ ## Cross-database migration
552
98
 
553
- Use either a **full database URL** or **separate credentials**:
99
+ MongoDB PostgreSQL or PostgreSQL MongoDB with automatic schema mapping, dry runs, and incremental sync.
554
100
 
555
101
  ```bash
556
- # Option 1: Full URL (env-only config, no password var needed)
557
- DBDOCK_DB_URL=postgresql://user:password@host:5432/database
558
- # or DATABASE_URL=postgresql://user:password@host:5432/database
559
-
560
- # Option 2: Separate credentials (with or without dbdock.config.json)
561
- DBDOCK_DB_PASSWORD=your-database-password # Required if not using URL
562
- DBDOCK_STORAGE_ACCESS_KEY=your-access-key # For cloud storage
563
- DBDOCK_STORAGE_SECRET_KEY=your-secret-key # For cloud storage
564
- DBDOCK_ENCRYPTION_SECRET=64-char-hex-string # If encryption enabled
565
- DBDOCK_SMTP_USER=your-email@example.com # For email alerts
566
- DBDOCK_SMTP_PASS=your-app-password # For email alerts
567
- DBDOCK_SLACK_WEBHOOK=https://hooks.slack.com/... # For Slack alerts
102
+ npx dbdock analyze "mongodb://localhost:27017/myapp" # Inspect first
103
+ npx dbdock migrate "mongo_url" "postgres_url" --dry-run # Validate
104
+ npx dbdock migrate "mongo_url" "postgres_url" # Run it
568
105
  ```
569
106
 
570
- When `DBDOCK_DB_URL` or `DATABASE_URL` is set, it overrides database settings from `dbdock.config.json`. You can run with **env-only** (no config file) by setting the URL plus storage/encryption/alert vars in `.env`. Reads from both `.env` and `.env.local` (`.env.local` takes priority).
107
+ See [docs.dbdock.xyz/migration](https://docs.dbdock.xyz/migration/overview).
571
108
 
572
- ### .pgpass Support
109
+ ## Storage providers
573
110
 
574
- If you prefer `.pgpass` over env vars for the database password, DBDock detects and uses it automatically:
111
+ DBdock writes backups to your storage of choice:
575
112
 
576
- ```bash
577
- touch ~/.pgpass && chmod 600 ~/.pgpass
578
- echo "localhost:5432:myapp:postgres:my-secure-password" >> ~/.pgpass
579
- ```
580
-
581
- ### What's Built In
582
-
583
- - AES-256 encryption at rest
584
- - Automatic credential masking in all logs
585
- - File permission warnings for insecure configs
586
- - Strict mode (`DBDOCK_STRICT_MODE=true`) — enforces env-only secrets
587
-
588
- ---
589
-
590
- ## Retention Policy
591
-
592
- Backups pile up fast. Retention handles it automatically:
593
-
594
- ```json
595
- {
596
- "backup": {
597
- "retention": {
598
- "enabled": true,
599
- "maxBackups": 100,
600
- "maxAgeDays": 30,
601
- "minBackups": 5,
602
- "runAfterBackup": true
603
- }
604
- }
605
- }
606
- ```
607
-
608
- - Keeps at least `minBackups` recent backups (safety net — these never get deleted)
609
- - Removes anything over `maxBackups` (oldest first)
610
- - Removes anything older than `maxAgeDays`
611
- - Runs automatically after each backup if `runAfterBackup` is on
612
- - Or run manually: `npx dbdock cleanup`
613
-
614
- ---
615
-
616
- ## Alerts
617
-
618
- Get notified on Slack or Email when backups succeed or fail. Set it up during `dbdock init` or add to your config:
619
-
620
- ```json
621
- {
622
- "alerts": {
623
- "email": {
624
- "enabled": true,
625
- "smtp": {
626
- "host": "smtp.gmail.com",
627
- "port": 587,
628
- "secure": false
629
- },
630
- "from": "backups@yourapp.com",
631
- "to": ["admin@yourapp.com"]
632
- },
633
- "slack": {
634
- "enabled": true,
635
- "webhookUrl": "https://hooks.slack.com/services/..."
636
- }
637
- }
638
- }
639
- ```
640
-
641
- SMTP credentials go in env vars (`DBDOCK_SMTP_USER`, `DBDOCK_SMTP_PASS`).
642
-
643
- **Works with:** Gmail, SendGrid, AWS SES, Mailgun — anything that speaks SMTP.
644
-
645
- > For Gmail, use an [App Password](https://support.google.com/accounts/answer/185833), not your regular password.
646
-
647
- Alerts fire automatically on both CLI and programmatic backups. They include backup ID, database name, size, duration, and storage location. Failure alerts include the error message and troubleshooting tips.
648
-
649
- ---
113
+ - [Local disk](https://docs.dbdock.xyz/storage/local) — fastest, single server
114
+ - [AWS S3](https://docs.dbdock.xyz/storage/s3) industry standard, any S3-compatible service
115
+ - [Cloudflare R2](https://docs.dbdock.xyz/storage/r2) — zero egress fees
116
+ - [Cloudinary](https://docs.dbdock.xyz/storage/cloudinary) — generous free tier
650
117
 
651
- ## Programmatic Usage
118
+ Swap providers by changing one line in `dbdock.config.json`.
652
119
 
653
- Don't just use the CLI — drop DBDock into your Node.js app and trigger backups from code. Works with any backend (Express, Fastify, NestJS, whatever).
654
-
655
- ```bash
656
- npm install dbdock
657
- ```
658
-
659
- Use `dbdock.config.json` (run `npx dbdock init`) or configure entirely via env vars (`DBDOCK_DB_URL` / `DATABASE_URL` plus storage and other env vars).
660
-
661
- ### Create a Backup
120
+ ## Programmatic usage
662
121
 
663
122
  ```javascript
664
123
  const { createDBDock, BackupService } = require('dbdock');
665
124
 
666
- async function backup() {
667
- const dbdock = await createDBDock();
668
- const backupService = dbdock.get(BackupService);
669
-
670
- const result = await backupService.createBackup({
671
- format: 'plain',
672
- compress: true,
673
- encrypt: true,
674
- });
675
-
676
- console.log(`Done: ${result.metadata.id} (${result.metadata.formattedSize})`);
677
- }
678
-
679
- backup();
680
- ```
681
-
682
- **Options:** `compress`, `encrypt`, `format` (`'custom'` | `'plain'` | `'directory'` | `'tar'`), `type` (`'full'` | `'schema'` | `'data'`)
683
-
684
- ### List Backups
685
-
686
- ```javascript
687
- const { createDBDock, BackupService } = require('dbdock');
125
+ const dbdock = await createDBDock();
126
+ const backups = dbdock.get(BackupService);
688
127
 
689
- async function list() {
690
- const dbdock = await createDBDock();
691
- const backups = await dbdock.get(BackupService).listBackups();
692
- backups.forEach(b => console.log(`${b.id} — ${b.formattedSize} — ${b.startTime}`));
693
- }
128
+ const result = await backups.createBackup({
129
+ compress: true,
130
+ encrypt: true,
131
+ });
694
132
 
695
- list();
133
+ console.log(`Backup ${result.metadata.id} — ${result.metadata.formattedSize}`);
696
134
  ```
697
135
 
698
- ### Get Backup Info
699
-
700
- ```javascript
701
- const { createDBDock, BackupService } = require('dbdock');
702
-
703
- async function info(id) {
704
- const dbdock = await createDBDock();
705
- const metadata = await dbdock.get(BackupService).getBackupMetadata(id);
706
- if (!metadata) return console.log('Not found');
707
- console.log({ id: metadata.id, size: metadata.size, encrypted: !!metadata.encryption });
708
- }
136
+ Full SDK reference at [docs.dbdock.xyz/sdk](https://docs.dbdock.xyz/sdk/overview).
709
137
 
710
- info('your-backup-id');
711
- ```
712
-
713
- Restore is CLI-only for now (`npx dbdock restore`). Programmatic restore is coming.
714
-
715
- ### Schedule Backups with node-cron
716
-
717
- DBDock stays lightweight — no built-in daemon. Use `node-cron` to schedule:
718
-
719
- ```bash
720
- npm install node-cron
721
- ```
722
-
723
- ```typescript
724
- import { createDBDock, BackupService } from 'dbdock';
725
- import * as cron from 'node-cron';
726
-
727
- async function start() {
728
- const dbdock = await createDBDock();
729
- const backupService = dbdock.get(BackupService);
730
-
731
- cron.schedule('0 2 * * *', async () => {
732
- const result = await backupService.createBackup({ compress: true, encrypt: true });
733
- console.log(`Backup done: ${result.metadata.id}`);
734
- });
735
-
736
- console.log('Scheduler running — daily at 2 AM');
737
- }
738
-
739
- start();
740
- ```
741
-
742
- ---
743
-
744
- ## Troubleshooting
745
-
746
- **First step, always:**
747
-
748
- ```bash
749
- npx dbdock test
750
- ```
751
-
752
- This tests your database, storage, and alert config in one go.
753
-
754
- ### pg_dump / pg_restore / psql not found
755
-
756
- You need PostgreSQL client tools installed:
757
-
758
- ```bash
759
- brew install postgresql # macOS
760
- sudo apt-get install postgresql-client # Ubuntu/Debian
761
- ```
762
-
763
- ### Can't connect to database
764
-
765
- - Double-check `host`, `port`, `username`, `password`, `database` in config
766
- - Test manually: `psql -h HOST -p PORT -U USERNAME -d DATABASE`
767
- - Make sure the PostgreSQL server is actually running
768
- - Check firewalls / security groups if it's a remote database
769
-
770
- ### Storage errors
771
-
772
- **S3:** Check credentials, bucket name, region. IAM user needs `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject`.
773
-
774
- **R2:** Check endpoint format (`https://ACCOUNT_ID.r2.cloudflarestorage.com`), verify API token and bucket exist.
775
-
776
- **Cloudinary:** Verify cloud name, API key, API secret. Make sure the account is active.
138
+ ## Security
777
139
 
778
- ### Encryption key issues
140
+ - Secrets live in environment variables, never in `dbdock.config.json`
141
+ - AES-256-GCM encryption with PBKDF2 key derivation
142
+ - Credential masking in logs
143
+ - `.pgpass` support for host-level credential isolation
144
+ - Strict mode (`DBDOCK_STRICT_MODE=true`) refuses any config file that contains secrets
779
145
 
780
- Key must be exactly 64 hex characters. Generate a valid one:
146
+ See [docs.dbdock.xyz/core/security](https://docs.dbdock.xyz/core/security) and [SECURITY.md](SECURITY.md) for vulnerability reporting.
781
147
 
782
- ```bash
783
- node -e "console.log(require('crypto').randomBytes(32).toString('hex'))"
784
- ```
148
+ ## Contributing
785
149
 
786
- ### No backups found during restore
150
+ Contributions welcome! See [CONTRIBUTING.md](CONTRIBUTING.md) for development setup and guidelines.
787
151
 
788
- - **Local:** Check the configured path has files
789
- - **S3/R2:** Files should be in `dbdock_backups/` folder
790
- - **Cloudinary:** Check Media Library for `dbdock_backups` folder
791
- - Files should match: `backup-*.sql`
152
+ ### Contributors
792
153
 
793
- ---
154
+ Thanks to everyone who has contributed to DBdock:
794
155
 
795
- ## Links
156
+ <a href="https://github.com/dbdock/dbdock/graphs/contributors">
157
+ <img src="https://contrib.rocks/image?repo=dbdock/dbdock" alt="DBdock contributors" />
158
+ </a>
796
159
 
797
- - [npm Package](https://www.npmjs.com/package/dbdock)
798
- - [Full Documentation](https://dbdock.mintlify.app)
799
- - [GitHub](https://github.com/naheemolaide/dbdock-support)
800
- - [Discussions](https://github.com/naheemolaide/dbdock-support/discussions)
801
- - [Report Issues](https://github.com/naheemolaide/dbdock-support/issues)
160
+ Created and maintained by [Naheem Olaide](https://github.com/appdever01).
802
161
 
803
162
  ## License
804
163
 
805
- MIT
164
+ [MIT](LICENSE) — free forever, self-hosted, no vendor lock-in.