@memberjunction/db-auto-doc 5.14.0 → 5.15.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +169 -29
- package/bin/run.js +1 -1
- package/dist/commands/analyze.d.ts +3 -0
- package/dist/commands/analyze.d.ts.map +1 -1
- package/dist/commands/analyze.js +33 -3
- package/dist/commands/analyze.js.map +1 -1
- package/dist/commands/prune.d.ts +17 -0
- package/dist/commands/prune.d.ts.map +1 -0
- package/dist/commands/prune.js +153 -0
- package/dist/commands/prune.js.map +1 -0
- package/dist/core/AnalysisEngine.d.ts +44 -0
- package/dist/core/AnalysisEngine.d.ts.map +1 -1
- package/dist/core/AnalysisEngine.js +427 -1
- package/dist/core/AnalysisEngine.js.map +1 -1
- package/dist/core/AnalysisOrchestrator.d.ts.map +1 -1
- package/dist/core/AnalysisOrchestrator.js +33 -10
- package/dist/core/AnalysisOrchestrator.js.map +1 -1
- package/dist/discovery/FKDetector.d.ts +6 -0
- package/dist/discovery/FKDetector.d.ts.map +1 -1
- package/dist/discovery/FKDetector.js +101 -4
- package/dist/discovery/FKDetector.js.map +1 -1
- package/dist/discovery/PKDetector.d.ts +7 -0
- package/dist/discovery/PKDetector.d.ts.map +1 -1
- package/dist/discovery/PKDetector.js +121 -6
- package/dist/discovery/PKDetector.js.map +1 -1
- package/dist/drivers/MySQLDriver.d.ts.map +1 -1
- package/dist/drivers/MySQLDriver.js +2 -0
- package/dist/drivers/MySQLDriver.js.map +1 -1
- package/dist/drivers/PostgreSQLDriver.d.ts.map +1 -1
- package/dist/drivers/PostgreSQLDriver.js +2 -0
- package/dist/drivers/PostgreSQLDriver.js.map +1 -1
- package/dist/drivers/SQLServerDriver.d.ts.map +1 -1
- package/dist/drivers/SQLServerDriver.js +2 -0
- package/dist/drivers/SQLServerDriver.js.map +1 -1
- package/dist/prompts/PromptEngine.d.ts +19 -0
- package/dist/prompts/PromptEngine.d.ts.map +1 -1
- package/dist/prompts/PromptEngine.js +91 -7
- package/dist/prompts/PromptEngine.js.map +1 -1
- package/dist/types/analysis.d.ts +10 -0
- package/dist/types/analysis.d.ts.map +1 -1
- package/dist/types/config.d.ts +47 -0
- package/dist/types/config.d.ts.map +1 -1
- package/dist/types/config.js.map +1 -1
- package/dist/types/prompts.d.ts +26 -0
- package/dist/types/prompts.d.ts.map +1 -1
- package/dist/utils/config-loader.js +2 -2
- package/dist/utils/config-loader.js.map +1 -1
- package/dist/utils/ensureArray.d.ts +13 -0
- package/dist/utils/ensureArray.d.ts.map +1 -0
- package/dist/utils/ensureArray.js +39 -0
- package/dist/utils/ensureArray.js.map +1 -0
- package/package.json +5 -5
- package/prompts/fk-evaluation.md +94 -0
- package/prompts/fk-pruning-holistic.md +57 -0
- package/prompts/fk-pruning-table.md +51 -0
- package/prompts/pk-pruning-holistic.md +26 -0
- package/prompts/pk-pruning-table.md +35 -0
- package/prompts/table-analysis.md +28 -3
package/README.md
CHANGED
|
@@ -94,6 +94,43 @@ npm install @memberjunction/db-auto-doc
|
|
|
94
94
|
npm install @memberjunction/db-auto-doc --save
|
|
95
95
|
```
|
|
96
96
|
|
|
97
|
+
|
|
98
|
+
## Benchmark Results
|
|
99
|
+
|
|
100
|
+
DBAutoDoc has been extensively benchmarked across multiple databases and LLM providers. Full details in the [research paper](research/v1/paper.md).
|
|
101
|
+
|
|
102
|
+
### AdventureWorks2022 (71 tables, all constraints stripped)
|
|
103
|
+
|
|
104
|
+
| Model Configuration | PK F1 | FK F1 | Overall |
|
|
105
|
+
|---------------------|-------|-------|---------|
|
|
106
|
+
| Gemini 3 Flash / 3.1 Pro | 95.0% | 94.2% | 96.1% (A+) |
|
|
107
|
+
| Claude Sonnet 4.6 / Opus 4.6 | 95.0% | 93.0% | 96.1% (A+) |
|
|
108
|
+
| GPT-5.4-mini / GPT-5.4 | 89.4% | 77.9% | 87.9% (B+) |
|
|
109
|
+
|
|
110
|
+
### Cross-Database Results (Gemini Flash)
|
|
111
|
+
|
|
112
|
+
| Database | Tables | PK F1 | FK F1 | Descriptions |
|
|
113
|
+
|----------|--------|-------|-------|-------------|
|
|
114
|
+
| AdventureWorks | 71 | 95.0% | 94.2% | 99% |
|
|
115
|
+
| Chinook | 11 | 95.2% | 95.2% | 100% |
|
|
116
|
+
| LousyDB (dark DB) | 20 | 97.6% | 77.2% | 100% |
|
|
117
|
+
| Northwind | 13 | 72.7% | 75.0% | 100% |
|
|
118
|
+
|
|
119
|
+
### Key Discovery Pipeline
|
|
120
|
+
|
|
121
|
+
DBAutoDoc uses a 4-phase pipeline for relationship discovery:
|
|
122
|
+
|
|
123
|
+
1. **Statistical Discovery** — PK/FK candidates via uniqueness analysis, value overlap, cardinality ratios. Six deterministic gates filter false positives with zero correct-key loss.
|
|
124
|
+
2. **LLM Iterative Analysis** — The LLM creates new FK/PK proposals based on semantic understanding (89% precision on LLM-created FKs). Cross-table statistics provided in prompt context.
|
|
125
|
+
3. **Ground Truth Locking** — High-confidence keys (≥90%) become immutable to protect correct discoveries.
|
|
126
|
+
4. **Two-Pass Pruning** — A stronger model evaluates remaining candidates per-table, then holistically reviews all proposals.
|
|
127
|
+
|
|
128
|
+
### Research Paper
|
|
129
|
+
|
|
130
|
+
See [research/v1/](research/v1/) for the full paper, benchmark results, and comparison scripts:
|
|
131
|
+
- **[Paper](research/v1/paper.md)** — "DBAutoDoc: Automated Discovery and Documentation of Undocumented Database Schemas via Statistical Analysis and Iterative LLM Refinement"
|
|
132
|
+
- **[Results](research/v1/results/)** — Full exports (HTML, Markdown, ERD, SQL, CSV) for all public benchmarks
|
|
133
|
+
|
|
97
134
|
## Quick Start
|
|
98
135
|
|
|
99
136
|
### 1. Initialize
|
|
@@ -219,7 +256,7 @@ Generate SQL
|
|
|
219
256
|
- Custom value - Generate queries for top N tables
|
|
220
257
|
|
|
221
258
|
- **`tokenBudget`**: Controls LLM token usage and cost
|
|
222
|
-
- `100000` (default) - Limit to 100K tokens (~$0.50-1.00 with
|
|
259
|
+
- `100000` (default) - Limit to 100K tokens (~$0.50-1.00 with Gemini Flash)
|
|
223
260
|
- `0` - **Unlimited** token budget (useful with `maxTables: 0`)
|
|
224
261
|
- Custom value - Set specific token limit for cost control
|
|
225
262
|
|
|
@@ -249,11 +286,11 @@ Generate SQL
|
|
|
249
286
|
}
|
|
250
287
|
```
|
|
251
288
|
|
|
252
|
-
**Model Recommendations:**
|
|
253
|
-
- ✅ **
|
|
254
|
-
- ✅ **Claude
|
|
255
|
-
- ⚠️ **GPT-5**
|
|
256
|
-
-
|
|
289
|
+
**Model Recommendations (based on benchmark results):**
|
|
290
|
+
- ✅ **Gemini 3 Flash** — Best overall: 96.1% (A+) on AdventureWorks, 1M context window, lowest cost (~$0.50/100 tables). Recommended default.
|
|
291
|
+
- ✅ **Claude Sonnet 4.6** — Highest token efficiency: 96.1% (A+) with only 471K tokens, 100% description coverage. Best quality-per-token.
|
|
292
|
+
- ⚠️ **GPT-5.4-mini** — Good but limited: 87.9% (B+), 272K context window causes FK misses on large tables.
|
|
293
|
+
- 💡 **For pruning**: Use a stronger model (Gemini 3.1 Pro, Claude Opus 4.6, or GPT-5.4) via `modelOverrides` config for the precision-critical pruning pass.
|
|
257
294
|
|
|
258
295
|
### 4. Export
|
|
259
296
|
|
|
@@ -498,14 +535,69 @@ Tables and columns marked `userApproved: true` (via ground truth or manual appro
|
|
|
498
535
|
Resume analysis with fine-grained control:
|
|
499
536
|
|
|
500
537
|
```bash
|
|
538
|
+
# Resume from a previous run's state file
|
|
539
|
+
db-auto-doc analyze --resume ./output/run-1/state.json
|
|
540
|
+
|
|
501
541
|
# Resume and re-analyze tables with confidence below 70%
|
|
502
|
-
db-auto-doc analyze --resume ./state.json --reanalyze-below
|
|
542
|
+
db-auto-doc analyze --resume ./state.json --reanalyze-below 0.7
|
|
503
543
|
|
|
504
544
|
# Resume with a different iteration limit
|
|
505
545
|
db-auto-doc analyze --resume ./state.json --max-iterations 20
|
|
546
|
+
|
|
547
|
+
# Resume with additional iterations (e.g., ran 5 originally, now want 3 more)
|
|
548
|
+
db-auto-doc analyze --resume ./state.json --max-iterations 3
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
The `--reanalyze-below` flag clears `userApproved` on non-ground-truth tables whose latest confidence is below the threshold, allowing them to be re-analyzed while protecting authoritative descriptions.
|
|
552
|
+
|
|
553
|
+
### Pruning-Only Mode
|
|
554
|
+
|
|
555
|
+
Run just the FK pruning pass on an existing state file, skipping discovery and analysis iterations. Useful when you want to apply a stronger model to clean up FK false positives without re-running the full analysis:
|
|
556
|
+
|
|
557
|
+
```bash
|
|
558
|
+
# Run only the FK pruning pass on an existing state
|
|
559
|
+
db-auto-doc analyze --resume ./output/run-1/state.json --pruning-only
|
|
560
|
+
|
|
561
|
+
# Combine with a config that specifies a stronger pruning model
|
|
562
|
+
db-auto-doc analyze --resume ./output/run-1/state.json --pruning-only --config ./config-with-pro-pruning.json
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
Requires `--resume` pointing to a state file that has already completed discovery and at least one analysis iteration. The pruning pass uses the `ai.modelOverrides.fkPruning` config to select a potentially stronger model (e.g., Gemini Pro, Claude Opus) for the precision-critical FK filtering.
|
|
566
|
+
|
|
567
|
+
### CLI Flags Reference
|
|
568
|
+
|
|
569
|
+
| Flag | Short | Description |
|
|
570
|
+
|------|-------|-------------|
|
|
571
|
+
| `--config` | `-c` | Path to config file (default: `./config.json`) |
|
|
572
|
+
| `--resume` | `-r` | Resume from an existing state file |
|
|
573
|
+
| `--max-iterations` | `-n` | Override max iterations from config |
|
|
574
|
+
| `--reanalyze-below` | | Re-analyze tables with confidence below threshold (0-1) |
|
|
575
|
+
| `--pruning-only` | | Skip discovery/iterations, run only PK/FK pruning (requires `--resume`) |
|
|
576
|
+
|
|
577
|
+
|
|
578
|
+
### Standalone Pruning Command
|
|
579
|
+
|
|
580
|
+
The `prune` command runs PK/FK pruning on an existing state file without re-running analysis. It provides interactive confirmation before applying changes.
|
|
581
|
+
|
|
582
|
+
```bash
|
|
583
|
+
# Interactive mode (shows proposals, asks for confirmation)
|
|
584
|
+
db-auto-doc prune --state ./output/run-1/state.json --config ./config.json
|
|
585
|
+
|
|
586
|
+
# Silent mode (applies all pruning automatically)
|
|
587
|
+
db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --silent
|
|
588
|
+
|
|
589
|
+
# PK-only or FK-only pruning
|
|
590
|
+
db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --pk-only
|
|
591
|
+
db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --fk-only
|
|
506
592
|
```
|
|
507
593
|
|
|
508
|
-
|
|
594
|
+
| Flag | Description |
|
|
595
|
+
|------|-------------|
|
|
596
|
+
| `--state` | Path to existing state.json file (required) |
|
|
597
|
+
| `--config` | Path to config file with AI settings (required) |
|
|
598
|
+
| `--silent` | Skip interactive confirmation |
|
|
599
|
+
| `--pk-only` | Only prune primary keys |
|
|
600
|
+
| `--fk-only` | Only prune foreign keys |
|
|
509
601
|
|
|
510
602
|
### CodeGen Integration (Additional Schema Info)
|
|
511
603
|
|
|
@@ -632,7 +724,7 @@ This rich context enables AI to make accurate inferences.
|
|
|
632
724
|
},
|
|
633
725
|
"ai": {
|
|
634
726
|
"provider": "openai",
|
|
635
|
-
"model": "gpt-4-
|
|
727
|
+
"model": "gpt-5.4-mini",
|
|
636
728
|
"apiKey": "sk-...",
|
|
637
729
|
"temperature": 0.1,
|
|
638
730
|
"maxTokens": 8000,
|
|
@@ -757,7 +849,7 @@ This rich context enables AI to make accurate inferences.
|
|
|
757
849
|
},
|
|
758
850
|
"ai": {
|
|
759
851
|
"provider": "openai",
|
|
760
|
-
"model": "gpt-4-
|
|
852
|
+
"model": "gpt-5.4-mini",
|
|
761
853
|
"apiKey": "sk-...",
|
|
762
854
|
"temperature": 0.1,
|
|
763
855
|
"maxTokens": 8000
|
|
@@ -798,7 +890,7 @@ This rich context enables AI to make accurate inferences.
|
|
|
798
890
|
},
|
|
799
891
|
"ai": {
|
|
800
892
|
"provider": "openai",
|
|
801
|
-
"model": "gpt-4-
|
|
893
|
+
"model": "gpt-5.4-mini",
|
|
802
894
|
"apiKey": "sk-...",
|
|
803
895
|
"temperature": 0.1,
|
|
804
896
|
"maxTokens": 8000
|
|
@@ -824,23 +916,71 @@ This rich context enables AI to make accurate inferences.
|
|
|
824
916
|
}
|
|
825
917
|
```
|
|
826
918
|
|
|
919
|
+
|
|
920
|
+
### Retry and Rate Limiting
|
|
921
|
+
|
|
922
|
+
```json
|
|
923
|
+
{
|
|
924
|
+
"ai": {
|
|
925
|
+
"retry": {
|
|
926
|
+
"maxRetries": 5,
|
|
927
|
+
"initialDelayMs": 30000,
|
|
928
|
+
"maxDelayMs": 480000,
|
|
929
|
+
"backoffMultiplier": 2
|
|
930
|
+
},
|
|
931
|
+
"rateLimits": {
|
|
932
|
+
"requestsPerMinute": 60,
|
|
933
|
+
"maxParallelRequests": 1
|
|
934
|
+
}
|
|
935
|
+
}
|
|
936
|
+
}
|
|
937
|
+
```
|
|
938
|
+
|
|
939
|
+
Handles 429 (rate limit) and transient network errors with exponential backoff. Configure based on your API provider's limits.
|
|
940
|
+
|
|
941
|
+
|
|
942
|
+
### Multi-Model Configuration
|
|
943
|
+
|
|
944
|
+
Use different models for different pipeline phases. A cheaper/faster model for bulk analysis, a stronger model for precision-critical pruning:
|
|
945
|
+
|
|
946
|
+
```json
|
|
947
|
+
{
|
|
948
|
+
"ai": {
|
|
949
|
+
"provider": "gemini",
|
|
950
|
+
"model": "gemini-3-flash-preview",
|
|
951
|
+
"modelOverrides": {
|
|
952
|
+
"fkPruning": {
|
|
953
|
+
"model": "gemini-3.1-pro-preview",
|
|
954
|
+
"temperature": 0.05,
|
|
955
|
+
"maxTokens": 16000
|
|
956
|
+
},
|
|
957
|
+
"pkPruning": {
|
|
958
|
+
"model": "gemini-3.1-pro-preview",
|
|
959
|
+
"temperature": 0.05
|
|
960
|
+
}
|
|
961
|
+
}
|
|
962
|
+
}
|
|
963
|
+
}
|
|
964
|
+
```
|
|
965
|
+
|
|
966
|
+
|
|
827
967
|
## Supported AI Providers
|
|
828
968
|
|
|
829
|
-
DBAutoDoc integrates with MemberJunction's AI provider system. Supported providers:
|
|
969
|
+
DBAutoDoc integrates with [MemberJunction's AI provider system](../../packages/AI/). Supported providers:
|
|
830
970
|
|
|
831
971
|
| Config Provider | Driver Class | Description |
|
|
832
972
|
|-----------------|--------------|-------------|
|
|
833
|
-
| `gemini` (default) | GeminiLLM | Google Gemini |
|
|
834
|
-
| `openai` | OpenAILLM | OpenAI |
|
|
835
|
-
| `anthropic` | AnthropicLLM | Anthropic Claude |
|
|
836
|
-
| `groq` | GroqLLM | Groq |
|
|
837
|
-
| `mistral` | MistralLLM | Mistral AI |
|
|
973
|
+
| `gemini` (default) | [GeminiLLM](../../packages/AI/Providers/Gemini/) | Google Gemini |
|
|
974
|
+
| `openai` | [OpenAILLM](../../packages/AI/Providers/OpenAI/) | OpenAI |
|
|
975
|
+
| `anthropic` | [AnthropicLLM](../../packages/AI/Providers/Anthropic/) | Anthropic Claude |
|
|
976
|
+
| `groq` | [GroqLLM](../../packages/AI/Providers/Groq/) | Groq |
|
|
977
|
+
| `mistral` | [MistralLLM](../../packages/AI/Providers/Mistral/) | Mistral AI |
|
|
838
978
|
| `vertex` | VertexLLM | Google Vertex AI |
|
|
839
979
|
| `azure` | AzureLLM | Azure OpenAI |
|
|
840
|
-
| `cerebras` | CerebrasLLM | Cerebras |
|
|
841
|
-
| `openrouter` | OpenRouterLLM | OpenRouter (multi-model) |
|
|
842
|
-
| `xai` | xAILLM | xAI (Grok) |
|
|
843
|
-
| `bedrock` | BedrockLLM | AWS Bedrock |
|
|
980
|
+
| `cerebras` | [CerebrasLLM](../../packages/AI/Providers/Cerebras/) | Cerebras |
|
|
981
|
+
| `openrouter` | [OpenRouterLLM](../../packages/AI/Providers/OpenRouter/) | OpenRouter (multi-model) |
|
|
982
|
+
| `xai` | [xAILLM](../../packages/AI/Providers/xAI/) | xAI (Grok) |
|
|
983
|
+
| `bedrock` | [BedrockLLM](../../packages/AI/Providers/Bedrock/) | AWS Bedrock |
|
|
844
984
|
|
|
845
985
|
### Gemini (Default)
|
|
846
986
|
```json
|
|
@@ -855,7 +995,7 @@ DBAutoDoc integrates with MemberJunction's AI provider system. Supported provide
|
|
|
855
995
|
```json
|
|
856
996
|
{
|
|
857
997
|
"provider": "openai",
|
|
858
|
-
"model": "gpt-4-
|
|
998
|
+
"model": "gpt-5.4-mini",
|
|
859
999
|
"apiKey": "sk-..."
|
|
860
1000
|
}
|
|
861
1001
|
```
|
|
@@ -864,7 +1004,7 @@ DBAutoDoc integrates with MemberJunction's AI provider system. Supported provide
|
|
|
864
1004
|
```json
|
|
865
1005
|
{
|
|
866
1006
|
"provider": "anthropic",
|
|
867
|
-
"model": "claude-
|
|
1007
|
+
"model": "claude-sonnet-4-6",
|
|
868
1008
|
"apiKey": "sk-ant-..."
|
|
869
1009
|
}
|
|
870
1010
|
```
|
|
@@ -873,7 +1013,7 @@ DBAutoDoc integrates with MemberJunction's AI provider system. Supported provide
|
|
|
873
1013
|
```json
|
|
874
1014
|
{
|
|
875
1015
|
"provider": "groq",
|
|
876
|
-
"model": "llama-
|
|
1016
|
+
"model": "llama-4-scout-17b-16e-instruct",
|
|
877
1017
|
"apiKey": "gsk_..."
|
|
878
1018
|
}
|
|
879
1019
|
```
|
|
@@ -1050,7 +1190,7 @@ const schemaInfo = schemaInfoGen.generate(state, {
|
|
|
1050
1190
|
|
|
1051
1191
|
Typical costs (will vary by database size and complexity):
|
|
1052
1192
|
|
|
1053
|
-
| Database Size | Tables | Iterations | Tokens | Cost (
|
|
1193
|
+
| Database Size | Tables | Iterations | Tokens | Cost (Gemini Flash) | Cost (Sonnet 4.6) |
|
|
1054
1194
|
|---------------|--------|------------|--------|--------------|-------------|
|
|
1055
1195
|
| Small | 10-20 | 2-3 | ~50K | $0.50 | $0.02 |
|
|
1056
1196
|
| Medium | 50-100 | 3-5 | ~200K | $2.00 | $0.08 |
|
|
@@ -1059,7 +1199,7 @@ Typical costs (will vary by database size and complexity):
|
|
|
1059
1199
|
|
|
1060
1200
|
**With Relationship Discovery**: Add 25-40% to token/cost estimates for databases with missing constraints.
|
|
1061
1201
|
|
|
1062
|
-
**With Sample Query Generation** (5 queries/table,
|
|
1202
|
+
**With Sample Query Generation** (5 queries/table, Gemini Flash):
|
|
1063
1203
|
|
|
1064
1204
|
| Database Size | Tables | Additional Tokens | Additional Cost |
|
|
1065
1205
|
|---------------|--------|-------------------|-----------------|
|
|
@@ -1081,13 +1221,13 @@ Note: Sample query generation uses ~6× more API calls than description generati
|
|
|
1081
1221
|
6. **Filter exports** - Use `--confidence-threshold` to only apply high-confidence descriptions
|
|
1082
1222
|
7. **Iterate** - Run analysis multiple times if first pass isn't satisfactory
|
|
1083
1223
|
8. **Resume from checkpoints** - Save costs by continuing previous runs
|
|
1084
|
-
9. **Use appropriate models** - Balance cost vs. quality (
|
|
1224
|
+
9. **Use appropriate models** - Balance cost vs. quality (see benchmark results)
|
|
1085
1225
|
10. **Export multiple formats** - HTML for browsing, CSV for analysis, SQL for database
|
|
1086
1226
|
|
|
1087
1227
|
### Sample Query Generation Best Practices
|
|
1088
1228
|
|
|
1089
1229
|
**Configuration:**
|
|
1090
|
-
1. **Use
|
|
1230
|
+
1. **Use Gemini 3 Flash or Claude Sonnet 4.6** - Best balance of quality, speed, and cost (see benchmark results)
|
|
1091
1231
|
2. **Set token budget** - Prevents runaway costs (default: 100K tokens)
|
|
1092
1232
|
3. **Start with 5 queries/table** - Good balance of coverage and cost
|
|
1093
1233
|
4. **Enable query fix** (`enableQueryFix: true`, default) - Auto-fixes broken queries (up to 3 attempts)
|
|
@@ -1134,7 +1274,7 @@ Note: Sample query generation uses ~6× more API calls than description generati
|
|
|
1134
1274
|
- Enable guardrails with appropriate limits
|
|
1135
1275
|
- Reduce `maxTokens` per prompt
|
|
1136
1276
|
- Filter schemas/tables to focus on subset
|
|
1137
|
-
- Use cheaper model (
|
|
1277
|
+
- Use cheaper model (Gemini Flash is already very cost-effective)
|
|
1138
1278
|
- Disable relationship discovery if not needed
|
|
1139
1279
|
|
|
1140
1280
|
### "Guardrail limits exceeded"
|
package/bin/run.js
CHANGED
|
@@ -10,7 +10,7 @@ dotenvConfig({ path: resolve(__dirname, '../../../.env') });
|
|
|
10
10
|
dotenvConfig();
|
|
11
11
|
|
|
12
12
|
// Commands that need the full MJ server bootstrap (database connectivity, AI providers, etc.)
|
|
13
|
-
const heavyCommands = ['analyze', 'generate-queries', 'export-sample-queries'];
|
|
13
|
+
const heavyCommands = ['analyze', 'generate-queries', 'export-sample-queries', 'prune'];
|
|
14
14
|
const commandArg = process.argv[2];
|
|
15
15
|
const needsBootstrap = heavyCommands.includes(commandArg);
|
|
16
16
|
|
|
@@ -8,6 +8,9 @@ export default class Analyze extends Command {
|
|
|
8
8
|
static flags: {
|
|
9
9
|
resume: import("@oclif/core/lib/interfaces/parser.js").OptionFlag<string, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
10
10
|
config: import("@oclif/core/lib/interfaces/parser.js").OptionFlag<string, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
11
|
+
'max-iterations': import("@oclif/core/lib/interfaces/parser.js").OptionFlag<number, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
12
|
+
'reanalyze-below': import("@oclif/core/lib/interfaces/parser.js").OptionFlag<string, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
13
|
+
'pruning-only': import("@oclif/core/lib/interfaces/parser.js").BooleanFlag<boolean>;
|
|
11
14
|
};
|
|
12
15
|
static args: {};
|
|
13
16
|
run(): Promise<void>;
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"analyze.d.ts","sourceRoot":"","sources":["../../src/commands/analyze.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAS,MAAM,aAAa,CAAC;AAM7C,MAAM,CAAC,OAAO,OAAO,OAAQ,SAAQ,OAAO;IAC1C,MAAM,CAAC,WAAW,SAAiD;IAEnE,MAAM,CAAC,QAAQ,
|
|
1
|
+
{"version":3,"file":"analyze.d.ts","sourceRoot":"","sources":["../../src/commands/analyze.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAS,MAAM,aAAa,CAAC;AAM7C,MAAM,CAAC,OAAO,OAAO,OAAQ,SAAQ,OAAO;IAC1C,MAAM,CAAC,WAAW,SAAiD;IAEnE,MAAM,CAAC,QAAQ,WAOb;IAEF,MAAM,CAAC,KAAK;;;;;;MAwBV;IAEF,MAAM,CAAC,IAAI,KAAM;IAEX,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;CAqE3B"}
|
package/dist/commands/analyze.js
CHANGED
|
@@ -10,34 +10,64 @@ export default class Analyze extends Command {
|
|
|
10
10
|
static { this.description = 'Analyze database and generate documentation'; }
|
|
11
11
|
static { this.examples = [
|
|
12
12
|
'$ db-auto-doc analyze',
|
|
13
|
-
'$ db-auto-doc analyze --
|
|
14
|
-
'$ db-auto-doc analyze --
|
|
13
|
+
'$ db-auto-doc analyze --config ./my-config.json',
|
|
14
|
+
'$ db-auto-doc analyze --resume ./output/run-6/run-1/state.json',
|
|
15
|
+
'$ db-auto-doc analyze --resume ./output/run-6/run-1/state.json --max-iterations 3',
|
|
16
|
+
'$ db-auto-doc analyze --resume ./output/run-6/run-1/state.json --reanalyze-below 0.7',
|
|
17
|
+
'$ db-auto-doc analyze --resume ./output/run-6/run-1/state.json --pruning-only'
|
|
15
18
|
]; }
|
|
16
19
|
static { this.flags = {
|
|
17
20
|
resume: Flags.string({
|
|
18
21
|
char: 'r',
|
|
19
|
-
description: 'Resume from an existing state file',
|
|
22
|
+
description: 'Resume from an existing state file. Skips schema introspection and discovery if state already has data.',
|
|
20
23
|
required: false
|
|
21
24
|
}),
|
|
22
25
|
config: Flags.string({
|
|
23
26
|
char: 'c',
|
|
24
27
|
description: 'Path to config file',
|
|
25
28
|
default: './config.json'
|
|
29
|
+
}),
|
|
30
|
+
'max-iterations': Flags.integer({
|
|
31
|
+
char: 'n',
|
|
32
|
+
description: 'Override max iterations from config for this run',
|
|
33
|
+
required: false
|
|
34
|
+
}),
|
|
35
|
+
'reanalyze-below': Flags.string({
|
|
36
|
+
description: 'Only re-analyze tables with confidence below this threshold (0-1). Use with --resume to refine low-confidence results.',
|
|
37
|
+
required: false
|
|
38
|
+
}),
|
|
39
|
+
'pruning-only': Flags.boolean({
|
|
40
|
+
description: 'Skip discovery and analysis iterations. Only run the FK pruning pass on an existing state file. Requires --resume.',
|
|
41
|
+
default: false
|
|
26
42
|
})
|
|
27
43
|
}; }
|
|
28
44
|
static { this.args = {}; }
|
|
29
45
|
async run() {
|
|
30
46
|
const { flags } = await this.parse(Analyze);
|
|
31
47
|
const spinner = ora();
|
|
48
|
+
// Validate flag combinations
|
|
49
|
+
if (flags['pruning-only'] && !flags.resume) {
|
|
50
|
+
this.error('--pruning-only requires --resume pointing to an existing state file');
|
|
51
|
+
}
|
|
32
52
|
try {
|
|
33
53
|
// Load configuration
|
|
34
54
|
spinner.start('Loading configuration');
|
|
35
55
|
const config = await ConfigLoader.load(flags.config);
|
|
36
56
|
spinner.succeed('Configuration loaded');
|
|
57
|
+
// Parse reanalyze-below as number if provided
|
|
58
|
+
const reanalyzeBelowConfidence = flags['reanalyze-below'] != null
|
|
59
|
+
? parseFloat(flags['reanalyze-below'])
|
|
60
|
+
: undefined;
|
|
61
|
+
// For pruning-only, set max iterations to 0 so iteration loop is skipped
|
|
62
|
+
const maxIterations = flags['pruning-only']
|
|
63
|
+
? 0
|
|
64
|
+
: flags['max-iterations'];
|
|
37
65
|
// Create orchestrator
|
|
38
66
|
const orchestrator = new AnalysisOrchestrator({
|
|
39
67
|
config,
|
|
40
68
|
resumeFromState: flags.resume,
|
|
69
|
+
reanalyzeBelowConfidence,
|
|
70
|
+
maxIterations,
|
|
41
71
|
onProgress: (message, data) => {
|
|
42
72
|
if (data) {
|
|
43
73
|
spinner.succeed(`${message}: ${JSON.stringify(data)}`);
|
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"analyze.js","sourceRoot":"","sources":["../../src/commands/analyze.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,GAAG,MAAM,KAAK,CAAC;AACtB,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,oBAAoB,EAAE,MAAM,iCAAiC,CAAC;AAEvE,MAAM,CAAC,OAAO,OAAO,OAAQ,SAAQ,OAAO;aACnC,gBAAW,GAAG,6CAA6C,CAAC;aAE5D,aAAQ,GAAG;QAChB,uBAAuB;QACvB,
|
|
1
|
+
{"version":3,"file":"analyze.js","sourceRoot":"","sources":["../../src/commands/analyze.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,GAAG,MAAM,KAAK,CAAC;AACtB,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,oBAAoB,EAAE,MAAM,iCAAiC,CAAC;AAEvE,MAAM,CAAC,OAAO,OAAO,OAAQ,SAAQ,OAAO;aACnC,gBAAW,GAAG,6CAA6C,CAAC;aAE5D,aAAQ,GAAG;QAChB,uBAAuB;QACvB,iDAAiD;QACjD,gEAAgE;QAChE,mFAAmF;QACnF,sFAAsF;QACtF,+EAA+E;KAChF,CAAC;aAEK,UAAK,GAAG;QACb,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC;YACnB,IAAI,EAAE,GAAG;YACT,WAAW,EAAE,yGAAyG;YACtH,QAAQ,EAAE,KAAK;SAChB,CAAC;QACF,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC;YACnB,IAAI,EAAE,GAAG;YACT,WAAW,EAAE,qBAAqB;YAClC,OAAO,EAAE,eAAe;SACzB,CAAC;QACF,gBAAgB,EAAE,KAAK,CAAC,OAAO,CAAC;YAC9B,IAAI,EAAE,GAAG;YACT,WAAW,EAAE,kDAAkD;YAC/D,QAAQ,EAAE,KAAK;SAChB,CAAC;QACF,iBAAiB,EAAE,KAAK,CAAC,MAAM,CAAC;YAC9B,WAAW,EAAE,wHAAwH;YACrI,QAAQ,EAAE,KAAK;SAChB,CAAC;QACF,cAAc,EAAE,KAAK,CAAC,OAAO,CAAC;YAC5B,WAAW,EAAE,oHAAoH;YACjI,OAAO,EAAE,KAAK;SACf,CAAC;KACH,CAAC;aAEK,SAAI,GAAG,EAAE,CAAC;IAEjB,KAAK,CAAC,GAAG;QACP,MAAM,EAAE,KAAK,EAAE,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC;QAC5C,MAAM,OAAO,GAAG,GAAG,EAAE,CAAC;QAEtB,6BAA6B;QAC7B,IAAI,KAAK,CAAC,cAAc,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC;YAC3C,IAAI,CAAC,KAAK,CAAC,qEAAqE,CAAC,CAAC;QACpF,CAAC;QAED,IAAI,CAAC;YACH,qBAAqB;YACrB,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;YACvC,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YACrD,OAAO,CAAC,OAAO,CAAC,sBAAsB,CAAC,CAAC;YAExC,8CAA8C;YAC9C,MAAM,wBAAwB,GAAG,KAAK,CAAC,iBAAiB,CAAC,IAAI,IAAI;gBAC/D,CAAC,CAAC,UAAU,CAAC,KAAK,CAAC,iBAAiB,CAAC,CAAC;gBACtC,CAAC,CAAC,SAAS,CAAC;YAEd,yEAAyE;YACzE,MAAM,aAAa,GAAG,KAAK,CAAC,cAAc,CAAC;gBACzC,CAAC,CAAC,CAAC;gBACH,CAAC,CAAC,KAAK,CAAC,gBAAgB,CAAC,CAAC;YAE5B,sBAAsB;YACtB,MAAM,YAAY,GAAG,IAAI,oBAAoB,CAAC;gBAC5C,MAAM;gBACN,eAAe,EAAE,KAAK,CAAC,MAAM;gBAC7B,wBAAwB;gBACxB,aAAa;gBACb,UAAU,EAAE,CAAC,OAAO,EAAE,IAAI,EAAE,EAAE;oBAC5B,IAAI,IAAI,EAAE,CAAC;wBACT,OAAO,CAAC,OAAO,CAAC,GAAG,OAAO,KAAK,IAAI,CAAC,SAAS,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;oBACzD,CAAC;yBAAM,CAAC;wBACN,OAAO,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;oBAC3B,CAAC;gBACH,CAAC;aACF,CAAC,CAAC;YAEH,mBAAmB;YACnB,OAAO,CAAC,KAAK,CAAC,mBAAmB,CAAC,CAAC;YACnC,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,OAAO,EAAE,CAAC;YAE5C,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;gBACnB,OAAO,CAAC,OAAO,CAAC,oBAAoB,CAAC,CAAC;gBACtC,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,wBAAwB,CAAC,CAAC,CAAC;gBAChD,IAAI,CAAC,GAAG,CAAC,iBAAiB,MAAM,CAAC,GAAG,CAAC,mBAAmB,EAAE,CAAC,CAAC;gBAC5D,IAAI,CAAC,GAAG,CAAC,kBAAkB,MAAM,CAAC,GAAG,CAAC,eAAe,EAAE,cAAc,EAAE,IAAI,CAAC,YAAY,CAAC,MAAM,CAAC,GAAG,CAAC,gBAAgB,IAAI,CAAC,CAAC,CAAC,cAAc,EAAE,aAAa,CAAC,MAAM,CAAC,GAAG,CAAC,iBAAiB,IAAI,CAAC,CAAC,CAAC,cAAc,EAAE,GAAG,CAAC,CAAC;gBACjN,IAAI,CAAC,GAAG,CAAC,sBAAsB,MAAM,CAAC,GAAG,CAAC,aAAa,EAAE,OAAO,CAAC,CAAC,CAAC,IAAI,MAAM,EAAE,CAAC,CAAC;gBACjF,IAAI,CAAC,GAAG,CAAC,oBAAoB,MAAM,CAAC,YAAY,EAAE,CAAC,CAAC;gBACpD,IAAI,CAAC,GAAG,CAAC,UAAU,CAAC,CAAC;gBACrB,IAAI,CAAC,GAAG,CAAC,kBAAkB,CAAC,CAAC;gBAC7B,IAAI,CAAC,GAAG,CAAC,0BAA0B,CAAC,CAAC;gBACrC,IAAI,CAAC,GAAG,CAAC,kBAAkB,CAAC,CAAC;gBAE7B,IAAI,KAAK,CAAC,MAAM,EAAE,CAAC;oBACjB,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,qBAAqB,KAAK,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC;gBAC5D,CAAC;YACH,CAAC;iBAAM,CAAC;gBACN,OAAO,CAAC,IAAI,CAAC,iBAAiB,CAAC,CAAC;gBAChC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,OAAO,IAAI,eAAe,CAAC,CAAC;YAChD,CAAC;QAEH,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,IAAI,CAAC,iBAAiB,CAAC,CAAC;YAChC,IAAI,CAAC,KAAK,CAAE,KAAe,CAAC,OAAO,CAAC,CAAC;QACvC,CAAC;IACH,CAAC"}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Prune command - Standalone PK/FK pruning on existing state
|
|
3
|
+
*/
|
|
4
|
+
import { Command } from '@oclif/core';
|
|
5
|
+
export default class Prune extends Command {
|
|
6
|
+
static description: string;
|
|
7
|
+
static examples: string[];
|
|
8
|
+
static flags: {
|
|
9
|
+
state: import("@oclif/core/lib/interfaces/parser.js").OptionFlag<string, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
10
|
+
config: import("@oclif/core/lib/interfaces/parser.js").OptionFlag<string, import("@oclif/core/lib/interfaces/parser.js").CustomOptions>;
|
|
11
|
+
silent: import("@oclif/core/lib/interfaces/parser.js").BooleanFlag<boolean>;
|
|
12
|
+
'pk-only': import("@oclif/core/lib/interfaces/parser.js").BooleanFlag<boolean>;
|
|
13
|
+
'fk-only': import("@oclif/core/lib/interfaces/parser.js").BooleanFlag<boolean>;
|
|
14
|
+
};
|
|
15
|
+
run(): Promise<void>;
|
|
16
|
+
}
|
|
17
|
+
//# sourceMappingURL=prune.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"prune.d.ts","sourceRoot":"","sources":["../../src/commands/prune.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAS,MAAM,aAAa,CAAC;AAiB7C,MAAM,CAAC,OAAO,OAAO,KAAM,SAAQ,OAAO;IACxC,MAAM,CAAC,WAAW,SAA6E;IAE/F,MAAM,CAAC,QAAQ,WAKb;IAEF,MAAM,CAAC,KAAK;;;;;;MAuBV;IAEI,GAAG,IAAI,OAAO,CAAC,IAAI,CAAC;CAkH3B"}
|
|
@@ -0,0 +1,153 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Prune command - Standalone PK/FK pruning on existing state
|
|
3
|
+
*/
|
|
4
|
+
import { Command, Flags } from '@oclif/core';
|
|
5
|
+
import ora from 'ora';
|
|
6
|
+
import chalk from 'chalk';
|
|
7
|
+
import * as fs from 'fs';
|
|
8
|
+
import * as readline from 'readline';
|
|
9
|
+
import { ConfigLoader } from '../utils/config-loader.js';
|
|
10
|
+
import { AnalysisEngine } from '../core/AnalysisEngine.js';
|
|
11
|
+
import { StateManager } from '../state/StateManager.js';
|
|
12
|
+
import * as path from 'path';
|
|
13
|
+
import { fileURLToPath } from 'node:url';
|
|
14
|
+
const __filename = fileURLToPath(import.meta.url);
|
|
15
|
+
const __dirname = path.dirname(__filename);
|
|
16
|
+
import { PromptEngine } from '../prompts/PromptEngine.js';
|
|
17
|
+
import { IterationTracker } from '../state/IterationTracker.js';
|
|
18
|
+
export default class Prune extends Command {
|
|
19
|
+
static { this.description = 'Run PK/FK pruning on an existing state file without re-running analysis'; }
|
|
20
|
+
static { this.examples = [
|
|
21
|
+
'$ db-auto-doc prune --state ./output/run-1/state.json --config ./config.json',
|
|
22
|
+
'$ db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --pk-only',
|
|
23
|
+
'$ db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --fk-only',
|
|
24
|
+
'$ db-auto-doc prune --state ./output/run-1/state.json --config ./config.json --silent'
|
|
25
|
+
]; }
|
|
26
|
+
static { this.flags = {
|
|
27
|
+
state: Flags.string({
|
|
28
|
+
char: 's',
|
|
29
|
+
description: 'Path to existing state.json file',
|
|
30
|
+
required: true
|
|
31
|
+
}),
|
|
32
|
+
config: Flags.string({
|
|
33
|
+
char: 'c',
|
|
34
|
+
description: 'Path to config file',
|
|
35
|
+
default: './config.json'
|
|
36
|
+
}),
|
|
37
|
+
silent: Flags.boolean({
|
|
38
|
+
description: 'Skip interactive confirmation, apply all pruning proposals automatically',
|
|
39
|
+
default: false
|
|
40
|
+
}),
|
|
41
|
+
'pk-only': Flags.boolean({
|
|
42
|
+
description: 'Only prune primary keys',
|
|
43
|
+
default: false
|
|
44
|
+
}),
|
|
45
|
+
'fk-only': Flags.boolean({
|
|
46
|
+
description: 'Only prune foreign keys',
|
|
47
|
+
default: false
|
|
48
|
+
})
|
|
49
|
+
}; }
|
|
50
|
+
async run() {
|
|
51
|
+
const { flags } = await this.parse(Prune);
|
|
52
|
+
const spinner = ora();
|
|
53
|
+
try {
|
|
54
|
+
// Load config
|
|
55
|
+
spinner.start('Loading configuration');
|
|
56
|
+
const config = await ConfigLoader.load(flags.config);
|
|
57
|
+
spinner.succeed('Configuration loaded');
|
|
58
|
+
// Load state
|
|
59
|
+
spinner.start('Loading state file');
|
|
60
|
+
if (!fs.existsSync(flags.state)) {
|
|
61
|
+
this.error(`State file not found: ${flags.state}`);
|
|
62
|
+
}
|
|
63
|
+
const stateContent = fs.readFileSync(flags.state, 'utf-8');
|
|
64
|
+
const state = JSON.parse(stateContent);
|
|
65
|
+
spinner.succeed(`State loaded: ${state.schemas?.length || 0} schemas`);
|
|
66
|
+
// Create engine components
|
|
67
|
+
const stateManager = new StateManager(flags.state);
|
|
68
|
+
const promptsDir = path.join(__dirname, '../../prompts');
|
|
69
|
+
const promptEngine = new PromptEngine(config.ai, promptsDir);
|
|
70
|
+
const iterationTracker = new IterationTracker();
|
|
71
|
+
await promptEngine.initialize();
|
|
72
|
+
const analysisEngine = new AnalysisEngine(config, promptEngine, stateManager, iterationTracker);
|
|
73
|
+
// Create a minimal run object for tracking
|
|
74
|
+
const run = {
|
|
75
|
+
runId: 'prune-' + Date.now(),
|
|
76
|
+
startedAt: new Date().toISOString(),
|
|
77
|
+
status: 'in_progress',
|
|
78
|
+
levelsProcessed: 0,
|
|
79
|
+
iterationsPerformed: 0,
|
|
80
|
+
backpropagationCount: 0,
|
|
81
|
+
sanityCheckCount: 0,
|
|
82
|
+
converged: false,
|
|
83
|
+
modelUsed: config.ai.model,
|
|
84
|
+
vendor: config.ai.provider || 'gemini',
|
|
85
|
+
temperature: config.ai.temperature || 0.1,
|
|
86
|
+
totalTokensUsed: 0,
|
|
87
|
+
totalInputTokens: 0,
|
|
88
|
+
totalOutputTokens: 0,
|
|
89
|
+
estimatedCost: 0,
|
|
90
|
+
warnings: [],
|
|
91
|
+
errors: [],
|
|
92
|
+
processingLog: [],
|
|
93
|
+
sanityChecks: []
|
|
94
|
+
};
|
|
95
|
+
let pkResults = { removed: 0, kept: 0 };
|
|
96
|
+
let fkResults = { removed: 0, kept: 0 };
|
|
97
|
+
// PK pruning
|
|
98
|
+
if (!flags['fk-only'] && state.phases.keyDetection) {
|
|
99
|
+
spinner.start('Running PK pruning...');
|
|
100
|
+
const { locked: pkLocked, unlocked: pkUnlocked } = analysisEngine.lockInterimPKGroundTruth(state);
|
|
101
|
+
spinner.succeed(`PK ground truth: ${pkLocked} locked, ${pkUnlocked} unlocked`);
|
|
102
|
+
if (pkUnlocked > 0) {
|
|
103
|
+
pkResults = await analysisEngine.prunePrimaryKeys(state, run);
|
|
104
|
+
this.log(chalk.yellow(` PK pruning: ${pkResults.removed} removed, ${pkResults.kept} kept`));
|
|
105
|
+
}
|
|
106
|
+
else {
|
|
107
|
+
this.log(chalk.green(' No unlocked PKs to prune'));
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
// FK pruning
|
|
111
|
+
if (!flags['pk-only'] && state.phases.keyDetection) {
|
|
112
|
+
spinner.start('Running FK pruning...');
|
|
113
|
+
const { locked: fkLocked, unlocked: fkUnlocked } = analysisEngine.lockInterimGroundTruth(state);
|
|
114
|
+
spinner.succeed(`FK ground truth: ${fkLocked} locked, ${fkUnlocked} unlocked`);
|
|
115
|
+
if (fkUnlocked > 0) {
|
|
116
|
+
fkResults = await analysisEngine.pruneForeignKeys(state, run);
|
|
117
|
+
this.log(chalk.yellow(` FK pruning: ${fkResults.removed} removed, ${fkResults.kept} kept`));
|
|
118
|
+
}
|
|
119
|
+
else {
|
|
120
|
+
this.log(chalk.green(' No unlocked FKs to prune'));
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
// Summary
|
|
124
|
+
this.log('');
|
|
125
|
+
this.log(chalk.bold('Pruning Summary:'));
|
|
126
|
+
this.log(` PKs: ${pkResults.removed} removed, ${pkResults.kept} kept`);
|
|
127
|
+
this.log(` FKs: ${fkResults.removed} removed, ${fkResults.kept} kept`);
|
|
128
|
+
// Interactive confirmation (unless --silent)
|
|
129
|
+
if (!flags.silent && (pkResults.removed > 0 || fkResults.removed > 0)) {
|
|
130
|
+
const rl = readline.createInterface({ input: process.stdin, output: process.stdout });
|
|
131
|
+
const answer = await new Promise(resolve => {
|
|
132
|
+
rl.question(chalk.cyan('\nApply these changes? (Y/N): '), resolve);
|
|
133
|
+
});
|
|
134
|
+
rl.close();
|
|
135
|
+
if (answer.toLowerCase() !== 'y') {
|
|
136
|
+
this.log(chalk.red('Changes discarded.'));
|
|
137
|
+
return;
|
|
138
|
+
}
|
|
139
|
+
}
|
|
140
|
+
// Save updated state
|
|
141
|
+
spinner.start('Saving updated state');
|
|
142
|
+
stateManager.updateSummary(state);
|
|
143
|
+
fs.writeFileSync(flags.state, JSON.stringify(state, null, 2));
|
|
144
|
+
spinner.succeed(`State saved to ${flags.state}`);
|
|
145
|
+
this.log(chalk.green('\n✓ Pruning complete!'));
|
|
146
|
+
}
|
|
147
|
+
catch (error) {
|
|
148
|
+
spinner.fail('Pruning failed');
|
|
149
|
+
this.error(error.message);
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
}
|
|
153
|
+
//# sourceMappingURL=prune.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"prune.js","sourceRoot":"","sources":["../../src/commands/prune.ts"],"names":[],"mappings":"AAAA;;GAEG;AAEH,OAAO,EAAE,OAAO,EAAE,KAAK,EAAE,MAAM,aAAa,CAAC;AAC7C,OAAO,GAAG,MAAM,KAAK,CAAC;AACtB,OAAO,KAAK,MAAM,OAAO,CAAC;AAC1B,OAAO,KAAK,EAAE,MAAM,IAAI,CAAC;AACzB,OAAO,KAAK,QAAQ,MAAM,UAAU,CAAC;AACrC,OAAO,EAAE,YAAY,EAAE,MAAM,2BAA2B,CAAC;AACzD,OAAO,EAAE,cAAc,EAAE,MAAM,2BAA2B,CAAC;AAC3D,OAAO,EAAE,YAAY,EAAE,MAAM,0BAA0B,CAAC;AACxD,OAAO,KAAK,IAAI,MAAM,MAAM,CAAC;AAC7B,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AAEzC,MAAM,UAAU,GAAG,aAAa,CAAC,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AAClD,MAAM,SAAS,GAAG,IAAI,CAAC,OAAO,CAAC,UAAU,CAAC,CAAC;AAC3C,OAAO,EAAE,YAAY,EAAE,MAAM,4BAA4B,CAAC;AAC1D,OAAO,EAAE,gBAAgB,EAAE,MAAM,8BAA8B,CAAC;AAGhE,MAAM,CAAC,OAAO,OAAO,KAAM,SAAQ,OAAO;aACjC,gBAAW,GAAG,yEAAyE,CAAC;aAExF,aAAQ,GAAG;QAChB,8EAA8E;QAC9E,wFAAwF;QACxF,wFAAwF;QACxF,uFAAuF;KACxF,CAAC;aAEK,UAAK,GAAG;QACb,KAAK,EAAE,KAAK,CAAC,MAAM,CAAC;YAClB,IAAI,EAAE,GAAG;YACT,WAAW,EAAE,kCAAkC;YAC/C,QAAQ,EAAE,IAAI;SACf,CAAC;QACF,MAAM,EAAE,KAAK,CAAC,MAAM,CAAC;YACnB,IAAI,EAAE,GAAG;YACT,WAAW,EAAE,qBAAqB;YAClC,OAAO,EAAE,eAAe;SACzB,CAAC;QACF,MAAM,EAAE,KAAK,CAAC,OAAO,CAAC;YACpB,WAAW,EAAE,0EAA0E;YACvF,OAAO,EAAE,KAAK;SACf,CAAC;QACF,SAAS,EAAE,KAAK,CAAC,OAAO,CAAC;YACvB,WAAW,EAAE,yBAAyB;YACtC,OAAO,EAAE,KAAK;SACf,CAAC;QACF,SAAS,EAAE,KAAK,CAAC,OAAO,CAAC;YACvB,WAAW,EAAE,yBAAyB;YACtC,OAAO,EAAE,KAAK;SACf,CAAC;KACH,CAAC;IAEF,KAAK,CAAC,GAAG;QACP,MAAM,EAAE,KAAK,EAAE,GAAG,MAAM,IAAI,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;QAC1C,MAAM,OAAO,GAAG,GAAG,EAAE,CAAC;QAEtB,IAAI,CAAC;YACH,cAAc;YACd,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;YACvC,MAAM,MAAM,GAAG,MAAM,YAAY,CAAC,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC;YACrD,OAAO,CAAC,OAAO,CAAC,sBAAsB,CAAC,CAAC;YAExC,aAAa;YACb,OAAO,CAAC,KAAK,CAAC,oBAAoB,CAAC,CAAC;YACpC,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,KAAK,CAAC,KAAK,CAAC,EAAE,CAAC;gBAChC,IAAI,CAAC,KAAK,CAAC,yBAAyB,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;YACrD,CAAC;YACD,MAAM,YAAY,GAAG,EAAE,CAAC,YAAY,CAAC,KAAK,CAAC,KAAK,EAAE,OAAO,CAAC,CAAC;YAC3D,MAAM,KAAK,GAA0B,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC;YAC9D,OAAO,CAAC,OAAO,CAAC,iBAAiB,KAAK,CAAC,OAAO,EAAE,MAAM,IAAI,CAAC,UAAU,CAAC,CAAC;YAEvE,2BAA2B;YAC3B,MAAM,YAAY,GAAG,IAAI,YAAY,CAAC,KAAK,CAAC,KAAK,CAAC,CAAC;YACnD,MAAM,UAAU,GAAG,IAAI,CAAC,IAAI,CAAC,SAAS,EAAE,eAAe,CAAC,CAAC;YACzD,MAAM,YAAY,GAAG,IAAI,YAAY,CAAC,MAAM,CAAC,EAAE,EAAE,UAAU,CAAC,CAAC;YAC7D,MAAM,gBAAgB,GAAG,IAAI,gBAAgB,EAAE,CAAC;YAChD,MAAM,YAAY,CAAC,UAAU,EAAE,CAAC;YAChC,MAAM,cAAc,GAAG,IAAI,cAAc,CAAC,MAAM,EAAE,YAAY,EAAE,YAAY,EAAE,gBAAgB,CAAC,CAAC;YAEhG,2CAA2C;YAC3C,MAAM,GAAG,GAAgB;gBACvB,KAAK,EAAE,QAAQ,GAAG,IAAI,CAAC,GAAG,EAAE;gBAC5B,SAAS,EAAE,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE;gBACnC,MAAM,EAAE,aAAa;gBACrB,eAAe,EAAE,CAAC;gBAClB,mBAAmB,EAAE,CAAC;gBACtB,oBAAoB,EAAE,CAAC;gBACvB,gBAAgB,EAAE,CAAC;gBACnB,SAAS,EAAE,KAAK;gBAChB,SAAS,EAAE,MAAM,CAAC,EAAE,CAAC,KAAK;gBAC1B,MAAM,EAAE,MAAM,CAAC,EAAE,CAAC,QAAQ,IAAI,QAAQ;gBACtC,WAAW,EAAE,MAAM,CAAC,EAAE,CAAC,WAAW,IAAI,GAAG;gBACzC,eAAe,EAAE,CAAC;gBAClB,gBAAgB,EAAE,CAAC;gBACnB,iBAAiB,EAAE,CAAC;gBACpB,aAAa,EAAE,CAAC;gBAChB,QAAQ,EAAE,EAAE;gBACZ,MAAM,EAAE,EAAE;gBACV,aAAa,EAAE,EAAE;gBACjB,YAAY,EAAE,EAAE;aACjB,CAAC;YAEF,IAAI,SAAS,GAAG,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC;YACxC,IAAI,SAAS,GAAG,EAAE,OAAO,EAAE,CAAC,EAAE,IAAI,EAAE,CAAC,EAAE,CAAC;YAExC,aAAa;YACb,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,YAAY,EAAE,CAAC;gBACnD,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;gBACvC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,EAAE,UAAU,EAAE,GAAG,cAAc,CAAC,wBAAwB,CAAC,KAAK,CAAC,CAAC;gBAClG,OAAO,CAAC,OAAO,CAAC,oBAAoB,QAAQ,YAAY,UAAU,WAAW,CAAC,CAAC;gBAE/E,IAAI,UAAU,GAAG,CAAC,EAAE,CAAC;oBACnB,SAAS,GAAG,MAAM,cAAc,CAAC,gBAAgB,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;oBAC9D,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,iBAAiB,SAAS,CAAC,OAAO,aAAa,SAAS,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC;gBAC/F,CAAC;qBAAM,CAAC;oBACN,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC,CAAC;gBACtD,CAAC;YACH,CAAC;YAED,aAAa;YACb,IAAI,CAAC,KAAK,CAAC,SAAS,CAAC,IAAI,KAAK,CAAC,MAAM,CAAC,YAAY,EAAE,CAAC;gBACnD,OAAO,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC;gBACvC,MAAM,EAAE,MAAM,EAAE,QAAQ,EAAE,QAAQ,EAAE,UAAU,EAAE,GAAG,cAAc,CAAC,sBAAsB,CAAC,KAAK,CAAC,CAAC;gBAChG,OAAO,CAAC,OAAO,CAAC,oBAAoB,QAAQ,YAAY,UAAU,WAAW,CAAC,CAAC;gBAE/E,IAAI,UAAU,GAAG,CAAC,EAAE,CAAC;oBACnB,SAAS,GAAG,MAAM,cAAc,CAAC,gBAAgB,CAAC,KAAK,EAAE,GAAG,CAAC,CAAC;oBAC9D,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,MAAM,CAAC,iBAAiB,SAAS,CAAC,OAAO,aAAa,SAAS,CAAC,IAAI,OAAO,CAAC,CAAC,CAAC;gBAC/F,CAAC;qBAAM,CAAC;oBACN,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,4BAA4B,CAAC,CAAC,CAAC;gBACtD,CAAC;YACH,CAAC;YAED,UAAU;YACV,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;YACb,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,IAAI,CAAC,kBAAkB,CAAC,CAAC,CAAC;YACzC,IAAI,CAAC,GAAG,CAAC,UAAU,SAAS,CAAC,OAAO,aAAa,SAAS,CAAC,IAAI,OAAO,CAAC,CAAC;YACxE,IAAI,CAAC,GAAG,CAAC,UAAU,SAAS,CAAC,OAAO,aAAa,SAAS,CAAC,IAAI,OAAO,CAAC,CAAC;YAExE,6CAA6C;YAC7C,IAAI,CAAC,KAAK,CAAC,MAAM,IAAI,CAAC,SAAS,CAAC,OAAO,GAAG,CAAC,IAAI,SAAS,CAAC,OAAO,GAAG,CAAC,CAAC,EAAE,CAAC;gBACtE,MAAM,EAAE,GAAG,QAAQ,CAAC,eAAe,CAAC,EAAE,KAAK,EAAE,OAAO,CAAC,KAAK,EAAE,MAAM,EAAE,OAAO,CAAC,MAAM,EAAE,CAAC,CAAC;gBACtF,MAAM,MAAM,GAAG,MAAM,IAAI,OAAO,CAAS,OAAO,CAAC,EAAE;oBACjD,EAAE,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,gCAAgC,CAAC,EAAE,OAAO,CAAC,CAAC;gBACrE,CAAC,CAAC,CAAC;gBACH,EAAE,CAAC,KAAK,EAAE,CAAC;gBAEX,IAAI,MAAM,CAAC,WAAW,EAAE,KAAK,GAAG,EAAE,CAAC;oBACjC,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,oBAAoB,CAAC,CAAC,CAAC;oBAC1C,OAAO;gBACT,CAAC;YACH,CAAC;YAED,qBAAqB;YACrB,OAAO,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;YACtC,YAAY,CAAC,aAAa,CAAC,KAAK,CAAC,CAAC;YAClC,EAAE,CAAC,aAAa,CAAC,KAAK,CAAC,KAAK,EAAE,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;YAC9D,OAAO,CAAC,OAAO,CAAC,kBAAkB,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC;YAEjD,IAAI,CAAC,GAAG,CAAC,KAAK,CAAC,KAAK,CAAC,uBAAuB,CAAC,CAAC,CAAC;QAEjD,CAAC;QAAC,OAAO,KAAK,EAAE,CAAC;YACf,OAAO,CAAC,IAAI,CAAC,gBAAgB,CAAC,CAAC;YAC/B,IAAI,CAAC,KAAK,CAAE,KAAe,CAAC,OAAO,CAAC,CAAC;QACvC,CAAC;IACH,CAAC"}
|