@caracal-lynx/sluice 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +1822 -1822
- package/README.md +12 -21
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -301,8 +301,8 @@ sluice/
|
|
|
301
301
|
│ ├── unit/ ← unit tests (all I/O mocked)
|
|
302
302
|
│ └── integration/ ← real DuckDB :memory: + CSV fixtures
|
|
303
303
|
└── clients/ ← 🙈 gitignored — each client has their own repo
|
|
304
|
-
├──
|
|
305
|
-
└──
|
|
304
|
+
├── acme-corp/ ← Acme Corp pipelines
|
|
305
|
+
└── style-co/ ← Style Co pipelines
|
|
306
306
|
```
|
|
307
307
|
|
|
308
308
|
---
|
|
@@ -313,7 +313,7 @@ Connection strings and credentials live in `.env` (never in YAML files, never in
|
|
|
313
313
|
|
|
314
314
|
```bash
|
|
315
315
|
# .env
|
|
316
|
-
|
|
316
|
+
SOURCE_MSSQL=mssql://user:password@serverlegacy.example.local/LegacyDB
|
|
317
317
|
BC_BASE_URL=https://api.businesscentral.dynamics.com/v2.0
|
|
318
318
|
BC_TENANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
|
319
319
|
BC_CLIENT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
|
|
@@ -326,7 +326,7 @@ Reference them in YAML with `${ENV_VAR}` — resolved at runtime, never stored i
|
|
|
326
326
|
```yaml
|
|
327
327
|
source:
|
|
328
328
|
adapter: mssql
|
|
329
|
-
connection: ${
|
|
329
|
+
connection: ${SOURCE_MSSQL}
|
|
330
330
|
```
|
|
331
331
|
|
|
332
332
|
---
|
|
@@ -342,7 +342,7 @@ Name a bundle of checks in a shared rules file and reference them like built-ins
|
|
|
342
342
|
```yaml
|
|
343
343
|
# shared/rules.yaml
|
|
344
344
|
rules:
|
|
345
|
-
- id:
|
|
345
|
+
- id: style-coStyleNo
|
|
346
346
|
checks:
|
|
347
347
|
- { type: notNull, severity: critical }
|
|
348
348
|
- { type: pattern, value: "^[A-Z]{2}[0-9]{4}$", severity: critical }
|
|
@@ -356,7 +356,7 @@ dq:
|
|
|
356
356
|
rules:
|
|
357
357
|
- field: STYLE_NO
|
|
358
358
|
checks:
|
|
359
|
-
- { type:
|
|
359
|
+
- { type: style-coStyleNo } # expands to the three checks above ✨
|
|
360
360
|
```
|
|
361
361
|
|
|
362
362
|
### Tier 2 — Plugin Files (TypeScript) 🔌
|
|
@@ -432,7 +432,7 @@ Detailed guide: **[PLUGINS.md](./PLUGINS.md)**
|
|
|
432
432
|
|
|
433
433
|
## 🔀 Multi-Source Merge
|
|
434
434
|
|
|
435
|
-
Phase 3 lets a single pipeline extract from **2+ sources** and merge them on a key column before DQ and transform. Useful when the master record for an entity is scattered across systems — master data in SQL Server, pricing enrichment in an Excel sheet, product descriptions in
|
|
435
|
+
Phase 3 lets a single pipeline extract from **2+ sources** and merge them on a key column before DQ and transform. Useful when the master record for an entity is scattered across systems — master data in SQL Server, pricing enrichment in an Excel sheet, product descriptions in a REST API, and so on.
|
|
436
436
|
|
|
437
437
|
### Built-in merge strategies
|
|
438
438
|
|
|
@@ -449,8 +449,8 @@ Custom strategies can be dropped in as `*.merge.ts` plugins or shipped as npm pa
|
|
|
449
449
|
|
|
450
450
|
```yaml
|
|
451
451
|
pipeline:
|
|
452
|
-
name:
|
|
453
|
-
client:
|
|
452
|
+
name: style-co-products-merged
|
|
453
|
+
client: style-co
|
|
454
454
|
version: "1.0"
|
|
455
455
|
entity: Style
|
|
456
456
|
|
|
@@ -458,7 +458,7 @@ sources:
|
|
|
458
458
|
- id: sql-server # staging table: stg_raw_sql-server
|
|
459
459
|
priority: 1 # lower = higher precedence
|
|
460
460
|
adapter: mssql
|
|
461
|
-
connection: ${
|
|
461
|
+
connection: ${SOURCE_2_MSSQL}
|
|
462
462
|
query: "SELECT STYLE_NO, STYLE_DESC, COST_PRICE FROM dbo.Styles WHERE Active = 1"
|
|
463
463
|
|
|
464
464
|
- id: excel
|
|
@@ -478,7 +478,7 @@ merge:
|
|
|
478
478
|
fieldStrategies: # per-field overrides
|
|
479
479
|
- { field: FIBRE_CONTENT, source: excel } # pin to one source
|
|
480
480
|
- { field: COST_PRICE, strategy: priority-override }
|
|
481
|
-
conflictLog: ./output/
|
|
481
|
+
conflictLog: ./output/style-co-products-conflicts.csv # optional CSV of field disagreements
|
|
482
482
|
|
|
483
483
|
dq:
|
|
484
484
|
stopOnCritical: true
|
|
@@ -514,16 +514,7 @@ sluice merge list-strategies # ids + descriptions for all registered stra
|
|
|
514
514
|
sluice merge info coalesce # details for one strategy
|
|
515
515
|
```
|
|
516
516
|
|
|
517
|
-
A full working example lives at [tests/fixtures/
|
|
518
|
-
|
|
519
|
-
---
|
|
520
|
-
|
|
521
|
-
## 🤝 Known Clients
|
|
522
|
-
|
|
523
|
-
| Client | Source | Target | Adapter |
|
|
524
|
-
|--------|--------|--------|---------|
|
|
525
|
-
| Cochran Group (Annan) | MSSQL legacy DB | IFS ERP | `ifs` |
|
|
526
|
-
| Eribé Knitwear | MSSQL / CSV exports | BlueCherry ERP | `bluecherry` |
|
|
517
|
+
A full working example lives at [tests/fixtures/style-co-products-merged.pipeline.yaml](tests/fixtures/style-co-products-merged.pipeline.yaml).
|
|
527
518
|
|
|
528
519
|
---
|
|
529
520
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@caracal-lynx/sluice",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.1",
|
|
4
4
|
"description": "Config-driven ETL toolkit for ERP data migrations",
|
|
5
5
|
"license": "Elastic-2.0",
|
|
6
6
|
"author": "Caracal Lynx Ltd. <michael.scott@caracallynx.com> (https://caracallynx.com)",
|