@caracal-lynx/sluice 0.1.1 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +1822 -1822
- package/LICENCE-FAQ.md +74 -74
- package/LICENSE +92 -92
- package/PLUGINS.md +294 -0
- package/README.md +681 -573
- package/dist/multi-source-runner.js +16 -16
- package/dist/runner.js +10 -10
- package/package.json +98 -92
- package/dist/adapters/source/rest.types.d.ts +0 -15
- package/dist/adapters/source/rest.types.d.ts.map +0 -1
- package/dist/adapters/source/rest.types.js +0 -6
- package/dist/adapters/source/rest.types.js.map +0 -1
- package/dist/merge/strategies/registry.d.ts +0 -8
- package/dist/merge/strategies/registry.d.ts.map +0 -1
- package/dist/merge/strategies/registry.js +0 -19
- package/dist/merge/strategies/registry.js.map +0 -1
package/LICENCE-FAQ.md
CHANGED
|
@@ -1,74 +1,74 @@
|
|
|
1
|
-
# Sluice Licence FAQ
|
|
2
|
-
|
|
3
|
-
Sluice is published under the **Elastic Licence 2.0 (ELv2)**. This page explains in plain English what that means for you. If in doubt, read the [full licence text](LICENSE) or contact us.
|
|
4
|
-
|
|
5
|
-
---
|
|
6
|
-
|
|
7
|
-
## Can I use Sluice to run data migrations for my own business?
|
|
8
|
-
|
|
9
|
-
**Yes, absolutely.** If you are running Sluice internally — your own team, your own data, your own migration project — you are welcome to use it free of charge with no restrictions. That is exactly what it was built for.
|
|
10
|
-
|
|
11
|
-
---
|
|
12
|
-
|
|
13
|
-
## Can I use Sluice as part of an internal IT project or toolchain?
|
|
14
|
-
|
|
15
|
-
**Yes.** Embedding Sluice in internal scripts, automation pipelines, or tooling used within your own organisation is permitted.
|
|
16
|
-
|
|
17
|
-
---
|
|
18
|
-
|
|
19
|
-
## Can I modify Sluice for my own use?
|
|
20
|
-
|
|
21
|
-
**Yes.** You can fork the repository, modify the code, and run your modified version internally. You are not required to publish your changes, though contributions back to the project are always welcome.
|
|
22
|
-
|
|
23
|
-
---
|
|
24
|
-
|
|
25
|
-
## Can I use Sluice to deliver a data migration project for a client?
|
|
26
|
-
|
|
27
|
-
**No.** Using Sluice as the basis of a commercial service delivered to third parties — for example, running Sluice pipelines on behalf of a paying client as part of a consultancy engagement — requires a commercial licence from Caracal Lynx Ltd.
|
|
28
|
-
|
|
29
|
-
This restriction exists because Sluice is the core product of Caracal Lynx's data migration practice. We are happy for businesses to use it for themselves; we are not able to allow other consultancies to compete directly using our tool without an agreement in place.
|
|
30
|
-
|
|
31
|
-
---
|
|
32
|
-
|
|
33
|
-
## I am a consultant. Can I recommend Sluice to my clients and help them set it up?
|
|
34
|
-
|
|
35
|
-
**Yes, with a distinction.** You may recommend Sluice to clients and help them understand it. The restriction applies when Sluice is central to a commercial service *you* are delivering — i.e. you are operating or running pipelines on the client's behalf as a paid engagement. If the client is operating Sluice themselves after your initial help, that is their internal use and is permitted.
|
|
36
|
-
|
|
37
|
-
If you are unsure whether your situation requires a commercial licence, please contact us — we would rather clarify upfront than create problems later.
|
|
38
|
-
|
|
39
|
-
---
|
|
40
|
-
|
|
41
|
-
## Can I resell Sluice or white-label it as my own product?
|
|
42
|
-
|
|
43
|
-
**No.** Reselling, repackaging, or white-labelling Sluice — with or without modifications — is not permitted under ELv2.
|
|
44
|
-
|
|
45
|
-
---
|
|
46
|
-
|
|
47
|
-
## Can I write and publish plugins or adapters for Sluice?
|
|
48
|
-
|
|
49
|
-
**Yes.** Building and publishing `sluice-adapter-*` or `sluice-plugin-*` packages on npm is encouraged. The restriction on commercial use applies to Sluice itself; it does not prevent you from building and selling plugins or adapters as separate products, provided those products do not effectively replicate Sluice's core functionality.
|
|
50
|
-
|
|
51
|
-
---
|
|
52
|
-
|
|
53
|
-
## Can I contribute to the Sluice codebase?
|
|
54
|
-
|
|
55
|
-
**Yes, and thank you.** Pull requests are welcome. By submitting a contribution you agree that your code is licenced under the same ELv2 terms as the rest of the project, and that Caracal Lynx Ltd. retains the right to include it in future releases (including any future commercial releases).
|
|
56
|
-
|
|
57
|
-
---
|
|
58
|
-
|
|
59
|
-
## What if I want to use Sluice commercially?
|
|
60
|
-
|
|
61
|
-
Contact us. We offer commercial licences for consultancies and service providers who want to use Sluice as part of a paid offering. Pricing is by arrangement and depends on the nature and scale of use.
|
|
62
|
-
|
|
63
|
-
📧 **michael.scott@caracallynx.com**
|
|
64
|
-
🌐 **Caracal Lynx Ltd.** — registered in Scotland, SC826823
|
|
65
|
-
|
|
66
|
-
---
|
|
67
|
-
|
|
68
|
-
## Is this "open source"?
|
|
69
|
-
|
|
70
|
-
Technically, no. The Elastic Licence 2.0 is a *source available* licence — the code is fully public, you can read it, fork it, and use it for internal purposes, but it does not meet the OSI definition of "open source" because of the restriction on commercial service use. We have chosen this approach to keep Sluice accessible to businesses while protecting the consultancy practice that funds its development.
|
|
71
|
-
|
|
72
|
-
---
|
|
73
|
-
|
|
74
|
-
*This FAQ is a plain-English guide and does not constitute legal advice. The [LICENSE](LICENSE) file is the authoritative legal text. Last updated: April 2026.*
|
|
1
|
+
# Sluice Licence FAQ
|
|
2
|
+
|
|
3
|
+
Sluice is published under the **Elastic Licence 2.0 (ELv2)**. This page explains in plain English what that means for you. If in doubt, read the [full licence text](LICENSE) or contact us.
|
|
4
|
+
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
## Can I use Sluice to run data migrations for my own business?
|
|
8
|
+
|
|
9
|
+
**Yes, absolutely.** If you are running Sluice internally — your own team, your own data, your own migration project — you are welcome to use it free of charge with no restrictions. That is exactly what it was built for.
|
|
10
|
+
|
|
11
|
+
---
|
|
12
|
+
|
|
13
|
+
## Can I use Sluice as part of an internal IT project or toolchain?
|
|
14
|
+
|
|
15
|
+
**Yes.** Embedding Sluice in internal scripts, automation pipelines, or tooling used within your own organisation is permitted.
|
|
16
|
+
|
|
17
|
+
---
|
|
18
|
+
|
|
19
|
+
## Can I modify Sluice for my own use?
|
|
20
|
+
|
|
21
|
+
**Yes.** You can fork the repository, modify the code, and run your modified version internally. You are not required to publish your changes, though contributions back to the project are always welcome.
|
|
22
|
+
|
|
23
|
+
---
|
|
24
|
+
|
|
25
|
+
## Can I use Sluice to deliver a data migration project for a client?
|
|
26
|
+
|
|
27
|
+
**No.** Using Sluice as the basis of a commercial service delivered to third parties — for example, running Sluice pipelines on behalf of a paying client as part of a consultancy engagement — requires a commercial licence from Caracal Lynx Ltd.
|
|
28
|
+
|
|
29
|
+
This restriction exists because Sluice is the core product of Caracal Lynx's data migration practice. We are happy for businesses to use it for themselves; we are not able to allow other consultancies to compete directly using our tool without an agreement in place.
|
|
30
|
+
|
|
31
|
+
---
|
|
32
|
+
|
|
33
|
+
## I am a consultant. Can I recommend Sluice to my clients and help them set it up?
|
|
34
|
+
|
|
35
|
+
**Yes, with a distinction.** You may recommend Sluice to clients and help them understand it. The restriction applies when Sluice is central to a commercial service *you* are delivering — i.e. you are operating or running pipelines on the client's behalf as a paid engagement. If the client is operating Sluice themselves after your initial help, that is their internal use and is permitted.
|
|
36
|
+
|
|
37
|
+
If you are unsure whether your situation requires a commercial licence, please contact us — we would rather clarify upfront than create problems later.
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Can I resell Sluice or white-label it as my own product?
|
|
42
|
+
|
|
43
|
+
**No.** Reselling, repackaging, or white-labelling Sluice — with or without modifications — is not permitted under ELv2.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Can I write and publish plugins or adapters for Sluice?
|
|
48
|
+
|
|
49
|
+
**Yes.** Building and publishing `sluice-adapter-*` or `sluice-plugin-*` packages on npm is encouraged. The restriction on commercial use applies to Sluice itself; it does not prevent you from building and selling plugins or adapters as separate products, provided those products do not effectively replicate Sluice's core functionality.
|
|
50
|
+
|
|
51
|
+
---
|
|
52
|
+
|
|
53
|
+
## Can I contribute to the Sluice codebase?
|
|
54
|
+
|
|
55
|
+
**Yes, and thank you.** Pull requests are welcome. By submitting a contribution you agree that your code is licenced under the same ELv2 terms as the rest of the project, and that Caracal Lynx Ltd. retains the right to include it in future releases (including any future commercial releases).
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## What if I want to use Sluice commercially?
|
|
60
|
+
|
|
61
|
+
Contact us. We offer commercial licences for consultancies and service providers who want to use Sluice as part of a paid offering. Pricing is by arrangement and depends on the nature and scale of use.
|
|
62
|
+
|
|
63
|
+
📧 **michael.scott@caracallynx.com**
|
|
64
|
+
🌐 **Caracal Lynx Ltd.** — registered in Scotland, SC826823
|
|
65
|
+
|
|
66
|
+
---
|
|
67
|
+
|
|
68
|
+
## Is this "open source"?
|
|
69
|
+
|
|
70
|
+
Technically, no. The Elastic Licence 2.0 is a *source available* licence — the code is fully public, you can read it, fork it, and use it for internal purposes, but it does not meet the OSI definition of "open source" because of the restriction on commercial service use. We have chosen this approach to keep Sluice accessible to businesses while protecting the consultancy practice that funds its development.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
*This FAQ is a plain-English guide and does not constitute legal advice. The [LICENSE](LICENSE) file is the authoritative legal text. Last updated: April 2026.*
|
package/LICENSE
CHANGED
|
@@ -1,92 +1,92 @@
|
|
|
1
|
-
Elastic License 2.0
|
|
2
|
-
|
|
3
|
-
URL: https://www.elastic.co/licensing/elastic-license
|
|
4
|
-
|
|
5
|
-
## Acceptance
|
|
6
|
-
|
|
7
|
-
By using the software, you agree to all of the terms and conditions below.
|
|
8
|
-
|
|
9
|
-
## Copyright License
|
|
10
|
-
|
|
11
|
-
The licensor grants you a non-exclusive, royalty-free, worldwide,
|
|
12
|
-
non-sublicensable, non-transferable license to use, copy, distribute, make
|
|
13
|
-
available, and prepare derivative works of the software, in each case subject
|
|
14
|
-
to the limitations and conditions below.
|
|
15
|
-
|
|
16
|
-
## Limitations
|
|
17
|
-
|
|
18
|
-
You may not provide the software to third parties as a hosted or managed
|
|
19
|
-
service, where the service provides users with access to any substantial set
|
|
20
|
-
of the features or functionality of the software.
|
|
21
|
-
|
|
22
|
-
You may not move, change, disable, or circumvent the license key functionality
|
|
23
|
-
in the software, and you may not remove or obscure any functionality in the
|
|
24
|
-
software that is protected by the license key.
|
|
25
|
-
|
|
26
|
-
You may not alter, remove, or obscure any licensing, copyright, or other
|
|
27
|
-
notices of the licensor in the software. Any use of the licensor's trademarks
|
|
28
|
-
is subject to applicable law.
|
|
29
|
-
|
|
30
|
-
## Patents
|
|
31
|
-
|
|
32
|
-
The licensor grants you a license, under any patent claims the licensor can
|
|
33
|
-
license, or becomes able to license, to make, have made, use, sell, offer for
|
|
34
|
-
sale, import and have imported the software, in each case subject to the
|
|
35
|
-
limitations and conditions in this license. This license does not cover any
|
|
36
|
-
patent claims that you cause to be infringed by modifications or additions to
|
|
37
|
-
the software. If you or your company make any written claim that the software
|
|
38
|
-
infringes or contributes to infringement of any patent, your patent license
|
|
39
|
-
for the software granted under these terms ends immediately. If your company
|
|
40
|
-
makes such a claim, your patent license ends immediately for work on behalf
|
|
41
|
-
of your company.
|
|
42
|
-
|
|
43
|
-
## Notices
|
|
44
|
-
|
|
45
|
-
You must ensure that anyone who gets a copy of any part of the software from
|
|
46
|
-
you also gets a copy of these terms.
|
|
47
|
-
|
|
48
|
-
If you modify the software, you must include in any modified copies of the
|
|
49
|
-
software prominent notices stating that you have modified the software.
|
|
50
|
-
|
|
51
|
-
## No Other Rights
|
|
52
|
-
|
|
53
|
-
These terms do not imply any licenses other than those expressly granted in
|
|
54
|
-
these terms.
|
|
55
|
-
|
|
56
|
-
## Termination
|
|
57
|
-
|
|
58
|
-
If you use the software in violation of these terms, such use is not licensed,
|
|
59
|
-
and your licenses will automatically terminate. If the licensor provides you
|
|
60
|
-
with a notice of your violation, and you cease all violation of this license
|
|
61
|
-
no later than 30 days after you receive that notice, your licenses will be
|
|
62
|
-
reinstated retroactively. However, if you violate these terms after such
|
|
63
|
-
reinstatement, any additional violation of these terms will cause your
|
|
64
|
-
licenses to terminate automatically and permanently.
|
|
65
|
-
|
|
66
|
-
## No Liability
|
|
67
|
-
|
|
68
|
-
As far as the law allows, the software comes as is, without any warranty or
|
|
69
|
-
condition, and the licensor will not be liable to you for any damages arising
|
|
70
|
-
out of these terms or the use or nature of the software, under any kind of
|
|
71
|
-
legal claim.
|
|
72
|
-
|
|
73
|
-
## Definitions
|
|
74
|
-
|
|
75
|
-
The **licensor** is the entity offering these terms, and the **software** is
|
|
76
|
-
the software the licensor makes available under these terms.
|
|
77
|
-
|
|
78
|
-
**You** refers to the individual or entity agreeing to these terms.
|
|
79
|
-
|
|
80
|
-
**Your company** is any legal entity, sole proprietorship, or other kind of
|
|
81
|
-
organization that you work for, plus all organizations that have control over,
|
|
82
|
-
are under the control of, or are under common control with that organization.
|
|
83
|
-
**Control** means ownership of substantially all the assets of an entity, or
|
|
84
|
-
the power to direct its management and policies by vote, contract, or
|
|
85
|
-
otherwise. Control can be direct or indirect.
|
|
86
|
-
|
|
87
|
-
**Your licenses** are all the licenses granted to you for the software under
|
|
88
|
-
these terms.
|
|
89
|
-
|
|
90
|
-
**Use** means anything you do with the software requiring one of your licenses.
|
|
91
|
-
|
|
92
|
-
**Trademark** means trademarks, service marks, and similar rights.
|
|
1
|
+
Elastic License 2.0
|
|
2
|
+
|
|
3
|
+
URL: https://www.elastic.co/licensing/elastic-license
|
|
4
|
+
|
|
5
|
+
## Acceptance
|
|
6
|
+
|
|
7
|
+
By using the software, you agree to all of the terms and conditions below.
|
|
8
|
+
|
|
9
|
+
## Copyright License
|
|
10
|
+
|
|
11
|
+
The licensor grants you a non-exclusive, royalty-free, worldwide,
|
|
12
|
+
non-sublicensable, non-transferable license to use, copy, distribute, make
|
|
13
|
+
available, and prepare derivative works of the software, in each case subject
|
|
14
|
+
to the limitations and conditions below.
|
|
15
|
+
|
|
16
|
+
## Limitations
|
|
17
|
+
|
|
18
|
+
You may not provide the software to third parties as a hosted or managed
|
|
19
|
+
service, where the service provides users with access to any substantial set
|
|
20
|
+
of the features or functionality of the software.
|
|
21
|
+
|
|
22
|
+
You may not move, change, disable, or circumvent the license key functionality
|
|
23
|
+
in the software, and you may not remove or obscure any functionality in the
|
|
24
|
+
software that is protected by the license key.
|
|
25
|
+
|
|
26
|
+
You may not alter, remove, or obscure any licensing, copyright, or other
|
|
27
|
+
notices of the licensor in the software. Any use of the licensor's trademarks
|
|
28
|
+
is subject to applicable law.
|
|
29
|
+
|
|
30
|
+
## Patents
|
|
31
|
+
|
|
32
|
+
The licensor grants you a license, under any patent claims the licensor can
|
|
33
|
+
license, or becomes able to license, to make, have made, use, sell, offer for
|
|
34
|
+
sale, import and have imported the software, in each case subject to the
|
|
35
|
+
limitations and conditions in this license. This license does not cover any
|
|
36
|
+
patent claims that you cause to be infringed by modifications or additions to
|
|
37
|
+
the software. If you or your company make any written claim that the software
|
|
38
|
+
infringes or contributes to infringement of any patent, your patent license
|
|
39
|
+
for the software granted under these terms ends immediately. If your company
|
|
40
|
+
makes such a claim, your patent license ends immediately for work on behalf
|
|
41
|
+
of your company.
|
|
42
|
+
|
|
43
|
+
## Notices
|
|
44
|
+
|
|
45
|
+
You must ensure that anyone who gets a copy of any part of the software from
|
|
46
|
+
you also gets a copy of these terms.
|
|
47
|
+
|
|
48
|
+
If you modify the software, you must include in any modified copies of the
|
|
49
|
+
software prominent notices stating that you have modified the software.
|
|
50
|
+
|
|
51
|
+
## No Other Rights
|
|
52
|
+
|
|
53
|
+
These terms do not imply any licenses other than those expressly granted in
|
|
54
|
+
these terms.
|
|
55
|
+
|
|
56
|
+
## Termination
|
|
57
|
+
|
|
58
|
+
If you use the software in violation of these terms, such use is not licensed,
|
|
59
|
+
and your licenses will automatically terminate. If the licensor provides you
|
|
60
|
+
with a notice of your violation, and you cease all violation of this license
|
|
61
|
+
no later than 30 days after you receive that notice, your licenses will be
|
|
62
|
+
reinstated retroactively. However, if you violate these terms after such
|
|
63
|
+
reinstatement, any additional violation of these terms will cause your
|
|
64
|
+
licenses to terminate automatically and permanently.
|
|
65
|
+
|
|
66
|
+
## No Liability
|
|
67
|
+
|
|
68
|
+
As far as the law allows, the software comes as is, without any warranty or
|
|
69
|
+
condition, and the licensor will not be liable to you for any damages arising
|
|
70
|
+
out of these terms or the use or nature of the software, under any kind of
|
|
71
|
+
legal claim.
|
|
72
|
+
|
|
73
|
+
## Definitions
|
|
74
|
+
|
|
75
|
+
The **licensor** is the entity offering these terms, and the **software** is
|
|
76
|
+
the software the licensor makes available under these terms.
|
|
77
|
+
|
|
78
|
+
**You** refers to the individual or entity agreeing to these terms.
|
|
79
|
+
|
|
80
|
+
**Your company** is any legal entity, sole proprietorship, or other kind of
|
|
81
|
+
organization that you work for, plus all organizations that have control over,
|
|
82
|
+
are under the control of, or are under common control with that organization.
|
|
83
|
+
**Control** means ownership of substantially all the assets of an entity, or
|
|
84
|
+
the power to direct its management and policies by vote, contract, or
|
|
85
|
+
otherwise. Control can be direct or indirect.
|
|
86
|
+
|
|
87
|
+
**Your licenses** are all the licenses granted to you for the software under
|
|
88
|
+
these terms.
|
|
89
|
+
|
|
90
|
+
**Use** means anything you do with the software requiring one of your licenses.
|
|
91
|
+
|
|
92
|
+
**Trademark** means trademarks, service marks, and similar rights.
|
package/PLUGINS.md
ADDED
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
# Sluice — Plugin Author Guide
|
|
2
|
+
|
|
3
|
+
Sluice's pipeline configs are intentionally constrained — the schema is fixed, the field types are enumerated, the DQ check types are an explicit list. That keeps configs readable and reviewable. But every real migration eventually needs *something* the built-ins don't cover: a regex pattern that only makes sense for one client's data, a date format only one ERP uses, a merge strategy with custom precedence rules.
|
|
4
|
+
|
|
5
|
+
Plugins fill that gap without forcing you to fork the engine. Sluice exposes a **three-tier extension model** that scales from "I just want a reusable composite rule for this client" to "I want to publish a paid adapter package on npm."
|
|
6
|
+
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
## The three tiers at a glance
|
|
10
|
+
|
|
11
|
+
| Tier | What it is | Where it lives | Who writes it | Distribution |
|
|
12
|
+
|---|---|---|---|---|
|
|
13
|
+
| **Tier 1 — Composite YAML rules** | A named bundle of built-in DQ checks | YAML files in your project | Anyone — no code | In your repo |
|
|
14
|
+
| **Tier 2 — File-based plugins** | TypeScript module exporting a `RulePlugin`, `TransformPlugin`, or `MergeStrategyPlugin` | `plugins/*.{rule,transform,merge}.ts` in your project | Anyone with TypeScript | In your repo |
|
|
15
|
+
| **Tier 3 — npm package plugins** | A published npm package with a `register()` function | Anywhere installable via `npm install` | Plugin authors | npmjs.com (public or private) |
|
|
16
|
+
|
|
17
|
+
You can mix tiers freely — a single pipeline can pull composite rules (Tier 1), a local dev's file plugin (Tier 2), and an installed npm rule pack (Tier 3) all at once.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Tier 1 — Composite YAML rules
|
|
22
|
+
|
|
23
|
+
A composite rule is a **named bundle of built-in checks** that you reference in pipeline DQ rules by a single ID. Useful when the same combination of checks (e.g., `notNull` + `pattern` + `maxLength` for an internal style number) repeats across many fields and many pipelines.
|
|
24
|
+
|
|
25
|
+
### Library file
|
|
26
|
+
|
|
27
|
+
Composite rules live in a YAML file referenced by `dq.rulesFile`. Convention: `shared/rules.yaml` at the repo root.
|
|
28
|
+
|
|
29
|
+
```yaml
|
|
30
|
+
# shared/rules.yaml
|
|
31
|
+
version: "1.0"
|
|
32
|
+
|
|
33
|
+
rules:
|
|
34
|
+
- id: ukVatNumber
|
|
35
|
+
description: UK VAT registration number — format only (existence check is a separate concern)
|
|
36
|
+
checks:
|
|
37
|
+
- { type: pattern, value: "^GB([0-9]{9}|[0-9]{12}|(GD|HA)[0-9]{3})$", severity: warning }
|
|
38
|
+
|
|
39
|
+
- id: positivePrice
|
|
40
|
+
description: Price must be a non-negative number with sensible upper bound
|
|
41
|
+
checks:
|
|
42
|
+
- { type: notNull, severity: critical }
|
|
43
|
+
- { type: min, value: 0, severity: critical }
|
|
44
|
+
- { type: max, value: 99999.99, severity: warning }
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
### Use in a pipeline
|
|
48
|
+
|
|
49
|
+
```yaml
|
|
50
|
+
# customers.pipeline.yaml
|
|
51
|
+
dq:
|
|
52
|
+
rulesFile: ./shared/rules.yaml # tell ConfigLoader where to find composite rules
|
|
53
|
+
rules:
|
|
54
|
+
- field: VAT_NUMBER
|
|
55
|
+
checks:
|
|
56
|
+
- { type: ukVatNumber } # expands to the pattern check above at load time
|
|
57
|
+
|
|
58
|
+
- field: COST_PRICE
|
|
59
|
+
checks:
|
|
60
|
+
- { type: positivePrice } # expands to all three checks at load time
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
`ConfigLoader` expands composite-rule references into their underlying built-in checks **before** Zod validation runs, so the DQ engine only ever sees standard check types.
|
|
64
|
+
|
|
65
|
+
### When to reach for Tier 1
|
|
66
|
+
|
|
67
|
+
- The same combination of checks repeats across multiple fields or pipelines.
|
|
68
|
+
- The combination doesn't need any custom code — built-in checks suffice.
|
|
69
|
+
- You want non-developers (data analysts, project managers) to be able to read and edit the rules.
|
|
70
|
+
|
|
71
|
+
### Constraints
|
|
72
|
+
|
|
73
|
+
- Composite rule IDs must be valid identifiers (`^[a-zA-Z][a-zA-Z0-9_-]*$`) and must not collide with built-in check type names (`notNull`, `unique`, `pattern`, etc.).
|
|
74
|
+
- Composite rules can only contain built-in checks — they cannot reference other composite rules (no nesting).
|
|
75
|
+
|
|
76
|
+
---
|
|
77
|
+
|
|
78
|
+
## Tier 2 — File-based plugins
|
|
79
|
+
|
|
80
|
+
When you need actual logic — a check that calls a custom regex with side conditions, a transform that does business-specific date parsing, a merge strategy that picks the maximum value across sources — write a TypeScript plugin file.
|
|
81
|
+
|
|
82
|
+
Plugins are auto-discovered from a `plugins/` directory next to your pipeline YAML (or from any directory passed via `--plugins`).
|
|
83
|
+
|
|
84
|
+
### File naming convention
|
|
85
|
+
|
|
86
|
+
| Filename suffix | Plugin type | Exported symbol |
|
|
87
|
+
|---|---|---|
|
|
88
|
+
| `*.rule.ts` (or `.rule.js`) | DQ rule | `export const rule: RulePlugin` |
|
|
89
|
+
| `*.transform.ts` (or `.transform.js`) | Field transform | `export const transform: TransformPlugin` |
|
|
90
|
+
| `*.merge.ts` (or `.merge.js`) | Merge strategy | `export const mergeStrategy: MergeStrategyPlugin` |
|
|
91
|
+
|
|
92
|
+
### Example — DQ rule plugin
|
|
93
|
+
|
|
94
|
+
```typescript
|
|
95
|
+
// plugins/ifs-customer-no.rule.ts
|
|
96
|
+
import type { RulePlugin } from '@caracal-lynx/sluice';
|
|
97
|
+
|
|
98
|
+
export const rule: RulePlugin = {
|
|
99
|
+
id: 'ifsCustomerNo',
|
|
100
|
+
description: 'IFS customer number — three uppercase letters followed by 4–7 digits',
|
|
101
|
+
|
|
102
|
+
validate(value, config, rowIndex, field) {
|
|
103
|
+
if (typeof value !== 'string') return null;
|
|
104
|
+
if (/^[A-Z]{3}[0-9]{4,7}$/.test(value)) return null;
|
|
105
|
+
return {
|
|
106
|
+
field,
|
|
107
|
+
rowIndex,
|
|
108
|
+
value,
|
|
109
|
+
rule: 'ifsCustomerNo',
|
|
110
|
+
severity: config.severity,
|
|
111
|
+
message: config.message ?? `${value} is not a valid IFS customer number`,
|
|
112
|
+
};
|
|
113
|
+
},
|
|
114
|
+
};
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
Use it in a pipeline:
|
|
118
|
+
|
|
119
|
+
```yaml
|
|
120
|
+
dq:
|
|
121
|
+
rules:
|
|
122
|
+
- field: CUSTOMER_NO
|
|
123
|
+
checks:
|
|
124
|
+
- { type: ifsCustomerNo, severity: critical }
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
### Example — transform plugin
|
|
128
|
+
|
|
129
|
+
```typescript
|
|
130
|
+
// plugins/season-from-date.transform.ts
|
|
131
|
+
import type { TransformPlugin } from '@caracal-lynx/sluice';
|
|
132
|
+
|
|
133
|
+
export const transform: TransformPlugin = {
|
|
134
|
+
id: 'seasonFromDate',
|
|
135
|
+
description: 'Derives a fashion season code (SS25, AW25 …) from a YYYY-MM-DD launch date',
|
|
136
|
+
|
|
137
|
+
apply(value, row, config) {
|
|
138
|
+
if (typeof value !== 'string') return null;
|
|
139
|
+
const match = /^(\d{4})-(\d{2})-/.exec(value);
|
|
140
|
+
if (!match) return null;
|
|
141
|
+
const [, year, month] = match;
|
|
142
|
+
const yy = year!.slice(2);
|
|
143
|
+
const mm = parseInt(month!, 10);
|
|
144
|
+
return mm >= 1 && mm <= 6 ? `SS${yy}` : `AW${yy}`;
|
|
145
|
+
},
|
|
146
|
+
};
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
Use it in a pipeline:
|
|
150
|
+
|
|
151
|
+
```yaml
|
|
152
|
+
transform:
|
|
153
|
+
fields:
|
|
154
|
+
- { from: LAUNCH_DATE, to: Season, type: custom, customOp: seasonFromDate }
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
### Example — merge strategy plugin
|
|
158
|
+
|
|
159
|
+
```typescript
|
|
160
|
+
// plugins/max-cost.merge.ts
|
|
161
|
+
import type { MergeStrategyPlugin } from '@caracal-lynx/sluice';
|
|
162
|
+
|
|
163
|
+
export const mergeStrategy: MergeStrategyPlugin = {
|
|
164
|
+
id: 'max-cost',
|
|
165
|
+
description: 'Coalesce by key, picking the highest COST_PRICE across all sources',
|
|
166
|
+
async merge(store, sources, config) {
|
|
167
|
+
// Implementation uses the StagingStore SQL surface — see docs for full example
|
|
168
|
+
// ...
|
|
169
|
+
return { rowsMerged: 0, conflicts: 0, unmatched: 0, tableName: 'stg_merged' };
|
|
170
|
+
},
|
|
171
|
+
};
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
### Loading & discovery
|
|
175
|
+
|
|
176
|
+
By default, the runner scans `{cwd}/plugins/` for files matching the suffixes above. You can pass extra directories with `--plugins`:
|
|
177
|
+
|
|
178
|
+
```bash
|
|
179
|
+
sluice run customers.pipeline.yaml --plugins ./shared/plugins --plugins ./team/plugins
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
All discovered plugins are registered before any pipeline phase runs. Duplicate IDs (across files, directories, or with built-ins) raise a `ConfigError` at startup — fail fast.
|
|
183
|
+
|
|
184
|
+
### Constraints
|
|
185
|
+
|
|
186
|
+
- **Plugins must be pure.** No I/O, no async, no mutation of the input row, no global state. The DQ and transform engines call them in tight loops — side effects break determinism.
|
|
187
|
+
- **Errors must be predictable.** `RulePlugin.validate` returns `null` for "valid"; throw only on unrecoverable bugs. `TransformPlugin.apply` returns the transformed value; throw `TransformError` to fail the row.
|
|
188
|
+
- **Plugin IDs are global.** `ifsCustomerNo` lives in the same namespace as the built-in `notNull`, `unique`, etc. Choose distinctive IDs.
|
|
189
|
+
|
|
190
|
+
---
|
|
191
|
+
|
|
192
|
+
## Tier 3 — npm package plugins
|
|
193
|
+
|
|
194
|
+
When you want to **distribute** a plugin — to other projects in your organisation, to a client engagement, or to the public — package it as an npm module and register it via Sluice's package-discovery mechanism.
|
|
195
|
+
|
|
196
|
+
### Package shape
|
|
197
|
+
|
|
198
|
+
A plugin package exports a `register()` function that registers any combination of rules, transforms, and merge strategies:
|
|
199
|
+
|
|
200
|
+
```typescript
|
|
201
|
+
// @your-org/sluice-rules-uk/src/index.ts
|
|
202
|
+
import type {
|
|
203
|
+
PluginPackage,
|
|
204
|
+
RuleRegistry,
|
|
205
|
+
TransformRegistry,
|
|
206
|
+
MergeStrategyRegistry,
|
|
207
|
+
} from '@caracal-lynx/sluice';
|
|
208
|
+
import { ukVatNumber } from './rules/uk-vat-number.js';
|
|
209
|
+
import { ukPostcode } from './rules/uk-postcode-strict.js';
|
|
210
|
+
import { sortCodeAccount } from './rules/sort-code-account.js';
|
|
211
|
+
|
|
212
|
+
export const plugin: PluginPackage = {
|
|
213
|
+
register(rules: RuleRegistry, transforms: TransformRegistry, options, merges) {
|
|
214
|
+
rules.register(ukVatNumber);
|
|
215
|
+
rules.register(ukPostcode);
|
|
216
|
+
rules.register(sortCodeAccount);
|
|
217
|
+
// transforms.register(...) and merges?.register(...) work the same way
|
|
218
|
+
},
|
|
219
|
+
};
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
`package.json`:
|
|
223
|
+
|
|
224
|
+
```json
|
|
225
|
+
{
|
|
226
|
+
"name": "@your-org/sluice-rules-uk",
|
|
227
|
+
"version": "1.0.0",
|
|
228
|
+
"main": "dist/index.js",
|
|
229
|
+
"types": "dist/index.d.ts",
|
|
230
|
+
"peerDependencies": {
|
|
231
|
+
"@caracal-lynx/sluice": "^0.1.0"
|
|
232
|
+
}
|
|
233
|
+
}
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
### Wiring it into a pipeline project
|
|
237
|
+
|
|
238
|
+
Each Sluice project can declare its npm plugin packages in a top-level `sluice.config.yaml`:
|
|
239
|
+
|
|
240
|
+
```yaml
|
|
241
|
+
# sluice.config.yaml — alongside your pipeline YAMLs
|
|
242
|
+
version: "1.0"
|
|
243
|
+
|
|
244
|
+
plugins:
|
|
245
|
+
- package: '@your-org/sluice-rules-uk'
|
|
246
|
+
options: # passed verbatim to register()
|
|
247
|
+
enableExperimental: false
|
|
248
|
+
- package: '@your-org/sluice-rules-fashion'
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
Then `npm install` the packages and `sluice run` will load them automatically:
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
npm install @your-org/sluice-rules-uk @your-org/sluice-rules-fashion
|
|
255
|
+
sluice run customers.pipeline.yaml
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
### When to reach for Tier 3
|
|
259
|
+
|
|
260
|
+
- You want to share a plugin across multiple projects or teams.
|
|
261
|
+
- You want to publish a commercial plugin (paid adapter, domain rule pack).
|
|
262
|
+
- You want versioning and changelogs separate from your pipeline configs.
|
|
263
|
+
|
|
264
|
+
### Distribution
|
|
265
|
+
|
|
266
|
+
Public plugins go on the public npm registry. Private plugins use a private registry (npmjs.com Pro plan, GitHub Packages, Verdaccio, etc.) — Sluice doesn't care which.
|
|
267
|
+
|
|
268
|
+
If you publish a public plugin, please mention `@caracal-lynx/sluice` as a `peerDependency` rather than a direct dependency, and declare a SemVer range that matches the Sluice public API surface you depend on.
|
|
269
|
+
|
|
270
|
+
---
|
|
271
|
+
|
|
272
|
+
## Plugin contracts (the rules)
|
|
273
|
+
|
|
274
|
+
Regardless of tier, every Sluice plugin must obey these:
|
|
275
|
+
|
|
276
|
+
1. **Pure.** No filesystem, network, database, or environment access. No timers, no `Math.random()` without a seed, no `Date.now()`. Plugins run inside tight loops — side effects break determinism and reproducibility.
|
|
277
|
+
2. **Synchronous.** `RulePlugin.validate`, `TransformPlugin.apply`, and `MergeStrategyPlugin.merge` are called via the engine's synchronous (or `async`-but-deterministic) hot path. (`MergeStrategyPlugin` is `async` because merge strategies query the staging DB; their async-ness is bounded to that.)
|
|
278
|
+
3. **Idempotent and stateless.** Calling a plugin twice with the same input must produce the same output. No instance state, no closures over external mutables.
|
|
279
|
+
4. **Throw `TransformError` / return `RuleViolation`** — don't throw raw strings, don't return `undefined`. The engine catches at the pipeline boundary.
|
|
280
|
+
5. **Don't mutate the row.** Plugins receive the source row by reference for cross-field reads; treat it as read-only.
|
|
281
|
+
|
|
282
|
+
The one exception to Rule 1 is the **enrich phase** (Phase 4) — `EnrichPlugin` is async and may call external APIs. That's a separate plugin interface (`@caracal-lynx/sluice-enrich`) with its own contract; see the enrich-phase docs for details.
|
|
283
|
+
|
|
284
|
+
---
|
|
285
|
+
|
|
286
|
+
## More
|
|
287
|
+
|
|
288
|
+
- The full schema reference for pipeline YAML, including how built-in checks and transforms work, lives in [CLAUDE.md](CLAUDE.md).
|
|
289
|
+
- The runtime types (`RulePlugin`, `TransformPlugin`, `MergeStrategyPlugin`, `PluginPackage`) are exported from the package root: `import type { RulePlugin } from '@caracal-lynx/sluice'`.
|
|
290
|
+
- Working examples of all three tiers ship in this repo's `tests/fixtures/plugins/` and `tests/fixtures/shared-rules.yaml`.
|
|
291
|
+
|
|
292
|
+
Questions, gaps, or contributions to this guide? Open a Discussion or send a PR.
|
|
293
|
+
|
|
294
|
+
— Caracal Lynx Ltd.
|