@tuned-tensor/cli 0.28.0 → 0.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -0
- package/dist/chunk-CT2RXESA.js +1272 -0
- package/dist/chunk-CT2RXESA.js.map +1 -0
- package/dist/chunk-G3KT4AYL.js +1623 -0
- package/dist/chunk-G3KT4AYL.js.map +1 -0
- package/dist/chunk-HKWPQPOJ.js +260 -0
- package/dist/chunk-HKWPQPOJ.js.map +1 -0
- package/dist/chunk-MM75IAY3.js +138 -0
- package/dist/chunk-MM75IAY3.js.map +1 -0
- package/dist/{chunk-7CO5DBCY.js → chunk-UY3SU2XA.js} +4949 -4843
- package/dist/chunk-UY3SU2XA.js.map +1 -0
- package/dist/contracts-4MEIRTIN.js +72 -0
- package/dist/dataset-Y5NEUKI5.js +22 -0
- package/dist/dataset-Y5NEUKI5.js.map +1 -0
- package/dist/index.js +812 -331
- package/dist/index.js.map +1 -1
- package/dist/local-runtime.js +84 -467
- package/dist/local-runtime.js.map +1 -1
- package/dist/model-server-7BNWDXHP.js +18 -0
- package/dist/model-server-7BNWDXHP.js.map +1 -0
- package/dist/{prefetch-LBIOO7ZO.js → prefetch-ZFCK4IBD.js} +3 -2
- package/dist/prefetch-ZFCK4IBD.js.map +1 -0
- package/docs/local-runtime/model-quality.md +103 -0
- package/docs/spec-workflow.md +9 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/LICENSE.md +55 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/README.md +364 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.d.ts +123 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.js +778 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.min.js +3 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/index.min.js.map +7 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/numeric-range.d.ts +15 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/numeric-range.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/numeric-range.js +76 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/numeric-range.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/package.json +3 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/parse-class.d.ts +4 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/parse-class.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/parse-class.js +142 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/commonjs/parse-class.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.d.ts +123 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.js +774 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.min.js +3 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/index.min.js.map +7 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/numeric-range.d.ts +15 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/numeric-range.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/numeric-range.js +72 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/numeric-range.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/package.json +3 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/parse-class.d.ts +4 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/parse-class.d.ts.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/parse-class.js +138 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/dist/esm/parse-class.js.map +1 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/package.json +56 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/src/index.ts +932 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/src/numeric-range.ts +79 -0
- package/node_modules/@isaacs/nonrecursive-glob-match/src/parse-class.ts +158 -0
- package/node_modules/minimatch/README.md +125 -107
- package/node_modules/minimatch/dist/commonjs/index.d.ts +49 -38
- package/node_modules/minimatch/dist/commonjs/index.d.ts.map +1 -1
- package/node_modules/minimatch/dist/commonjs/index.js +782 -359
- package/node_modules/minimatch/dist/commonjs/index.js.map +1 -1
- package/node_modules/minimatch/dist/esm/index.d.ts +49 -38
- package/node_modules/minimatch/dist/esm/index.d.ts.map +1 -1
- package/node_modules/minimatch/dist/esm/index.js +781 -356
- package/node_modules/minimatch/dist/esm/index.js.map +1 -1
- package/node_modules/minimatch/package.json +5 -4
- package/package.json +3 -2
- package/training/adapter/src/train.py +16 -1
- package/dist/chunk-7CO5DBCY.js.map +0 -1
- package/dist/chunk-FSVTMX4L.js +0 -2820
- package/dist/chunk-FSVTMX4L.js.map +0 -1
- package/node_modules/minimatch/dist/commonjs/ast.d.ts +0 -22
- package/node_modules/minimatch/dist/commonjs/ast.d.ts.map +0 -1
- package/node_modules/minimatch/dist/commonjs/ast.js +0 -845
- package/node_modules/minimatch/dist/commonjs/ast.js.map +0 -1
- package/node_modules/minimatch/dist/commonjs/brace-expressions.d.ts +0 -8
- package/node_modules/minimatch/dist/commonjs/brace-expressions.d.ts.map +0 -1
- package/node_modules/minimatch/dist/commonjs/brace-expressions.js +0 -150
- package/node_modules/minimatch/dist/commonjs/brace-expressions.js.map +0 -1
- package/node_modules/minimatch/dist/esm/ast.d.ts +0 -22
- package/node_modules/minimatch/dist/esm/ast.d.ts.map +0 -1
- package/node_modules/minimatch/dist/esm/ast.js +0 -841
- package/node_modules/minimatch/dist/esm/ast.js.map +0 -1
- package/node_modules/minimatch/dist/esm/brace-expressions.d.ts +0 -8
- package/node_modules/minimatch/dist/esm/brace-expressions.d.ts.map +0 -1
- package/node_modules/minimatch/dist/esm/brace-expressions.js +0 -146
- package/node_modules/minimatch/dist/esm/brace-expressions.js.map +0 -1
- /package/dist/{prefetch-LBIOO7ZO.js.map → contracts-4MEIRTIN.js.map} +0 -0
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
import {
|
|
3
|
+
buildFoundationModelServerLaunch,
|
|
4
|
+
buildLocalBaseModelServerLaunch,
|
|
5
|
+
buildLocalModelServerLaunch,
|
|
6
|
+
serveLocalModel,
|
|
7
|
+
verifyLocalModelServerLaunch
|
|
8
|
+
} from "./chunk-HKWPQPOJ.js";
|
|
9
|
+
import "./chunk-CT2RXESA.js";
|
|
10
|
+
import "./chunk-G3KT4AYL.js";
|
|
11
|
+
export {
|
|
12
|
+
buildFoundationModelServerLaunch,
|
|
13
|
+
buildLocalBaseModelServerLaunch,
|
|
14
|
+
buildLocalModelServerLaunch,
|
|
15
|
+
serveLocalModel,
|
|
16
|
+
verifyLocalModelServerLaunch
|
|
17
|
+
};
|
|
18
|
+
//# sourceMappingURL=model-server-7BNWDXHP.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}
|
|
@@ -6,7 +6,8 @@ import {
|
|
|
6
6
|
safeName,
|
|
7
7
|
verifyLocalBaseModel,
|
|
8
8
|
verifyModelPrefetchCacheReport
|
|
9
|
-
} from "./chunk-
|
|
9
|
+
} from "./chunk-CT2RXESA.js";
|
|
10
|
+
import "./chunk-G3KT4AYL.js";
|
|
10
11
|
export {
|
|
11
12
|
buildModelPrefetchPayload,
|
|
12
13
|
buildModelSourcePrefetchPayload,
|
|
@@ -15,4 +16,4 @@ export {
|
|
|
15
16
|
verifyLocalBaseModel,
|
|
16
17
|
verifyModelPrefetchCacheReport
|
|
17
18
|
};
|
|
18
|
-
//# sourceMappingURL=prefetch-
|
|
19
|
+
//# sourceMappingURL=prefetch-ZFCK4IBD.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"sources":[],"sourcesContent":[],"mappings":"","names":[]}
|
|
@@ -0,0 +1,103 @@
|
|
|
1
|
+
# Review and use a customized model through the agent
|
|
2
|
+
|
|
3
|
+
Adapter projects can now inspect data, review completed runs, activate a reviewed
|
|
4
|
+
model, serve it on the current CUDA machine, stop that service, and roll back
|
|
5
|
+
through conversation. Each change uses the existing reviewed-action approval
|
|
6
|
+
flow; inspection needs no approval.
|
|
7
|
+
|
|
8
|
+
## Agree on success before training
|
|
9
|
+
|
|
10
|
+
Keep acceptance criteria in `tunedtensor.json`, alongside the existing scoring
|
|
11
|
+
and general-regression settings. These values are illustrative, not universal
|
|
12
|
+
quality standards:
|
|
13
|
+
|
|
14
|
+
```json
|
|
15
|
+
{
|
|
16
|
+
"evaluation": {
|
|
17
|
+
"scoring": { "mode": "json_fields" },
|
|
18
|
+
"taskQuality": {
|
|
19
|
+
"minScore": 0.85,
|
|
20
|
+
"minPassRate": 0.85,
|
|
21
|
+
"minExamples": 40,
|
|
22
|
+
"minScoreDelta": 0
|
|
23
|
+
}
|
|
24
|
+
}
|
|
25
|
+
}
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Choose the scorer and thresholds for the actual task, using representative,
|
|
29
|
+
disjoint held-out examples. Keep a separate test set when selecting recipes
|
|
30
|
+
on validation results. `minScore` and `minPassRate` range from 0 to 1;
|
|
31
|
+
`minExamples` is the required evaluated count; `minScoreDelta` is the required
|
|
32
|
+
improvement over the baseline and defaults to zero. A pass-rate regression is
|
|
33
|
+
always rejected. Baseline and candidate must use the same prompts/references.
|
|
34
|
+
Training examples and dry-run output cannot establish acceptance.
|
|
35
|
+
|
|
36
|
+
The report records the policy used by that run. Agent activation and serving
|
|
37
|
+
require that policy, passing task and general-regression checks, unchanged
|
|
38
|
+
current criteria, and verified artifacts. Editing the spec cannot attach new
|
|
39
|
+
criteria to an old report. CLI activation of older reports without a policy
|
|
40
|
+
remains supported, but now also rejects task-score/pass-rate regressions.
|
|
41
|
+
Explicit CLI serving by model ID remains a diagnostic path without this quality
|
|
42
|
+
gate; integrity verification alone is not a quality assessment.
|
|
43
|
+
|
|
44
|
+
These policy and agent lifecycle features currently apply to adapter projects.
|
|
45
|
+
Foundation and decision engines retain their existing engine-specific workflows.
|
|
46
|
+
|
|
47
|
+
## Conversation workflow
|
|
48
|
+
|
|
49
|
+
1. “Inspect training.jsonl and validation.jsonl for duplicate prompts, conflicting
|
|
50
|
+
answers and overlap.” `assess_local_dataset` scans supported single-turn chat
|
|
51
|
+
JSONL, including pinned cached split sources. It reports exact normalized
|
|
52
|
+
duplicates, length distributions and bounded answer counts. This does not
|
|
53
|
+
establish semantic accuracy, license suitability, PII safety or business coverage.
|
|
54
|
+
2. Agree on the task criteria, update the spec through its reviewed edit flow,
|
|
55
|
+
and preview/run training using the existing pipeline tools.
|
|
56
|
+
3. “Review my latest run. Activate it if it meets our agreed criteria.”
|
|
57
|
+
`list_local_runs` discovers actual run IDs and `inspect_local_run` returns
|
|
58
|
+
verified scores, the recorded evaluation split, gates and a bounded set of failures. The activation proposal
|
|
59
|
+
seals the selected run, report, manifest, project configuration and current
|
|
60
|
+
active selection. Approval rechecks these; changes require a new proposal.
|
|
61
|
+
4. “Serve this reviewed model on port 8000.” Approval starts loading; the agent
|
|
62
|
+
checks `get_local_service` for health and the exact `/v1/models` identity.
|
|
63
|
+
Starting a process is not reported as a ready service. The evaluated system
|
|
64
|
+
instruction is restored from the saved run, not taken from a later spec edit.
|
|
65
|
+
5. “Stop this service” releases its process group. “Roll back the active model”
|
|
66
|
+
restores the previous verified selection, or clears it if none existed.
|
|
67
|
+
|
|
68
|
+
Activation changes the configured store's active pointer. If projects explicitly
|
|
69
|
+
share a store, undoing this project's activation may restore the exact previous
|
|
70
|
+
model from another project; its identity and evidence are verified again.
|
|
71
|
+
Unrelated projects cannot be selected through the report or activation tools.
|
|
72
|
+
|
|
73
|
+
Activation changes the active pointer. Starting/stopping a service is a separate
|
|
74
|
+
operation; rollback does not replace a running server. The service binds to
|
|
75
|
+
`127.0.0.1` only, needs Linux/CUDA and uses the pinned serving runtime. Its preview
|
|
76
|
+
shows a 2,048-token context, 256-token output default, one concurrent request and
|
|
77
|
+
GPU memory utilization 0.3. These conservative defaults are not hardware-optimal
|
|
78
|
+
benchmarks; use explicit CLI serving settings for larger contexts or deployment.
|
|
79
|
+
|
|
80
|
+
The agent owns one server per CLI process. It does not adopt an existing server
|
|
81
|
+
or arbitrary PID. Normal process exit stops its server; this is an interactive
|
|
82
|
+
local session, not a persistent deployment manager. The run's `agent-serving.log`
|
|
83
|
+
contains startup errors. Port conflicts and identity mismatch never establish
|
|
84
|
+
readiness. Approval results persist so subsequent turns can use the actual
|
|
85
|
+
model and server IDs without asking the user to paste terminal output.
|
|
86
|
+
|
|
87
|
+
An abrupt OS kill (for example SIGKILL) cannot run application cleanup; this
|
|
88
|
+
session manager is not a crash-recovery supervisor.
|
|
89
|
+
|
|
90
|
+
## Repeating a training experiment
|
|
91
|
+
|
|
92
|
+
Adapter specs accept `hyperparameters.seed` (an integer from 0 through
|
|
93
|
+
4,294,967,295; default 42). The runner seeds before loading the model and
|
|
94
|
+
initializing LoRA, then uses the same seed for Trainer and data sampling.
|
|
95
|
+
`training-metrics.json` records the effective seed, trainable parameter count
|
|
96
|
+
and peak CUDA allocated/reserved bytes. These are PyTorch allocator peaks,
|
|
97
|
+
not total device or Spark unified-memory usage. A seed improves repeatability;
|
|
98
|
+
it does not guarantee identical GPU kernels across hardware or library versions.
|
|
99
|
+
Keep model revision, data hashes, runtime versions and evaluation policy fixed
|
|
100
|
+
when comparing backends. Never treat training speed alone as model quality.
|
|
101
|
+
|
|
102
|
+
For a measured Spark comparison with Unsloth, including startup cost, memory,
|
|
103
|
+
quality and repeatability limits, see the [backend review](https://github.com/tunedtensor/tuned-tensor-cli/blob/main/docs/reviews/2026-11-10-unsloth-comparison.md).
|
package/docs/spec-workflow.md
CHANGED
|
@@ -233,3 +233,12 @@ Splitting a multi-gigabyte master does not remove that training-loader limit.
|
|
|
233
233
|
Do not treat a successful split preview as proof that an arbitrarily large split
|
|
234
234
|
can train; streaming training ingestion or bounded dataset selection is separate
|
|
235
235
|
work. Use `tt validate` on the saved spec to validate the actual training inputs.
|
|
236
|
+
|
|
237
|
+
## Task acceptance and conversational model use
|
|
238
|
+
|
|
239
|
+
Adapter specs can set `evaluation.taskQuality` before training. The report records
|
|
240
|
+
the criteria; agent activation and serving require passing task and general
|
|
241
|
+
regression checks, an unchanged policy, and verified artifacts. The agent can
|
|
242
|
+
inspect local datasets and completed runs directly, then prepare activation,
|
|
243
|
+
serving, stopping and rollback. See [model quality](local-runtime/model-quality.md)
|
|
244
|
+
for policy fields, lifecycle behavior and limits.
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Blue Oak Model License
|
|
2
|
+
|
|
3
|
+
Version 1.0.0
|
|
4
|
+
|
|
5
|
+
## Purpose
|
|
6
|
+
|
|
7
|
+
This license gives everyone as much permission to work with
|
|
8
|
+
this software as possible, while protecting contributors
|
|
9
|
+
from liability.
|
|
10
|
+
|
|
11
|
+
## Acceptance
|
|
12
|
+
|
|
13
|
+
In order to receive this license, you must agree to its
|
|
14
|
+
rules. The rules of this license are both obligations
|
|
15
|
+
under that agreement and conditions to your license.
|
|
16
|
+
You must not do anything with this software that triggers
|
|
17
|
+
a rule that you cannot or will not follow.
|
|
18
|
+
|
|
19
|
+
## Copyright
|
|
20
|
+
|
|
21
|
+
Each contributor licenses you to do everything with this
|
|
22
|
+
software that would otherwise infringe that contributor's
|
|
23
|
+
copyright in it.
|
|
24
|
+
|
|
25
|
+
## Notices
|
|
26
|
+
|
|
27
|
+
You must ensure that everyone who gets a copy of
|
|
28
|
+
any part of this software from you, with or without
|
|
29
|
+
changes, also gets the text of this license or a link to
|
|
30
|
+
<https://blueoakcouncil.org/license/1.0.0>.
|
|
31
|
+
|
|
32
|
+
## Excuse
|
|
33
|
+
|
|
34
|
+
If anyone notifies you in writing that you have not
|
|
35
|
+
complied with [Notices](#notices), you can keep your
|
|
36
|
+
license by taking all practical steps to comply within 30
|
|
37
|
+
days after the notice. If you do not do so, your license
|
|
38
|
+
ends immediately.
|
|
39
|
+
|
|
40
|
+
## Patent
|
|
41
|
+
|
|
42
|
+
Each contributor licenses you to do everything with this
|
|
43
|
+
software that would otherwise infringe any patent claims
|
|
44
|
+
they can license or become able to license.
|
|
45
|
+
|
|
46
|
+
## Reliability
|
|
47
|
+
|
|
48
|
+
No contributor can revoke this license.
|
|
49
|
+
|
|
50
|
+
## No Liability
|
|
51
|
+
|
|
52
|
+
***As far as the law allows, this software comes as is,
|
|
53
|
+
without any warranty or condition, and no contributor
|
|
54
|
+
will be liable to anyone for any damages related to this
|
|
55
|
+
software or this license, under any kind of legal claim.***
|
|
@@ -0,0 +1,364 @@
|
|
|
1
|
+
# @isaacs/nonrecursive-glob-match
|
|
2
|
+
|
|
3
|
+
This is a glob matcher that can be applied to a single path
|
|
4
|
+
portion (ie, not `/`), which completes in polynomial time because
|
|
5
|
+
it doesn't compile to Regular Expressions or any other
|
|
6
|
+
backtrack-dependent algorithm.
|
|
7
|
+
|
|
8
|
+
## USAGE
|
|
9
|
+
|
|
10
|
+
```ts
|
|
11
|
+
import { Matcher } from '@isaacs/nonrecursive-glob-match'
|
|
12
|
+
|
|
13
|
+
const m = new Matcher('a*a!(y|z)')
|
|
14
|
+
|
|
15
|
+
console.log(m.test('abcay')) // false
|
|
16
|
+
console.log(m.test('abcax')) // true
|
|
17
|
+
console.log(m.test('aa')) // true
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
The pattern and the tested string must be a single path portion.
|
|
21
|
+
If either one contains a `/` character, the matcher throws an
|
|
22
|
+
error.
|
|
23
|
+
|
|
24
|
+
If you already split a path on `/`, use `m.testUnchecked(s)`. It
|
|
25
|
+
gives the same result as `m.test(s)`, but it does not check `s`
|
|
26
|
+
for a `/` character. This can make it faster, because some
|
|
27
|
+
patterns, such as `*.js`, then do not need to look at each
|
|
28
|
+
character. If `s` has a `/`, the result is not defined.
|
|
29
|
+
|
|
30
|
+
## Why
|
|
31
|
+
|
|
32
|
+
Patterns like this frustrate most glob matchers: `a*a*a*a*a*a*a`.
|
|
33
|
+
They can be used to take down servers if applied to untrusted
|
|
34
|
+
input. Fixing these vulns is like playing whack-a-mole with the
|
|
35
|
+
whole internet.
|
|
36
|
+
|
|
37
|
+
If compiled to a regular expression like
|
|
38
|
+
`/^a[^/]*a[^/]*a[^/]*a[^/]*a[^/]*a[^/]*a$/`, and then applied to
|
|
39
|
+
a path portion like `aaaaaaaaaaaaaaaaaaab`, the engine must check
|
|
40
|
+
every possible path through the pattern, constantly rewinding
|
|
41
|
+
back to the first character each time and trying a different
|
|
42
|
+
number of `a` characters matched against the `[^/]*` to see if it
|
|
43
|
+
matches. Because nothing matches, it must test every possible
|
|
44
|
+
path before ultimately giving up.
|
|
45
|
+
|
|
46
|
+
In contrast, the algorithm presented here is based on the
|
|
47
|
+
Thompson NFA paper from 1968, which tests every possible path
|
|
48
|
+
through the string at the same time, so that it only has to walk
|
|
49
|
+
the string one time. This algorithm is also implemented in the
|
|
50
|
+
RegExp engines used by `egrep` and `awk`, which is why those
|
|
51
|
+
tools are so fast, even when matching complicated RegExps against
|
|
52
|
+
extremely large strings.
|
|
53
|
+
|
|
54
|
+
The `Matcher` objects have a `test` method which takes a string,
|
|
55
|
+
just like a `RegExp` object, and can be a drop-in replacement for
|
|
56
|
+
the regular expressions used in minimatch and other matchers.
|
|
57
|
+
However, they are _not_ a complete drop-in replacement for a
|
|
58
|
+
RegExp, so be careful when replacing one with the other. (For
|
|
59
|
+
example, they do not capture, do not track match indexes, no
|
|
60
|
+
fancy RegExp features, etc.)
|
|
61
|
+
|
|
62
|
+
## Patterns Supported
|
|
63
|
+
|
|
64
|
+
Here are the patterns supported, and their effective state
|
|
65
|
+
machines drawn in lovely ASCII diagrams.
|
|
66
|
+
|
|
67
|
+
### `*` 0 or more of any character, not starting with `.`
|
|
68
|
+
|
|
69
|
+
```
|
|
70
|
+
--+-------------------+->
|
|
71
|
+
v ^
|
|
72
|
+
+<------------------+
|
|
73
|
+
v ^
|
|
74
|
+
+-->any character-->+
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
### `?`
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
-->any character-->
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
### `[<set>]`
|
|
84
|
+
|
|
85
|
+
```
|
|
86
|
+
-->character in set-->
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
This covers named posix sets like `[[:alpha:]]` as well as
|
|
90
|
+
character classes like `[a-b]`. (In this case, a regular
|
|
91
|
+
expression actually is compiled, but it's not subject to ReDoS,
|
|
92
|
+
becuase it's just a `/[a-z]/` style pattern tested against one
|
|
93
|
+
character at a time.)
|
|
94
|
+
|
|
95
|
+
### `@(e1|e2|...)`
|
|
96
|
+
|
|
97
|
+
"One" extglob.
|
|
98
|
+
|
|
99
|
+
```
|
|
100
|
+
--+--> e1 -->+-->
|
|
101
|
+
v ^
|
|
102
|
+
+--> e2 -->+
|
|
103
|
+
v ^
|
|
104
|
+
+--> e3 -->+
|
|
105
|
+
v ... ^
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
### `?(e1|e2|...)`
|
|
109
|
+
|
|
110
|
+
"Zero or one" extglob.
|
|
111
|
+
|
|
112
|
+
```
|
|
113
|
+
--+--------->+-->
|
|
114
|
+
v ^
|
|
115
|
+
+--> e1 -->+
|
|
116
|
+
v ^
|
|
117
|
+
+--> e2 -->+
|
|
118
|
+
v ^
|
|
119
|
+
+--> e3 -->+
|
|
120
|
+
v ... ^
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
### `*(e1|e2|...)`
|
|
124
|
+
|
|
125
|
+
"Zero or more" extglob.
|
|
126
|
+
|
|
127
|
+
```
|
|
128
|
+
--+--------->+-->
|
|
129
|
+
v ^
|
|
130
|
+
+<---------+
|
|
131
|
+
v ^
|
|
132
|
+
+--> e1 -->+
|
|
133
|
+
v ^
|
|
134
|
+
+--> e2 -->+
|
|
135
|
+
v ^
|
|
136
|
+
+--> e3 -->+
|
|
137
|
+
v ... ^
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
### `+(e1|e2|...)`
|
|
141
|
+
|
|
142
|
+
"One or more" extglob.
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
+<---------+
|
|
146
|
+
v ^
|
|
147
|
+
--+--> e1 -->+-->
|
|
148
|
+
v ^
|
|
149
|
+
+--> e2 -->+
|
|
150
|
+
v ^
|
|
151
|
+
+--> e3 -->+
|
|
152
|
+
v ... ^
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
### `!(e1|e2|...)`
|
|
156
|
+
|
|
157
|
+
"Negative" extglob.
|
|
158
|
+
|
|
159
|
+
The challenge with negation is that it opens the door to a whole
|
|
160
|
+
heap of edge cases. Especially, _any_ expression can be nested
|
|
161
|
+
inside the negation, and it needs to match anything at all, as
|
|
162
|
+
long as it _doesn't_ match one of the supplied patterns. This is
|
|
163
|
+
where it gets complicated, and past attempts have run into
|
|
164
|
+
problems.
|
|
165
|
+
|
|
166
|
+
We handle this by starting a "Run" that technically does recurse,
|
|
167
|
+
and thus can push the performance out from `O(n * m)` to
|
|
168
|
+
`O(n * m * r)` where `n` is the string length, `m` is the pattern
|
|
169
|
+
length, and `r` is the number of negative extglob portions it
|
|
170
|
+
contains. In practice, this is limited by the pattern length, and
|
|
171
|
+
still doesn't allow ReDOS-style explosive recursive backtracking,
|
|
172
|
+
so this is fine.
|
|
173
|
+
|
|
174
|
+
## Unicode Normalization
|
|
175
|
+
|
|
176
|
+
A char such as `é` can be one code point (U+00E9), or `e`
|
|
177
|
+
followed by a combining accent (U+0301). Some file systems, such
|
|
178
|
+
as on macOS, can give file names in the second form. These two
|
|
179
|
+
forms do not match by default.
|
|
180
|
+
|
|
181
|
+
Set the `normalize` option to normalize the pattern and each
|
|
182
|
+
tested string to the same form. Use `'NFC'`, `'NFD'`, `'NFKC'`,
|
|
183
|
+
or `'NFKD'`. `true` means `'NFC'`.
|
|
184
|
+
|
|
185
|
+
```ts
|
|
186
|
+
const m = new Matcher('café', { normalize: true })
|
|
187
|
+
m.test('café') // true
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
- Use NFC if you are not sure. With NFD and NFKD, `é` is two
|
|
191
|
+
code points. Then `?` does not match `é`, and `[é]` can not
|
|
192
|
+
match it.
|
|
193
|
+
- NFKC and NFKD also fold compatibility chars. For example, `fi`
|
|
194
|
+
matches `fi`, and full-width `A` matches `A`.
|
|
195
|
+
- Syntax is parsed before normalization. So, a char that
|
|
196
|
+
normalizes to a syntax char stays a literal. For example, with
|
|
197
|
+
NFKC, full-width `*` matches only `*` or `*`, and it is not a
|
|
198
|
+
wildcard.
|
|
199
|
+
- Each run of literal chars normalizes as one string. Each member
|
|
200
|
+
of a `[...]` class normalizes on its own, and stays as it is if
|
|
201
|
+
its normal form is more than one code point.
|
|
202
|
+
- A combining mark after a wildcard is its own literal, and does
|
|
203
|
+
not compose with the char that the wildcard matches. So, with
|
|
204
|
+
NFC, `?́` does not match `é`. With NFD, it does match,
|
|
205
|
+
because `é` is also two code points in the string. Patterns
|
|
206
|
+
match code points, not grapheme clusters.
|
|
207
|
+
- With `nocase`, normalization comes first, then case folding.
|
|
208
|
+
- The `/` check uses the strings as given. So, full-width `/`
|
|
209
|
+
does not cause an error.
|
|
210
|
+
|
|
211
|
+
## Numeric Ranges
|
|
212
|
+
|
|
213
|
+
Bash brace expansion makes `{1..1000}` into 1000 words. A glob
|
|
214
|
+
library that expands braces before matching, such as
|
|
215
|
+
[minimatch](http://npm.im/minimatch), can use the `rangeMark`
|
|
216
|
+
option to match a range in one pattern instead.
|
|
217
|
+
|
|
218
|
+
Set `rangeMark` to a char that is not otherwise in the pattern.
|
|
219
|
+
Then `<mark><lo>..<hi><mark>` matches the same strings as the
|
|
220
|
+
brace expansion `{<lo>..<hi>}`, where `lo` and `hi` are integers.
|
|
221
|
+
If either end has a leading zero, each number has leading zeros to
|
|
222
|
+
make it as long as the longer end, counting a `-` sign.
|
|
223
|
+
|
|
224
|
+
```ts
|
|
225
|
+
const M = '\uE000'
|
|
226
|
+
const m = new Matcher(`file-${M}001..500${M}.txt`, {
|
|
227
|
+
rangeMark: M,
|
|
228
|
+
})
|
|
229
|
+
m.test('file-042.txt') // true
|
|
230
|
+
m.test('file-42.txt') // false
|
|
231
|
+
m.test('file-501.txt') // false
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
The range is matched with a few digit classes, so a large range
|
|
235
|
+
costs about the same as a small one.
|
|
236
|
+
|
|
237
|
+
- A `\` before the mark does not escape it.
|
|
238
|
+
- Inside a `[...]` class, the mark is a char of the class.
|
|
239
|
+
- A mark without a valid range after it, or without a closing
|
|
240
|
+
mark, is a literal char.
|
|
241
|
+
|
|
242
|
+
## Differences from Bash
|
|
243
|
+
|
|
244
|
+
Bash 5 is the reference for this module, with two known
|
|
245
|
+
deviations.
|
|
246
|
+
|
|
247
|
+
### Dotfiles
|
|
248
|
+
|
|
249
|
+
If `dot` is not set, a string that starts with `.` only matches
|
|
250
|
+
if a literal `.` in the pattern matches that first character.
|
|
251
|
+
Extglobs before that `.` can match the empty string, and the
|
|
252
|
+
pattern can still match. For example, `!(.a|js)@(.*)` matches
|
|
253
|
+
`.a`, `.a.js`, and `.js`, because `!(.a|js)` matches the empty
|
|
254
|
+
string, and `@(.*)` matches the rest. Bash does not match these
|
|
255
|
+
dotfiles. See [this bug
|
|
256
|
+
report](https://lists.gnu.org/archive/html/bug-bash/2026-03/msg00034.html).
|
|
257
|
+
|
|
258
|
+
A `*` can never match at the start of a dotfile, even if it
|
|
259
|
+
matches the empty string. So, `*.js` does not match `.js`, as in
|
|
260
|
+
Bash.
|
|
261
|
+
|
|
262
|
+
### Case-insensitive matching
|
|
263
|
+
|
|
264
|
+
Bash compares chars with the case functions of the C library for
|
|
265
|
+
the current locale. So, the results can change with the locale
|
|
266
|
+
and the platform.
|
|
267
|
+
|
|
268
|
+
With `nocase`, this module uses Unicode simple case folding, one
|
|
269
|
+
code point at a time. This is the same folding that a JavaScript
|
|
270
|
+
RegExp uses with the `iu` flags. It does not change with the
|
|
271
|
+
locale, and literals, `[...]` classes, and ranges all use it.
|
|
272
|
+
|
|
273
|
+
- `Σ`, `σ`, and final sigma `ς` all match each other.
|
|
274
|
+
- The micro sign `µ` matches `μ` and `Μ`. The Kelvin sign `K`
|
|
275
|
+
matches `k`. The Angstrom sign `Å` matches `å`.
|
|
276
|
+
- `ẞ` matches `ß`, but `ß` does not match `ss`. A fold never
|
|
277
|
+
changes the number of chars, so ligatures such as `fi` do not
|
|
278
|
+
match `fi`.
|
|
279
|
+
- There are no Turkish rules. `I` matches `i`, but dotted `İ`
|
|
280
|
+
and dotless `ı` only match themselves.
|
|
281
|
+
|
|
282
|
+
## Algorithm
|
|
283
|
+
|
|
284
|
+
A glob pattern is parsed and compiled into a series of attached
|
|
285
|
+
state machines.
|
|
286
|
+
|
|
287
|
+
For example, the glob patttern `*(a|[c-d]|+(x|y|z))z` would
|
|
288
|
+
compile to a state machine something like the following flow
|
|
289
|
+
diagram:
|
|
290
|
+
|
|
291
|
+
```
|
|
292
|
+
-->+----------------------->+-->8 LITERAL(z)-->9 end
|
|
293
|
+
v ^
|
|
294
|
+
+<-----------------------6
|
|
295
|
+
v ^
|
|
296
|
+
0-->LITERAL(a)---------->|
|
|
297
|
+
v |
|
|
298
|
+
1-->CLASS(c-d)---------->|
|
|
299
|
+
v |
|
|
300
|
+
2-->+<---------------+-->+
|
|
301
|
+
v ^
|
|
302
|
+
3-->LITERAL(x)-->7
|
|
303
|
+
v |
|
|
304
|
+
4-->LITERAL(y)-->|
|
|
305
|
+
v |
|
|
306
|
+
5-->LITERAL(z)-->+
|
|
307
|
+
```
|
|
308
|
+
|
|
309
|
+
Then, for each character in the string, we walk through the
|
|
310
|
+
resulting graph until we find a test node.
|
|
311
|
+
|
|
312
|
+
These test nodes are all executed on the current character, and
|
|
313
|
+
for any tests pass, their output nodes are added to the walk, and
|
|
314
|
+
we proceed to walk until hitting a test node again.
|
|
315
|
+
|
|
316
|
+
Because all routes eventually get to the exit, which only passes
|
|
317
|
+
when the string is at its last character, we are guaranteed to
|
|
318
|
+
eventually either fail (and have no more states to process), or
|
|
319
|
+
exit with a match.
|
|
320
|
+
|
|
321
|
+
At each branching point, effectively we increase the memory usage
|
|
322
|
+
by a tiny bit to track which nodes are being processed, but
|
|
323
|
+
importantly, we _don't_ have to try all the way to the exit and
|
|
324
|
+
then backtrack recursively, storing multiple states of the string
|
|
325
|
+
in an exponential explosion.
|
|
326
|
+
|
|
327
|
+
A pattern like `a*a*a*a*a*a*a*a*a*a*a`, which could be naively
|
|
328
|
+
converted into a Regular Expression
|
|
329
|
+
`^a.*a.*a.*a.*a.*a.*a.*a.*a.*a.*a$`, would take over 33 seconds
|
|
330
|
+
to match the string `'a'.repeat(50) + 'b'` using standard
|
|
331
|
+
JavaScript regular expressions, because it must attempt every
|
|
332
|
+
possible grouping of characters, backtracking after each failure.
|
|
333
|
+
|
|
334
|
+
By contrast, this implementation can solve this match for either
|
|
335
|
+
a pass or failure in roughly the same amount of time, which
|
|
336
|
+
is bound by `C * N`, where `C` is the number of states in the
|
|
337
|
+
machine, and `N` is the length of the string, finishing both
|
|
338
|
+
failure and success cases in polynomial time.
|
|
339
|
+
|
|
340
|
+
Even on the first rather naive implementation in JavaScript, with
|
|
341
|
+
no optimizations for repeated `*` or other common tweaks, this
|
|
342
|
+
finishes in 300-400 µs for the pathological case described, and
|
|
343
|
+
as the numbers increase, this time rises roughly linearly.
|
|
344
|
+
|
|
345
|
+
That's somewhat worse than RegExp's performance in the _success_
|
|
346
|
+
case, which is around 40µs, but as it's around 100,000x faster in
|
|
347
|
+
the failure case, this is already quite promising. And, future
|
|
348
|
+
iterations of this can likely bring that performance closer to
|
|
349
|
+
the native RegExp implementation.
|
|
350
|
+
|
|
351
|
+
## Acknowledgement
|
|
352
|
+
|
|
353
|
+
The algorithm and ideas in this implementation are inspired,
|
|
354
|
+
borrowed, and at times outright stolen from [Russ
|
|
355
|
+
Cox](https://research.swtch.com)'s explanation of Ken Thompson's
|
|
356
|
+
work in [Regular Expression Matching Can Be Simple And
|
|
357
|
+
Fast](https://swtch.com/~rsc/regexp/regexp1.html). I am both
|
|
358
|
+
impressed by and grateful for their work.
|
|
359
|
+
|
|
360
|
+
The code here in this library was written entirely by me, [Isaac
|
|
361
|
+
Z. Schlueter](https://izs.me), and so you should not blame either
|
|
362
|
+
of them (or anyone else) for any problems it might have. You
|
|
363
|
+
should also not blame me. It's open source, after all, no
|
|
364
|
+
warrantees express or implied, etc.
|