@sema-agent/primitives 7.67.2 → 7.68.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +39 -21
- package/dist/index.d.ts +2 -1
- package/dist/index.js +2 -1
- package/dist/redaction-marks.d.ts +49 -0
- package/dist/redaction-marks.js +40 -0
- package/dist/secret-scan.d.ts +69 -52
- package/dist/secret-scan.js +210 -98
- package/dist/secret-vocabulary.d.ts +39 -5
- package/dist/secret-vocabulary.js +28 -21
- package/package.json +2 -2
package/README.md
CHANGED
|
@@ -96,8 +96,9 @@ form and token head an environment value is read by), the credential words a nam
|
|
|
96
96
|
credentials but are none, and `names`: the names that mark a credential in an assignment (`places: ["assignment"]`) or as a
|
|
97
97
|
URL query or fragment parameter (`"url-param"`).
|
|
98
98
|
Every layer that looks for a secret reads this table, so a shape added to it is found everywhere at once. The table holds
|
|
99
|
-
no layer's policy: which form is read, how a text is read first
|
|
100
|
-
|
|
99
|
+
no layer's policy: which form is read, how a text is read first and how a URL's username is treated are the caller's,
|
|
100
|
+
passed in whole on every call. Nothing here has a default. Each value family row names the mark its hit is written as
|
|
101
|
+
(`mark`, below).
|
|
101
102
|
|
|
102
103
|
```ts
|
|
103
104
|
interface SecretScanPolicy {
|
|
@@ -107,32 +108,49 @@ interface SecretScanPolicy {
|
|
|
107
108
|
urlUserinfo: "keep-username" | "whole"; // a URL userinfo: keep the user and withhold the password, or withhold all of it
|
|
108
109
|
onUnresolvedReading: "secret" | "ignore"; // what a reading that cannot read a text to its end answers
|
|
109
110
|
}
|
|
110
|
-
interface SecretNotation {
|
|
111
|
-
value: string; // the mark a hit is written as
|
|
112
|
-
byKind?: Record<string, string>; // ...or, for a row of this kind, this mark
|
|
113
|
-
byFamily?: Record<string, string>; // ...or, for this row, this mark
|
|
114
|
-
overStrip: string; // the whole text, when no reading can afford it
|
|
115
|
-
}
|
|
116
111
|
```
|
|
117
112
|
|
|
118
113
|
| Function | Answers |
|
|
119
114
|
| --- | --- |
|
|
120
115
|
| `scanSecrets(text, policy)` | `{ outcome: "spans", spans }` — every hit as `{ start, end, family, kind }` in `text`'s own coordinates, sorted (empty = none found) — or `{ outcome: "whole", reason }` with `reason` one of `scan-budget` (more than 64 places where format characters stood), `reading-unresolved` (the percent layers ran out, under `onUnresolvedReading: "secret"`), `mapping-failed`; a hit in a derived text is reported over the written units it was read from |
|
|
121
116
|
| `isSecretShaped(text, policy)` | `scanSecrets` found a span or answered the whole text |
|
|
122
|
-
| `userinfoHolds(userinfo)` | a URL userinfo (no trailing `@`) holds a user or a password — an empty one and the bare separator `:` hold neither (the reading the `keep-username` rendering withholds by) |
|
|
123
|
-
| `renderSecrets(text, policy,
|
|
117
|
+
| `userinfoHolds(userinfo)` | a URL userinfo (no trailing `@`) holds a user or a password — an empty one and the bare separator `:` hold neither (the reading the `keep-username` rendering withholds by; a mark is one token, its `:` splits nothing) |
|
|
118
|
+
| `renderSecrets(text, policy, { passes?, report? }?)` | `text` with every hit replaced by its row's mark (the fragment grade's by that mark's `:partial` member; a text no reading can afford, whole, by `«redacted:oversized»`); `passes` are the caller's own rules, run after the table's in the same pipeline, each `marker` a member of `REDACTION_MARKS`; `report.findings` receives one finding per hit (`report.preexistingMarkers` is the caller's to set) |
|
|
119
|
+
| `renderWithheld(text, spans)` | `text` with each `{ start, end, mark }` span withheld by the same law `renderSecrets` writes its own hits by — for a layer that finds its regions itself |
|
|
124
120
|
| `isSensitiveRecordKey(key)` / `isSensitiveHeaderName(name)` | a record member / an HTTP header name names a credential |
|
|
125
121
|
| `envNameView()` | the readings of environment variable names (exact names, credential words, a namespace's names `anySegmentUnder(namespace)`) |
|
|
126
122
|
| `envValueView()` | the readings of environment values, off the rows: `forms` (each row's env value form, in table order), `tokenHead` (each row's token head, one anchored alternation), `flagWords` |
|
|
127
123
|
| `secretReadingViews(text, readings)` | the texts the readings derive — the written text first, each once — each with `toOriginalSpan(start, end)` (the whole written units a span was read from), and `unresolved` (the percent layers ran out while one more would still decode) |
|
|
128
124
|
|
|
129
|
-
A policy
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
a
|
|
125
|
+
A policy that is not whole (a missing or unknown member), a caller's pass whose `marker` is no member of the marks table,
|
|
126
|
+
or whose string `replace` writes anything but that member and `$n` references, throws a `TypeError`; so does a
|
|
127
|
+
`renderWithheld` span that is not a range of the text or whose mark is no member. The policy is read on every call; what
|
|
128
|
+
it reads as picks a compilation kept across calls. `scanSecrets` answers where its forms match, a mark the text
|
|
129
|
+
holds included (`password=«redacted:key»` is a span); `renderSecrets` leaves a hit whose withheld part is exactly a mark
|
|
130
|
+
the text already holds (below). `URL_OF_ANY_SCHEME` is the URL grammar (a global regex: read it with
|
|
131
|
+
`replace` / `matchAll`), for a layer that withholds whole URLs. `CREDENTIAL_MIN_CHARS` is the fewest characters a string
|
|
132
|
+
can have and still be read as a credential.
|
|
133
|
+
|
|
134
|
+
### Redaction marks
|
|
135
|
+
|
|
136
|
+
`REDACTION_MARKS` is the one table of what stands in a text where a layer withheld part of it — fifteen members, frozen:
|
|
137
|
+
|
|
138
|
+
| Name | Member | Written for |
|
|
139
|
+
| --- | --- | --- |
|
|
140
|
+
| `bare` | `«redacted»` | a value whose place in the text already says what it was: a credential name's value, an assignment, a header value, a scheme word's credential (`Authorization: «redacted»`), the password after a kept username (`https://alice:«redacted»@host`), a URL query or fragment value |
|
|
141
|
+
| `key` | `«redacted:key»` | a credential read by its shape alone (`sk-…`, `AIza…`, a JWT, …) |
|
|
142
|
+
| `gh-token` / `aws-key` / `slack-token` / `private-key` | `«redacted:gh-token»` … | a credential read by its shape alone, of that vendor or kind |
|
|
143
|
+
| `<kind>:partial` | `«redacted:key:partial»` … | the same five kinds, found by the `fragment` grade (a cut residue) |
|
|
144
|
+
| `userinfo` | `«redacted:userinfo»` | a URL userinfo withheld whole |
|
|
145
|
+
| `url` / `path` | `«redacted:url»` / `«redacted:path»` | a whole URL / an absolute path, withheld by a layer that hides where things are |
|
|
146
|
+
| `oversized` | `«redacted:oversized»` | a stretch withheld because it could not be read |
|
|
147
|
+
|
|
148
|
+
`isRedactionMark(s)`, `redactionMarkerSpans(s)` and `eachRedactionMarkerSpan(s)` (lazy) are the one reader: they
|
|
149
|
+
recognize the members exactly — `«redacted:evil»` and square-bracket spellings (`[redacted]`, `[redacted-url]`) are
|
|
150
|
+
ordinary text. A mark is one token to the renderer: a hit whose withheld part is a mark the text already holds is no hit
|
|
151
|
+
(`password=«redacted:key»` stays as written), a hit reaching into or inside a mark takes the whole mark, and hits that
|
|
152
|
+
overlap or touch write one mark — the widest hit's (of two over one region, the later pass's). Rendering a rendered text
|
|
153
|
+
again changes nothing.
|
|
136
154
|
|
|
137
155
|
Every count in the table follows one character or one character class and is closed — "`n` or more" is written as `n`
|
|
138
156
|
of them and then a plain loop, and a group's count is written out — so a reading keeps no regex backtrack state per
|
|
@@ -148,8 +166,7 @@ exported table changes only that caller's reading.
|
|
|
148
166
|
import { renderSecrets, scanSecrets } from "@sema-agent/primitives";
|
|
149
167
|
|
|
150
168
|
const policy = Object.freeze({ grade: "wide", readings: Object.freeze([Object.freeze({ kind: "raw" })]), urlUserinfo: "keep-username", onUnresolvedReading: "secret" });
|
|
151
|
-
|
|
152
|
-
renderSecrets("clone https://me:pa55word@git.example/r", policy, notation); // "clone https://me:[hidden]@git.example/r"
|
|
169
|
+
renderSecrets("clone https://me:pa55word@git.example/r", policy); // "clone https://me:«redacted»@git.example/r"
|
|
153
170
|
scanSecrets("clone https://me:pa55word@git.example/r", policy); // { outcome: "spans", spans: [{ start: 17, end: 25, family: "url-userinfo", kind: "url-credentials" }] }
|
|
154
171
|
```
|
|
155
172
|
|
|
@@ -164,5 +181,6 @@ removed or re-kinded export is named as breaking in the `@sema-agent/core` relea
|
|
|
164
181
|
| --- | --- |
|
|
165
182
|
| Credential reference | `readCredentialRef` `CREDENTIAL_REF_KINDS` `CREDENTIAL_REF_TRUST` `CREDENTIAL_REF_TEXT_MAX` `CREDENTIAL_REF_MAP_MAX` `CREDENTIAL_REF_PROBLEM_REASONS` |
|
|
166
183
|
| Types | `CredentialRef` `CredentialRefKind` `CredentialRefRequestAdmission` `CredentialRefProblem` `CredentialRefProblemReason` `CredentialRefReading` |
|
|
167
|
-
| Secret-shape detection | `CREDENTIAL_VOCABULARY` `CREDENTIAL_MIN_CHARS` `URL_OF_ANY_SCHEME` `SECRET_READING_KINDS` `scanSecrets` `isSecretShaped` `renderSecrets` `isSensitiveRecordKey` `isSensitiveHeaderName` `envNameView` `envValueView` `secretReadingViews` `splitCamelCase` |
|
|
168
|
-
|
|
|
184
|
+
| Secret-shape detection | `CREDENTIAL_VOCABULARY` `CREDENTIAL_MIN_CHARS` `URL_OF_ANY_SCHEME` `SECRET_READING_KINDS` `scanSecrets` `isSecretShaped` `renderSecrets` `renderWithheld` `isSensitiveRecordKey` `isSensitiveHeaderName` `envNameView` `envValueView` `secretReadingViews` `splitCamelCase` |
|
|
185
|
+
| Redaction marks | `REDACTION_MARKS` `isRedactionMark` `redactionMarkerSpans` `eachRedactionMarkerSpan` |
|
|
186
|
+
| Types | `CredentialVocabulary` `ValueFamilyRow` `CredentialWordRow` `CredentialWordPosition` `CredentialNameRow` `CredentialNamePlace` `EnvNameView` `EnvValueView` `RecordKeyView` `RedactionConfidence` `RedactionSource` `SecretGrade` `SecretReading` `SecretReadingKind` `SecretReadingView` `SecretReadingViews` `SecretScanPolicy` `SecretScan` `SecretSpan` `RenderOptions` `RedactionPass` `RedactionFinding` `RedactionReport` `SummarizableFinding` `WithheldSpan` `RedactionMark` `RedactionMarkName` |
|
package/dist/index.d.ts
CHANGED
|
@@ -2,4 +2,5 @@
|
|
|
2
2
|
export { CREDENTIAL_REF_KINDS, CREDENTIAL_REF_TRUST, CREDENTIAL_REF_TEXT_MAX, CREDENTIAL_REF_MAP_MAX, CREDENTIAL_REF_PROBLEM_REASONS, readCredentialRef, type CredentialRef, type CredentialRefKind, type CredentialRefRequestAdmission, type CredentialRefProblem, type CredentialRefProblemReason, type CredentialRefReading, } from "./credential-ref.js";
|
|
3
3
|
export { CREDENTIAL_VOCABULARY, CREDENTIAL_MIN_CHARS, URL_OF_ANY_SCHEME, SCHEME_FOLLOWER_WORDS, envNameView, envValueView, splitCamelCase, isSensitiveRecordKey, isSensitiveHeaderName, type CredentialVocabulary, type ValueFamilyRow, type CredentialFamilyId, type CredentialWordRow, type CredentialWordPosition, type CredentialNameRow, type CredentialNamePlace, type CredentialNameStrength, type EnvNameView, type EnvValueView, type RecordKeyView, type RedactionConfidence, type RedactionSource, } from "./secret-vocabulary.js";
|
|
4
4
|
export { SECRET_READING_KINDS, secretReadingViews, type SecretReading, type SecretReadingKind, type SecretReadingView, type SecretReadingViews, } from "./secret-readings.js";
|
|
5
|
-
export {
|
|
5
|
+
export { REDACTION_MARKS, isRedactionMark, eachRedactionMarkerSpan, redactionMarkerSpans, type RedactionMark, type RedactionMarkName, } from "./redaction-marks.js";
|
|
6
|
+
export { scanSecrets, isSecretShaped, renderSecrets, renderWithheld, userinfoHolds, type SecretGrade, type SecretScanPolicy, type SecretScan, type SecretSpan, type RenderOptions, type RedactionPass, type RedactionFinding, type RedactionReport, type SummarizableFinding, type WithheldSpan, } from "./secret-scan.js";
|
package/dist/index.js
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
1
|
export { CREDENTIAL_REF_KINDS, CREDENTIAL_REF_TRUST, CREDENTIAL_REF_TEXT_MAX, CREDENTIAL_REF_MAP_MAX, CREDENTIAL_REF_PROBLEM_REASONS, readCredentialRef, } from "./credential-ref.js";
|
|
2
2
|
export { CREDENTIAL_VOCABULARY, CREDENTIAL_MIN_CHARS, URL_OF_ANY_SCHEME, SCHEME_FOLLOWER_WORDS, envNameView, envValueView, splitCamelCase, isSensitiveRecordKey, isSensitiveHeaderName, } from "./secret-vocabulary.js";
|
|
3
3
|
export { SECRET_READING_KINDS, secretReadingViews, } from "./secret-readings.js";
|
|
4
|
-
export {
|
|
4
|
+
export { REDACTION_MARKS, isRedactionMark, eachRedactionMarkerSpan, redactionMarkerSpans, } from "./redaction-marks.js";
|
|
5
|
+
export { scanSecrets, isSecretShaped, renderSecrets, renderWithheld, userinfoHolds, } from "./secret-scan.js";
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* The redaction marks — the ONE table of what stands in a text where a layer withheld part of it, and the ONE reader of that
|
|
3
|
+
* table. Every layer that writes a mark (the engine's renderer, a host's own redactor, a display) takes a member of
|
|
4
|
+
* {@link REDACTION_MARKS}; every layer that must tell "a layer withheld this" from the text's own words reads
|
|
5
|
+
* {@link eachRedactionMarkerSpan} / {@link redactionMarkerSpans} / {@link isRedactionMark}. A mark spelled anywhere else is a
|
|
6
|
+
* second source that drifts the first time this table changes.
|
|
7
|
+
*
|
|
8
|
+
* One rule picks the member a layer writes: where the text around the mark already says what was withheld (a credential
|
|
9
|
+
* name's value, an assignment, a header value, a scheme word's credential, the password after a kept username, a URL query or
|
|
10
|
+
* fragment value, a sent credential a remote echoed), the mark is the bare `«redacted»`; where it does not, the mark names the
|
|
11
|
+
* kind — a credential read by its shape (`key`, or the vendor's own word), a URL userinfo withheld whole, a whole URL, an
|
|
12
|
+
* absolute path, or a stretch withheld because it could not be read (`oversized`). A credential residue found by the fragment
|
|
13
|
+
* grade adds `:partial`. Who withheld the text is not in the mark: a host that needs it reads the report's findings.
|
|
14
|
+
*
|
|
15
|
+
* The reader recognizes the members, exactly — not the shape they share: `«redacted:evil»` and the retired square-bracket
|
|
16
|
+
* spellings (`[redacted]`, `[redacted-url]` …) are ordinary text. Zero imports: every layer and package above reads it.
|
|
17
|
+
*/
|
|
18
|
+
/** The marks, by name: `bare`, a kind, or a kind and `:partial`. The value is the member as written; the name is the
|
|
19
|
+
* member's words after `redacted` (the bare mark has none). Frozen: a write by a reader would change what every writer
|
|
20
|
+
* writes. */
|
|
21
|
+
export declare const REDACTION_MARKS: Readonly<{
|
|
22
|
+
readonly bare: "«redacted»";
|
|
23
|
+
readonly key: "«redacted:key»";
|
|
24
|
+
readonly "key:partial": "«redacted:key:partial»";
|
|
25
|
+
readonly "gh-token": "«redacted:gh-token»";
|
|
26
|
+
readonly "gh-token:partial": "«redacted:gh-token:partial»";
|
|
27
|
+
readonly "aws-key": "«redacted:aws-key»";
|
|
28
|
+
readonly "aws-key:partial": "«redacted:aws-key:partial»";
|
|
29
|
+
readonly "slack-token": "«redacted:slack-token»";
|
|
30
|
+
readonly "slack-token:partial": "«redacted:slack-token:partial»";
|
|
31
|
+
readonly "private-key": "«redacted:private-key»";
|
|
32
|
+
readonly "private-key:partial": "«redacted:private-key:partial»";
|
|
33
|
+
readonly userinfo: "«redacted:userinfo»";
|
|
34
|
+
readonly url: "«redacted:url»";
|
|
35
|
+
readonly path: "«redacted:path»";
|
|
36
|
+
readonly oversized: "«redacted:oversized»";
|
|
37
|
+
}>;
|
|
38
|
+
/** A mark's name in {@link REDACTION_MARKS}. */
|
|
39
|
+
export type RedactionMarkName = keyof typeof REDACTION_MARKS;
|
|
40
|
+
/** A member of {@link REDACTION_MARKS}: the only strings a layer writes where it withheld text. */
|
|
41
|
+
export type RedactionMark = (typeof REDACTION_MARKS)[RedactionMarkName];
|
|
42
|
+
/** Is `s`, whole, a member of {@link REDACTION_MARKS}? */
|
|
43
|
+
export declare function isRedactionMark(s: string): s is RedactionMark;
|
|
44
|
+
/** Where the marks sit in `s`, one [start, end) span at a time, leftmost first. Lazy: a reader that stops early (a display cut
|
|
45
|
+
* stops at its bound) pays for the marks it looked at, not for every mark an input of a million of them holds. Members never
|
|
46
|
+
* overlap: each opens with the only `«` and closes with the only `»` it holds. */
|
|
47
|
+
export declare function eachRedactionMarkerSpan(s: string): Generator<[number, number], void, undefined>;
|
|
48
|
+
/** Where the marks sit in `s`, all of them, as [start, end) spans, leftmost first. */
|
|
49
|
+
export declare function redactionMarkerSpans(s: string): Array<[number, number]>;
|
|
@@ -0,0 +1,40 @@
|
|
|
1
|
+
export const REDACTION_MARKS = Object.freeze({
|
|
2
|
+
bare: "«redacted»",
|
|
3
|
+
key: "«redacted:key»",
|
|
4
|
+
"key:partial": "«redacted:key:partial»",
|
|
5
|
+
"gh-token": "«redacted:gh-token»",
|
|
6
|
+
"gh-token:partial": "«redacted:gh-token:partial»",
|
|
7
|
+
"aws-key": "«redacted:aws-key»",
|
|
8
|
+
"aws-key:partial": "«redacted:aws-key:partial»",
|
|
9
|
+
"slack-token": "«redacted:slack-token»",
|
|
10
|
+
"slack-token:partial": "«redacted:slack-token:partial»",
|
|
11
|
+
"private-key": "«redacted:private-key»",
|
|
12
|
+
"private-key:partial": "«redacted:private-key:partial»",
|
|
13
|
+
userinfo: "«redacted:userinfo»",
|
|
14
|
+
url: "«redacted:url»",
|
|
15
|
+
path: "«redacted:path»",
|
|
16
|
+
oversized: "«redacted:oversized»",
|
|
17
|
+
});
|
|
18
|
+
const MEMBER_SHAPE = /^«redacted(?::[a-z-]{1,16})?(?::[a-z-]{1,16})?»$/;
|
|
19
|
+
const MEMBERS = new Set(Object.values(REDACTION_MARKS));
|
|
20
|
+
for (const member of MEMBERS) {
|
|
21
|
+
if (!MEMBER_SHAPE.test(member))
|
|
22
|
+
throw new Error(`redaction marks: ${member} is not of the marks' shape`);
|
|
23
|
+
for (const other of MEMBERS)
|
|
24
|
+
if (other !== member && other.startsWith(member))
|
|
25
|
+
throw new Error(`redaction marks: ${member} is the head of ${other}`);
|
|
26
|
+
}
|
|
27
|
+
const escapeLiteral = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
28
|
+
const MEMBER_RE = new RegExp([...MEMBERS].map(escapeLiteral).join("|"), "g");
|
|
29
|
+
export function isRedactionMark(s) {
|
|
30
|
+
return MEMBERS.has(s);
|
|
31
|
+
}
|
|
32
|
+
export function* eachRedactionMarkerSpan(s) {
|
|
33
|
+
if (!s.includes("«redacted"))
|
|
34
|
+
return;
|
|
35
|
+
for (const m of s.matchAll(MEMBER_RE))
|
|
36
|
+
yield [m.index, m.index + m[0].length];
|
|
37
|
+
}
|
|
38
|
+
export function redactionMarkerSpans(s) {
|
|
39
|
+
return [...eachRedactionMarkerSpan(s)];
|
|
40
|
+
}
|
package/dist/secret-scan.d.ts
CHANGED
|
@@ -1,32 +1,35 @@
|
|
|
1
1
|
/**
|
|
2
2
|
* The matcher and the renderer of the credential vocabulary (secret-vocabulary.ts), each reading the caller's policy
|
|
3
|
-
* whole — the grade of form, the readings a text is read through, the URL userinfo rendering
|
|
4
|
-
* caller's notation (which marker a hit is written as):
|
|
3
|
+
* whole — the grade of form, the readings a text is read through, the URL userinfo rendering:
|
|
5
4
|
* - {@link scanSecrets} answers WHERE the vocabulary sees a credential (original-text spans, or "the whole text" when a
|
|
6
5
|
* reading cannot be afforded), and {@link isSecretShaped} whether it sees one at all;
|
|
7
|
-
* - {@link renderSecrets} answers what the text reads as with every credential withheld,
|
|
8
|
-
* that also withholds what is not a credential (a layer's own
|
|
9
|
-
* same pipeline, through the same readings.
|
|
10
|
-
* No layer's policy
|
|
6
|
+
* - {@link renderSecrets} answers what the text reads as with every credential withheld, each hit written as its row's
|
|
7
|
+
* member of the marks table (redaction-marks.ts). A caller that also withholds what is not a credential (a layer's own
|
|
8
|
+
* URL or path rule) appends its passes, which run in the same pipeline, through the same readings.
|
|
9
|
+
* No layer's policy lives here: every one is an argument, every argument is required. No mark is spelled here: every one
|
|
10
|
+
* is a member of the marks table.
|
|
11
11
|
*
|
|
12
12
|
* The renderer is one SEQUENTIAL multi-pass engine: each pass scans the text the earlier passes left, so a hit that runs
|
|
13
|
-
* into an earlier marker reads
|
|
14
|
-
*
|
|
15
|
-
*
|
|
16
|
-
*
|
|
17
|
-
*
|
|
18
|
-
*
|
|
13
|
+
* into an earlier marker reads that marker. A row whose wide form captures a label keeps it and replaces only the value
|
|
14
|
+
* (`GITHUB_TOKEN=ghp_xxx` keeps `GITHUB_TOKEN=`), and a second run over already-rendered text is a no-op. A mark is
|
|
15
|
+
* atomic: a hit whose withheld part IS a mark is no hit (an earlier layer's mark stands as that layer wrote it), and a hit
|
|
16
|
+
* that reaches into a mark takes the whole mark with it. Hits that touch or overlap write one mark, whichever pass made each
|
|
17
|
+
* ({@link renderWithheld} is that one law, exported). Every pass declares what it detects (kind), how sure it is
|
|
18
|
+
* (confidence) and the marker it writes; a report collector, when given, receives one finding per hit in the ORIGINAL
|
|
19
|
+
* input's coordinates, mapped back through the earlier passes' edits (best-effort where redactions overlap — a span landing
|
|
20
|
+
* inside an earlier replacement anchors to that edit's start).
|
|
19
21
|
*/
|
|
20
|
-
import { type
|
|
22
|
+
import { type CredentialFamilyId, type RedactionConfidence, type RedactionSource, type ValueFamilyRow } from "./secret-vocabulary.js";
|
|
21
23
|
import { type ReadingSet, type SecretReading } from "./secret-readings.js";
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
*
|
|
24
|
+
import { type RedactionMark } from "./redaction-marks.js";
|
|
25
|
+
/** One redaction hit: what was detected, where (ORIGINAL-input coordinates), how confidently, the mark written in its
|
|
26
|
+
* place, which rule table minted it, and — present when a vocabulary row minted it — the value family (the row's stable
|
|
27
|
+
* id; a host-leak hit such as a URL or a path names none). */
|
|
25
28
|
export interface RedactionFinding {
|
|
26
29
|
kind: string;
|
|
27
30
|
span: readonly [number, number];
|
|
28
31
|
confidence: RedactionConfidence;
|
|
29
|
-
marker:
|
|
32
|
+
marker: RedactionMark;
|
|
30
33
|
source: RedactionSource;
|
|
31
34
|
family?: string;
|
|
32
35
|
}
|
|
@@ -37,10 +40,10 @@ export interface SummarizableFinding {
|
|
|
37
40
|
confidence: RedactionConfidence;
|
|
38
41
|
}
|
|
39
42
|
/** The optional collector the renderer accepts. `findings` receives one entry per hit. `preexistingMarkers` counts the
|
|
40
|
-
* marks ALREADY present in the input before any pass ran — a nonzero value means the text
|
|
41
|
-
*
|
|
42
|
-
*
|
|
43
|
-
*
|
|
43
|
+
* marks ALREADY present in the input before any pass ran (the marks table's reader) — a nonzero value means the text
|
|
44
|
+
* arrived pre-redacted or carries marks another layer wrote, either way "a mark is present" is not proof the pipeline
|
|
45
|
+
* ran. The caller sets it; the renderer never writes it. Caller-owned: pass `{ findings: [] }` and read it back after
|
|
46
|
+
* the call. */
|
|
44
47
|
export interface RedactionReport {
|
|
45
48
|
findings: RedactionFinding[];
|
|
46
49
|
preexistingMarkers?: number;
|
|
@@ -50,17 +53,37 @@ export interface RedactionReport {
|
|
|
50
53
|
export interface RedactionPass {
|
|
51
54
|
kind: string;
|
|
52
55
|
confidence: RedactionConfidence;
|
|
53
|
-
|
|
56
|
+
/** The member of the marks table this pass's replacement writes. A replacer function may write another member per hit
|
|
57
|
+
* (the vocabulary's userinfo row writes the bare mark where it keeps a username; its vendor group writes each hit's row
|
|
58
|
+
* mark); a finding records the member its hit wrote. */
|
|
59
|
+
marker: RedactionMark;
|
|
54
60
|
/** The rule table that owns this pass's rule — copied verbatim onto every finding it mints (ANG-11), whoever
|
|
55
61
|
* runs the pass. */
|
|
56
62
|
source: RedactionSource;
|
|
57
63
|
re: RegExp;
|
|
58
|
-
/** A literal replacement (only `$1` substitution is used by these passes) or a replacer fn
|
|
64
|
+
/** A literal replacement (only `$1` substitution is used by these passes) or a replacer fn; either writes its member of
|
|
65
|
+
* the marks table in place of what it withholds. A caller's literal replacement is checked when the renderer is called:
|
|
66
|
+
* its text outside the group references is `marker`, or the call is refused (a `TypeError`). */
|
|
59
67
|
replace: string | ((match: string, ...groups: string[]) => string);
|
|
60
68
|
/** The value family a hit belongs to: one id, or — for a pass joining several rows — the id of the row that
|
|
61
69
|
* matches at the hit's offset in the pass input. Absent = the pass is no vocabulary row. */
|
|
62
70
|
family?: string | ((input: string, at: number) => string);
|
|
63
71
|
}
|
|
72
|
+
/** One region of a text a layer withholds, and the member of the marks table written in its place. */
|
|
73
|
+
export interface WithheldSpan {
|
|
74
|
+
readonly start: number;
|
|
75
|
+
readonly end: number;
|
|
76
|
+
readonly mark: RedactionMark;
|
|
77
|
+
}
|
|
78
|
+
/**
|
|
79
|
+
* `text` with every span withheld, by the one law the renderer writes its own hits by — for a layer that finds its regions
|
|
80
|
+
* itself (the engine's eraser of the credentials a request carried): a span that is, exactly, a mark `text` already holds
|
|
81
|
+
* is no span (the mark stands as written); one that reaches into a mark takes the whole mark; spans that overlap or touch
|
|
82
|
+
* are one region, written as the mark of its widest span (of two over the same region, the later one in `spans`). `text`
|
|
83
|
+
* itself when nothing is withheld. Throws a `TypeError` on a span that is not a range of `text` or whose mark is no member
|
|
84
|
+
* of the marks table.
|
|
85
|
+
*/
|
|
86
|
+
export declare function renderWithheld(text: string, spans: readonly WithheldSpan[]): string;
|
|
64
87
|
/** The `format-free` reading of a text that holds a format character: the text with every `\p{Cf}` removed (the view),
|
|
65
88
|
* where the format characters stood (`cuts`: the view offset of every view character that has format characters right
|
|
66
89
|
* before it, the first character excepted), and the way back to the text's own coordinates. */
|
|
@@ -78,11 +101,10 @@ export interface FormatFreeReading {
|
|
|
78
101
|
}
|
|
79
102
|
/** The reading of `input` (a text that holds a format character). Not on the package barrel; the module's own pins read it. */
|
|
80
103
|
export declare function readFormatFree(input: string): FormatFreeReading;
|
|
81
|
-
/** One run of the engine: the
|
|
82
|
-
* (the policy's
|
|
83
|
-
*
|
|
104
|
+
/** One run of the engine: the readings (the policy's, as a set), what a text the percent layers cannot read to its end
|
|
105
|
+
* answers (the policy's `onUnresolvedReading`; read only when the readings hold `percent-layers`), and the optional report.
|
|
106
|
+
* A text no reading can afford is replaced whole by the `oversized` mark. */
|
|
84
107
|
export interface EngineRun {
|
|
85
|
-
readonly overStrip: string;
|
|
86
108
|
readonly readings: ReadingSet;
|
|
87
109
|
readonly onUnresolvedReading?: SecretScanPolicy["onUnresolvedReading"] | undefined;
|
|
88
110
|
readonly report?: RedactionReport | undefined;
|
|
@@ -118,16 +140,6 @@ export interface SecretScanPolicy {
|
|
|
118
140
|
readonly urlUserinfo: "keep-username" | "whole";
|
|
119
141
|
readonly onUnresolvedReading: "secret" | "ignore";
|
|
120
142
|
}
|
|
121
|
-
/** The marks a hit is written as — the caller's own spellings, none of them a default here. A row's hit is written as
|
|
122
|
-
* `byFamily[row id]`, else `byKind[row kind]`, else `value`; a text no reading can afford is replaced whole by
|
|
123
|
-
* `overStrip`. Rows that run as one pass (they share a `pass`) are one alternation and write one mark: a `byFamily` that
|
|
124
|
-
* spells two of them differently is refused. A mark is a non-empty string. */
|
|
125
|
-
export interface SecretNotation {
|
|
126
|
-
readonly value: string;
|
|
127
|
-
readonly byKind?: Readonly<Record<string, string>>;
|
|
128
|
-
readonly byFamily?: Readonly<Partial<Record<CredentialFamilyId, string>>>;
|
|
129
|
-
readonly overStrip: string;
|
|
130
|
-
}
|
|
131
143
|
/** Where the vocabulary sees a credential: the withheld part of one hit (a kept label or username outside it), in the
|
|
132
144
|
* text's own coordinates, with the row that found it. */
|
|
133
145
|
export interface SecretSpan {
|
|
@@ -149,37 +161,42 @@ export type SecretScan = {
|
|
|
149
161
|
readonly outcome: "whole";
|
|
150
162
|
readonly reason: "reading-unresolved" | "scan-budget" | "mapping-failed";
|
|
151
163
|
};
|
|
152
|
-
/** What {@link renderSecrets} takes beside the policy
|
|
153
|
-
*
|
|
164
|
+
/** What {@link renderSecrets} takes beside the policy: the caller's own passes, run after the vocabulary's in the same
|
|
165
|
+
* pipeline and through the same readings (each writes its own `marker`, a member of the marks table), and a report
|
|
166
|
+
* collector. */
|
|
154
167
|
export interface RenderOptions {
|
|
155
168
|
readonly passes?: readonly RedactionPass[] | undefined;
|
|
156
169
|
readonly report?: RedactionReport | undefined;
|
|
157
170
|
}
|
|
158
|
-
/** Does a URL userinfo (no trailing `@`) hold a user (before its first `:`) or a password (after it)? An empty
|
|
159
|
-
* nothing — `scheme://@host` and `scheme://:@host` hold neither — as {@link userinfoWithheld} reads under
|
|
171
|
+
/** Does a URL userinfo (no trailing `@`) hold a user (before its first `:` outside its marks) or a password (after it)? An empty
|
|
172
|
+
* interval holds nothing — `scheme://@host` and `scheme://:@host` hold neither — as {@link userinfoWithheld} reads under
|
|
173
|
+
* `keep-username`. */
|
|
160
174
|
export declare function userinfoHolds(userinfo: string): boolean;
|
|
161
175
|
/**
|
|
162
176
|
* The wide forms rendered as passes, in the vocabulary's order: one pass per group (a group of several rows is ONE
|
|
163
|
-
* leftmost alternation, its hits attributed to the row that matches at the hit's offset
|
|
164
|
-
* captures a label keeps it ahead of the marker; the URL userinfo is split per `urlUserinfo
|
|
177
|
+
* leftmost alternation, its hits attributed to the row that matches at the hit's offset, each written as that row's
|
|
178
|
+
* mark). A row whose wide form captures a label keeps it ahead of the marker; the URL userinfo is split per `urlUserinfo`
|
|
179
|
+
* — a userinfo withheld whole writes the row's `userinfo` mark, a password withheld behind a kept username the bare mark
|
|
180
|
+
* (the username says what follows it).
|
|
165
181
|
*/
|
|
166
|
-
export declare function egressPasses(groups: ReadonlyArray<readonly ValueFamilyRow[]>,
|
|
182
|
+
export declare function egressPasses(groups: ReadonlyArray<readonly ValueFamilyRow[]>, urlUserinfo: SecretScanPolicy["urlUserinfo"]): RedactionPass[];
|
|
167
183
|
/**
|
|
168
|
-
* `text` with every credential the vocabulary finds under `policy` withheld,
|
|
169
|
-
*
|
|
170
|
-
*
|
|
171
|
-
*
|
|
172
|
-
*
|
|
184
|
+
* `text` with every credential the vocabulary finds under `policy` withheld, each hit written as its row's member of the
|
|
185
|
+
* marks table (the fragment grade's as the `:partial` member), and the caller's own passes ({@link RenderOptions.passes})
|
|
186
|
+
* run after the vocabulary's in the same pipeline. One finding per hit goes to `options.report` when given
|
|
187
|
+
* (`report.preexistingMarkers` is the caller's to set). The policy and the caller's passes are read and checked on every
|
|
188
|
+
* call; what the policy reads as picks a compilation kept across calls. Throws a `TypeError` on a policy that is not whole
|
|
189
|
+
* or a caller's pass whose marker is no member of the marks table.
|
|
173
190
|
*/
|
|
174
|
-
export declare function renderSecrets(text: string, policy: SecretScanPolicy,
|
|
191
|
+
export declare function renderSecrets(text: string, policy: SecretScanPolicy, options?: RenderOptions): string;
|
|
175
192
|
/**
|
|
176
193
|
* Where the vocabulary sees a credential in `text` under `policy`: every hit of the grade's form of every row, on every
|
|
177
194
|
* reading of the policy — each text the text readings derive, scanned by the character readings — as the hit's withheld
|
|
178
195
|
* part in `text`'s own coordinates (the whole written units a derived hit was read from; deduplicated, sorted by start) —
|
|
179
196
|
* or the whole text, when a reading cannot be afforded (more than 64 places where format characters stood: `scan-budget`)
|
|
180
197
|
* or the percent layers ran out before the text did under `onUnresolvedReading: "secret"` (`reading-unresolved`). The
|
|
181
|
-
* matcher answers where
|
|
182
|
-
*
|
|
198
|
+
* matcher answers where its forms match, a mark the text holds included (`password=«redacted:key»` is a span here); the renderer
|
|
199
|
+
* leaves a hit whose withheld part is exactly a mark the text holds. Throws a `TypeError` on a policy that is not whole.
|
|
183
200
|
*/
|
|
184
201
|
export declare function scanSecrets(text: string, policy: SecretScanPolicy): SecretScan;
|
|
185
202
|
/** Does the vocabulary see a credential in `text` under `policy`? {@link scanSecrets} answers the whole text or a span. */
|