text-sanctifier 1.0.18 โ 1.0.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/ADDITIONAL_TERMS.md +122 -0
- package/LICENSE +694 -0
- package/README.md +163 -167
- package/package.json +65 -63
- package/robots.txt +79 -0
- package/src/index.js +12 -12
- package/src/inspectText.js +174 -174
- package/src/sanctifyText.js +507 -507
- package/LICENSE.md +0 -285
package/README.md
CHANGED
|
@@ -1,167 +1,163 @@
|
|
|
1
|
-
# text-sanctifier
|
|
2
|
-
|
|
3
|
-
[](https://www.npmjs.com/package/text-sanctifier)
|
|
4
|
-
[](https://www.npmjs.com/package/text-sanctifier)
|
|
4
|
+
[](https://www.npmjs.com/package/text-sanctifier)
|
|
5
|
+
[](https://bundlephobia.com/package/text-sanctifier)
|
|
6
|
+
[](https://github.com/iWhatty/text-sanctifier/blob/main/LICENSE)
|
|
7
|
+
[](https://github.com/iWhatty/text-sanctifier)
|
|
8
|
+
|
|
9
|
+
Brutal text normalizer and invisible Unicode scrubber for modern web projects. ~1.66 KB gzipped.
|
|
10
|
+
|
|
11
|
+
## Features
|
|
12
|
+
|
|
13
|
+
- Purges zero-width Unicode garbage
|
|
14
|
+
- Normalizes line endings (CRLF, CR, LF) to LF
|
|
15
|
+
- Collapses unwanted spaces and paragraphs
|
|
16
|
+
- Nukes control characters (if enabled)
|
|
17
|
+
- Smart normalization of typographic junk (quotes, dashes, bullets, full-width punctuation)
|
|
18
|
+
- Keyboard-only filtering (retain printable ASCII + full emoji sequences)
|
|
19
|
+
- Preserves ZWJ emoji clusters (๐จโ๐ฉโ๐งโ๐ฆ)
|
|
20
|
+
- Preserves VS16 emoji presentation variants (โ๏ธ, โผ๏ธ)
|
|
21
|
+
- Configurable via fine-grained flags or ready-made presets
|
|
22
|
+
- Includes strict, loose, and keyboard-only modes
|
|
23
|
+
- Deterministic RegExp usage (no global `lastIndex` state leaks)
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Install
|
|
28
|
+
|
|
29
|
+
```sh
|
|
30
|
+
pnpm add text-sanctifier
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
---
|
|
34
|
+
|
|
35
|
+
## Quick start
|
|
36
|
+
|
|
37
|
+
**Custom config:**
|
|
38
|
+
|
|
39
|
+
```js
|
|
40
|
+
import { summonSanctifier } from 'text-sanctifier';
|
|
41
|
+
|
|
42
|
+
const clean = summonSanctifier({
|
|
43
|
+
purgeInvisibleChars: true,
|
|
44
|
+
purgeEmojis: true,
|
|
45
|
+
collapseSpaces: true,
|
|
46
|
+
collapseNewLines: true,
|
|
47
|
+
preserveParagraphs: true,
|
|
48
|
+
finalTrim: true,
|
|
49
|
+
});
|
|
50
|
+
|
|
51
|
+
const output = clean(rawText);
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
**Strict preset:**
|
|
55
|
+
|
|
56
|
+
```js
|
|
57
|
+
const output = summonSanctifier.strict(rawText);
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
**Loose preset:**
|
|
61
|
+
|
|
62
|
+
```js
|
|
63
|
+
const output = summonSanctifier.loose(rawText);
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
**Keyboard-only (no emojis):**
|
|
67
|
+
|
|
68
|
+
```js
|
|
69
|
+
const output = summonSanctifier.keyboardOnly(userInput);
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
**Keyboard-only (with emojis):**
|
|
73
|
+
|
|
74
|
+
```js
|
|
75
|
+
const output = summonSanctifier.keyboardOnlyEmoji(commentText);
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
---
|
|
79
|
+
|
|
80
|
+
## Security notes
|
|
81
|
+
|
|
82
|
+
- **Not an HTML/XSS sanitizer.** This library normalizes and filters plain text.
|
|
83
|
+
- If you need to render **untrusted content**, render it as text (e.g. `textContent`), not HTML (`innerHTML`).
|
|
84
|
+
- If you need to sanitize **HTML**, use a dedicated HTML sanitizer (e.g. DOMPurify / sanitize-html).
|
|
85
|
+
- Like any text-processing library, extremely large untrusted inputs can be used for CPU/DoS pressure; consider input size limits in high-risk environments.
|
|
86
|
+
|
|
87
|
+
---
|
|
88
|
+
|
|
89
|
+
## API
|
|
90
|
+
|
|
91
|
+
### `summonSanctifier(options?: SanctifyOptions): (text: string) => string`
|
|
92
|
+
|
|
93
|
+
Creates a reusable sanitizer from an option object.
|
|
94
|
+
|
|
95
|
+
### `summonSanctifier.strict`
|
|
96
|
+
|
|
97
|
+
Aggressively purges: emojis, control characters, extra spacing, and newlines.
|
|
98
|
+
|
|
99
|
+
### `summonSanctifier.loose`
|
|
100
|
+
|
|
101
|
+
Gently normalizes spacing and newlines while preserving emojis and paragraphs.
|
|
102
|
+
|
|
103
|
+
### `summonSanctifier.keyboardOnly`
|
|
104
|
+
|
|
105
|
+
Restricts to printable ASCII only (removes emojis).
|
|
106
|
+
|
|
107
|
+
### `summonSanctifier.keyboardOnlyEmoji`
|
|
108
|
+
|
|
109
|
+
Restricts to printable ASCII + full emoji sequences. Preserves ZWJ emoji clusters and emoji presentation variants.
|
|
110
|
+
|
|
111
|
+
### `inspectText(text: string): UnicodeTrashReport`
|
|
112
|
+
|
|
113
|
+
Returns a structural report of control codes, invisible chars, newline styles, and more.
|
|
114
|
+
|
|
115
|
+
```js
|
|
116
|
+
import { inspectText } from 'text-sanctifier';
|
|
117
|
+
|
|
118
|
+
const report = inspectText(input);
|
|
119
|
+
|
|
120
|
+
/*
|
|
121
|
+
{
|
|
122
|
+
hasControlChars: true,
|
|
123
|
+
hasInvisibleChars: true,
|
|
124
|
+
hasMixedNewlines: false,
|
|
125
|
+
newlineStyle: 'LF',
|
|
126
|
+
hasEmojis: true,
|
|
127
|
+
hasNonKeyboardChars: false,
|
|
128
|
+
summary: [
|
|
129
|
+
'Control characters detected.',
|
|
130
|
+
'Invisible Unicode characters detected.',
|
|
131
|
+
'Emojis detected.',
|
|
132
|
+
'Consistent newline style: LF'
|
|
133
|
+
]
|
|
134
|
+
}
|
|
135
|
+
*/
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
Use `inspectText` to preflight text content before rendering, storing, or linting. It's a diagnostic tool to help inform sanitization needs. Pass the report to `getRecommendedSanctifierOptions(report)` to auto-generate config flags for `summonSanctifier()`.
|
|
139
|
+
|
|
140
|
+
---
|
|
141
|
+
|
|
142
|
+
## Notes
|
|
143
|
+
|
|
144
|
+
### Runtime requirements
|
|
145
|
+
|
|
146
|
+
Requires a modern JavaScript runtime with ES2020+ support:
|
|
147
|
+
|
|
148
|
+
- Node.js 14+
|
|
149
|
+
- Modern evergreen browsers
|
|
150
|
+
|
|
151
|
+
### Package and build info
|
|
152
|
+
|
|
153
|
+
- **Source (`src/`):** ES2020+ ESM modules with JSDoc
|
|
154
|
+
- **Browser build (`dist/`):** Minified ESM bundle for `<script type="module">`
|
|
155
|
+
- **Tree-shake friendly:** ships `sideEffects: false`
|
|
156
|
+
- **Zero transpilation:** no built-in polyfills or runtime overhead
|
|
157
|
+
- **Bundler ready:** works with Vite, Rollup, Webpack, Parcel, esbuild
|
|
158
|
+
|
|
159
|
+
---
|
|
160
|
+
|
|
161
|
+
## License
|
|
162
|
+
|
|
163
|
+
Licensed under AGPL-3.0 with WATT3D Additional Terms. See [LICENSE](./LICENSE) and [ADDITIONAL_TERMS.md](./ADDITIONAL_TERMS.md). Commercial AI/model-training use requires compliance with those terms or a separate WATT3D license. ยฉ WATT3D.
|
package/package.json
CHANGED
|
@@ -1,63 +1,65 @@
|
|
|
1
|
-
{
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
"
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
"
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
"
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
"
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
"
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
"
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
1
|
+
{
|
|
2
|
+
"name": "text-sanctifier",
|
|
3
|
+
"version": "1.0.19",
|
|
4
|
+
"type": "module",
|
|
5
|
+
"description": "A brutal text normalizer and invisible trash scrubber for modern web projects.",
|
|
6
|
+
"main": "./src/index.js",
|
|
7
|
+
"module": "./src/index.js",
|
|
8
|
+
"browser": "./dist/text-sanctifier.min.js",
|
|
9
|
+
"files": [
|
|
10
|
+
"src",
|
|
11
|
+
"dist/text-sanctifier.min.js",
|
|
12
|
+
"dist/types",
|
|
13
|
+
"LICENSE",
|
|
14
|
+
"README.md",
|
|
15
|
+
"ADDITIONAL_TERMS.md",
|
|
16
|
+
"robots.txt"
|
|
17
|
+
],
|
|
18
|
+
"exports": {
|
|
19
|
+
".": {
|
|
20
|
+
"types": "./dist/types/index.d.ts",
|
|
21
|
+
"default": "./src/index.js"
|
|
22
|
+
},
|
|
23
|
+
"./browser": {
|
|
24
|
+
"default": "./dist/text-sanctifier.min.js"
|
|
25
|
+
}
|
|
26
|
+
},
|
|
27
|
+
"types": "./dist/types/index.d.ts",
|
|
28
|
+
"sideEffects": false,
|
|
29
|
+
"scripts": {
|
|
30
|
+
"build": "node scripts/buildc.js",
|
|
31
|
+
"build:types": "tsc -p tsconfig.types.json",
|
|
32
|
+
"test": "node tests/sanctifyText.test.js",
|
|
33
|
+
"test-min": "node tests/sanctifyText.test.min.js",
|
|
34
|
+
"ci": "npm run test && npm run build && npm run build:types && npm run test-min"
|
|
35
|
+
},
|
|
36
|
+
"keywords": [
|
|
37
|
+
"text",
|
|
38
|
+
"sanitize",
|
|
39
|
+
"normalize",
|
|
40
|
+
"invisible",
|
|
41
|
+
"scrubber",
|
|
42
|
+
"unicode",
|
|
43
|
+
"clean",
|
|
44
|
+
"sanctify"
|
|
45
|
+
],
|
|
46
|
+
"author": "WATT3D <WATT3D@protonmail.com>",
|
|
47
|
+
"license": "SEE LICENSE IN LICENSE",
|
|
48
|
+
"repository": {
|
|
49
|
+
"type": "git",
|
|
50
|
+
"url": "git+https://github.com/iWhatty/text-sanctifier.git"
|
|
51
|
+
},
|
|
52
|
+
"bugs": {
|
|
53
|
+
"url": "https://github.com/iWhatty/text-sanctifier/issues"
|
|
54
|
+
},
|
|
55
|
+
"homepage": "https://github.com/iWhatty/text-sanctifier#readme",
|
|
56
|
+
"directories": {
|
|
57
|
+
"test": "tests"
|
|
58
|
+
},
|
|
59
|
+
"devDependencies": {
|
|
60
|
+
"esbuild": "^0.25.3",
|
|
61
|
+
"google-closure-compiler": "^20240317.0.0",
|
|
62
|
+
"terser": "^5.39.0",
|
|
63
|
+
"typescript": "^5.9.3"
|
|
64
|
+
}
|
|
65
|
+
}
|
package/robots.txt
ADDED
|
@@ -0,0 +1,79 @@
|
|
|
1
|
+
# WATT3D โ AI scraping policy
|
|
2
|
+
#
|
|
3
|
+
# This file enumerates known AI / LLM training and inference crawlers
|
|
4
|
+
# that the WATT3D AI Training Rider does not consent to. See
|
|
5
|
+
# ./ADDITIONAL_TERMS.md for the legal scope and definitions.
|
|
6
|
+
#
|
|
7
|
+
# Honoring robots.txt is voluntary. The rider applies whether or not
|
|
8
|
+
# a bot respects this file. The enumeration is non-exhaustive โ bots
|
|
9
|
+
# not listed here are still bound by the rider if they perform
|
|
10
|
+
# Training Use as defined in ADDITIONAL_TERMS.md ยง1.
|
|
11
|
+
|
|
12
|
+
User-agent: GPTBot
|
|
13
|
+
Disallow: /
|
|
14
|
+
|
|
15
|
+
User-agent: ChatGPT-User
|
|
16
|
+
Disallow: /
|
|
17
|
+
|
|
18
|
+
User-agent: OAI-SearchBot
|
|
19
|
+
Disallow: /
|
|
20
|
+
|
|
21
|
+
User-agent: anthropic-ai
|
|
22
|
+
Disallow: /
|
|
23
|
+
|
|
24
|
+
User-agent: ClaudeBot
|
|
25
|
+
Disallow: /
|
|
26
|
+
|
|
27
|
+
User-agent: Claude-Web
|
|
28
|
+
Disallow: /
|
|
29
|
+
|
|
30
|
+
User-agent: Claude-SearchBot
|
|
31
|
+
Disallow: /
|
|
32
|
+
|
|
33
|
+
User-agent: Google-Extended
|
|
34
|
+
Disallow: /
|
|
35
|
+
|
|
36
|
+
User-agent: Applebot-Extended
|
|
37
|
+
Disallow: /
|
|
38
|
+
|
|
39
|
+
User-agent: Meta-ExternalAgent
|
|
40
|
+
Disallow: /
|
|
41
|
+
|
|
42
|
+
User-agent: Meta-ExternalFetcher
|
|
43
|
+
Disallow: /
|
|
44
|
+
|
|
45
|
+
User-agent: FacebookBot
|
|
46
|
+
Disallow: /
|
|
47
|
+
|
|
48
|
+
User-agent: CCBot
|
|
49
|
+
Disallow: /
|
|
50
|
+
|
|
51
|
+
User-agent: PerplexityBot
|
|
52
|
+
Disallow: /
|
|
53
|
+
|
|
54
|
+
User-agent: Perplexity-User
|
|
55
|
+
Disallow: /
|
|
56
|
+
|
|
57
|
+
User-agent: cohere-ai
|
|
58
|
+
Disallow: /
|
|
59
|
+
|
|
60
|
+
User-agent: Bytespider
|
|
61
|
+
Disallow: /
|
|
62
|
+
|
|
63
|
+
User-agent: Amazonbot
|
|
64
|
+
Disallow: /
|
|
65
|
+
|
|
66
|
+
User-agent: Omgilibot
|
|
67
|
+
Disallow: /
|
|
68
|
+
|
|
69
|
+
User-agent: AI2Bot
|
|
70
|
+
Disallow: /
|
|
71
|
+
|
|
72
|
+
User-agent: Diffbot
|
|
73
|
+
Disallow: /
|
|
74
|
+
|
|
75
|
+
# Everyone else: full access. Search engines and human-driven crawlers
|
|
76
|
+
# remain welcome.
|
|
77
|
+
|
|
78
|
+
User-agent: *
|
|
79
|
+
Allow: /
|
package/src/index.js
CHANGED
|
@@ -1,12 +1,12 @@
|
|
|
1
|
-
// src/index.js
|
|
2
|
-
|
|
3
|
-
|
|
4
|
-
import { inspectText, getRecommendedSanctifierOptions } from './inspectText.js';
|
|
5
|
-
export { inspectText, getRecommendedSanctifierOptions };
|
|
6
|
-
|
|
7
|
-
import { summonSanctifier } from './sanctifyText.js';
|
|
8
|
-
export { summonSanctifier };
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
// In Future: export helpers for power users
|
|
12
|
-
// export { purgeInvisibleTrash, purgeEmojisCharacters, normalizeNewlines, trimSpacesAroundNewlines, collapseParagraphs, collapseExtraSpaces, purgeControlCharacters } from './sanctifyText.js';
|
|
1
|
+
// src/index.js
|
|
2
|
+
|
|
3
|
+
|
|
4
|
+
import { inspectText, getRecommendedSanctifierOptions } from './inspectText.js';
|
|
5
|
+
export { inspectText, getRecommendedSanctifierOptions };
|
|
6
|
+
|
|
7
|
+
import { summonSanctifier } from './sanctifyText.js';
|
|
8
|
+
export { summonSanctifier };
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
// In Future: export helpers for power users
|
|
12
|
+
// export { purgeInvisibleTrash, purgeEmojisCharacters, normalizeNewlines, trimSpacesAroundNewlines, collapseParagraphs, collapseExtraSpaces, purgeControlCharacters } from './sanctifyText.js';
|