@dpuse/dpuse-shared 1.0.106 → 1.0.110
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +119 -67
- package/dist/dpuse-shared.es.js +412 -323
- package/dist/dpuse-shared.es.js.map +1 -1
- package/dist/types/src/encoding/index.d.ts +329 -244
- package/package.json +3 -3
package/README.md
CHANGED
|
@@ -83,65 +83,101 @@ Character encoding types with detection and decodability flags, a catalogue of e
|
|
|
83
83
|
|
|
84
84
|
<!-- ENCODINGS_START -->
|
|
85
85
|
|
|
86
|
-
The table below lists every
|
|
87
|
-
|
|
88
|
-
- **
|
|
89
|
-
- **
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
|
95
|
-
|
|
|
96
|
-
| — |
|
|
97
|
-
| — |
|
|
98
|
-
| Arabic |
|
|
99
|
-
| Arabic |
|
|
100
|
-
|
|
|
101
|
-
|
|
|
102
|
-
|
|
|
103
|
-
|
|
|
104
|
-
|
|
|
105
|
-
|
|
|
106
|
-
|
|
|
107
|
-
|
|
|
108
|
-
|
|
|
109
|
-
|
|
|
110
|
-
|
|
|
111
|
-
|
|
|
112
|
-
|
|
|
113
|
-
|
|
|
114
|
-
|
|
|
115
|
-
| Cyrillic |
|
|
116
|
-
|
|
|
117
|
-
|
|
|
118
|
-
|
|
|
119
|
-
|
|
|
120
|
-
|
|
|
121
|
-
|
|
|
122
|
-
|
|
|
123
|
-
|
|
|
124
|
-
|
|
|
125
|
-
|
|
|
126
|
-
|
|
|
127
|
-
|
|
|
128
|
-
|
|
|
129
|
-
|
|
|
130
|
-
|
|
|
131
|
-
|
|
|
132
|
-
|
|
|
133
|
-
|
|
|
134
|
-
|
|
|
135
|
-
|
|
|
136
|
-
|
|
|
137
|
-
|
|
|
138
|
-
|
|
|
139
|
-
|
|
|
140
|
-
|
|
|
141
|
-
|
|
|
142
|
-
|
|
|
143
|
-
|
|
|
144
|
-
|
|
|
86
|
+
The table below lists every encoding [jschardet](https://github.com/aadsm/jschardet) can identify from a file's contents, under the name it reports. A test checks the table against jschardet's own list of encodings, so it fails if a jschardet upgrade adds, drops, renames or remaps one.
|
|
87
|
+
|
|
88
|
+
- **Decodable** — the browser's built-in `TextDecoder` can decode it. Those that cannot are refused by all the major browsers: `HZ-GB-2312` and `ISO-2022-KR` are blocked by the WHATWG Encoding Standard for security reasons, and the rest, such as UTF-32, UTF-7 and the DOS, Mac and mainframe code pages, are not part of the standard. A file in one can still be named, even though it cannot be read. `UTF-16` is marked not decodable because its byte-order mark decides which way round it is read, so it has no single browser name.
|
|
89
|
+
- **Decoded As** — the name browsers decode it with, taken from jschardet's own map. Browsers treat some names as aliases, so for example `ascii` and `ISO-8859-1` are decoded as `windows-1252`. The practical effects: bytes 0x80–0x9F decode as Windows-1252 characters (such as `€` and curly quotes) rather than control codes, and `ascii` never rejects bytes above 0x7F.
|
|
90
|
+
|
|
91
|
+
The major browsers all follow the Encoding Standard, so these columns hold for each of them. A test checks that Node's `TextDecoder`, which follows the same standard, accepts every Decoded As name as its own. They were also confirmed in Chrome 154, Edge 154, Firefox 156 and Safari 26.6 (October 2026), and all four gave identical results. To re-check them on a Mac, run `npm run check:browsers` after a build.
|
|
92
|
+
|
|
93
|
+
| Group | Encoding | Decodable | Decoded As |
|
|
94
|
+
| :------------------ | :-------------- | :-------: | :------------- |
|
|
95
|
+
| — | ascii | ✅ | windows-1252 |
|
|
96
|
+
| — | utf-8 | ✅ | utf-8 |
|
|
97
|
+
| — | UTF-8-SIG | ✅ | utf-8 |
|
|
98
|
+
| Arabic | cp1006 | ❌ | — |
|
|
99
|
+
| Arabic | cp720 | ❌ | — |
|
|
100
|
+
| Arabic | cp864 | ❌ | — |
|
|
101
|
+
| Arabic | ISO-8859-6 | ✅ | iso-8859-6 |
|
|
102
|
+
| Arabic | Windows-1256 | ✅ | windows-1256 |
|
|
103
|
+
| Baltic | cp775 | ❌ | — |
|
|
104
|
+
| Baltic | ISO-8859-13 | ✅ | iso-8859-13 |
|
|
105
|
+
| Baltic | iso8859-4 | ✅ | iso-8859-4 |
|
|
106
|
+
| Baltic | Windows-1257 | ✅ | windows-1257 |
|
|
107
|
+
| Celtic | iso8859-14 | ✅ | iso-8859-14 |
|
|
108
|
+
| Central European | cp852 | ❌ | — |
|
|
109
|
+
| Central European | ISO-8859-2 | ✅ | iso-8859-2 |
|
|
110
|
+
| Central European | MacLatin2 | ❌ | — |
|
|
111
|
+
| Central European | Windows-1250 | ✅ | windows-1250 |
|
|
112
|
+
| Chinese Simplified | GB18030 | ✅ | gb18030 |
|
|
113
|
+
| Chinese Simplified | HZ-GB-2312 | ❌ | — |
|
|
114
|
+
| Chinese Traditional | Big5 | ✅ | big5 |
|
|
115
|
+
| Cyrillic | cp1125 | ❌ | — |
|
|
116
|
+
| Cyrillic | IBM855 | ❌ | — |
|
|
117
|
+
| Cyrillic | IBM866 | ✅ | ibm866 |
|
|
118
|
+
| Cyrillic | ISO-8859-5 | ✅ | iso-8859-5 |
|
|
119
|
+
| Cyrillic | KOI8-R | ✅ | koi8-r |
|
|
120
|
+
| Cyrillic | koi8-t | ❌ | — |
|
|
121
|
+
| Cyrillic | koi8-u | ✅ | koi8-u |
|
|
122
|
+
| Cyrillic | KZ1048 | ❌ | — |
|
|
123
|
+
| Cyrillic | MacCyrillic | ✅ | x-mac-cyrillic |
|
|
124
|
+
| Cyrillic | ptcp154 | ❌ | — |
|
|
125
|
+
| Cyrillic | Windows-1251 | ✅ | windows-1251 |
|
|
126
|
+
| Greek | cp737 | ❌ | — |
|
|
127
|
+
| Greek | cp869 | ❌ | — |
|
|
128
|
+
| Greek | cp875 | ❌ | — |
|
|
129
|
+
| Greek | ISO-8859-7 | ✅ | iso-8859-7 |
|
|
130
|
+
| Greek | MacGreek | ❌ | — |
|
|
131
|
+
| Greek | Windows-1253 | ✅ | windows-1253 |
|
|
132
|
+
| Hebrew | cp424 | ❌ | — |
|
|
133
|
+
| Hebrew | cp856 | ❌ | — |
|
|
134
|
+
| Hebrew | cp862 | ❌ | — |
|
|
135
|
+
| Hebrew | ISO-8859-8 | ✅ | iso-8859-8-i |
|
|
136
|
+
| Hebrew | Windows-1255 | ✅ | windows-1255 |
|
|
137
|
+
| Japanese | CP932 | ✅ | shift_jis |
|
|
138
|
+
| Japanese | EUC-JP | ✅ | euc-jp |
|
|
139
|
+
| Japanese | ISO-2022-JP | ✅ | iso-2022-jp |
|
|
140
|
+
| Japanese | iso2022_jp_2004 | ❌ | — |
|
|
141
|
+
| Japanese | iso2022_jp_ext | ❌ | — |
|
|
142
|
+
| Japanese | SHIFT_JIS | ✅ | shift_jis |
|
|
143
|
+
| Korean | CP949 | ✅ | euc-kr |
|
|
144
|
+
| Korean | EUC-KR | ✅ | euc-kr |
|
|
145
|
+
| Korean | ISO-2022-KR | ❌ | — |
|
|
146
|
+
| Korean | Johab | ❌ | — |
|
|
147
|
+
| Nordic | cp861 | ❌ | — |
|
|
148
|
+
| Nordic | cp865 | ❌ | — |
|
|
149
|
+
| Nordic | iso8859-10 | ✅ | iso-8859-10 |
|
|
150
|
+
| Nordic | MacIceland | ❌ | — |
|
|
151
|
+
| Other | utf-7 | ❌ | — |
|
|
152
|
+
| Romanian | iso8859-16 | ✅ | iso-8859-16 |
|
|
153
|
+
| Southern European | iso8859-3 | ✅ | iso-8859-3 |
|
|
154
|
+
| Thai | CP874 | ✅ | windows-874 |
|
|
155
|
+
| Thai | TIS-620 | ✅ | windows-874 |
|
|
156
|
+
| Turkish | cp1026 | ❌ | — |
|
|
157
|
+
| Turkish | cp857 | ❌ | — |
|
|
158
|
+
| Turkish | ISO-8859-9 | ✅ | windows-1254 |
|
|
159
|
+
| Turkish | MacTurkish | ❌ | — |
|
|
160
|
+
| Turkish | Windows-1254 | ✅ | windows-1254 |
|
|
161
|
+
| Unicode 16 | utf-16-be | ✅ | utf-16be |
|
|
162
|
+
| Unicode 16 | utf-16-le | ✅ | utf-16le |
|
|
163
|
+
| Unicode 16 | UTF-16 | ❌ | — |
|
|
164
|
+
| Unicode 32 | utf-32-be | ❌ | — |
|
|
165
|
+
| Unicode 32 | utf-32-le | ❌ | — |
|
|
166
|
+
| Unicode 32 | UTF-32 | ❌ | — |
|
|
167
|
+
| Vietnamese | cp1258 | ✅ | windows-1258 |
|
|
168
|
+
| Western | cp1140 | ❌ | — |
|
|
169
|
+
| Western | cp273 | ❌ | — |
|
|
170
|
+
| Western | cp437 | ❌ | — |
|
|
171
|
+
| Western | cp500 | ❌ | — |
|
|
172
|
+
| Western | cp850 | ❌ | — |
|
|
173
|
+
| Western | cp858 | ❌ | — |
|
|
174
|
+
| Western | cp860 | ❌ | — |
|
|
175
|
+
| Western | cp863 | ❌ | — |
|
|
176
|
+
| Western | hp-roman8 | ❌ | — |
|
|
177
|
+
| Western | ISO-8859-1 | ✅ | windows-1252 |
|
|
178
|
+
| Western | iso8859-15 | ✅ | iso-8859-15 |
|
|
179
|
+
| Western | MacRoman | ✅ | macintosh |
|
|
180
|
+
| Western | Windows-1252 | ✅ | windows-1252 |
|
|
145
181
|
|
|
146
182
|
<!-- ENCODINGS_END -->
|
|
147
183
|
|
|
@@ -161,6 +197,22 @@ OData-to-internal type conversion, file path name and extension extraction, numb
|
|
|
161
197
|
|
|
162
198
|
See [API_REFERENCE.md](./API_REFERENCE.md) for the complete API reference, including all exported schemas, types, classes, constants, and actions, grouped by topic.
|
|
163
199
|
|
|
200
|
+
## Test Data
|
|
201
|
+
|
|
202
|
+
The encoding tests use sample files in [`tests/fixtures/encodings`](./tests/fixtures/encodings). They're used only to test and evaluate character encoding detection and decoding.
|
|
203
|
+
|
|
204
|
+
- **Universal Declaration of Human Rights:** from NLTK's "Universal Declaration of Human Rights Corpus (Unicode Version)" ([udhr2.zip](https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/packages/corpora/udhr2.zip)), which holds the declaration in 388 languages as UTF-8 text files named by language code, such as `rus.txt`.
|
|
205
|
+
- It's in the public domain, as recorded in [NLTK's data index](https://raw.githubusercontent.com/nltk/nltk_data/gh-pages/index.xml) (the `udhr2` entry).
|
|
206
|
+
- NLTK's copy comes from the [Unicode UDHR project](https://unicode.org/udhr/). We use NLTK's copy because the Unicode project's own file links no longer work.
|
|
207
|
+
- We use excerpts of about 1,000 characters in 29 languages, re-encoded into each of the 86 encodings.
|
|
208
|
+
- Characters an encoding can't hold were removed.
|
|
209
|
+
- Arabic letters in `cp864` and `cp1006` are stored as presentation forms, as those encodings require.
|
|
210
|
+
- **Written for these tests:** the seven CSV files, and two Welsh sentences added to `iso8859-14.txt`. They're under this project's licence.
|
|
211
|
+
|
|
212
|
+
A second set of tests uses chardet's own test files, in [`tests/fixtures/encodings/chardet`](./tests/fixtures/encodings/chardet): its [unit-test files](https://github.com/runk/node-chardet/tree/master/src/test/data/encodings) and [corpus](https://github.com/runk/node-chardet/tree/master/corpus), copyright 2024 Dmitry Shirokov, under the MIT licence in that folder's `LICENSE` file.
|
|
213
|
+
|
|
214
|
+
The October 2026 comparison of chardet and jschardet also used [chardet/test-data](https://github.com/chardet/test-data), which isn't included here, because each of its files is copyright its publisher.
|
|
215
|
+
|
|
164
216
|
<!-- DEPENDENCY_LICENSES_START -->
|
|
165
217
|
|
|
166
218
|
## Dependency Licenses
|
|
@@ -189,14 +241,14 @@ _Note: Sonda's Vite reports currently exclude CSS files, since Vite does not gen
|
|
|
189
241
|
|
|
190
242
|
| Chunk/Module/File | Composition |
|
|
191
243
|
| :-------------------------------------------------------------------------- | :------------------------------------------ |
|
|
192
|
-
| **dist/dpuse-shared.es.js** |
|
|
193
|
-
| src | `█████████████░░░░░░░` 65.
|
|
194
|
-
| ↳ index.ts | `▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░`
|
|
195
|
-
| ↳ connectorConfig.schema.ts | `▒░░░░░░░░░░░░░░░░░░░` 3.
|
|
196
|
-
| ↳ componentConfig.schema.ts | `▒░░░░░░░░░░░░░░░░░░░` 2.
|
|
197
|
-
| ↳ 7 smaller files | `▒░░░░░░░░░░░░░░░░░░░` 4.
|
|
198
|
-
| valibot → dist/index.mjs | `████░░░░░░░░░░░░░░░░` 20.
|
|
199
|
-
| (bundler output, whitespace & JSON) | `███░░░░░░░░░░░░░░░░░` 14.
|
|
244
|
+
| **dist/dpuse-shared.es.js** | 35.1 kB · gzip 9.2 kB · 100.0% of the build |
|
|
245
|
+
| src | `█████████████░░░░░░░` 65.6% · 23.0 kB |
|
|
246
|
+
| ↳ index.ts | `▒▒▒▒▒▒▒▒▒▒▒░░░░░░░░░` 55.7% · 19.5 kB |
|
|
247
|
+
| ↳ connectorConfig.schema.ts | `▒░░░░░░░░░░░░░░░░░░░` 3.4% · 1.2 kB |
|
|
248
|
+
| ↳ componentConfig.schema.ts | `▒░░░░░░░░░░░░░░░░░░░` 2.6% · 936 B |
|
|
249
|
+
| ↳ 7 smaller files | `▒░░░░░░░░░░░░░░░░░░░` 4.0% · 1.4 kB |
|
|
250
|
+
| valibot → dist/index.mjs | `████░░░░░░░░░░░░░░░░` 20.0% · 7.0 kB |
|
|
251
|
+
| (bundler output, whitespace & JSON) | `███░░░░░░░░░░░░░░░░░` 14.3% · 5.0 kB |
|
|
200
252
|
|
|
201
253
|
Bars show each row's share of its output file. ↳ rows are part of the row above.
|
|
202
254
|
|