urdu-text-utils 0.1.2 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +11 -0
- package/README.md +39 -2
- package/dist/index.cjs +2053 -292
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +64 -4
- package/dist/index.d.ts +64 -4
- package/dist/index.js +2050 -293
- package/dist/index.js.map +1 -1
- package/package.json +8 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,16 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.4
|
|
4
|
+
|
|
5
|
+
- Add Urdu stop words module (`URDU_STOP_WORDS`, `isStopWord`, `filterStopWords`, `removeStopWords`) with a curated set of 130+ canonical functional words.
|
|
6
|
+
- Enhance sentence segmentation in `splitSentences` and `countSentences` with abbreviation & honorific protection (e.g. `ڈاکٹر.`, `پروفیسر.`, `صاحب.`), numeric decimal protection, and optional `preserveTerminators` option.
|
|
7
|
+
|
|
8
|
+
## 0.1.3
|
|
9
|
+
|
|
10
|
+
- Significantly expand transliteration dictionary coverage across pronouns & oblique forms (`مجھے`, `ہمیں`, `انہیں`, `اسے`, `جسے`), everyday verbs and inflections, calendar & weekdays, numbers & ordinals, food, household objects, health, geography, and common Pakistani names.
|
|
11
|
+
- Expanded Roman Urdu variants (`ROMAN_VARIANTS`) for common texting and phonetically ambiguous spellings (`mjhe`, `humein`, `unhein`, `isay`, `chahye`, etc.).
|
|
12
|
+
- Enhanced dictionary normalization and clean token mapping for slug generation and reverse transliteration.
|
|
13
|
+
|
|
3
14
|
## 0.1.2
|
|
4
15
|
|
|
5
16
|
- Dictionary grows to ~650 entries: news and public-life vocabulary (the densest register in Urdu media), technology, education, health, colours, animals, extended family, food and place names.
|
package/README.md
CHANGED
|
@@ -1,9 +1,18 @@
|
|
|
1
1
|
# urdu-text-utils
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/urdu-text-utils)
|
|
4
|
+
[](https://github.com/Zaid-maker/urdu-text-utils/actions/workflows/ci.yml)
|
|
5
|
+
[](https://bundlejs.com/?q=urdu-text-utils)
|
|
6
|
+
[](https://www.npmjs.com/package/urdu-text-utils?activeTab=dependencies)
|
|
7
|
+
[](https://www.npmjs.com/package/urdu-text-utils)
|
|
8
|
+
[](./LICENSE)
|
|
9
|
+
|
|
3
10
|
Urdu text processing toolkit for JavaScript and TypeScript. Normalization, detection, digits, diacritics, collation, search, statistics and transliteration.
|
|
4
11
|
|
|
5
12
|
Zero runtime dependencies. ESM + CJS. Ships its own types.
|
|
6
13
|
|
|
14
|
+
**[Documentation and live playground →](https://zaid-maker.github.io/urdu-text-utils/)**
|
|
15
|
+
|
|
7
16
|
```bash
|
|
8
17
|
npm install urdu-text-utils
|
|
9
18
|
```
|
|
@@ -67,14 +76,30 @@ hasUrduSpecificLetters("لڑکی"); // true — ڑ does not exist in Arabic
|
|
|
67
76
|
hasUrduSpecificLetters("كتاب مدرسة"); // false
|
|
68
77
|
```
|
|
69
78
|
|
|
70
|
-
## Word and sentence
|
|
79
|
+
## Word and sentence tokenization
|
|
71
80
|
|
|
72
81
|
```ts
|
|
73
82
|
countWords("پاکستان ایک خوبصورت ملک ہے"); // 5
|
|
74
83
|
countWords("آپ کیسے ہیں؟"); // 3 — attached punctuation is not a word
|
|
75
84
|
countSentences("یہ پہلا جملہ ہے۔ یہ دوسرا ہے۔"); // 2
|
|
76
85
|
splitWords(text); // string[]
|
|
77
|
-
splitSentences(text); // string[]
|
|
86
|
+
splitSentences(text); // string[] — protects abbreviations like ڈاکٹر. and decimals
|
|
87
|
+
splitSentences(text, { preserveTerminators: true }); // preserves ending punctuation
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Stop words
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
import { isStopWord, filterStopWords, removeStopWords, URDU_STOP_WORDS } from "urdu-text-utils";
|
|
94
|
+
|
|
95
|
+
isStopWord("اور"); // true
|
|
96
|
+
isStopWord("کتاب"); // false
|
|
97
|
+
|
|
98
|
+
filterStopWords(["یہ", "ایک", "بہترین", "کتاب", "ہے"]);
|
|
99
|
+
// ["بہترین", "کتاب"]
|
|
100
|
+
|
|
101
|
+
removeStopWords("پاکستان ایک خوبصورت ملک ہے");
|
|
102
|
+
// "پاکستان خوبصورت ملک"
|
|
78
103
|
```
|
|
79
104
|
|
|
80
105
|
## Urdu numbers
|
|
@@ -182,6 +207,18 @@ npm run typecheck
|
|
|
182
207
|
npm run build
|
|
183
208
|
```
|
|
184
209
|
|
|
210
|
+
### Docs site
|
|
211
|
+
|
|
212
|
+
The site is VitePress, in `docs/`. Its playground imports the library from `src/` through a Vite alias, so the examples can never drift from the code.
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
npm run docs:dev # local server with hot reload
|
|
216
|
+
npm run docs:build # static output in docs/.vitepress/dist
|
|
217
|
+
npm run docs:preview
|
|
218
|
+
```
|
|
219
|
+
|
|
220
|
+
CI builds the docs on every push. VitePress fails on dead links, so a renamed page breaks the build rather than shipping a 404.
|
|
221
|
+
|
|
185
222
|
## Releasing
|
|
186
223
|
|
|
187
224
|
Publishing is automated and tag-driven. CI runs tests on Node 18/20/22 for every push and PR; nothing reaches npm until a version tag exists.
|