@desert-ant-labs/emo 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE.md +136 -0
- package/README.md +117 -0
- package/dist/cache-node.d.ts +5 -0
- package/dist/cache-node.js +32 -0
- package/dist/cache-web.d.ts +3 -0
- package/dist/cache-web.js +24 -0
- package/dist/hub.d.ts +31 -0
- package/dist/hub.js +42 -0
- package/dist/index.browser.d.ts +17 -0
- package/dist/index.browser.js +37 -0
- package/dist/index.node.d.ts +17 -0
- package/dist/index.node.js +44 -0
- package/dist/model.d.ts +33 -0
- package/dist/model.js +163 -0
- package/dist/skin-tone.d.ts +8 -0
- package/dist/skin-tone.js +51 -0
- package/dist/tokenizer.d.ts +16 -0
- package/dist/tokenizer.js +229 -0
- package/package.json +78 -0
package/LICENSE.md
ADDED
|
@@ -0,0 +1,136 @@
|
|
|
1
|
+
# Desert Ant Labs Source-Available License
|
|
2
|
+
|
|
3
|
+
Version 1.0 · 2026-06-15
|
|
4
|
+
|
|
5
|
+
Copyright © 2026 Studio Company B.V., trading as Desert Ant Labs (`desertant.ai`).
|
|
6
|
+
|
|
7
|
+
This license governs Your use of the software, model weights, datasets, documentation, and associated assets in this repository (collectively, the **"Licensed Materials"**). It is a **source-available license intended for professional and business use**; it is not an OSI-approved open-source license. The rights granted here are subject to a per-Model usage cap and the other terms set out below.
|
|
8
|
+
|
|
9
|
+
## Definitions
|
|
10
|
+
|
|
11
|
+
- **"Desert Ant Labs"** (also "We", "Us", "Our") — Studio Company B.V., a private limited liability company (*besloten vennootschap*) incorporated under the laws of the Netherlands, with statutory seat in Amsterdam, registered with the Dutch Chamber of Commerce (KvK) under number **81228112**, trading as Desert Ant Labs.
|
|
12
|
+
- **"Model"** — a set of trained weights published by Desert Ant Labs under a single HuggingFace repository (for example, `huggingface.co/desert-ant-labs/eye`). Each Model is a distinct work under this license.
|
|
13
|
+
- **"Monthly Active User"** or **"MAU"** — a unique end user who interacts with a feature powered by a Model during a calendar month. Internal use by Your own employees and contractors does not count.
|
|
14
|
+
- **"You"** — the individual or legal entity exercising rights under this license.
|
|
15
|
+
|
|
16
|
+
## 0. Acceptance
|
|
17
|
+
|
|
18
|
+
By downloading, accessing, using, copying, modifying, or distributing the Licensed Materials, You agree to be bound by this license. If You do not agree, You may not use the Licensed Materials.
|
|
19
|
+
|
|
20
|
+
## 1. Grant
|
|
21
|
+
|
|
22
|
+
Subject to Your continued compliance with this license, Desert Ant Labs grants You a worldwide, royalty-free, non-exclusive, non-transferable, non-sublicensable license to use, reproduce, modify, and distribute the Licensed Materials.
|
|
23
|
+
|
|
24
|
+
## 2. Free-tier use
|
|
25
|
+
|
|
26
|
+
You may use the Licensed Materials free of charge, including for commercial purposes, subject to the following per-Model threshold:
|
|
27
|
+
|
|
28
|
+
> **For each Desert Ant Labs Model that Your products incorporate, the aggregate Monthly Active Users across all Your products must not exceed 100,000 in any calendar month.**
|
|
29
|
+
|
|
30
|
+
The threshold applies independently to each Model. Two Models at 50,000 MAU each do not combine into 100,000 MAU; each is independently within its own free-tier limit.
|
|
31
|
+
|
|
32
|
+
If a repository contains only software (no trained Model weights), no MAU threshold applies to that software on its own. The threshold attaches to the Models the software is used with.
|
|
33
|
+
|
|
34
|
+
## 3. Research and academic use
|
|
35
|
+
|
|
36
|
+
Use of the Licensed Materials for non-commercial scientific research and academic publications is exempt from the MAU threshold in Section 2, provided that any resulting publication credits "Desert Ant Labs" and links to the relevant Model repository.
|
|
37
|
+
|
|
38
|
+
## 4. Commercial license
|
|
39
|
+
|
|
40
|
+
If Your aggregate MAU for any Model exceeds 100,000 (or You expect it to within the next calendar month), You must obtain a commercial license from Desert Ant Labs before continuing to distribute or operate any product that incorporates that Model. A commercial license covers only the Models named in it; other Models remain under the free tier subject to their own thresholds.
|
|
41
|
+
|
|
42
|
+
Contact: <licensing@desertant.ai>.
|
|
43
|
+
|
|
44
|
+
## 5. Patent grant
|
|
45
|
+
|
|
46
|
+
Each contributor to the Licensed Materials grants You a perpetual, worldwide, non-exclusive, royalty-free, irrevocable (except as stated in this Section) patent license to make, use, sell, offer to sell, and import the Licensed Materials, where such license applies only to those patent claims licensable by that contributor that are necessarily infringed by the Licensed Materials.
|
|
47
|
+
|
|
48
|
+
**If You institute patent litigation against any entity** (including a cross-claim or counterclaim in a lawsuit) alleging that the Licensed Materials infringe a patent, **all patent licenses granted to You under this license terminate as of the date such litigation is filed.**
|
|
49
|
+
|
|
50
|
+
## 6. Restrictions
|
|
51
|
+
|
|
52
|
+
You may not:
|
|
53
|
+
|
|
54
|
+
1. Sell, sublicense, or otherwise distribute the Licensed Materials themselves, or any substantially unmodified derivative of them, as a standalone product or hosted service.
|
|
55
|
+
2. Use the Licensed Materials to train, distill, fine-tune, or otherwise derive a model that You release or distribute under any license more permissive than this license, including any OSI-approved open-source license. Internal research, personal fine-tuning, and use of derivative models that remain under this license are not restricted.
|
|
56
|
+
3. Remove or alter the copyright, license, attribution, or third-party notices in the Licensed Materials.
|
|
57
|
+
|
|
58
|
+
You may freely embed the Licensed Materials in Your own product, subject to Section 2 and the rest of this license.
|
|
59
|
+
|
|
60
|
+
## 7. Attribution
|
|
61
|
+
|
|
62
|
+
You must retain this LICENSE file and the copyright notice in all copies and substantial portions of the Licensed Materials. Where Your product surfaces acknowledgements, third-party notices, or "About" / "Credits" screens, please credit **"Desert Ant Labs (`desertant.ai`)"**.
|
|
63
|
+
|
|
64
|
+
## 8. Modifications
|
|
65
|
+
|
|
66
|
+
You may modify the Licensed Materials. If You distribute modified Licensed Materials, You must:
|
|
67
|
+
|
|
68
|
+
(a) include a prominent notice stating that You have modified them, with the date of modification; and
|
|
69
|
+
(b) ensure that Your distributed modifications are licensed under this license.
|
|
70
|
+
|
|
71
|
+
This license governs the Licensed Materials and modifications to them. Applications and products that merely incorporate, link to, or call the Licensed Materials are Your own work; their structure, source, and license are not constrained by this license, subject only to Sections 2, 6, and 7.
|
|
72
|
+
|
|
73
|
+
## 9. Reporting
|
|
74
|
+
|
|
75
|
+
To enable Desert Ant Labs to administer the free tier, You agree:
|
|
76
|
+
|
|
77
|
+
(a) to maintain reasonable records of Your aggregate MAU per Model; and
|
|
78
|
+
(b) upon written request from Desert Ant Labs, to provide a reasonable summary of those records sufficient to confirm compliance with Section 2. Such requests will be made no more than once per calendar year per licensee, except in the case of a reasonable, specific suspicion of breach.
|
|
79
|
+
|
|
80
|
+
You are encouraged — but not required — to notify Desert Ant Labs at <licensing@desertant.ai> when Your aggregate MAU for any Model first exceeds 80,000.
|
|
81
|
+
|
|
82
|
+
## 10. Trademarks
|
|
83
|
+
|
|
84
|
+
This license does not grant You any right to use the trade names, trademarks, service marks, or product names of Desert Ant Labs, except for the limited attribution use specified in Section 7. All goodwill arising from such attribution use inures to the benefit of Desert Ant Labs.
|
|
85
|
+
|
|
86
|
+
## 11. Third-party components
|
|
87
|
+
|
|
88
|
+
The Licensed Materials may include or derive from components licensed by third parties. Those components remain subject to their own licenses; nothing in this license overrides those terms. Notices for such components are provided in the model card or in a `THIRD_PARTY_NOTICES.md` file alongside the Licensed Materials.
|
|
89
|
+
|
|
90
|
+
## 12. Termination
|
|
91
|
+
|
|
92
|
+
This license terminates automatically and immediately upon any of the following:
|
|
93
|
+
|
|
94
|
+
(a) Your aggregate MAU for any Model exceeds 100,000 and You do not obtain a commercial license within thirty (30) days;
|
|
95
|
+
(b) Your breach of Section 6 (Restrictions); or
|
|
96
|
+
(c) Your institution of patent litigation as described in Section 5.
|
|
97
|
+
|
|
98
|
+
For any other curable breach, Desert Ant Labs will provide written notice; if the breach is not cured within thirty (30) days of receipt of that notice, this license terminates as of the date the notice was sent.
|
|
99
|
+
|
|
100
|
+
Upon termination, You must cease using and distributing the Licensed Materials. Sections 7 (Attribution, with respect to copies already in the field), 10 (Trademarks), 11 (Third-Party Components), 13 (No Warranty), 14 (Limitation of Liability), and 15 (Governing Law) survive termination.
|
|
101
|
+
|
|
102
|
+
## 13. No warranty
|
|
103
|
+
|
|
104
|
+
THE LICENSED MATERIALS ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT. Desert Ant Labs makes no representation that the Licensed Materials are error-free, secure, or suitable for any particular use.
|
|
105
|
+
|
|
106
|
+
## 14. Limitation of liability
|
|
107
|
+
|
|
108
|
+
IN NO EVENT SHALL DESERT ANT LABS BE LIABLE FOR ANY INDIRECT, INCIDENTAL, CONSEQUENTIAL, SPECIAL, OR EXEMPLARY DAMAGES (INCLUDING LOST PROFITS, LOSS OF DATA, OR BUSINESS INTERRUPTION) ARISING FROM OR IN CONNECTION WITH THE LICENSED MATERIALS, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
|
|
109
|
+
|
|
110
|
+
DESERT ANT LABS' TOTAL CUMULATIVE LIABILITY UNDER OR IN CONNECTION WITH THIS LICENSE SHALL NOT EXCEED THE GREATER OF (A) ONE HUNDRED EUROS (€100) OR (B) THE TOTAL FEES PAID BY YOU TO DESERT ANT LABS UNDER A SEPARATE COMMERCIAL LICENSE IN THE TWELVE (12) MONTHS PRECEDING THE CLAIM.
|
|
111
|
+
|
|
112
|
+
Nothing in this license limits or excludes liability to the extent such limitation or exclusion is not permitted by applicable mandatory law, including liability for willful misconduct (*opzet*) or gross negligence (*bewuste roekeloosheid*) and any mandatory provision of Dutch consumer-protection law.
|
|
113
|
+
|
|
114
|
+
## 15. Governing law and jurisdiction
|
|
115
|
+
|
|
116
|
+
This license is governed by the laws of the Netherlands, without regard to its conflict-of-laws provisions. The competent courts of Amsterdam, the Netherlands, have exclusive jurisdiction over any dispute arising under or in connection with this license, subject to mandatory provisions of applicable consumer-protection law.
|
|
117
|
+
|
|
118
|
+
The United Nations Convention on Contracts for the International Sale of Goods (CISG, 1980) does not apply to this license.
|
|
119
|
+
|
|
120
|
+
## 16. Miscellaneous
|
|
121
|
+
|
|
122
|
+
(a) **Severability.** If any provision of this license is held unenforceable by a court of competent jurisdiction, the remaining provisions remain in full force and effect, and the unenforceable provision shall be interpreted to give effect to its intent to the greatest extent permitted by law.
|
|
123
|
+
|
|
124
|
+
(b) **No waiver.** Failure or delay by Desert Ant Labs to enforce any right under this license is not a waiver of that right.
|
|
125
|
+
|
|
126
|
+
(c) **Assignment.** You may not assign this license without Desert Ant Labs' prior written consent. Desert Ant Labs may freely assign this license to a successor in interest. This license binds and benefits each party's successors and permitted assigns.
|
|
127
|
+
|
|
128
|
+
(d) **Entire agreement.** This license, together with any separate commercial license referenced in Section 4, is the entire agreement between You and Desert Ant Labs regarding the Licensed Materials and supersedes all prior agreements on the same subject matter.
|
|
129
|
+
|
|
130
|
+
(e) **Language.** The authoritative version of this license is in English. Any translation is for convenience only; in the event of any conflict, the English version controls.
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
Commercial licensing inquiries: <licensing@desertant.ai>.
|
|
135
|
+
|
|
136
|
+
© 2026 Studio Company B.V., trading as Desert Ant Labs. Statutory seat: Amsterdam, the Netherlands. KvK: **81228112**.
|
package/README.md
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# @desert-ant-labs/emo
|
|
2
|
+
|
|
3
|
+
On-device emoji suggestions from text. Suggests the best-matching emoji for short
|
|
4
|
+
tasks, calendar entries, notes, or message drafts across **23 languages** — fully
|
|
5
|
+
in-process, no inference runtime.
|
|
6
|
+
|
|
7
|
+
```ts
|
|
8
|
+
import { suggestions } from "@desert-ant-labs/emo";
|
|
9
|
+
|
|
10
|
+
const results = await suggestions("Pay my bills");
|
|
11
|
+
// [{ emoji: "💰", confidence: 0.62 }, ...]
|
|
12
|
+
|
|
13
|
+
const emoji = (await suggestions("犬の散歩", 1))[0]?.emoji; // "🐕"
|
|
14
|
+
const toned = (await suggestions("go for a run", 1, { skinTone: "medium" }))[0]?.emoji; // "🏃🏽"
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
## Features
|
|
18
|
+
|
|
19
|
+
- Pure-JS inference (no ONNX/WASM runtime) — prediction is sub-millisecond
|
|
20
|
+
- Suggests from a data-driven vocabulary of ~300 task/calendar/message emojis
|
|
21
|
+
- Supports 23 languages (incl. CJK, Arabic, Thai, Hindi, …)
|
|
22
|
+
- Model (~5.0 MB, 4-bit palettized) is fetched from the Hugging Face Hub at a **pinned revision**, then
|
|
23
|
+
cached — to the **filesystem** on Node and to **Cache Storage** in the browser, so
|
|
24
|
+
it loads once and runs offline after
|
|
25
|
+
|
|
26
|
+
## Install
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
npm install @desert-ant-labs/emo
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Importing
|
|
33
|
+
|
|
34
|
+
Pure ESM and fully tree-shakeable. The **same import works everywhere** — Node, bundlers,
|
|
35
|
+
browsers, and edge/worker runtimes — the right build is selected automatically:
|
|
36
|
+
|
|
37
|
+
```ts
|
|
38
|
+
import { suggestions } from "@desert-ant-labs/emo";
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
CommonJS consumers use dynamic import (`const { suggestions } = await import("@desert-ant-labs/emo")`);
|
|
42
|
+
native `require()` works on Node ≥ 22.12.
|
|
43
|
+
|
|
44
|
+
**Minimal footprint / bring-your-own-bytes.** If you load the model files yourself, import the
|
|
45
|
+
hub-free core from `@desert-ant-labs/emo/core` — only the ~6 KB inference engine, with zero
|
|
46
|
+
network/filesystem code:
|
|
47
|
+
|
|
48
|
+
```ts
|
|
49
|
+
import { createEmo } from "@desert-ant-labs/emo/core";
|
|
50
|
+
|
|
51
|
+
// weights and tokenizer are Uint8Array buffers; meta is the parsed emo_meta.json
|
|
52
|
+
const emo = createEmo({ weights, tokenizer, meta });
|
|
53
|
+
emo.suggestions("walk the dog", 1)[0]?.emoji; // "🐕"
|
|
54
|
+
```
|
|
55
|
+
|
|
56
|
+
## Loading model
|
|
57
|
+
|
|
58
|
+
Model files are fetched from the Hugging Face Hub
|
|
59
|
+
([`desert-ant-labs/emo`](https://huggingface.co/desert-ant-labs/emo)) at a pinned
|
|
60
|
+
revision and cached. The first call downloads `emo.safetensors`, `emo_tokenizer.bin`,
|
|
61
|
+
and `emo_meta.json`; subsequent calls read from the cache.
|
|
62
|
+
|
|
63
|
+
- **Node**: `suggestions()` works zero-config; files cache under `~/.cache/emo`.
|
|
64
|
+
To run fully offline, ship the files yourself and point at a folder with
|
|
65
|
+
`env.localModelPath` (or `EMO_LOCAL_PATH`).
|
|
66
|
+
- **Browser**: same API; files cache in Cache Storage.
|
|
67
|
+
|
|
68
|
+
```ts
|
|
69
|
+
import { env, load, suggestions } from "@desert-ant-labs/emo";
|
|
70
|
+
|
|
71
|
+
// optional global config (set before first use)
|
|
72
|
+
env.revision = "main"; // or a commit SHA / tag
|
|
73
|
+
env.cacheDir = "/var/cache/emo"; // Node only
|
|
74
|
+
env.localModelPath = "./emo-model"; // Node: use local files, skip the Hugging Face Hub
|
|
75
|
+
|
|
76
|
+
// or load an explicit instance (synchronous inference after it resolves)
|
|
77
|
+
const emo = await load({ revision: "main" });
|
|
78
|
+
emo.suggestions("book a flight to Tokyo", 1)[0]?.emoji; // "✈️"
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
## API
|
|
82
|
+
|
|
83
|
+
```ts
|
|
84
|
+
export function suggestions(text: string, limit?: number, options?: EmoSuggestionOptions): Promise<EmoSuggestion[]>;
|
|
85
|
+
export function load(options?: Partial<EmoEnv>): Promise<EmoModel>;
|
|
86
|
+
export function createEmo(buffers: { weights; tokenizer; meta }): EmoModel; // raw buffers
|
|
87
|
+
export const env: EmoEnv;
|
|
88
|
+
export function reset(): void; // clear the memoized model so the next suggestions() re-reads env
|
|
89
|
+
|
|
90
|
+
export interface EmoSuggestion {
|
|
91
|
+
emoji: string;
|
|
92
|
+
confidence: number;
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
export interface EmoSuggestionOptions {
|
|
96
|
+
skinTone?: EmojiSkinTone; // default: "default"
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
export type EmojiSkinTone = "default" | "light" | "mediumLight" | "medium" | "mediumDark" | "dark";
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
`suggestions(text, limit = 3, options)` returns up to `limit` emojis, most likely first;
|
|
103
|
+
empty input returns `[]`. `skinTone` post-processes skin-tone-capable emoji; the default is `"default"` (no modifier). `EmoModel.suggestions` is synchronous once loaded.
|
|
104
|
+
|
|
105
|
+
## Example
|
|
106
|
+
|
|
107
|
+
[`examples/EmoExample`](examples/EmoExample) is a small todo-list web app that predicts an
|
|
108
|
+
emoji for each task on-device. Run it with `node server.js` from that folder and open
|
|
109
|
+
<http://localhost:5173>.
|
|
110
|
+
|
|
111
|
+
## Model
|
|
112
|
+
|
|
113
|
+
Published at [`desert-ant-labs/emo`](https://huggingface.co/desert-ant-labs/emo) on Hugging Face.
|
|
114
|
+
|
|
115
|
+
## License
|
|
116
|
+
|
|
117
|
+
See [`LICENSE.md`](LICENSE.md) — Desert Ant Labs Source-Available License v1.0. Free for commercial use up to 100,000 MAU per Model; <licensing@desertant.ai> above that.
|
|
@@ -0,0 +1,5 @@
|
|
|
1
|
+
import type { FileCache } from "./hub.js";
|
|
2
|
+
/** Filesystem-backed cache (Node): downloaded files persist across runs. */
|
|
3
|
+
export declare function fsCache(cacheDir: string): FileCache;
|
|
4
|
+
/** Reads model files from a local directory (for `localModelPath`). */
|
|
5
|
+
export declare function localReader(dir: string): (name: string) => Promise<Uint8Array | null>;
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
import { mkdir, readFile, writeFile } from "node:fs/promises";
|
|
2
|
+
import { dirname, join } from "node:path";
|
|
3
|
+
const keyToPath = (cacheDir, key) => join(cacheDir, key.replace(/^https?:\/\//, "").replace(/[^a-zA-Z0-9._/-]/g, "_"));
|
|
4
|
+
/** Filesystem-backed cache (Node): downloaded files persist across runs. */
|
|
5
|
+
export function fsCache(cacheDir) {
|
|
6
|
+
return {
|
|
7
|
+
async get(key) {
|
|
8
|
+
try {
|
|
9
|
+
return new Uint8Array(await readFile(keyToPath(cacheDir, key)));
|
|
10
|
+
}
|
|
11
|
+
catch {
|
|
12
|
+
return null;
|
|
13
|
+
}
|
|
14
|
+
},
|
|
15
|
+
async put(key, data) {
|
|
16
|
+
const path = keyToPath(cacheDir, key);
|
|
17
|
+
await mkdir(dirname(path), { recursive: true });
|
|
18
|
+
await writeFile(path, data);
|
|
19
|
+
},
|
|
20
|
+
};
|
|
21
|
+
}
|
|
22
|
+
/** Reads model files from a local directory (for `localModelPath`). */
|
|
23
|
+
export function localReader(dir) {
|
|
24
|
+
return async (name) => {
|
|
25
|
+
try {
|
|
26
|
+
return new Uint8Array(await readFile(join(dir, name)));
|
|
27
|
+
}
|
|
28
|
+
catch {
|
|
29
|
+
return null;
|
|
30
|
+
}
|
|
31
|
+
};
|
|
32
|
+
}
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
/** Cache Storage-backed cache (browser): downloaded files persist across visits. */
|
|
2
|
+
export function webCache(name = "emo-model") {
|
|
3
|
+
return {
|
|
4
|
+
async get(key) {
|
|
5
|
+
try {
|
|
6
|
+
const cache = await caches.open(name);
|
|
7
|
+
const res = await cache.match(key);
|
|
8
|
+
return res ? new Uint8Array(await res.arrayBuffer()) : null;
|
|
9
|
+
}
|
|
10
|
+
catch {
|
|
11
|
+
return null;
|
|
12
|
+
}
|
|
13
|
+
},
|
|
14
|
+
async put(key, data) {
|
|
15
|
+
try {
|
|
16
|
+
const cache = await caches.open(name);
|
|
17
|
+
await cache.put(key, new Response(data));
|
|
18
|
+
}
|
|
19
|
+
catch {
|
|
20
|
+
/* caching is best-effort */
|
|
21
|
+
}
|
|
22
|
+
},
|
|
23
|
+
};
|
|
24
|
+
}
|
package/dist/hub.d.ts
ADDED
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
import { EmoModel } from "./model.js";
|
|
2
|
+
export declare const DEFAULT_HOST = "https://huggingface.co";
|
|
3
|
+
export declare const DEFAULT_REPO = "desert-ant-labs/emo";
|
|
4
|
+
/** Pinned revision of the model repo (a commit SHA). */
|
|
5
|
+
export declare const DEFAULT_REVISION = "bcbd42b62f8d6c0bad5d2b4a76598a908f102235";
|
|
6
|
+
/** Resolution + caching configuration (mutate the exported `env` to change defaults). */
|
|
7
|
+
export interface EmoEnv {
|
|
8
|
+
/** Hugging Face host serving the model repo. */
|
|
9
|
+
host: string;
|
|
10
|
+
/** Model repo id, e.g. `"desert-ant-labs/emo"`. */
|
|
11
|
+
repo: string;
|
|
12
|
+
/** Pinned revision (commit SHA, tag, or branch). */
|
|
13
|
+
revision: string;
|
|
14
|
+
/** Allow fetching from the Hugging Face Hub. Set `false` to require a local copy. */
|
|
15
|
+
allowRemote: boolean;
|
|
16
|
+
/** Cache downloaded files (filesystem on Node, Cache Storage in the browser). */
|
|
17
|
+
useCache: boolean;
|
|
18
|
+
/** Directory of pre-downloaded model files to use instead of the Hugging Face Hub (Node). */
|
|
19
|
+
localModelPath?: string;
|
|
20
|
+
/** Filesystem cache directory (Node). */
|
|
21
|
+
cacheDir?: string;
|
|
22
|
+
/** Optional Hugging Face access token (Node). */
|
|
23
|
+
token?: string;
|
|
24
|
+
}
|
|
25
|
+
/** A key/value store for cached file bytes, keyed by resolve URL. */
|
|
26
|
+
export interface FileCache {
|
|
27
|
+
get(key: string): Promise<Uint8Array | null>;
|
|
28
|
+
put(key: string, data: Uint8Array): Promise<void>;
|
|
29
|
+
}
|
|
30
|
+
/** Resolves all model files (local dir → cache → Hugging Face Hub) and builds an {@link EmoModel}. */
|
|
31
|
+
export declare function loadModel(env: EmoEnv, cache: FileCache | null, readLocal?: (name: string) => Promise<Uint8Array | null>): Promise<EmoModel>;
|
package/dist/hub.js
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
import { createEmo } from "./model.js";
|
|
2
|
+
export const DEFAULT_HOST = "https://huggingface.co";
|
|
3
|
+
export const DEFAULT_REPO = "desert-ant-labs/emo";
|
|
4
|
+
/** Pinned revision of the model repo (a commit SHA). */
|
|
5
|
+
export const DEFAULT_REVISION = "bcbd42b62f8d6c0bad5d2b4a76598a908f102235";
|
|
6
|
+
const FILES = ["emo.safetensors", "emo_tokenizer.bin", "emo_meta.json"];
|
|
7
|
+
function resolveUrl(env, name) {
|
|
8
|
+
return `${env.host}/${env.repo}/resolve/${env.revision}/${name}`;
|
|
9
|
+
}
|
|
10
|
+
async function fetchFile(env, name, cache, readLocal) {
|
|
11
|
+
if (readLocal) {
|
|
12
|
+
const local = await readLocal(name);
|
|
13
|
+
if (local)
|
|
14
|
+
return local;
|
|
15
|
+
}
|
|
16
|
+
const url = resolveUrl(env, name);
|
|
17
|
+
if (cache && env.useCache) {
|
|
18
|
+
const hit = await cache.get(url);
|
|
19
|
+
if (hit)
|
|
20
|
+
return hit;
|
|
21
|
+
}
|
|
22
|
+
if (!env.allowRemote)
|
|
23
|
+
throw new Error(`emo: ${name} unavailable locally and remote loading is disabled`);
|
|
24
|
+
const res = await fetch(url, env.token ? { headers: { Authorization: `Bearer ${env.token}` } } : undefined);
|
|
25
|
+
if (!res.ok)
|
|
26
|
+
throw new Error(`emo: failed to fetch ${name} from ${url} (${res.status} ${res.statusText})`);
|
|
27
|
+
const data = new Uint8Array(await res.arrayBuffer());
|
|
28
|
+
if (cache && env.useCache) {
|
|
29
|
+
try {
|
|
30
|
+
await cache.put(url, data);
|
|
31
|
+
}
|
|
32
|
+
catch {
|
|
33
|
+
/* caching is best-effort */
|
|
34
|
+
}
|
|
35
|
+
}
|
|
36
|
+
return data;
|
|
37
|
+
}
|
|
38
|
+
/** Resolves all model files (local dir → cache → Hugging Face Hub) and builds an {@link EmoModel}. */
|
|
39
|
+
export async function loadModel(env, cache, readLocal) {
|
|
40
|
+
const [weights, tokenizer, meta] = await Promise.all(FILES.map((name) => fetchFile(env, name, cache, readLocal)));
|
|
41
|
+
return createEmo({ weights, tokenizer, meta: JSON.parse(new TextDecoder().decode(meta)) });
|
|
42
|
+
}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import { type EmoEnv } from "./hub.js";
|
|
2
|
+
import { type EmoModel, type EmoSuggestion } from "./model.js";
|
|
3
|
+
import { type EmoSuggestionOptions } from "./skin-tone.js";
|
|
4
|
+
export { createEmo, EmoModel, type EmoSuggestion, type EmoMeta } from "./model.js";
|
|
5
|
+
export { applyEmojiSkinTone, type EmojiSkinTone, type EmoSuggestionOptions } from "./skin-tone.js";
|
|
6
|
+
export { loadModel, type EmoEnv, type FileCache } from "./hub.js";
|
|
7
|
+
/** Loading configuration. Mutate before the first call, or pass overrides to {@link load}. */
|
|
8
|
+
export declare const env: EmoEnv;
|
|
9
|
+
/** Loads the model from the Hugging Face Hub (cached in Cache Storage). */
|
|
10
|
+
export declare function load(options?: Partial<EmoEnv>): Promise<EmoModel>;
|
|
11
|
+
/**
|
|
12
|
+
* Returns emoji suggestions for a phrase, most likely first. The model is loaded
|
|
13
|
+
* (and cached) lazily on first call. Empty input returns `[]`.
|
|
14
|
+
*/
|
|
15
|
+
export declare function suggestions(text: string, limit?: number, options?: EmoSuggestionOptions): Promise<EmoSuggestion[]>;
|
|
16
|
+
/** Clears the memoized model so the next {@link suggestions} call re-reads `env`. */
|
|
17
|
+
export declare function reset(): void;
|
|
@@ -0,0 +1,37 @@
|
|
|
1
|
+
import { webCache } from "./cache-web.js";
|
|
2
|
+
import { DEFAULT_HOST, DEFAULT_REPO, DEFAULT_REVISION, loadModel } from "./hub.js";
|
|
3
|
+
export { createEmo, EmoModel } from "./model.js";
|
|
4
|
+
export { applyEmojiSkinTone } from "./skin-tone.js";
|
|
5
|
+
export { loadModel } from "./hub.js";
|
|
6
|
+
/** Loading configuration. Mutate before the first call, or pass overrides to {@link load}. */
|
|
7
|
+
export const env = {
|
|
8
|
+
host: DEFAULT_HOST,
|
|
9
|
+
repo: DEFAULT_REPO,
|
|
10
|
+
revision: DEFAULT_REVISION,
|
|
11
|
+
allowRemote: true,
|
|
12
|
+
useCache: true,
|
|
13
|
+
};
|
|
14
|
+
/** Loads the model from the Hugging Face Hub (cached in Cache Storage). */
|
|
15
|
+
export async function load(options = {}) {
|
|
16
|
+
const e = { ...env, ...options };
|
|
17
|
+
return loadModel(e, e.useCache ? webCache() : null);
|
|
18
|
+
}
|
|
19
|
+
let modelPromise = null;
|
|
20
|
+
/**
|
|
21
|
+
* Returns emoji suggestions for a phrase, most likely first. The model is loaded
|
|
22
|
+
* (and cached) lazily on first call. Empty input returns `[]`.
|
|
23
|
+
*/
|
|
24
|
+
export async function suggestions(text, limit = 3, options = {}) {
|
|
25
|
+
if (!modelPromise) {
|
|
26
|
+
// Don't cache a rejected load: clear the slot on failure so the next call retries.
|
|
27
|
+
modelPromise = load().catch((err) => {
|
|
28
|
+
modelPromise = null;
|
|
29
|
+
throw err;
|
|
30
|
+
});
|
|
31
|
+
}
|
|
32
|
+
return (await modelPromise).suggestions(text, limit, options);
|
|
33
|
+
}
|
|
34
|
+
/** Clears the memoized model so the next {@link suggestions} call re-reads `env`. */
|
|
35
|
+
export function reset() {
|
|
36
|
+
modelPromise = null;
|
|
37
|
+
}
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
import { type EmoEnv } from "./hub.js";
|
|
2
|
+
import { type EmoModel, type EmoSuggestion } from "./model.js";
|
|
3
|
+
import { type EmoSuggestionOptions } from "./skin-tone.js";
|
|
4
|
+
export { createEmo, EmoModel, type EmoSuggestion, type EmoMeta } from "./model.js";
|
|
5
|
+
export { applyEmojiSkinTone, type EmojiSkinTone, type EmoSuggestionOptions } from "./skin-tone.js";
|
|
6
|
+
export { loadModel, type EmoEnv, type FileCache } from "./hub.js";
|
|
7
|
+
/** Loading configuration. Mutate before the first call, or pass overrides to {@link load}. */
|
|
8
|
+
export declare const env: EmoEnv;
|
|
9
|
+
/** Loads the model: a local dir if configured, else the Hugging Face Hub (cached to disk). */
|
|
10
|
+
export declare function load(options?: Partial<EmoEnv>): Promise<EmoModel>;
|
|
11
|
+
/**
|
|
12
|
+
* Returns emoji suggestions for a phrase, most likely first. The model is loaded
|
|
13
|
+
* (and cached) lazily on first call. Empty input returns `[]`.
|
|
14
|
+
*/
|
|
15
|
+
export declare function suggestions(text: string, limit?: number, options?: EmoSuggestionOptions): Promise<EmoSuggestion[]>;
|
|
16
|
+
/** Clears the memoized model so the next {@link suggestions} call re-reads `env`. */
|
|
17
|
+
export declare function reset(): void;
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
import { homedir } from "node:os";
|
|
2
|
+
import { join } from "node:path";
|
|
3
|
+
import { fsCache, localReader } from "./cache-node.js";
|
|
4
|
+
import { DEFAULT_HOST, DEFAULT_REPO, DEFAULT_REVISION, loadModel } from "./hub.js";
|
|
5
|
+
export { createEmo, EmoModel } from "./model.js";
|
|
6
|
+
export { applyEmojiSkinTone } from "./skin-tone.js";
|
|
7
|
+
export { loadModel } from "./hub.js";
|
|
8
|
+
/** Loading configuration. Mutate before the first call, or pass overrides to {@link load}. */
|
|
9
|
+
export const env = {
|
|
10
|
+
host: DEFAULT_HOST,
|
|
11
|
+
repo: DEFAULT_REPO,
|
|
12
|
+
revision: DEFAULT_REVISION,
|
|
13
|
+
allowRemote: true,
|
|
14
|
+
useCache: true,
|
|
15
|
+
cacheDir: process.env.EMO_CACHE_DIR ?? join(homedir(), ".cache", "emo"),
|
|
16
|
+
localModelPath: process.env.EMO_LOCAL_PATH,
|
|
17
|
+
token: process.env.HF_TOKEN ?? process.env.HUGGING_FACE_HUB_TOKEN,
|
|
18
|
+
};
|
|
19
|
+
/** Loads the model: a local dir if configured, else the Hugging Face Hub (cached to disk). */
|
|
20
|
+
export async function load(options = {}) {
|
|
21
|
+
const e = { ...env, ...options };
|
|
22
|
+
const cache = e.useCache && e.cacheDir ? fsCache(e.cacheDir) : null;
|
|
23
|
+
const readLocal = e.localModelPath ? localReader(e.localModelPath) : undefined;
|
|
24
|
+
return loadModel(e, cache, readLocal);
|
|
25
|
+
}
|
|
26
|
+
let modelPromise = null;
|
|
27
|
+
/**
|
|
28
|
+
* Returns emoji suggestions for a phrase, most likely first. The model is loaded
|
|
29
|
+
* (and cached) lazily on first call. Empty input returns `[]`.
|
|
30
|
+
*/
|
|
31
|
+
export async function suggestions(text, limit = 3, options = {}) {
|
|
32
|
+
if (!modelPromise) {
|
|
33
|
+
// Don't cache a rejected load: clear the slot on failure so the next call retries.
|
|
34
|
+
modelPromise = load().catch((err) => {
|
|
35
|
+
modelPromise = null;
|
|
36
|
+
throw err;
|
|
37
|
+
});
|
|
38
|
+
}
|
|
39
|
+
return (await modelPromise).suggestions(text, limit, options);
|
|
40
|
+
}
|
|
41
|
+
/** Clears the memoized model so the next {@link suggestions} call re-reads `env`. */
|
|
42
|
+
export function reset() {
|
|
43
|
+
modelPromise = null;
|
|
44
|
+
}
|
package/dist/model.d.ts
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
import { type EmoSuggestionOptions } from "./skin-tone.js";
|
|
2
|
+
export { applyEmojiSkinTone, type EmojiSkinTone, type EmoSuggestionOptions } from "./skin-tone.js";
|
|
3
|
+
/** A single emoji suggestion. */
|
|
4
|
+
export interface EmoSuggestion {
|
|
5
|
+
/** The suggested emoji. */
|
|
6
|
+
emoji: string;
|
|
7
|
+
/** The model's normalized confidence, from `0` to `1`. */
|
|
8
|
+
confidence: number;
|
|
9
|
+
}
|
|
10
|
+
export interface EmoMeta {
|
|
11
|
+
labels: string[];
|
|
12
|
+
n_hashes: number;
|
|
13
|
+
n_buckets: number;
|
|
14
|
+
n_importance: number;
|
|
15
|
+
sem_dim: number;
|
|
16
|
+
sem_pad_index: number;
|
|
17
|
+
}
|
|
18
|
+
/** Loads the bundled model and runs emoji inference. Construct once and reuse. */
|
|
19
|
+
export declare class EmoModel {
|
|
20
|
+
private readonly meta;
|
|
21
|
+
private readonly tok;
|
|
22
|
+
private readonly w;
|
|
23
|
+
private readonly maxLen;
|
|
24
|
+
constructor(weights: Uint8Array, tokenizer: Uint8Array, meta: EmoMeta);
|
|
25
|
+
/** Returns up to `limit` emoji suggestions, most likely first. */
|
|
26
|
+
suggestions(text: string, limit?: number, options?: EmoSuggestionOptions): EmoSuggestion[];
|
|
27
|
+
}
|
|
28
|
+
/** Creates an {@link EmoModel} from raw buffers (lowest-level entry). */
|
|
29
|
+
export declare function createEmo(buffers: {
|
|
30
|
+
weights: Uint8Array;
|
|
31
|
+
tokenizer: Uint8Array;
|
|
32
|
+
meta: EmoMeta;
|
|
33
|
+
}): EmoModel;
|
package/dist/model.js
ADDED
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
import { applyEmojiSkinTone } from "./skin-tone.js";
|
|
2
|
+
import { ngramEncode, SemTokenizer } from "./tokenizer.js";
|
|
3
|
+
export { applyEmojiSkinTone } from "./skin-tone.js";
|
|
4
|
+
// A weight tensor, either raw F32 or a 4-bit k-means palette kept packed in memory
|
|
5
|
+
// (a U8 index per weight, 2 per byte) and decoded on access — ~8x less memory than
|
|
6
|
+
// expanding to floats, for ~0.1ms more per inference.
|
|
7
|
+
class Tensor {
|
|
8
|
+
rows;
|
|
9
|
+
cols;
|
|
10
|
+
floats;
|
|
11
|
+
packed;
|
|
12
|
+
palette;
|
|
13
|
+
constructor(rows, cols, floats, packed, palette) {
|
|
14
|
+
this.rows = rows;
|
|
15
|
+
this.cols = cols;
|
|
16
|
+
this.floats = floats;
|
|
17
|
+
this.packed = packed;
|
|
18
|
+
this.palette = palette;
|
|
19
|
+
}
|
|
20
|
+
get(i) {
|
|
21
|
+
const f = this.floats;
|
|
22
|
+
if (f !== null)
|
|
23
|
+
return f[i];
|
|
24
|
+
const byte = this.packed[i >> 1];
|
|
25
|
+
return this.palette[(i & 1) ? (byte >> 4) & 0xf : byte & 0xf];
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
function erf(x) {
|
|
29
|
+
const t = 1 / (1 + 0.3275911 * Math.abs(x));
|
|
30
|
+
const y = 1 - (((((1.061405429 * t - 1.453152027) * t) + 1.421413741) * t - 0.284496736) * t + 0.254829592) * t * Math.exp(-x * x);
|
|
31
|
+
return x >= 0 ? y : -y;
|
|
32
|
+
}
|
|
33
|
+
const gelu = (x) => 0.5 * x * (1 + erf(x / Math.SQRT2));
|
|
34
|
+
// Minimal safetensors reader: u64 header length, JSON header, then raw tensor bytes.
|
|
35
|
+
// Each weight is either raw F32, or a 4-bit k-means palette stored as a packed U8 index
|
|
36
|
+
// tensor plus a "<name>.palette" F32 tensor (logical 2-D shape kept in __metadata__).
|
|
37
|
+
// Palette weights stay packed and are decoded on access (see Tensor).
|
|
38
|
+
function parseWeights(bytes) {
|
|
39
|
+
const dv = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength);
|
|
40
|
+
const headerLen = Number(dv.getBigUint64(0, true));
|
|
41
|
+
const header = JSON.parse(new TextDecoder().decode(bytes.subarray(8, 8 + headerLen)));
|
|
42
|
+
const dataStart = 8 + headerLen;
|
|
43
|
+
const meta = header.__metadata__ ?? {};
|
|
44
|
+
const slice = (name) => {
|
|
45
|
+
const e = header[name];
|
|
46
|
+
if (!e)
|
|
47
|
+
throw new Error(`emo: missing tensor ${name}`);
|
|
48
|
+
const [a, b] = e.data_offsets;
|
|
49
|
+
const out = new Uint8Array(b - a);
|
|
50
|
+
out.set(bytes.subarray(dataStart + a, dataStart + b));
|
|
51
|
+
return { shape: e.shape, bytes: out };
|
|
52
|
+
};
|
|
53
|
+
const f32 = (name) => new Float32Array(slice(name).bytes.buffer);
|
|
54
|
+
const expand = (name) => {
|
|
55
|
+
if (header[name + ".palette"]) {
|
|
56
|
+
const [rows, cols] = meta["shape." + name].split(",").map(Number);
|
|
57
|
+
return new Tensor(rows, cols, null, slice(name).bytes, f32(name + ".palette"));
|
|
58
|
+
}
|
|
59
|
+
const [rows, cols = 1] = slice(name).shape;
|
|
60
|
+
return new Tensor(rows, cols, f32(name), null, null);
|
|
61
|
+
};
|
|
62
|
+
return {
|
|
63
|
+
embed: expand("embed"), sem: expand("sem"), importance: expand("importance"),
|
|
64
|
+
w1: expand("w1"), b1: expand("b1"), w2: expand("w2"), b2: expand("b2"),
|
|
65
|
+
};
|
|
66
|
+
}
|
|
67
|
+
/** Loads the bundled model and runs emoji inference. Construct once and reuse. */
|
|
68
|
+
export class EmoModel {
|
|
69
|
+
meta;
|
|
70
|
+
tok;
|
|
71
|
+
w;
|
|
72
|
+
maxLen = 1024;
|
|
73
|
+
constructor(weights, tokenizer, meta) {
|
|
74
|
+
this.meta = meta;
|
|
75
|
+
this.w = parseWeights(weights);
|
|
76
|
+
this.tok = new SemTokenizer(tokenizer);
|
|
77
|
+
}
|
|
78
|
+
/** Returns up to `limit` emoji suggestions, most likely first. */
|
|
79
|
+
suggestions(text, limit = 3, options = {}) {
|
|
80
|
+
const trimmed = text.trim();
|
|
81
|
+
if (!trimmed)
|
|
82
|
+
return [];
|
|
83
|
+
const { embed, sem, importance, w1, b1, w2, b2 } = this.w;
|
|
84
|
+
const ngDim = embed.cols;
|
|
85
|
+
const semDim = sem.cols;
|
|
86
|
+
const ng = new Float32Array(ngDim);
|
|
87
|
+
const f = ngramEncode(trimmed, this.meta.n_buckets, this.meta.n_hashes, this.meta.n_importance, this.maxLen);
|
|
88
|
+
for (let i = 0; i < f.buckets.length; i++) {
|
|
89
|
+
const im = f.importance[i];
|
|
90
|
+
for (let k = 0; k < this.meta.n_hashes; k++) {
|
|
91
|
+
const w = importance.get(im * importance.cols + k) * f.signs[i][k];
|
|
92
|
+
const base = f.buckets[i][k] * ngDim;
|
|
93
|
+
for (let c = 0; c < ngDim; c++)
|
|
94
|
+
ng[c] += w * embed.get(base + c);
|
|
95
|
+
}
|
|
96
|
+
}
|
|
97
|
+
const fcount = Math.max(1, f.buckets.length);
|
|
98
|
+
for (let c = 0; c < ngDim; c++)
|
|
99
|
+
ng[c] /= fcount;
|
|
100
|
+
let ids = this.tok.encode(trimmed);
|
|
101
|
+
if (ids.length > this.maxLen)
|
|
102
|
+
ids = ids.slice(0, this.maxLen);
|
|
103
|
+
if (ids.length === 0)
|
|
104
|
+
ids = [this.meta.sem_pad_index];
|
|
105
|
+
const sv = new Float32Array(semDim);
|
|
106
|
+
for (const id of ids) {
|
|
107
|
+
if (id >= sem.rows)
|
|
108
|
+
continue;
|
|
109
|
+
const base = id * semDim;
|
|
110
|
+
for (let c = 0; c < semDim; c++)
|
|
111
|
+
sv[c] += sem.get(base + c);
|
|
112
|
+
}
|
|
113
|
+
for (let c = 0; c < semDim; c++)
|
|
114
|
+
sv[c] /= ids.length;
|
|
115
|
+
let norm = 0;
|
|
116
|
+
for (let c = 0; c < semDim; c++)
|
|
117
|
+
norm += sv[c] * sv[c];
|
|
118
|
+
norm = Math.sqrt(norm) + 1e-9;
|
|
119
|
+
for (let c = 0; c < semDim; c++)
|
|
120
|
+
sv[c] /= norm;
|
|
121
|
+
const inDim = ngDim + semDim;
|
|
122
|
+
const x = new Float32Array(inDim);
|
|
123
|
+
x.set(ng, 0);
|
|
124
|
+
x.set(sv, ngDim);
|
|
125
|
+
const hid = w1.rows;
|
|
126
|
+
const h = new Float32Array(hid);
|
|
127
|
+
for (let o = 0; o < hid; o++) {
|
|
128
|
+
let acc = b1.get(o);
|
|
129
|
+
const base = o * inDim;
|
|
130
|
+
for (let i = 0; i < inDim; i++)
|
|
131
|
+
acc += w1.get(base + i) * x[i];
|
|
132
|
+
h[o] = gelu(acc);
|
|
133
|
+
}
|
|
134
|
+
const n = w2.rows;
|
|
135
|
+
const logits = new Float32Array(n);
|
|
136
|
+
let maxLogit = -Infinity;
|
|
137
|
+
for (let o = 0; o < n; o++) {
|
|
138
|
+
let acc = b2.get(o);
|
|
139
|
+
const base = o * hid;
|
|
140
|
+
for (let i = 0; i < hid; i++)
|
|
141
|
+
acc += w2.get(base + i) * h[i];
|
|
142
|
+
logits[o] = acc;
|
|
143
|
+
if (acc > maxLogit)
|
|
144
|
+
maxLogit = acc;
|
|
145
|
+
}
|
|
146
|
+
let sum = 0;
|
|
147
|
+
for (let o = 0; o < n; o++) {
|
|
148
|
+
logits[o] = Math.exp(logits[o] - maxLogit);
|
|
149
|
+
sum += logits[o];
|
|
150
|
+
}
|
|
151
|
+
const labels = this.meta.labels;
|
|
152
|
+
const order = Array.from({ length: n }, (_, i) => i).sort((a, b) => logits[b] - logits[a]);
|
|
153
|
+
const skinTone = options.skinTone ?? "default";
|
|
154
|
+
return order.slice(0, Math.max(0, limit)).map((i) => ({
|
|
155
|
+
emoji: applyEmojiSkinTone(labels[i], skinTone),
|
|
156
|
+
confidence: logits[i] / sum,
|
|
157
|
+
}));
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
/** Creates an {@link EmoModel} from raw buffers (lowest-level entry). */
|
|
161
|
+
export function createEmo(buffers) {
|
|
162
|
+
return new EmoModel(buffers.weights, buffers.tokenizer, buffers.meta);
|
|
163
|
+
}
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
/** Preferred emoji skin tone variant for skin-tone-capable emoji. */
|
|
2
|
+
export type EmojiSkinTone = "default" | "light" | "mediumLight" | "medium" | "mediumDark" | "dark";
|
|
3
|
+
/** Options for emoji suggestion post-processing. */
|
|
4
|
+
export interface EmoSuggestionOptions {
|
|
5
|
+
/** Preferred skin tone for skin-tone-capable emoji. Defaults to `"default"` (no modifier). */
|
|
6
|
+
skinTone?: EmojiSkinTone;
|
|
7
|
+
}
|
|
8
|
+
export declare function applyEmojiSkinTone(emoji: string, skinTone?: EmojiSkinTone): string;
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
const modifiers = {
|
|
2
|
+
light: "🏻",
|
|
3
|
+
mediumLight: "🏼",
|
|
4
|
+
medium: "🏽",
|
|
5
|
+
mediumDark: "🏾",
|
|
6
|
+
dark: "🏿",
|
|
7
|
+
};
|
|
8
|
+
const isModifier = (cp) => cp >= 0x1f3fb && cp <= 0x1f3ff;
|
|
9
|
+
const isVariationSelector16 = (cp) => cp === 0xfe0f;
|
|
10
|
+
function isEmojiModifierBase(cp) {
|
|
11
|
+
return cp === 0x261d || cp === 0x26f9 ||
|
|
12
|
+
(cp >= 0x270a && cp <= 0x270d) || cp === 0x1f385 ||
|
|
13
|
+
(cp >= 0x1f3c2 && cp <= 0x1f3c4) || cp === 0x1f3c7 ||
|
|
14
|
+
(cp >= 0x1f3ca && cp <= 0x1f3cc) || (cp >= 0x1f442 && cp <= 0x1f443) ||
|
|
15
|
+
(cp >= 0x1f446 && cp <= 0x1f450) || (cp >= 0x1f466 && cp <= 0x1f478) ||
|
|
16
|
+
cp === 0x1f47c || (cp >= 0x1f481 && cp <= 0x1f483) ||
|
|
17
|
+
(cp >= 0x1f485 && cp <= 0x1f487) || cp === 0x1f48f || cp === 0x1f491 ||
|
|
18
|
+
cp === 0x1f4aa || (cp >= 0x1f574 && cp <= 0x1f575) || cp === 0x1f57a ||
|
|
19
|
+
cp === 0x1f590 || (cp >= 0x1f595 && cp <= 0x1f596) ||
|
|
20
|
+
(cp >= 0x1f645 && cp <= 0x1f647) || (cp >= 0x1f64b && cp <= 0x1f64f) ||
|
|
21
|
+
cp === 0x1f6a3 || (cp >= 0x1f6b4 && cp <= 0x1f6b6) || cp === 0x1f6c0 ||
|
|
22
|
+
cp === 0x1f6cc || cp === 0x1f90c || (cp >= 0x1f918 && cp <= 0x1f91f) ||
|
|
23
|
+
cp === 0x1f926 || (cp >= 0x1f930 && cp <= 0x1f939) ||
|
|
24
|
+
(cp >= 0x1f93d && cp <= 0x1f93e) || cp === 0x1f977 ||
|
|
25
|
+
(cp >= 0x1f9b5 && cp <= 0x1f9b6) || (cp >= 0x1f9b8 && cp <= 0x1f9b9) ||
|
|
26
|
+
(cp >= 0x1f9cd && cp <= 0x1f9cf) || (cp >= 0x1f9d1 && cp <= 0x1f9dd) ||
|
|
27
|
+
(cp >= 0x1faf0 && cp <= 0x1faf8);
|
|
28
|
+
}
|
|
29
|
+
export function applyEmojiSkinTone(emoji, skinTone = "default") {
|
|
30
|
+
if (skinTone === "default")
|
|
31
|
+
return emoji;
|
|
32
|
+
const modifier = modifiers[skinTone];
|
|
33
|
+
const chars = Array.from(emoji);
|
|
34
|
+
let out = "";
|
|
35
|
+
for (let i = 0; i < chars.length; i++) {
|
|
36
|
+
const cp = chars[i].codePointAt(0);
|
|
37
|
+
if (isModifier(cp))
|
|
38
|
+
continue;
|
|
39
|
+
out += chars[i];
|
|
40
|
+
if (isEmojiModifierBase(cp)) {
|
|
41
|
+
out += modifier;
|
|
42
|
+
while (i + 1 < chars.length) {
|
|
43
|
+
const next = chars[i + 1].codePointAt(0);
|
|
44
|
+
if (!isModifier(next) && !isVariationSelector16(next))
|
|
45
|
+
break;
|
|
46
|
+
i++;
|
|
47
|
+
}
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
return out;
|
|
51
|
+
}
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
export interface NGramFeatures {
|
|
2
|
+
buckets: Int32Array[];
|
|
3
|
+
signs: Float32Array[];
|
|
4
|
+
importance: Int32Array;
|
|
5
|
+
}
|
|
6
|
+
export declare function ngramEncode(text: string, nBuckets: number, nHashes: number, nImportance: number, maxFeatures: number): NGramFeatures;
|
|
7
|
+
export declare class SemTokenizer {
|
|
8
|
+
private pieces;
|
|
9
|
+
private scores;
|
|
10
|
+
private index;
|
|
11
|
+
private unkId;
|
|
12
|
+
private unkScore;
|
|
13
|
+
private maxLen;
|
|
14
|
+
constructor(data: Uint8Array);
|
|
15
|
+
encode(text: string): number[];
|
|
16
|
+
}
|
|
@@ -0,0 +1,229 @@
|
|
|
1
|
+
const FNV_OFFSET = 0xcbf29ce484222325n;
|
|
2
|
+
const FNV_PRIME = 0x100000001b3n;
|
|
3
|
+
const MASK = (1n << 64n) - 1n;
|
|
4
|
+
function fnv64(s, seed) {
|
|
5
|
+
let h = (FNV_OFFSET ^ seed) & MASK;
|
|
6
|
+
for (const b of new TextEncoder().encode(s)) {
|
|
7
|
+
h = ((h ^ BigInt(b)) * FNV_PRIME) & MASK;
|
|
8
|
+
}
|
|
9
|
+
return h;
|
|
10
|
+
}
|
|
11
|
+
const BUCKET_SEEDS = [
|
|
12
|
+
0x9e3779b97f4a7c15n, 0xc2b2ae3d27d4eb4fn, 0x165667b19e3779f9n,
|
|
13
|
+
0x27d4eb2f165667c5n, 0x85ebca77c2b2ae63n,
|
|
14
|
+
];
|
|
15
|
+
const IMP_SEED = 0xff51afd7ed558ccdn;
|
|
16
|
+
const NA = [3, 5];
|
|
17
|
+
const NC = [1, 2];
|
|
18
|
+
const NJ = [2, 4];
|
|
19
|
+
const NS = [2, 4];
|
|
20
|
+
const NI = [1, 3];
|
|
21
|
+
const reWord = /\p{L}|\p{M}|\p{N}/u;
|
|
22
|
+
const reMark = /\p{M}/u;
|
|
23
|
+
const cp = (s) => s.codePointAt(0);
|
|
24
|
+
const isHangul = (s) => cp(s) >= 0xac00 && cp(s) <= 0xd7a3;
|
|
25
|
+
function isCJK(s) {
|
|
26
|
+
const v = cp(s);
|
|
27
|
+
return (v >= 0x4e00 && v <= 0x9fff) || (v >= 0x3400 && v <= 0x4dbf)
|
|
28
|
+
|| (v >= 0x20000 && v <= 0x2a6df) || (v >= 0xf900 && v <= 0xfaff)
|
|
29
|
+
|| (v >= 0x3040 && v <= 0x30ff) || (v >= 0x31f0 && v <= 0x31ff)
|
|
30
|
+
|| isHangul(s);
|
|
31
|
+
}
|
|
32
|
+
function isSEA(s) {
|
|
33
|
+
const v = cp(s);
|
|
34
|
+
return (v >= 0x0e00 && v <= 0x0eff) || (v >= 0x1000 && v <= 0x109f) || (v >= 0x1780 && v <= 0x17ff);
|
|
35
|
+
}
|
|
36
|
+
const isIndic = (s) => cp(s) >= 0x0900 && cp(s) <= 0x0dff;
|
|
37
|
+
function jamo(c) {
|
|
38
|
+
if (!isHangul(c))
|
|
39
|
+
return [c];
|
|
40
|
+
const s = cp(c) - 0xac00;
|
|
41
|
+
const r = [String.fromCodePoint(0x1100 + Math.floor(s / 588)), String.fromCodePoint(0x1161 + Math.floor((s % 588) / 28))];
|
|
42
|
+
if (s % 28 !== 0)
|
|
43
|
+
r.push(String.fromCodePoint(0x11a7 + (s % 28)));
|
|
44
|
+
return r;
|
|
45
|
+
}
|
|
46
|
+
function normalize(text) {
|
|
47
|
+
return text.normalize("NFKC").toLowerCase().split(/\s+/).filter(Boolean).join(" ");
|
|
48
|
+
}
|
|
49
|
+
function tokens(text) {
|
|
50
|
+
const out = [];
|
|
51
|
+
let cur = [];
|
|
52
|
+
for (const c of text) {
|
|
53
|
+
if (reWord.test(c))
|
|
54
|
+
cur.push(c);
|
|
55
|
+
else if (cur.length) {
|
|
56
|
+
out.push(cur);
|
|
57
|
+
cur = [];
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
if (cur.length)
|
|
61
|
+
out.push(cur);
|
|
62
|
+
return out;
|
|
63
|
+
}
|
|
64
|
+
function clusters(s) {
|
|
65
|
+
const out = [];
|
|
66
|
+
let cur = [];
|
|
67
|
+
for (const c of s) {
|
|
68
|
+
if (!cur.length) {
|
|
69
|
+
cur = [c];
|
|
70
|
+
continue;
|
|
71
|
+
}
|
|
72
|
+
const p = cp(cur[cur.length - 1]);
|
|
73
|
+
const vir = p >= 0x0900 && p <= 0x0dff && ((p & 0xff) === 0x4d || (p & 0xff) === 0xcd);
|
|
74
|
+
if (reMark.test(c) || vir)
|
|
75
|
+
cur.push(c);
|
|
76
|
+
else {
|
|
77
|
+
out.push(cur);
|
|
78
|
+
cur = [c];
|
|
79
|
+
}
|
|
80
|
+
}
|
|
81
|
+
if (cur.length)
|
|
82
|
+
out.push(cur);
|
|
83
|
+
return out;
|
|
84
|
+
}
|
|
85
|
+
function charGrams(s, lo, hi, tag) {
|
|
86
|
+
const r = [];
|
|
87
|
+
for (let n = lo; n <= hi; n++) {
|
|
88
|
+
for (let i = 0; i + n <= s.length; i++)
|
|
89
|
+
r.push(tag + s.slice(i, i + n).join(""));
|
|
90
|
+
}
|
|
91
|
+
return r;
|
|
92
|
+
}
|
|
93
|
+
function clusterGrams(cl, lo, hi, tag) {
|
|
94
|
+
const r = [];
|
|
95
|
+
for (let n = lo; n <= hi; n++) {
|
|
96
|
+
for (let i = 0; i + n <= cl.length; i++)
|
|
97
|
+
r.push(tag + cl.slice(i, i + n).flat().join(""));
|
|
98
|
+
}
|
|
99
|
+
return r;
|
|
100
|
+
}
|
|
101
|
+
function feats(text) {
|
|
102
|
+
const out = [];
|
|
103
|
+
for (const run of tokens(normalize(text))) {
|
|
104
|
+
if (run.some(isSEA)) {
|
|
105
|
+
out.push(...charGrams(run, NS[0], NS[1], "s:"));
|
|
106
|
+
}
|
|
107
|
+
else if (run.some(isIndic)) {
|
|
108
|
+
const cl = clusters(run);
|
|
109
|
+
out.push("a:" + run.join(""));
|
|
110
|
+
out.push(...clusterGrams(cl, 1, 1, "k:"));
|
|
111
|
+
out.push(...clusterGrams([["<"], ...cl, [">"]], 2, NI[1], "k:"));
|
|
112
|
+
}
|
|
113
|
+
else if (run.some(isCJK)) {
|
|
114
|
+
const ex = [];
|
|
115
|
+
for (const c of run) {
|
|
116
|
+
if (isHangul(c))
|
|
117
|
+
out.push(...charGrams(jamo(c), NJ[0], NJ[1], "j:"));
|
|
118
|
+
ex.push(c);
|
|
119
|
+
}
|
|
120
|
+
out.push(...charGrams(ex, NC[0], NC[1], "c:"));
|
|
121
|
+
}
|
|
122
|
+
else {
|
|
123
|
+
out.push("w:" + run.join(""));
|
|
124
|
+
out.push(...charGrams(["<", ...run, ">"], NA[0], NA[1], "g:"));
|
|
125
|
+
}
|
|
126
|
+
}
|
|
127
|
+
return out.length ? out : ["w:\u0000"];
|
|
128
|
+
}
|
|
129
|
+
export function ngramEncode(text, nBuckets, nHashes, nImportance, maxFeatures) {
|
|
130
|
+
let fs = feats(text);
|
|
131
|
+
if (fs.length > maxFeatures)
|
|
132
|
+
fs = fs.slice(0, maxFeatures);
|
|
133
|
+
const buckets = [];
|
|
134
|
+
const signs = [];
|
|
135
|
+
const importance = new Int32Array(fs.length);
|
|
136
|
+
const nb = BigInt(nBuckets);
|
|
137
|
+
const ni = BigInt(nImportance);
|
|
138
|
+
for (let f = 0; f < fs.length; f++) {
|
|
139
|
+
const bk = new Int32Array(nHashes);
|
|
140
|
+
const sg = new Float32Array(nHashes);
|
|
141
|
+
for (let k = 0; k < nHashes; k++) {
|
|
142
|
+
const h = fnv64(fs[f], BUCKET_SEEDS[k]);
|
|
143
|
+
bk[k] = Number(h % nb);
|
|
144
|
+
sg[k] = (h >> 63n) & 1n ? 1 : -1;
|
|
145
|
+
}
|
|
146
|
+
buckets.push(bk);
|
|
147
|
+
signs.push(sg);
|
|
148
|
+
importance[f] = Number(fnv64(fs[f], IMP_SEED) % ni);
|
|
149
|
+
}
|
|
150
|
+
return { buckets, signs, importance };
|
|
151
|
+
}
|
|
152
|
+
export class SemTokenizer {
|
|
153
|
+
pieces = [];
|
|
154
|
+
scores;
|
|
155
|
+
index = new Map();
|
|
156
|
+
unkId;
|
|
157
|
+
unkScore;
|
|
158
|
+
maxLen = 1;
|
|
159
|
+
constructor(data) {
|
|
160
|
+
const dv = new DataView(data.buffer, data.byteOffset, data.byteLength);
|
|
161
|
+
if (data[0] !== 0x45 || data[1] !== 0x4d || data[2] !== 0x54 || data[3] !== 0x4b) {
|
|
162
|
+
throw new Error("bad tokenizer file");
|
|
163
|
+
}
|
|
164
|
+
this.unkId = dv.getInt32(6, true);
|
|
165
|
+
const k = dv.getUint32(10, true);
|
|
166
|
+
let off = 14;
|
|
167
|
+
this.scores = new Float32Array(k);
|
|
168
|
+
for (let i = 0; i < k; i++) {
|
|
169
|
+
this.scores[i] = dv.getFloat32(off, true);
|
|
170
|
+
off += 4;
|
|
171
|
+
}
|
|
172
|
+
const lens = new Array(k);
|
|
173
|
+
for (let i = 0; i < k; i++) {
|
|
174
|
+
lens[i] = dv.getUint16(off, true);
|
|
175
|
+
off += 2;
|
|
176
|
+
}
|
|
177
|
+
const dec = new TextDecoder();
|
|
178
|
+
for (let i = 0; i < k; i++) {
|
|
179
|
+
const piece = dec.decode(data.subarray(off, off + lens[i]));
|
|
180
|
+
off += lens[i];
|
|
181
|
+
this.pieces.push(piece);
|
|
182
|
+
this.index.set(piece, i);
|
|
183
|
+
const n = [...piece].length;
|
|
184
|
+
if (n > this.maxLen)
|
|
185
|
+
this.maxLen = n;
|
|
186
|
+
}
|
|
187
|
+
this.maxLen = Math.min(this.maxLen, 24);
|
|
188
|
+
this.unkScore = this.scores[this.unkId];
|
|
189
|
+
}
|
|
190
|
+
encode(text) {
|
|
191
|
+
const norm = "\u2581" + text.toLowerCase().normalize("NFKC").replace(/ /g, "\u2581");
|
|
192
|
+
const s = [...norm];
|
|
193
|
+
const n = s.length;
|
|
194
|
+
if (n === 0)
|
|
195
|
+
return [];
|
|
196
|
+
const NEG = -1e18;
|
|
197
|
+
const best = new Float64Array(n + 1).fill(NEG);
|
|
198
|
+
best[0] = 0;
|
|
199
|
+
const backPos = new Int32Array(n + 1).fill(-1);
|
|
200
|
+
const backId = new Int32Array(n + 1).fill(-1);
|
|
201
|
+
for (let i = 1; i <= n; i++) {
|
|
202
|
+
const lo = Math.max(0, i - this.maxLen);
|
|
203
|
+
for (let j = lo; j < i; j++) {
|
|
204
|
+
const tid = this.index.get(s.slice(j, i).join(""));
|
|
205
|
+
if (tid !== undefined) {
|
|
206
|
+
const sc = best[j] + this.scores[tid];
|
|
207
|
+
if (sc > best[i]) {
|
|
208
|
+
best[i] = sc;
|
|
209
|
+
backPos[i] = j;
|
|
210
|
+
backId[i] = tid;
|
|
211
|
+
}
|
|
212
|
+
}
|
|
213
|
+
}
|
|
214
|
+
const cand = best[i - 1] + this.unkScore;
|
|
215
|
+
if (cand > best[i]) {
|
|
216
|
+
best[i] = cand;
|
|
217
|
+
backPos[i] = i - 1;
|
|
218
|
+
backId[i] = this.unkId;
|
|
219
|
+
}
|
|
220
|
+
}
|
|
221
|
+
const ids = [];
|
|
222
|
+
let i = n;
|
|
223
|
+
while (i > 0) {
|
|
224
|
+
ids.push(backId[i]);
|
|
225
|
+
i = backPos[i];
|
|
226
|
+
}
|
|
227
|
+
return ids.reverse();
|
|
228
|
+
}
|
|
229
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@desert-ant-labs/emo",
|
|
3
|
+
"version": "0.3.0",
|
|
4
|
+
"description": "On-device emoji suggestions from text.",
|
|
5
|
+
"type": "module",
|
|
6
|
+
"main": "./dist/index.node.js",
|
|
7
|
+
"types": "./dist/index.node.d.ts",
|
|
8
|
+
"exports": {
|
|
9
|
+
".": {
|
|
10
|
+
"browser": {
|
|
11
|
+
"types": "./dist/index.browser.d.ts",
|
|
12
|
+
"default": "./dist/index.browser.js"
|
|
13
|
+
},
|
|
14
|
+
"worker": {
|
|
15
|
+
"types": "./dist/index.browser.d.ts",
|
|
16
|
+
"default": "./dist/index.browser.js"
|
|
17
|
+
},
|
|
18
|
+
"workerd": {
|
|
19
|
+
"types": "./dist/index.browser.d.ts",
|
|
20
|
+
"default": "./dist/index.browser.js"
|
|
21
|
+
},
|
|
22
|
+
"edge-light": {
|
|
23
|
+
"types": "./dist/index.browser.d.ts",
|
|
24
|
+
"default": "./dist/index.browser.js"
|
|
25
|
+
},
|
|
26
|
+
"node": {
|
|
27
|
+
"types": "./dist/index.node.d.ts",
|
|
28
|
+
"default": "./dist/index.node.js"
|
|
29
|
+
},
|
|
30
|
+
"default": {
|
|
31
|
+
"types": "./dist/index.node.d.ts",
|
|
32
|
+
"default": "./dist/index.node.js"
|
|
33
|
+
}
|
|
34
|
+
},
|
|
35
|
+
"./core": {
|
|
36
|
+
"types": "./dist/model.d.ts",
|
|
37
|
+
"default": "./dist/model.js"
|
|
38
|
+
},
|
|
39
|
+
"./package.json": "./package.json"
|
|
40
|
+
},
|
|
41
|
+
"files": [
|
|
42
|
+
"dist"
|
|
43
|
+
],
|
|
44
|
+
"scripts": {
|
|
45
|
+
"build": "tsc",
|
|
46
|
+
"test": "node --import tsx --test test/*.test.ts",
|
|
47
|
+
"lint:publint": "publint --strict",
|
|
48
|
+
"lint:attw": "attw --pack . --profile esm-only",
|
|
49
|
+
"check:pkg": "npm run build && npm run lint:publint && npm run lint:attw",
|
|
50
|
+
"prepare": "npm run build"
|
|
51
|
+
},
|
|
52
|
+
"keywords": [
|
|
53
|
+
"emoji",
|
|
54
|
+
"on-device",
|
|
55
|
+
"multilingual",
|
|
56
|
+
"text-classification"
|
|
57
|
+
],
|
|
58
|
+
"license": "SEE LICENSE IN LICENSE.md",
|
|
59
|
+
"repository": {
|
|
60
|
+
"type": "git",
|
|
61
|
+
"url": "git+https://github.com/Desert-Ant-Labs/emo-js.git"
|
|
62
|
+
},
|
|
63
|
+
"homepage": "https://github.com/Desert-Ant-Labs/emo-js",
|
|
64
|
+
"engines": {
|
|
65
|
+
"node": ">=18"
|
|
66
|
+
},
|
|
67
|
+
"sideEffects": false,
|
|
68
|
+
"publishConfig": {
|
|
69
|
+
"access": "public"
|
|
70
|
+
},
|
|
71
|
+
"devDependencies": {
|
|
72
|
+
"@arethetypeswrong/cli": "^0.18",
|
|
73
|
+
"@types/node": "^22",
|
|
74
|
+
"publint": "^0.3",
|
|
75
|
+
"tsx": "^4",
|
|
76
|
+
"typescript": "^5"
|
|
77
|
+
}
|
|
78
|
+
}
|