@cyberlangke/tokkit-microsoft 1.1.0

package/COPYRIGHT

@@ -0,0 +1,15 @@
+Copyright (c) 2026 cyberlangke
+
+This package bundles tokenizer assets derived from official Hugging Face repositories under the microsoft organization:
+- microsoft/phi-1 tokenizer.json
+- microsoft/Phi-3-mini-4k-instruct tokenizer.json
+- microsoft/Phi-3-medium-4k-instruct tokenizer.json
+- microsoft/Phi-3.5-mini-instruct tokenizer.json
+- microsoft/Phi-mini-MoE-instruct tokenizer.json
+- microsoft/phi-4 tokenizer.json
+- microsoft/Phi-4-mini-instruct tokenizer.json
+- microsoft/Phi-4-mini-reasoning tokenizer.json
+- microsoft/Phi-4-mini-flash-reasoning tokenizer.json
+- microsoft/Phi-4-reasoning tokenizer.json
+
+See LICENSE for the full license text that applies to this package and its bundled assets.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 cyberlangke
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,21 @@
+# @cyberlangke/tokkit-microsoft
+
+Microsoft 官方 tokenizer 的 tokkit 子包，当前包含 Phi 主线内置 family。
+
+## 支持的模型
+
+- [`microsoft/Phi-4-mini-instruct`](https://huggingface.co/microsoft/Phi-4-mini-instruct)
+
+## 使用方法
+
+```bash
+npm install @cyberlangke/tokkit-microsoft
+```
+
+```ts
+import { getTokenizer } from "@cyberlangke/tokkit-microsoft"
+
+const tokenizer = await getTokenizer("phi-4-mini")
+
+console.log(tokenizer.vocabSize)
+```