gpt-tokenizer 1.0.0 → 1.0.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +11 -1
- package/data/bpeRanks.json +1 -0
- package/data/encoder.json +1 -0
- package/data/vocab.bpe +50001 -0
- package/package.json +4 -3
package/README.md
CHANGED
|
@@ -1,4 +1,6 @@
|
|
|
1
|
-
#
|
|
1
|
+
# gpt-tokenizer
|
|
2
|
+
|
|
3
|
+
[](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14&hidenavigation=1&theme=dark)
|
|
2
4
|
|
|
3
5
|
`gpt-tokenizer` is a highly optimized Token Byte Pair Encoder/Decoder for GPT-2, GPT-3, GPT-3.5 and GPT-4 designed for JavaScript applications. OpenAI's GPT models utilize byte pair encoding to transform text into a sequence of integers before feeding them into the model. This package is a JavaScript implementation of OpenAI's original Python encoder/decoder, which can be found [here](https://github.com/openai/gpt-2).
|
|
4
6
|
|
|
@@ -28,6 +30,14 @@ As an UMD module:
|
|
|
28
30
|
<script src="https://unpkg.com/gpt-tokenizer" />
|
|
29
31
|
```
|
|
30
32
|
|
|
33
|
+
## Playground
|
|
34
|
+
|
|
35
|
+
You can play with the package in the browser using the [Playground](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14&hidenavigation=1&theme=dark).
|
|
36
|
+
|
|
37
|
+
[](https://codesandbox.io/s/gpt-tokenizer-tjcjoz?fontsize=14&hidenavigation=1&theme=dark)
|
|
38
|
+
|
|
39
|
+
The playground mimics the official [OpenAI Tokenizer](https://platform.openai.com/tokenizer).
|
|
40
|
+
|
|
31
41
|
## Usage
|
|
32
42
|
|
|
33
43
|
```typescript
|