tokenometer 0.0.5 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +138 -17
- package/dist/args.d.ts +15 -0
- package/dist/args.d.ts.map +1 -1
- package/dist/args.js +102 -9
- package/dist/args.js.map +1 -1
- package/dist/auto-detect.d.ts +19 -0
- package/dist/auto-detect.d.ts.map +1 -0
- package/dist/auto-detect.js +41 -0
- package/dist/auto-detect.js.map +1 -0
- package/dist/config-merge.d.ts +18 -0
- package/dist/config-merge.d.ts.map +1 -0
- package/dist/config-merge.js +49 -0
- package/dist/config-merge.js.map +1 -0
- package/dist/index.d.ts +12 -1
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +207 -34
- package/dist/index.js.map +1 -1
- package/dist/render.d.ts +6 -1
- package/dist/render.d.ts.map +1 -1
- package/dist/render.js +55 -7
- package/dist/render.js.map +1 -1
- package/dist/vision.d.ts +26 -0
- package/dist/vision.d.ts.map +1 -0
- package/dist/vision.js +44 -0
- package/dist/vision.js.map +1 -0
- package/package.json +36 -8
package/README.md
CHANGED
|
@@ -1,6 +1,11 @@
|
|
|
1
1
|
# tokenometer
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://www.npmjs.com/package/tokenometer)
|
|
4
|
+
[](https://github.com/faraa2m/tokenometer/blob/main/LICENSE)
|
|
5
|
+
|
|
6
|
+
> Empirical token-cost + latency benchmarking for LLM prompts. Tells you what your prompt actually costs and how fast each provider responds across Claude, GPT-4o, Gemini, Mistral, and Cohere — in every format.
|
|
7
|
+
|
|
8
|
+
See the [root README](https://github.com/faraa2m/tokenometer#readme) for findings, methodology, and the full project overview.
|
|
4
9
|
|
|
5
10
|
[**Live playground: tokenometer.vercel.app**](https://tokenometer.vercel.app) · [Source](https://github.com/faraa2m/tokenometer) · MIT
|
|
6
11
|
|
|
@@ -20,36 +25,152 @@ Cheapest: gpt-4o as json ($0.000192)
|
|
|
20
25
|
Priciest: claude-opus-4-7 as yaml ($0.001260, 6.74x more)
|
|
21
26
|
```
|
|
22
27
|
|
|
23
|
-
A leading `~` marks an approximate count (offline mode for Claude / Gemini, since
|
|
28
|
+
A leading `~` marks an approximate count (offline mode for Claude / Gemini / Mistral-Tekken / Cohere, since none of those vendors publishes a public production tokenizer that ships in JS).
|
|
29
|
+
|
|
30
|
+
## Flags
|
|
31
|
+
|
|
32
|
+
| Flag | Default | Notes |
|
|
33
|
+
|---|---|---|
|
|
34
|
+
| `--model <id[,id…]>` | `claude-opus-4-7` (or auto-detected) | Any registered model id (63 across 5 providers). |
|
|
35
|
+
| `--format <fmt[,fmt…]>` | `json,yaml,xml,markdown,text` | Subset of supported formats. |
|
|
36
|
+
| `--output <fmt>` | `table` | `table` \| `json` \| `sarif`. |
|
|
37
|
+
| `--by-file` | _off_ | Append a per-file token/USD table (multi-file only). |
|
|
38
|
+
| `--image <path>` | _none_ | Add vision-token cost for the image (repeatable). |
|
|
39
|
+
| `--config <path>` | _none_ | Load this exact config file (skips walk-up). |
|
|
40
|
+
| `--no-config` | _off_ | Skip `.tokenometer.yml` loading entirely. |
|
|
41
|
+
| `--empirical` | _off_ | Use provider `countTokens` APIs (free, exact). |
|
|
42
|
+
| `--latency` | _off_ | Measure real generation latency (TTFT, total ms, tokens/sec). Implies `--empirical`. |
|
|
43
|
+
| `--latency-trials <n>` | `3` | Trials per cell when `--latency` is set (1–10). |
|
|
44
|
+
| `--max-spend <usd>` | `0.05` (or `0.25` with `--latency`) | Hard ceiling for empirical / latency mode. |
|
|
45
|
+
| `--offline` | _off_ | Force offline path (overrides `--empirical`). |
|
|
46
|
+
| `-h`, `--help` | | Print help. |
|
|
47
|
+
| `-v`, `--version` | | Print version. |
|
|
48
|
+
|
|
49
|
+
```
|
|
50
|
+
tokenometer <file> [options]
|
|
51
|
+
echo "prompt" | tokenometer - [options]
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Models supported
|
|
55
|
+
|
|
56
|
+
63 models across 5 providers. Run `tokenometer --help` for the full list at runtime, or browse the [Cost Atlas](https://tokenometer.vercel.app/models) for sortable per-model pages.
|
|
57
|
+
|
|
58
|
+
| Provider | Examples | Offline tokenizer | Empirical |
|
|
59
|
+
|---|---|---|---|
|
|
60
|
+
| Anthropic | `claude-opus-4-7`, `claude-sonnet-4-6`, `claude-haiku-4-5`, Claude 3.x family | `gpt-tokenizer` `cl100k_base` (approximate) | `messages.countTokens` (free, exact) |
|
|
61
|
+
| OpenAI | `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-3.5-turbo`, `o1` family | `gpt-tokenizer` `o200k_base` (exact) | same `o200k_base` (matches production) |
|
|
62
|
+
| Google | `gemini-2.5-pro`, `gemini-2.5-flash`, `gemini-1.5-pro`, `gemini-1.5-flash` | `chars / 4` (approximate) | `model.countTokens` (free, exact) |
|
|
63
|
+
| Mistral (19 models) | `open-mistral-7b`, `open-mixtral-8x22b`, `mistral-large-latest`, `codestral-latest`, `mistral-nemo`, `pixtral-large-latest`, `mistral-medium-2505`, `magistral-small`, `ministral-3b-latest`, `devstral-small-2505` | `mistral-tokenizer-js` for SentencePiece V1/V2/V3 (exact); `chars/4` for Tekken (approximate) | unsupported (no public token-count API) |
|
|
64
|
+
| Cohere | `command-r`, `command-r-plus` | `chars / 4` (approximate) | `POST /v1/tokenize` (free, exact, requires `COHERE_API_KEY`) |
|
|
65
|
+
|
|
66
|
+
Pricing comes from the [`tokenlens`](https://www.npmjs.com/package/tokenlens) registry with a small set of local overrides for bleeding-edge models. Cohere pricing lives entirely in `LOCAL_OVERRIDES` because `@tokenlens/models` doesn't yet ship a Cohere catalog at v1.3.0.
|
|
24
67
|
|
|
25
68
|
## Empirical mode
|
|
26
69
|
|
|
27
|
-
For exact, vendor-billed counts on Claude and
|
|
70
|
+
For exact, vendor-billed counts on Claude, Gemini, and Cohere, set the right env var and pass `--empirical`. The tool calls each provider's free `countTokens`-equivalent endpoint — no charge.
|
|
28
71
|
|
|
29
72
|
```bash
|
|
30
|
-
ANTHROPIC_API_KEY=… GOOGLE_API_KEY=… \
|
|
31
|
-
npx tokenometer ./prompt.md --empirical
|
|
73
|
+
ANTHROPIC_API_KEY=… GOOGLE_API_KEY=… COHERE_API_KEY=… \
|
|
74
|
+
npx tokenometer ./prompt.md --empirical --model claude-opus-4-7,gemini-2.5-pro,command-r-plus
|
|
32
75
|
```
|
|
33
76
|
|
|
34
|
-
|
|
77
|
+
OpenAI's empirical path uses tiktoken `o200k_base` locally — that encoding matches OpenAI's production count exactly, so no API call is needed. Mistral has no public token-count endpoint; the offline `mistral-tokenizer-js` path is used regardless.
|
|
35
78
|
|
|
36
|
-
|
|
79
|
+
## Auto provider detection
|
|
37
80
|
|
|
38
|
-
|
|
81
|
+
When `--model` is omitted, tokenometer picks a default based on which provider key is set in your environment:
|
|
82
|
+
|
|
83
|
+
- `ANTHROPIC_API_KEY` only → `claude-opus-4-7`
|
|
84
|
+
- `OPENAI_API_KEY` only → `gpt-4o`
|
|
85
|
+
- `GOOGLE_API_KEY` / `GEMINI_API_KEY` only → first known `gemini-*` model (falls back to `gemini-2.5-pro`)
|
|
86
|
+
- `MISTRAL_API_KEY` only → first known `mistral-*` model
|
|
87
|
+
- `COHERE_API_KEY` only → `command-r-plus`
|
|
88
|
+
- Multiple keys set → falls back to `claude-opus-4-7` and prints a stderr note. Pass `--model` to disambiguate.
|
|
89
|
+
- No keys set → existing default (`claude-opus-4-7`).
|
|
90
|
+
|
|
91
|
+
This means `npx tokenometer prompt.md` does the right thing in any of those environments without you having to remember model names.
|
|
92
|
+
|
|
93
|
+
## `.tokenometer.yml` config
|
|
94
|
+
|
|
95
|
+
Drop a `.tokenometer.yml` (or `.yaml`) at the project root and tokenometer will pick it up automatically (walks up from the cwd, stopping at `.git`):
|
|
39
96
|
|
|
97
|
+
```yaml
|
|
98
|
+
models: [claude-opus-4-7, gpt-4o, mistral-large-latest]
|
|
99
|
+
formats: [json, yaml, markdown]
|
|
100
|
+
paths: [prompts/**/*.md]
|
|
101
|
+
budgets:
|
|
102
|
+
total: 0.50
|
|
103
|
+
per-file: 0.10
|
|
40
104
|
```
|
|
41
|
-
tokenometer <file> [options]
|
|
42
|
-
echo "prompt" | tokenometer - [options]
|
|
43
105
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
-
|
|
106
|
+
User-passed CLI flags always win over config defaults. Use `--config <path>` to load an explicit file (skips the walk-up). Use `--no-config` to skip config loading entirely.
|
|
107
|
+
|
|
108
|
+
## Output formats
|
|
109
|
+
|
|
110
|
+
The `--output` flag picks the *display* format (separate from `--format`, which controls how the prompt body is converted before tokenization):
|
|
111
|
+
|
|
112
|
+
- `--output table` (default) — the human-readable per-cell table you've been seeing.
|
|
113
|
+
- `--output json` — emits a `TokenometerResult` JSON shape: `{ files: [{ path, results: [...] }] }`. One entry per input file. Pipe to `jq` for filtering.
|
|
114
|
+
- `--output sarif` — emits SARIF 2.1.0 with one result per (file, model, format) cell. Drop the file into GitHub Code Scanning or any SARIF viewer.
|
|
115
|
+
|
|
116
|
+
```bash
|
|
117
|
+
npx tokenometer ./prompt.md --output sarif > tokenometer.sarif
|
|
118
|
+
npx tokenometer ./prompt.md --output json | jq '.files[].results | map(.inputCost) | add'
|
|
51
119
|
```
|
|
52
120
|
|
|
121
|
+
## Latency
|
|
122
|
+
|
|
123
|
+
`--latency` measures real generation latency in addition to token cost. For each `(model, format)` cell, tokenometer streams `n` real chat completions (default `n=3`, override with `--latency-trials 1..10`) capped at `max_tokens=200`, and reports:
|
|
124
|
+
|
|
125
|
+
- **TTFT** — time to first streamed token (ms)
|
|
126
|
+
- **Total** — wall-clock from request start to stream end (ms)
|
|
127
|
+
- **tokens/sec** — `output_tokens / (total - ttft)`
|
|
128
|
+
|
|
129
|
+
Numbers are reported as **p50 / p95 / mean** over the trials. Full per-trial data is included in `--output json`.
|
|
130
|
+
|
|
131
|
+
```bash
|
|
132
|
+
ANTHROPIC_API_KEY=… OPENAI_API_KEY=… \
|
|
133
|
+
npx tokenometer ./prompt.md --latency --model claude-opus-4-7,gpt-4o
|
|
134
|
+
```
|
|
135
|
+
|
|
136
|
+
`--latency` implies `--empirical` (offline mode can't measure real latency). The default `--max-spend` ceiling is bumped from `$0.05` to `$0.25` to cover the `n × 200-token` generations; pass `--max-spend` explicitly to override.
|
|
137
|
+
|
|
138
|
+
Supported providers: Anthropic (`messages.stream`), OpenAI (`/v1/chat/completions` SSE), Google (`generateContentStream`), Cohere (`/v1/chat` NDJSON), Mistral (`/v1/chat/completions` SSE). Each trial retries once on transient failures.
|
|
139
|
+
|
|
140
|
+
## Per-file attribution
|
|
141
|
+
|
|
142
|
+
`--by-file` appends a per-file token + USD summary table when you pass multiple input files (single-file inputs are a no-op):
|
|
143
|
+
|
|
144
|
+
```
|
|
145
|
+
By file:
|
|
146
|
+
File Tokens USD
|
|
147
|
+
──────────────── ─────── ───────
|
|
148
|
+
prompts/agent.md 1,243 $0.0186
|
|
149
|
+
prompts/router.md 872 $0.0131
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Useful for figuring out which prompt files dominate the cost of a multi-file pipeline. The aggregator that produces this table is also what powers the GitHub Action's per-file Δ comment, and is unit-tested in [`packages/action`](https://github.com/faraa2m/tokenometer/tree/main/packages/action).
|
|
153
|
+
|
|
154
|
+
## Vision tokens
|
|
155
|
+
|
|
156
|
+
Pass `--image <path>` (repeatable) to factor image-based vision tokens into the cost estimate alongside your prompt text:
|
|
157
|
+
|
|
158
|
+
```bash
|
|
159
|
+
npx tokenometer ./prompt.md --image ./screenshot.png --image ./diagram.jpg
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
Each image's dimensions are read with `image-size` (no native deps), then dispatched to the provider-specific vision-token estimator:
|
|
163
|
+
|
|
164
|
+
- Claude → Anthropic's `(width × height) / 750`, capped at 1600 tokens.
|
|
165
|
+
- GPT-4o → OpenAI's high-detail tiling: `85 + 170 × ceil(w/512) × ceil(h/512)` after the 2048/768 resize step.
|
|
166
|
+
- Gemini → Google's `258 × ceil(w/768) × ceil(h/768)` (with a flat 258 for ≤384×384 images).
|
|
167
|
+
|
|
168
|
+
Mistral and Cohere don't have published vision-token formulas, so vision images are skipped for those providers (with a stderr note). Vision-token cells are always marked `approximate: true` since they're formula-derived. Each image also gets its own row in the `--by-file` table as a virtual file `<image-path> [vision]`.
|
|
169
|
+
|
|
170
|
+
## Why not just `tiktoken`?
|
|
171
|
+
|
|
172
|
+
`tiktoken`'s `cl100k_base` (the encoding most "Claude tokenizer" libraries fall back on) **under-counts Opus 4.7 by a median of +62%** across a 10-prompt benchmark. Sonnet 4.6 and Haiku 4.5 are closer (~17%). Format choice is a wash. Model choice swings cost by 12×. See [README](https://github.com/faraa2m/tokenometer#findings-anthropic-n150-cells-across-10-prompt-shapes) for the dataset findings.
|
|
173
|
+
|
|
53
174
|
## License
|
|
54
175
|
|
|
55
176
|
MIT
|
package/dist/args.d.ts
CHANGED
|
@@ -1,14 +1,29 @@
|
|
|
1
1
|
import type { Format } from '@tokenometer/core';
|
|
2
|
+
export type OutputFormat = 'table' | 'json' | 'sarif';
|
|
2
3
|
export interface ParsedArgs {
|
|
4
|
+
byFile: boolean;
|
|
5
|
+
configPath: string | null;
|
|
3
6
|
empirical: boolean;
|
|
4
7
|
formats: Format[];
|
|
8
|
+
formatsSet: boolean;
|
|
5
9
|
help: boolean;
|
|
10
|
+
imagePaths: string[];
|
|
6
11
|
inputPaths: string[];
|
|
12
|
+
inputPathsSet: boolean;
|
|
13
|
+
latency: boolean;
|
|
14
|
+
latencyTrials: number;
|
|
7
15
|
maxSpend: number;
|
|
16
|
+
/** True iff the user passed `--max-spend` explicitly (not the default). */
|
|
17
|
+
maxSpendSet: boolean;
|
|
8
18
|
modelIds: string[];
|
|
19
|
+
modelsSet: boolean;
|
|
20
|
+
noConfig: boolean;
|
|
9
21
|
offline: boolean;
|
|
22
|
+
output: OutputFormat;
|
|
10
23
|
version: boolean;
|
|
11
24
|
}
|
|
12
25
|
export declare const HELP_TEXT: string;
|
|
26
|
+
export declare const DEFAULT_LATENCY_MAX_SPEND_USD = 0.25;
|
|
27
|
+
export declare const DEFAULT_LATENCY_TRIALS = 3;
|
|
13
28
|
export declare const parseArgs: (argv: readonly string[]) => ParsedArgs;
|
|
14
29
|
//# sourceMappingURL=args.d.ts.map
|
package/dist/args.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"args.d.ts","sourceRoot":"","sources":["../src/args.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,mBAAmB,CAAC;AAEhD,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,IAAI,EAAE,OAAO,CAAC;IACd,UAAU,EAAE,MAAM,EAAE,CAAC;IACrB,QAAQ,EAAE,MAAM,CAAC;IACjB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,OAAO,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,OAAO,CAAC;CAClB;AAED,eAAO,MAAM,SAAS,
|
|
1
|
+
{"version":3,"file":"args.d.ts","sourceRoot":"","sources":["../src/args.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,mBAAmB,CAAC;AAEhD,MAAM,MAAM,YAAY,GAAG,OAAO,GAAG,MAAM,GAAG,OAAO,CAAC;AAEtD,MAAM,WAAW,UAAU;IACzB,MAAM,EAAE,OAAO,CAAC;IAChB,UAAU,EAAE,MAAM,GAAG,IAAI,CAAC;IAC1B,SAAS,EAAE,OAAO,CAAC;IACnB,OAAO,EAAE,MAAM,EAAE,CAAC;IAClB,UAAU,EAAE,OAAO,CAAC;IACpB,IAAI,EAAE,OAAO,CAAC;IACd,UAAU,EAAE,MAAM,EAAE,CAAC;IACrB,UAAU,EAAE,MAAM,EAAE,CAAC;IACrB,aAAa,EAAE,OAAO,CAAC;IACvB,OAAO,EAAE,OAAO,CAAC;IACjB,aAAa,EAAE,MAAM,CAAC;IACtB,QAAQ,EAAE,MAAM,CAAC;IACjB,2EAA2E;IAC3E,WAAW,EAAE,OAAO,CAAC;IACrB,QAAQ,EAAE,MAAM,EAAE,CAAC;IACnB,SAAS,EAAE,OAAO,CAAC;IACnB,QAAQ,EAAE,OAAO,CAAC;IAClB,OAAO,EAAE,OAAO,CAAC;IACjB,MAAM,EAAE,YAAY,CAAC;IACrB,OAAO,EAAE,OAAO,CAAC;CAClB;AAED,eAAO,MAAM,SAAS,QA2CrB,CAAC;AAIF,eAAO,MAAM,6BAA6B,OAAO,CAAC;AAClD,eAAO,MAAM,sBAAsB,IAAI,CAAC;AAQxC,eAAO,MAAM,SAAS,GAAI,MAAM,SAAS,MAAM,EAAE,KAAG,UAkJnD,CAAC"}
|
package/dist/args.js
CHANGED
|
@@ -6,37 +6,73 @@ USAGE
|
|
|
6
6
|
echo "prompt" | tokenometer - [options]
|
|
7
7
|
|
|
8
8
|
OPTIONS
|
|
9
|
-
--model <id[,id...]> Comma-separated model ids
|
|
9
|
+
--model <id[,id...]> Comma-separated model ids. Default: claude-opus-4-7,
|
|
10
|
+
or auto-detected from *_API_KEY env when omitted.
|
|
10
11
|
Known: ${KNOWN_MODELS.join(', ')}
|
|
11
12
|
--format <fmt[,fmt...]> Comma-separated formats (default: all).
|
|
12
13
|
Known: ${allFormats().join(', ')}
|
|
14
|
+
--output <fmt> Output format: table (default), json, or sarif.
|
|
15
|
+
--by-file With multi-file input, append a per-file token/cost table.
|
|
16
|
+
--image <path> Path to an image to factor into vision-token cost.
|
|
17
|
+
Repeatable.
|
|
18
|
+
--config <path> Load this exact config file (skip walk-up).
|
|
19
|
+
--no-config Skip .tokenometer.yml loading entirely.
|
|
13
20
|
--empirical Run sample API calls and report real charges.
|
|
14
21
|
Requires the matching <PROVIDER>_API_KEY env var.
|
|
15
|
-
--
|
|
22
|
+
--latency Measure real generation latency (TTFT, total ms,
|
|
23
|
+
tokens/sec) per (model × format) cell. Implies
|
|
24
|
+
--empirical and bumps the default --max-spend
|
|
25
|
+
ceiling to $0.25 (each trial is a metered
|
|
26
|
+
~200-token chat completion). Anthropic, OpenAI,
|
|
27
|
+
Google, Cohere, Mistral are supported.
|
|
28
|
+
--latency-trials <n> Trials per cell for --latency (1-10, default 3).
|
|
29
|
+
Each trial requests max_tokens=200 to keep cost
|
|
30
|
+
predictable while giving enough output to
|
|
31
|
+
stabilize tokens/sec.
|
|
32
|
+
--max-spend <usd> Hard ceiling for empirical mode (default: 0.05;
|
|
33
|
+
with --latency, default 0.25).
|
|
16
34
|
--offline Force offline mode (overrides --empirical).
|
|
17
35
|
-h, --help Show this help.
|
|
18
36
|
-v, --version Show CLI version.
|
|
19
37
|
|
|
20
38
|
EXAMPLES
|
|
21
39
|
tokenometer ./prompt.md
|
|
22
|
-
tokenometer ./prompt.md --model claude-opus-4-7,gpt-4o
|
|
40
|
+
tokenometer ./prompt.md --model claude-opus-4-7,gpt-4o --by-file
|
|
41
|
+
tokenometer ./prompt.md --output sarif > tokenometer.sarif
|
|
42
|
+
tokenometer ./prompt.md --image ./screenshot.png
|
|
23
43
|
tokenometer ./prompt.md --format yaml,json --empirical --max-spend 0.01
|
|
44
|
+
tokenometer ./prompt.md --latency --model gpt-4o,claude-opus-4-7
|
|
24
45
|
`;
|
|
25
46
|
const DEFAULT_MODELS = ['claude-opus-4-7'];
|
|
26
47
|
const DEFAULT_MAX_SPEND_USD = 0.05;
|
|
48
|
+
export const DEFAULT_LATENCY_MAX_SPEND_USD = 0.25;
|
|
49
|
+
export const DEFAULT_LATENCY_TRIALS = 3;
|
|
50
|
+
const MIN_LATENCY_TRIALS = 1;
|
|
51
|
+
const MAX_LATENCY_TRIALS = 10;
|
|
52
|
+
const OUTPUT_FORMATS = ['table', 'json', 'sarif'];
|
|
53
|
+
const isOutputFormat = (value) => OUTPUT_FORMATS.includes(value);
|
|
27
54
|
export const parseArgs = (argv) => {
|
|
28
55
|
const result = {
|
|
56
|
+
byFile: false,
|
|
57
|
+
configPath: null,
|
|
29
58
|
empirical: false,
|
|
30
59
|
formats: [...allFormats()],
|
|
60
|
+
formatsSet: false,
|
|
31
61
|
help: false,
|
|
62
|
+
imagePaths: [],
|
|
32
63
|
inputPaths: [],
|
|
64
|
+
inputPathsSet: false,
|
|
65
|
+
latency: false,
|
|
66
|
+
latencyTrials: DEFAULT_LATENCY_TRIALS,
|
|
33
67
|
maxSpend: DEFAULT_MAX_SPEND_USD,
|
|
68
|
+
maxSpendSet: false,
|
|
34
69
|
modelIds: [...DEFAULT_MODELS],
|
|
70
|
+
modelsSet: false,
|
|
71
|
+
noConfig: false,
|
|
35
72
|
offline: false,
|
|
73
|
+
output: 'table',
|
|
36
74
|
version: false,
|
|
37
75
|
};
|
|
38
|
-
let modelsSet = false;
|
|
39
|
-
let formatsSet = false;
|
|
40
76
|
for (let i = 0; i < argv.length; i++) {
|
|
41
77
|
const arg = argv[i];
|
|
42
78
|
if (!arg)
|
|
@@ -53,10 +89,59 @@ export const parseArgs = (argv) => {
|
|
|
53
89
|
result.empirical = true;
|
|
54
90
|
continue;
|
|
55
91
|
}
|
|
92
|
+
if (arg === '--latency') {
|
|
93
|
+
result.latency = true;
|
|
94
|
+
// --latency implies --empirical (offline mode can't measure real latency).
|
|
95
|
+
result.empirical = true;
|
|
96
|
+
continue;
|
|
97
|
+
}
|
|
98
|
+
if (arg === '--latency-trials') {
|
|
99
|
+
const next = argv[++i];
|
|
100
|
+
if (!next)
|
|
101
|
+
throw new Error('--latency-trials requires a value');
|
|
102
|
+
const parsed = Number.parseInt(next, 10);
|
|
103
|
+
if (!Number.isFinite(parsed) || parsed < MIN_LATENCY_TRIALS || parsed > MAX_LATENCY_TRIALS) {
|
|
104
|
+
throw new Error(`--latency-trials must be an integer between ${MIN_LATENCY_TRIALS} and ${MAX_LATENCY_TRIALS}, got "${next}".`);
|
|
105
|
+
}
|
|
106
|
+
result.latencyTrials = parsed;
|
|
107
|
+
continue;
|
|
108
|
+
}
|
|
56
109
|
if (arg === '--offline') {
|
|
57
110
|
result.offline = true;
|
|
58
111
|
continue;
|
|
59
112
|
}
|
|
113
|
+
if (arg === '--by-file') {
|
|
114
|
+
result.byFile = true;
|
|
115
|
+
continue;
|
|
116
|
+
}
|
|
117
|
+
if (arg === '--no-config') {
|
|
118
|
+
result.noConfig = true;
|
|
119
|
+
continue;
|
|
120
|
+
}
|
|
121
|
+
if (arg === '--config') {
|
|
122
|
+
const next = argv[++i];
|
|
123
|
+
if (!next)
|
|
124
|
+
throw new Error('--config requires a value');
|
|
125
|
+
result.configPath = next;
|
|
126
|
+
continue;
|
|
127
|
+
}
|
|
128
|
+
if (arg === '--output') {
|
|
129
|
+
const next = argv[++i];
|
|
130
|
+
if (!next)
|
|
131
|
+
throw new Error('--output requires a value');
|
|
132
|
+
if (!isOutputFormat(next)) {
|
|
133
|
+
throw new Error(`Unknown --output "${next}". Known: ${OUTPUT_FORMATS.join(', ')}.`);
|
|
134
|
+
}
|
|
135
|
+
result.output = next;
|
|
136
|
+
continue;
|
|
137
|
+
}
|
|
138
|
+
if (arg === '--image') {
|
|
139
|
+
const next = argv[++i];
|
|
140
|
+
if (!next)
|
|
141
|
+
throw new Error('--image requires a value');
|
|
142
|
+
result.imagePaths.push(next);
|
|
143
|
+
continue;
|
|
144
|
+
}
|
|
60
145
|
if (arg === '--model') {
|
|
61
146
|
const next = argv[++i];
|
|
62
147
|
if (!next)
|
|
@@ -65,7 +150,7 @@ export const parseArgs = (argv) => {
|
|
|
65
150
|
.split(',')
|
|
66
151
|
.map((s) => s.trim())
|
|
67
152
|
.filter(Boolean);
|
|
68
|
-
modelsSet = true;
|
|
153
|
+
result.modelsSet = true;
|
|
69
154
|
continue;
|
|
70
155
|
}
|
|
71
156
|
if (arg === '--format') {
|
|
@@ -82,7 +167,7 @@ export const parseArgs = (argv) => {
|
|
|
82
167
|
}
|
|
83
168
|
}
|
|
84
169
|
result.formats = formats;
|
|
85
|
-
formatsSet = true;
|
|
170
|
+
result.formatsSet = true;
|
|
86
171
|
continue;
|
|
87
172
|
}
|
|
88
173
|
if (arg === '--max-spend') {
|
|
@@ -94,19 +179,27 @@ export const parseArgs = (argv) => {
|
|
|
94
179
|
throw new Error(`--max-spend must be a positive number, got "${next}".`);
|
|
95
180
|
}
|
|
96
181
|
result.maxSpend = parsed;
|
|
182
|
+
result.maxSpendSet = true;
|
|
97
183
|
continue;
|
|
98
184
|
}
|
|
99
185
|
if (arg.startsWith('--')) {
|
|
100
186
|
throw new Error(`Unknown flag: ${arg}`);
|
|
101
187
|
}
|
|
102
188
|
result.inputPaths.push(arg);
|
|
189
|
+
result.inputPathsSet = true;
|
|
103
190
|
}
|
|
104
|
-
if (!modelsSet && result.modelIds.length === 0) {
|
|
191
|
+
if (!result.modelsSet && result.modelIds.length === 0) {
|
|
105
192
|
result.modelIds = [...DEFAULT_MODELS];
|
|
106
193
|
}
|
|
107
|
-
if (!formatsSet && result.formats.length === 0) {
|
|
194
|
+
if (!result.formatsSet && result.formats.length === 0) {
|
|
108
195
|
result.formats = [...allFormats()];
|
|
109
196
|
}
|
|
197
|
+
// `--latency` makes the default `--max-spend` ceiling more generous since
|
|
198
|
+
// each trial is a metered ~200-token chat completion. Only bump if the
|
|
199
|
+
// user did not explicitly set --max-spend (theirs always wins).
|
|
200
|
+
if (result.latency && !result.maxSpendSet) {
|
|
201
|
+
result.maxSpend = DEFAULT_LATENCY_MAX_SPEND_USD;
|
|
202
|
+
}
|
|
110
203
|
return result;
|
|
111
204
|
};
|
|
112
205
|
//# sourceMappingURL=args.js.map
|
package/dist/args.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"args.js","sourceRoot":"","sources":["../src/args.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,mBAAmB,CAAC;
|
|
1
|
+
{"version":3,"file":"args.js","sourceRoot":"","sources":["../src/args.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,QAAQ,EAAE,MAAM,mBAAmB,CAAC;AA4BvE,MAAM,CAAC,MAAM,SAAS,GAAG;;;;;;;;;oCASW,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC;;oCAEvB,UAAU,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;CAgC1D,CAAC;AAEF,MAAM,cAAc,GAAG,CAAC,iBAAiB,CAAC,CAAC;AAC3C,MAAM,qBAAqB,GAAG,IAAI,CAAC;AACnC,MAAM,CAAC,MAAM,6BAA6B,GAAG,IAAI,CAAC;AAClD,MAAM,CAAC,MAAM,sBAAsB,GAAG,CAAC,CAAC;AACxC,MAAM,kBAAkB,GAAG,CAAC,CAAC;AAC7B,MAAM,kBAAkB,GAAG,EAAE,CAAC;AAC9B,MAAM,cAAc,GAA4B,CAAC,OAAO,EAAE,MAAM,EAAE,OAAO,CAAC,CAAC;AAE3E,MAAM,cAAc,GAAG,CAAC,KAAa,EAAyB,EAAE,CAC7D,cAAoC,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC;AAExD,MAAM,CAAC,MAAM,SAAS,GAAG,CAAC,IAAuB,EAAc,EAAE;IAC/D,MAAM,MAAM,GAAe;QACzB,MAAM,EAAE,KAAK;QACb,UAAU,EAAE,IAAI;QAChB,SAAS,EAAE,KAAK;QAChB,OAAO,EAAE,CAAC,GAAG,UAAU,EAAE,CAAC;QAC1B,UAAU,EAAE,KAAK;QACjB,IAAI,EAAE,KAAK;QACX,UAAU,EAAE,EAAE;QACd,UAAU,EAAE,EAAE;QACd,aAAa,EAAE,KAAK;QACpB,OAAO,EAAE,KAAK;QACd,aAAa,EAAE,sBAAsB;QACrC,QAAQ,EAAE,qBAAqB;QAC/B,WAAW,EAAE,KAAK;QAClB,QAAQ,EAAE,CAAC,GAAG,cAAc,CAAC;QAC7B,SAAS,EAAE,KAAK;QAChB,QAAQ,EAAE,KAAK;QACf,OAAO,EAAE,KAAK;QACd,MAAM,EAAE,OAAO;QACf,OAAO,EAAE,KAAK;KACf,CAAC;IAEF,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACrC,MAAM,GAAG,GAAG,IAAI,CAAC,CAAC,CAAC,CAAC;QACpB,IAAI,CAAC,GAAG;YAAE,SAAS;QACnB,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,QAAQ,EAAE,CAAC;YACrC,MAAM,CAAC,IAAI,GAAG,IAAI,CAAC;YACnB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,IAAI,IAAI,GAAG,KAAK,WAAW,EAAE,CAAC;YACxC,MAAM,CAAC,OAAO,GAAG,IAAI,CAAC;YACtB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,aAAa,EAAE,CAAC;YAC1B,MAAM,CAAC,SAAS,GAAG,IAAI,CAAC;YACxB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,WAAW,EAAE,CAAC;YACxB,MAAM,CAAC,OAAO,GAAG,IAAI,CAAC;YACtB,2EAA2E;YAC3E,MAAM,CAAC,SAAS,GAAG,IAAI,CAAC;YACxB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,kBAAkB,EAAE,CAAC;YAC/B,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,mCAAmC,CAAC,CAAC;YAChE,MAAM,MAAM,GAAG,MAAM,CAAC,QAAQ,CAAC,IAAI,EAAE,EAAE,CAAC,CAAC;YACzC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,IAAI,MAAM,GAAG,kBAAkB,IAAI,MAAM,GAAG,kBAAkB,EAAE,CAAC;gBAC3F,MAAM,IAAI,KAAK,CACb,+CAA+C,kBAAkB,QAAQ,kBAAkB,UAAU,IAAI,IAAI,CAC9G,CAAC;YACJ,CAAC;YACD,MAAM,CAAC,aAAa,GAAG,MAAM,CAAC;YAC9B,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,WAAW,EAAE,CAAC;YACxB,MAAM,CAAC,OAAO,GAAG,IAAI,CAAC;YACtB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,WAAW,EAAE,CAAC;YACxB,MAAM,CAAC,MAAM,GAAG,IAAI,CAAC;YACrB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,aAAa,EAAE,CAAC;YAC1B,MAAM,CAAC,QAAQ,GAAG,IAAI,CAAC;YACvB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,UAAU,EAAE,CAAC;YACvB,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,2BAA2B,CAAC,CAAC;YACxD,MAAM,CAAC,UAAU,GAAG,IAAI,CAAC;YACzB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,UAAU,EAAE,CAAC;YACvB,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,2BAA2B,CAAC,CAAC;YACxD,IAAI,CAAC,cAAc,CAAC,IAAI,CAAC,EAAE,CAAC;gBAC1B,MAAM,IAAI,KAAK,CAAC,qBAAqB,IAAI,aAAa,cAAc,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;YACtF,CAAC;YACD,MAAM,CAAC,MAAM,GAAG,IAAI,CAAC;YACrB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtB,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;YACvD,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;YAC7B,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,SAAS,EAAE,CAAC;YACtB,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,0BAA0B,CAAC,CAAC;YACvD,MAAM,CAAC,QAAQ,GAAG,IAAI;iBACnB,KAAK,CAAC,GAAG,CAAC;iBACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;iBACpB,MAAM,CAAC,OAAO,CAAC,CAAC;YACnB,MAAM,CAAC,SAAS,GAAG,IAAI,CAAC;YACxB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,UAAU,EAAE,CAAC;YACvB,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,2BAA2B,CAAC,CAAC;YACxD,MAAM,OAAO,GAAG,IAAI;iBACjB,KAAK,CAAC,GAAG,CAAC;iBACV,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;iBACpB,MAAM,CAAC,OAAO,CAAC,CAAC;YACnB,KAAK,MAAM,GAAG,IAAI,OAAO,EAAE,CAAC;gBAC1B,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC;oBACnB,MAAM,IAAI,KAAK,CAAC,mBAAmB,GAAG,aAAa,UAAU,EAAE,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;gBACjF,CAAC;YACH,CAAC;YACD,MAAM,CAAC,OAAO,GAAG,OAAmB,CAAC;YACrC,MAAM,CAAC,UAAU,GAAG,IAAI,CAAC;YACzB,SAAS;QACX,CAAC;QACD,IAAI,GAAG,KAAK,aAAa,EAAE,CAAC;YAC1B,MAAM,IAAI,GAAG,IAAI,CAAC,EAAE,CAAC,CAAC,CAAC;YACvB,IAAI,CAAC,IAAI;gBAAE,MAAM,IAAI,KAAK,CAAC,8BAA8B,CAAC,CAAC;YAC3D,MAAM,MAAM,GAAG,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,CAAC;YACvC,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,IAAI,MAAM,IAAI,CAAC,EAAE,CAAC;gBAC5C,MAAM,IAAI,KAAK,CAAC,+CAA+C,IAAI,IAAI,CAAC,CAAC;YAC3E,CAAC;YACD,MAAM,CAAC,QAAQ,GAAG,MAAM,CAAC;YACzB,MAAM,CAAC,WAAW,GAAG,IAAI,CAAC;YAC1B,SAAS;QACX,CAAC;QACD,IAAI,GAAG,CAAC,UAAU,CAAC,IAAI,CAAC,EAAE,CAAC;YACzB,MAAM,IAAI,KAAK,CAAC,iBAAiB,GAAG,EAAE,CAAC,CAAC;QAC1C,CAAC;QACD,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;QAC5B,MAAM,CAAC,aAAa,GAAG,IAAI,CAAC;IAC9B,CAAC;IAED,IAAI,CAAC,MAAM,CAAC,SAAS,IAAI,MAAM,CAAC,QAAQ,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtD,MAAM,CAAC,QAAQ,GAAG,CAAC,GAAG,cAAc,CAAC,CAAC;IACxC,CAAC;IACD,IAAI,CAAC,MAAM,CAAC,UAAU,IAAI,MAAM,CAAC,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACtD,MAAM,CAAC,OAAO,GAAG,CAAC,GAAG,UAAU,EAAE,CAAC,CAAC;IACrC,CAAC;IACD,0EAA0E;IAC1E,uEAAuE;IACvE,gEAAgE;IAChE,IAAI,MAAM,CAAC,OAAO,IAAI,CAAC,MAAM,CAAC,WAAW,EAAE,CAAC;QAC1C,MAAM,CAAC,QAAQ,GAAG,6BAA6B,CAAC;IAClD,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC,CAAC"}
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
export interface AutoDetectInput {
|
|
2
|
+
env?: NodeJS.ProcessEnv;
|
|
3
|
+
}
|
|
4
|
+
export interface AutoDetectResult {
|
|
5
|
+
/** The chosen default model id. */
|
|
6
|
+
modelId: string;
|
|
7
|
+
/** Optional human-readable note for stderr (e.g. multi-key conflict). */
|
|
8
|
+
note: string | null;
|
|
9
|
+
}
|
|
10
|
+
/**
|
|
11
|
+
* When the user has not passed `--model`, pick a default based on which
|
|
12
|
+
* provider API key is present in the environment.
|
|
13
|
+
*
|
|
14
|
+
* - Exactly one provider key set → that provider's canonical model.
|
|
15
|
+
* - Multiple keys set → fall back to the existing default with a stderr note.
|
|
16
|
+
* - No keys set → existing default behavior.
|
|
17
|
+
*/
|
|
18
|
+
export declare const autoDetectDefaultModel: (input?: AutoDetectInput) => AutoDetectResult;
|
|
19
|
+
//# sourceMappingURL=auto-detect.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"auto-detect.d.ts","sourceRoot":"","sources":["../src/auto-detect.ts"],"names":[],"mappings":"AAaA,MAAM,WAAW,eAAe;IAC9B,GAAG,CAAC,EAAE,MAAM,CAAC,UAAU,CAAC;CACzB;AAED,MAAM,WAAW,gBAAgB;IAC/B,mCAAmC;IACnC,OAAO,EAAE,MAAM,CAAC;IAChB,yEAAyE;IACzE,IAAI,EAAE,MAAM,GAAG,IAAI,CAAC;CACrB;AAED;;;;;;;GAOG;AACH,eAAO,MAAM,sBAAsB,GAAI,QAAO,eAAoB,KAAG,gBAiBpE,CAAC"}
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
import { KNOWN_MODELS, MODELS } from '@tokenometer/core';
|
|
2
|
+
const DEFAULT_MODEL = 'claude-opus-4-7';
|
|
3
|
+
const firstGoogleGeminiModel = () => {
|
|
4
|
+
for (const id of KNOWN_MODELS) {
|
|
5
|
+
if (!id.startsWith('gemini-'))
|
|
6
|
+
continue;
|
|
7
|
+
const desc = MODELS[id];
|
|
8
|
+
if (desc?.provider === 'google')
|
|
9
|
+
return id;
|
|
10
|
+
}
|
|
11
|
+
return 'gemini-2.5-pro';
|
|
12
|
+
};
|
|
13
|
+
/**
|
|
14
|
+
* When the user has not passed `--model`, pick a default based on which
|
|
15
|
+
* provider API key is present in the environment.
|
|
16
|
+
*
|
|
17
|
+
* - Exactly one provider key set → that provider's canonical model.
|
|
18
|
+
* - Multiple keys set → fall back to the existing default with a stderr note.
|
|
19
|
+
* - No keys set → existing default behavior.
|
|
20
|
+
*/
|
|
21
|
+
export const autoDetectDefaultModel = (input = {}) => {
|
|
22
|
+
const env = input.env ?? process.env;
|
|
23
|
+
const hasAnthropic = Boolean(env.ANTHROPIC_API_KEY);
|
|
24
|
+
const hasOpenAi = Boolean(env.OPENAI_API_KEY);
|
|
25
|
+
const hasGoogle = Boolean(env.GOOGLE_API_KEY ?? env.GEMINI_API_KEY);
|
|
26
|
+
const setCount = [hasAnthropic, hasOpenAi, hasGoogle].filter(Boolean).length;
|
|
27
|
+
if (setCount === 0)
|
|
28
|
+
return { modelId: DEFAULT_MODEL, note: null };
|
|
29
|
+
if (setCount > 1) {
|
|
30
|
+
return {
|
|
31
|
+
modelId: DEFAULT_MODEL,
|
|
32
|
+
note: `Multiple provider API keys detected; defaulting to ${DEFAULT_MODEL}. Pass --model to override.`,
|
|
33
|
+
};
|
|
34
|
+
}
|
|
35
|
+
if (hasAnthropic)
|
|
36
|
+
return { modelId: 'claude-opus-4-7', note: null };
|
|
37
|
+
if (hasOpenAi)
|
|
38
|
+
return { modelId: 'gpt-4o', note: null };
|
|
39
|
+
return { modelId: firstGoogleGeminiModel(), note: null };
|
|
40
|
+
};
|
|
41
|
+
//# sourceMappingURL=auto-detect.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"auto-detect.js","sourceRoot":"","sources":["../src/auto-detect.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,YAAY,EAAE,MAAM,EAAE,MAAM,mBAAmB,CAAC;AAEzD,MAAM,aAAa,GAAG,iBAAiB,CAAC;AAExC,MAAM,sBAAsB,GAAG,GAAW,EAAE;IAC1C,KAAK,MAAM,EAAE,IAAI,YAAY,EAAE,CAAC;QAC9B,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,SAAS,CAAC;YAAE,SAAS;QACxC,MAAM,IAAI,GAAG,MAAM,CAAC,EAAE,CAAC,CAAC;QACxB,IAAI,IAAI,EAAE,QAAQ,KAAK,QAAQ;YAAE,OAAO,EAAE,CAAC;IAC7C,CAAC;IACD,OAAO,gBAAgB,CAAC;AAC1B,CAAC,CAAC;AAaF;;;;;;;GAOG;AACH,MAAM,CAAC,MAAM,sBAAsB,GAAG,CAAC,QAAyB,EAAE,EAAoB,EAAE;IACtF,MAAM,GAAG,GAAG,KAAK,CAAC,GAAG,IAAI,OAAO,CAAC,GAAG,CAAC;IACrC,MAAM,YAAY,GAAG,OAAO,CAAC,GAAG,CAAC,iBAAiB,CAAC,CAAC;IACpD,MAAM,SAAS,GAAG,OAAO,CAAC,GAAG,CAAC,cAAc,CAAC,CAAC;IAC9C,MAAM,SAAS,GAAG,OAAO,CAAC,GAAG,CAAC,cAAc,IAAI,GAAG,CAAC,cAAc,CAAC,CAAC;IAEpE,MAAM,QAAQ,GAAG,CAAC,YAAY,EAAE,SAAS,EAAE,SAAS,CAAC,CAAC,MAAM,CAAC,OAAO,CAAC,CAAC,MAAM,CAAC;IAC7E,IAAI,QAAQ,KAAK,CAAC;QAAE,OAAO,EAAE,OAAO,EAAE,aAAa,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;IAClE,IAAI,QAAQ,GAAG,CAAC,EAAE,CAAC;QACjB,OAAO;YACL,OAAO,EAAE,aAAa;YACtB,IAAI,EAAE,sDAAsD,aAAa,6BAA6B;SACvG,CAAC;IACJ,CAAC;IACD,IAAI,YAAY;QAAE,OAAO,EAAE,OAAO,EAAE,iBAAiB,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;IACpE,IAAI,SAAS;QAAE,OAAO,EAAE,OAAO,EAAE,QAAQ,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;IACxD,OAAO,EAAE,OAAO,EAAE,sBAAsB,EAAE,EAAE,IAAI,EAAE,IAAI,EAAE,CAAC;AAC3D,CAAC,CAAC"}
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
import { type TokenometerConfig } from '@tokenometer/core';
|
|
2
|
+
import type { ParsedArgs } from './args.js';
|
|
3
|
+
export interface ApplyConfigOptions {
|
|
4
|
+
/** Pre-loaded config (already parsed). When omitted, applyConfig is a no-op. */
|
|
5
|
+
config: TokenometerConfig | null;
|
|
6
|
+
}
|
|
7
|
+
/**
|
|
8
|
+
* Apply config defaults to ParsedArgs.
|
|
9
|
+
* User-passed flags ALWAYS win over config (we only fill in fields the user did not set).
|
|
10
|
+
* Returns a new ParsedArgs (does not mutate the input).
|
|
11
|
+
*/
|
|
12
|
+
export declare const applyConfig: (args: ParsedArgs, opts: ApplyConfigOptions) => ParsedArgs;
|
|
13
|
+
/**
|
|
14
|
+
* Read a config from a user-specified path (used by `--config <path>`).
|
|
15
|
+
* Throws on parse / validation failure with the offending path included.
|
|
16
|
+
*/
|
|
17
|
+
export declare const loadConfigFromPath: (path: string) => Promise<TokenometerConfig>;
|
|
18
|
+
//# sourceMappingURL=config-merge.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"config-merge.d.ts","sourceRoot":"","sources":["../src/config-merge.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,KAAK,iBAAiB,EAAe,MAAM,mBAAmB,CAAC;AACxE,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,WAAW,CAAC;AAE5C,MAAM,WAAW,kBAAkB;IACjC,gFAAgF;IAChF,MAAM,EAAE,iBAAiB,GAAG,IAAI,CAAC;CAClC;AAED;;;;GAIG;AACH,eAAO,MAAM,WAAW,GAAI,MAAM,UAAU,EAAE,MAAM,kBAAkB,KAAG,UAoBxE,CAAC;AAEF;;;GAGG;AACH,eAAO,MAAM,kBAAkB,GAAU,MAAM,MAAM,KAAG,OAAO,CAAC,iBAAiB,CAYhF,CAAC"}
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
import { readFile } from 'node:fs/promises';
|
|
2
|
+
import { parseConfig } from '@tokenometer/core';
|
|
3
|
+
/**
|
|
4
|
+
* Apply config defaults to ParsedArgs.
|
|
5
|
+
* User-passed flags ALWAYS win over config (we only fill in fields the user did not set).
|
|
6
|
+
* Returns a new ParsedArgs (does not mutate the input).
|
|
7
|
+
*/
|
|
8
|
+
export const applyConfig = (args, opts) => {
|
|
9
|
+
const cfg = opts.config;
|
|
10
|
+
if (!cfg)
|
|
11
|
+
return args;
|
|
12
|
+
const next = { ...args, formats: [...args.formats], modelIds: [...args.modelIds] };
|
|
13
|
+
// When the config provides a value AND the user didn't pass it on the CLI,
|
|
14
|
+
// adopt the config value AND set the *Set flag so downstream auto-detect
|
|
15
|
+
// doesn't overwrite the config-provided default.
|
|
16
|
+
if (!args.modelsSet && cfg.models && cfg.models.length > 0) {
|
|
17
|
+
next.modelIds = [...cfg.models];
|
|
18
|
+
next.modelsSet = true;
|
|
19
|
+
}
|
|
20
|
+
if (!args.formatsSet && cfg.formats && cfg.formats.length > 0) {
|
|
21
|
+
next.formats = [...cfg.formats];
|
|
22
|
+
next.formatsSet = true;
|
|
23
|
+
}
|
|
24
|
+
if (!args.inputPathsSet && cfg.paths && cfg.paths.length > 0) {
|
|
25
|
+
next.inputPaths = [...cfg.paths];
|
|
26
|
+
next.inputPathsSet = true;
|
|
27
|
+
}
|
|
28
|
+
return next;
|
|
29
|
+
};
|
|
30
|
+
/**
|
|
31
|
+
* Read a config from a user-specified path (used by `--config <path>`).
|
|
32
|
+
* Throws on parse / validation failure with the offending path included.
|
|
33
|
+
*/
|
|
34
|
+
export const loadConfigFromPath = async (path) => {
|
|
35
|
+
let text;
|
|
36
|
+
try {
|
|
37
|
+
text = await readFile(path, 'utf8');
|
|
38
|
+
}
|
|
39
|
+
catch (err) {
|
|
40
|
+
throw new Error(`Failed to read config "${path}": ${err.message}`);
|
|
41
|
+
}
|
|
42
|
+
try {
|
|
43
|
+
return parseConfig(text);
|
|
44
|
+
}
|
|
45
|
+
catch (err) {
|
|
46
|
+
throw new Error(`Invalid config at "${path}": ${err.message}`);
|
|
47
|
+
}
|
|
48
|
+
};
|
|
49
|
+
//# sourceMappingURL=config-merge.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"config-merge.js","sourceRoot":"","sources":["../src/config-merge.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAA0B,WAAW,EAAE,MAAM,mBAAmB,CAAC;AAQxE;;;;GAIG;AACH,MAAM,CAAC,MAAM,WAAW,GAAG,CAAC,IAAgB,EAAE,IAAwB,EAAc,EAAE;IACpF,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,CAAC;IACxB,IAAI,CAAC,GAAG;QAAE,OAAO,IAAI,CAAC;IACtB,MAAM,IAAI,GAAe,EAAE,GAAG,IAAI,EAAE,OAAO,EAAE,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC,EAAE,QAAQ,EAAE,CAAC,GAAG,IAAI,CAAC,QAAQ,CAAC,EAAE,CAAC;IAC/F,2EAA2E;IAC3E,yEAAyE;IACzE,iDAAiD;IACjD,IAAI,CAAC,IAAI,CAAC,SAAS,IAAI,GAAG,CAAC,MAAM,IAAI,GAAG,CAAC,MAAM,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC3D,IAAI,CAAC,QAAQ,GAAG,CAAC,GAAG,GAAG,CAAC,MAAM,CAAC,CAAC;QAChC,IAAI,CAAC,SAAS,GAAG,IAAI,CAAC;IACxB,CAAC;IACD,IAAI,CAAC,IAAI,CAAC,UAAU,IAAI,GAAG,CAAC,OAAO,IAAI,GAAG,CAAC,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC9D,IAAI,CAAC,OAAO,GAAG,CAAC,GAAG,GAAG,CAAC,OAAO,CAAC,CAAC;QAChC,IAAI,CAAC,UAAU,GAAG,IAAI,CAAC;IACzB,CAAC;IACD,IAAI,CAAC,IAAI,CAAC,aAAa,IAAI,GAAG,CAAC,KAAK,IAAI,GAAG,CAAC,KAAK,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QAC7D,IAAI,CAAC,UAAU,GAAG,CAAC,GAAG,GAAG,CAAC,KAAK,CAAC,CAAC;QACjC,IAAI,CAAC,aAAa,GAAG,IAAI,CAAC;IAC5B,CAAC;IACD,OAAO,IAAI,CAAC;AACd,CAAC,CAAC;AAEF;;;GAGG;AACH,MAAM,CAAC,MAAM,kBAAkB,GAAG,KAAK,EAAE,IAAY,EAA8B,EAAE;IACnF,IAAI,IAAY,CAAC;IACjB,IAAI,CAAC;QACH,IAAI,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;IACtC,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,KAAK,CAAC,0BAA0B,IAAI,MAAO,GAAa,CAAC,OAAO,EAAE,CAAC,CAAC;IAChF,CAAC;IACD,IAAI,CAAC;QACH,OAAO,WAAW,CAAC,IAAI,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,IAAI,KAAK,CAAC,sBAAsB,IAAI,MAAO,GAAa,CAAC,OAAO,EAAE,CAAC,CAAC;IAC5E,CAAC;AACH,CAAC,CAAC"}
|
package/dist/index.d.ts
CHANGED
|
@@ -1,3 +1,14 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
|
|
2
|
+
import type { LatencyResult, MeasureLatencyOptions } from '@tokenometer/core';
|
|
3
|
+
import { type ImageSizeReader } from './vision.js';
|
|
4
|
+
/** Test seam for `measureLatency`; production code uses the SDK-backed default. */
|
|
5
|
+
export type MeasureLatencyFn = (options: MeasureLatencyOptions) => Promise<LatencyResult>;
|
|
6
|
+
interface RunDeps {
|
|
7
|
+
imageSizeReader?: ImageSizeReader;
|
|
8
|
+
measureLatencyFn?: MeasureLatencyFn;
|
|
9
|
+
stderr?: NodeJS.WriteStream;
|
|
10
|
+
stdout?: NodeJS.WriteStream;
|
|
11
|
+
}
|
|
12
|
+
export declare const main: (argv: readonly string[], deps?: RunDeps) => Promise<number>;
|
|
13
|
+
export {};
|
|
3
14
|
//# sourceMappingURL=index.d.ts.map
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAeA,OAAO,KAAK,EAGV,aAAa,EACb,qBAAqB,EAEtB,MAAM,mBAAmB,CAAC;AAK3B,OAAO,EACL,KAAK,eAAe,EAIrB,MAAM,aAAa,CAAC;AAsDrB,mFAAmF;AACnF,MAAM,MAAM,gBAAgB,GAAG,CAAC,OAAO,EAAE,qBAAqB,KAAK,OAAO,CAAC,aAAa,CAAC,CAAC;AAE1F,UAAU,OAAO;IACf,eAAe,CAAC,EAAE,eAAe,CAAC;IAClC,gBAAgB,CAAC,EAAE,gBAAgB,CAAC;IACpC,MAAM,CAAC,EAAE,MAAM,CAAC,WAAW,CAAC;IAC5B,MAAM,CAAC,EAAE,MAAM,CAAC,WAAW,CAAC;CAC7B;AAuFD,eAAO,MAAM,IAAI,GAAU,MAAM,SAAS,MAAM,EAAE,EAAE,OAAM,OAAY,KAAG,OAAO,CAAC,MAAM,CAwJtF,CAAC"}
|
package/dist/index.js
CHANGED
|
@@ -2,18 +2,27 @@
|
|
|
2
2
|
import { realpathSync } from 'node:fs';
|
|
3
3
|
import { readFile } from 'node:fs/promises';
|
|
4
4
|
import { pathToFileURL } from 'node:url';
|
|
5
|
-
import { tokenizeMatrix, tokenizeMatrixEmpirical } from '@tokenometer/core';
|
|
5
|
+
import { getModel, getRate, loadConfig, measureLatency, toSarif, tokenizeMatrix, tokenizeMatrixEmpirical, } from '@tokenometer/core';
|
|
6
6
|
import { HELP_TEXT, parseArgs } from './args.js';
|
|
7
|
-
import {
|
|
7
|
+
import { autoDetectDefaultModel } from './auto-detect.js';
|
|
8
|
+
import { applyConfig, loadConfigFromPath } from './config-merge.js';
|
|
9
|
+
import { renderByFile, renderModelLimits, renderSummary, renderTable } from './render.js';
|
|
10
|
+
import { computeVisionTokens, defaultImageSizeReader, resolveImages, } from './vision.js';
|
|
8
11
|
const VERSION = '0.0.2';
|
|
9
12
|
const readEnv = () => {
|
|
10
13
|
const env = {};
|
|
11
|
-
const { ANTHROPIC_API_KEY, GEMINI_API_KEY, GOOGLE_API_KEY } = process.env;
|
|
14
|
+
const { ANTHROPIC_API_KEY, COHERE_API_KEY, GEMINI_API_KEY, GOOGLE_API_KEY, MISTRAL_API_KEY, OPENAI_API_KEY, } = process.env;
|
|
12
15
|
if (ANTHROPIC_API_KEY)
|
|
13
16
|
env.anthropicApiKey = ANTHROPIC_API_KEY;
|
|
17
|
+
if (COHERE_API_KEY)
|
|
18
|
+
env.cohereApiKey = COHERE_API_KEY;
|
|
14
19
|
const googleKey = GOOGLE_API_KEY ?? GEMINI_API_KEY;
|
|
15
20
|
if (googleKey)
|
|
16
21
|
env.googleApiKey = googleKey;
|
|
22
|
+
if (MISTRAL_API_KEY)
|
|
23
|
+
env.mistralApiKey = MISTRAL_API_KEY;
|
|
24
|
+
if (OPENAI_API_KEY)
|
|
25
|
+
env.openaiApiKey = OPENAI_API_KEY;
|
|
17
26
|
return env;
|
|
18
27
|
};
|
|
19
28
|
const readStdin = async () => {
|
|
@@ -23,64 +32,228 @@ const readStdin = async () => {
|
|
|
23
32
|
}
|
|
24
33
|
return Buffer.concat(chunks).toString('utf8');
|
|
25
34
|
};
|
|
26
|
-
const
|
|
35
|
+
const readJoinedPrompt = async (paths) => {
|
|
27
36
|
if (paths.length === 0 || paths[0] === '-') {
|
|
28
37
|
return readStdin();
|
|
29
38
|
}
|
|
30
39
|
const contents = await Promise.all(paths.map((p) => readFile(p, 'utf8')));
|
|
31
40
|
return contents.join('\n');
|
|
32
41
|
};
|
|
33
|
-
|
|
42
|
+
const readPerFilePrompts = async (paths) => {
|
|
43
|
+
if (paths.length === 0 || paths[0] === '-') {
|
|
44
|
+
const prompt = await readStdin();
|
|
45
|
+
return [{ path: '-', prompt }];
|
|
46
|
+
}
|
|
47
|
+
return Promise.all(paths.map(async (path) => {
|
|
48
|
+
const prompt = await readFile(path, 'utf8');
|
|
49
|
+
return { path, prompt };
|
|
50
|
+
}));
|
|
51
|
+
};
|
|
52
|
+
/**
|
|
53
|
+
* For each cell, run a real streaming generation and attach the resulting
|
|
54
|
+
* `LatencyResult` in-place. Skips cells whose provider doesn't yet support
|
|
55
|
+
* the latency path (currently: none — all 5 do).
|
|
56
|
+
*/
|
|
57
|
+
const augmentWithLatency = async (cells, prompt, parsed, measure) => {
|
|
58
|
+
// Sequential (not parallel) so we don't slam a single provider with N
|
|
59
|
+
// concurrent metered requests; rate-limits are real.
|
|
60
|
+
for (const cell of cells) {
|
|
61
|
+
cell.latency = await measure({
|
|
62
|
+
env: readEnv(),
|
|
63
|
+
modelId: cell.model,
|
|
64
|
+
prompt,
|
|
65
|
+
trials: parsed.latencyTrials,
|
|
66
|
+
});
|
|
67
|
+
}
|
|
68
|
+
};
|
|
69
|
+
const buildPerFileResults = async (parsed, useEmpirical, measureLatencyFn) => {
|
|
70
|
+
const inputs = await readPerFilePrompts(parsed.inputPaths);
|
|
71
|
+
const out = [];
|
|
72
|
+
for (const { path, prompt } of inputs) {
|
|
73
|
+
if (!prompt.trim())
|
|
74
|
+
continue;
|
|
75
|
+
const cells = useEmpirical
|
|
76
|
+
? await tokenizeMatrixEmpirical({
|
|
77
|
+
env: readEnv(),
|
|
78
|
+
formats: parsed.formats,
|
|
79
|
+
modelIds: parsed.modelIds,
|
|
80
|
+
prompt,
|
|
81
|
+
})
|
|
82
|
+
: tokenizeMatrix({
|
|
83
|
+
formats: parsed.formats,
|
|
84
|
+
modelIds: parsed.modelIds,
|
|
85
|
+
prompt,
|
|
86
|
+
});
|
|
87
|
+
if (parsed.latency) {
|
|
88
|
+
await augmentWithLatency(cells, prompt, parsed, measureLatencyFn ?? measureLatency);
|
|
89
|
+
}
|
|
90
|
+
out.push({ path, results: cells });
|
|
91
|
+
}
|
|
92
|
+
return out;
|
|
93
|
+
};
|
|
94
|
+
const buildImageResults = async (parsed, reader) => {
|
|
95
|
+
if (parsed.imagePaths.length === 0)
|
|
96
|
+
return [];
|
|
97
|
+
const resolved = await resolveImages(parsed.imagePaths, reader);
|
|
98
|
+
const results = [];
|
|
99
|
+
for (const img of resolved) {
|
|
100
|
+
const cells = parsed.modelIds.map((modelId) => {
|
|
101
|
+
const tokens = computeVisionTokens(modelId, img.dim, img.path);
|
|
102
|
+
return makeVisionCell(modelId, tokens);
|
|
103
|
+
});
|
|
104
|
+
results.push({ path: `${img.path} [vision]`, results: cells });
|
|
105
|
+
}
|
|
106
|
+
return results;
|
|
107
|
+
};
|
|
108
|
+
const makeVisionCell = (modelId, tokens) => {
|
|
109
|
+
// Vision tokens are formula-derived; we fold them under format='text' as a
|
|
110
|
+
// neutral synthetic axis (vision is format-agnostic).
|
|
111
|
+
const rate = getRate(modelId);
|
|
112
|
+
const provider = getModel(modelId).provider;
|
|
113
|
+
return {
|
|
114
|
+
approximate: true,
|
|
115
|
+
format: 'text',
|
|
116
|
+
inputCost: (tokens / 1000) * rate.inputPer1k,
|
|
117
|
+
inputTokens: tokens,
|
|
118
|
+
model: modelId,
|
|
119
|
+
provider,
|
|
120
|
+
tokenizer: 'heuristic',
|
|
121
|
+
};
|
|
122
|
+
};
|
|
123
|
+
export const main = async (argv, deps = {}) => {
|
|
124
|
+
const stdout = deps.stdout ?? process.stdout;
|
|
125
|
+
const stderr = deps.stderr ?? process.stderr;
|
|
126
|
+
const reader = deps.imageSizeReader ?? defaultImageSizeReader;
|
|
34
127
|
let parsed;
|
|
35
128
|
try {
|
|
36
129
|
parsed = parseArgs(argv);
|
|
37
130
|
}
|
|
38
131
|
catch (err) {
|
|
39
|
-
|
|
132
|
+
stderr.write(`${err.message}\n\n${HELP_TEXT}`);
|
|
40
133
|
return 2;
|
|
41
134
|
}
|
|
42
135
|
if (parsed.help) {
|
|
43
|
-
|
|
136
|
+
stdout.write(HELP_TEXT);
|
|
44
137
|
return 0;
|
|
45
138
|
}
|
|
46
139
|
if (parsed.version) {
|
|
47
|
-
|
|
140
|
+
stdout.write(`tokenometer ${VERSION}\n`);
|
|
48
141
|
return 0;
|
|
49
142
|
}
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
143
|
+
// Apply config defaults (before auto-detect).
|
|
144
|
+
if (!parsed.noConfig) {
|
|
145
|
+
try {
|
|
146
|
+
if (parsed.configPath) {
|
|
147
|
+
const cfg = await loadConfigFromPath(parsed.configPath);
|
|
148
|
+
parsed = applyConfig(parsed, { config: cfg });
|
|
149
|
+
}
|
|
150
|
+
else {
|
|
151
|
+
const cfg = await loadConfig();
|
|
152
|
+
parsed = applyConfig(parsed, { config: cfg });
|
|
153
|
+
}
|
|
154
|
+
}
|
|
155
|
+
catch (err) {
|
|
156
|
+
stderr.write(`${err.message}\n`);
|
|
157
|
+
return 1;
|
|
158
|
+
}
|
|
53
159
|
}
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
160
|
+
// Auto-detect default model when neither user nor config set --model.
|
|
161
|
+
if (!parsed.modelsSet) {
|
|
162
|
+
const detected = autoDetectDefaultModel();
|
|
163
|
+
parsed.modelIds = [detected.modelId];
|
|
164
|
+
if (detected.note)
|
|
165
|
+
stderr.write(`${detected.note}\n`);
|
|
57
166
|
}
|
|
58
|
-
|
|
59
|
-
|
|
167
|
+
// Validate that we have at least one input source. (Prompt files may come
|
|
168
|
+
// from positional args or config.paths; images are optional but if they're
|
|
169
|
+
// the only input, that's a misuse.)
|
|
170
|
+
if (parsed.inputPaths.length === 0 && parsed.imagePaths.length === 0) {
|
|
171
|
+
stderr.write('No input files. Pass a path, "-" for stdin, or set paths in .tokenometer.yml.\n');
|
|
60
172
|
return 1;
|
|
61
173
|
}
|
|
62
174
|
const useEmpirical = parsed.empirical && !parsed.offline;
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
prompt
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
175
|
+
// For json/sarif output, we MUST have per-file results.
|
|
176
|
+
// For table output, if --by-file or --image is on, we need per-file too.
|
|
177
|
+
// Otherwise the existing joined-prompt path is used.
|
|
178
|
+
if (parsed.output === 'json' || parsed.output === 'sarif') {
|
|
179
|
+
const fileResults = parsed.inputPaths.length > 0
|
|
180
|
+
? await buildPerFileResults(parsed, useEmpirical, deps.measureLatencyFn)
|
|
181
|
+
: [];
|
|
182
|
+
const imageResults = await buildImageResults(parsed, reader);
|
|
183
|
+
const result = { files: [...fileResults, ...imageResults] };
|
|
184
|
+
if (result.files.length === 0) {
|
|
185
|
+
stderr.write('Empty prompt — nothing to measure.\n');
|
|
186
|
+
return 1;
|
|
187
|
+
}
|
|
188
|
+
if (parsed.output === 'json') {
|
|
189
|
+
stdout.write(`${JSON.stringify(result, null, 2)}\n`);
|
|
190
|
+
}
|
|
191
|
+
else {
|
|
192
|
+
const sarif = toSarif(result, { toolVersion: VERSION });
|
|
193
|
+
stdout.write(`${JSON.stringify(sarif, null, 2)}\n`);
|
|
194
|
+
}
|
|
195
|
+
return 0;
|
|
196
|
+
}
|
|
197
|
+
// Table output path.
|
|
198
|
+
let prompt = '';
|
|
199
|
+
if (parsed.inputPaths.length > 0) {
|
|
200
|
+
try {
|
|
201
|
+
prompt = await readJoinedPrompt(parsed.inputPaths);
|
|
202
|
+
}
|
|
203
|
+
catch (err) {
|
|
204
|
+
stderr.write(`Failed to read input: ${err.message}\n`);
|
|
205
|
+
return 1;
|
|
206
|
+
}
|
|
207
|
+
}
|
|
208
|
+
// Allow image-only invocation when there are no prompt files.
|
|
209
|
+
if (parsed.inputPaths.length > 0 && !prompt.trim()) {
|
|
210
|
+
stderr.write('Empty prompt — nothing to measure.\n');
|
|
211
|
+
return 1;
|
|
212
|
+
}
|
|
213
|
+
let mainResults = [];
|
|
214
|
+
if (parsed.inputPaths.length > 0) {
|
|
215
|
+
mainResults = useEmpirical
|
|
216
|
+
? await tokenizeMatrixEmpirical({
|
|
217
|
+
env: readEnv(),
|
|
218
|
+
formats: parsed.formats,
|
|
219
|
+
modelIds: parsed.modelIds,
|
|
220
|
+
prompt,
|
|
221
|
+
})
|
|
222
|
+
: tokenizeMatrix({
|
|
223
|
+
formats: parsed.formats,
|
|
224
|
+
modelIds: parsed.modelIds,
|
|
225
|
+
prompt,
|
|
226
|
+
});
|
|
227
|
+
if (parsed.latency) {
|
|
228
|
+
await augmentWithLatency(mainResults, prompt, parsed, deps.measureLatencyFn ?? measureLatency);
|
|
229
|
+
}
|
|
230
|
+
}
|
|
231
|
+
// Compute image rows up front (needed for main table appendage and by-file).
|
|
232
|
+
const imageFileResults = await buildImageResults(parsed, reader);
|
|
233
|
+
const imageCells = imageFileResults.flatMap((f) => f.results);
|
|
234
|
+
const allMainCells = [...mainResults, ...imageCells];
|
|
235
|
+
stdout.write(`${renderTable(allMainCells)}\n`);
|
|
236
|
+
const limits = renderModelLimits(allMainCells);
|
|
77
237
|
if (limits)
|
|
78
|
-
|
|
79
|
-
const summary = renderSummary(
|
|
238
|
+
stdout.write(`${limits}\n`);
|
|
239
|
+
const summary = renderSummary(allMainCells);
|
|
80
240
|
if (summary)
|
|
81
|
-
|
|
241
|
+
stdout.write(`${summary}\n`);
|
|
242
|
+
// by-file table requires per-file results from prompt files plus image virtual files.
|
|
243
|
+
if (parsed.byFile) {
|
|
244
|
+
const perFile = parsed.inputPaths.length > 0
|
|
245
|
+
? await buildPerFileResults(parsed, useEmpirical, deps.measureLatencyFn)
|
|
246
|
+
: [];
|
|
247
|
+
const allFiles = [...perFile, ...imageFileResults];
|
|
248
|
+
const byFile = renderByFile(allFiles);
|
|
249
|
+
if (byFile)
|
|
250
|
+
stdout.write(`${byFile}\n`);
|
|
251
|
+
}
|
|
82
252
|
if (useEmpirical) {
|
|
83
|
-
|
|
253
|
+
stdout.write('\n(empirical: Anthropic / Google counts via provider countTokens API; OpenAI via tiktoken o200k_base)\n');
|
|
254
|
+
}
|
|
255
|
+
if (parsed.latency) {
|
|
256
|
+
stdout.write(`(latency: ${parsed.latencyTrials} streaming generation${parsed.latencyTrials === 1 ? '' : 's'} per cell, max_tokens=200; p50/p95/mean over trials)\n`);
|
|
84
257
|
}
|
|
85
258
|
return 0;
|
|
86
259
|
};
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,
|
|
1
|
+
{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AACA,OAAO,EAAE,YAAY,EAAE,MAAM,SAAS,CAAC;AACvC,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,MAAM,UAAU,CAAC;AACzC,OAAO,EAGL,QAAQ,EACR,OAAO,EACP,UAAU,EACV,cAAc,EACd,OAAO,EACP,cAAc,EACd,uBAAuB,GACxB,MAAM,mBAAmB,CAAC;AAQ3B,OAAO,EAAE,SAAS,EAAmB,SAAS,EAAE,MAAM,WAAW,CAAC;AAClE,OAAO,EAAE,sBAAsB,EAAE,MAAM,kBAAkB,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,kBAAkB,EAAE,MAAM,mBAAmB,CAAC;AACpE,OAAO,EAAE,YAAY,EAAE,iBAAiB,EAAE,aAAa,EAAE,WAAW,EAAE,MAAM,aAAa,CAAC;AAC1F,OAAO,EAEL,mBAAmB,EACnB,sBAAsB,EACtB,aAAa,GACd,MAAM,aAAa,CAAC;AAErB,MAAM,OAAO,GAAG,OAAO,CAAC;AAExB,MAAM,OAAO,GAAG,GAAiB,EAAE;IACjC,MAAM,GAAG,GAAiB,EAAE,CAAC;IAC7B,MAAM,EACJ,iBAAiB,EACjB,cAAc,EACd,cAAc,EACd,cAAc,EACd,eAAe,EACf,cAAc,GACf,GAAG,OAAO,CAAC,GAAG,CAAC;IAChB,IAAI,iBAAiB;QAAE,GAAG,CAAC,eAAe,GAAG,iBAAiB,CAAC;IAC/D,IAAI,cAAc;QAAE,GAAG,CAAC,YAAY,GAAG,cAAc,CAAC;IACtD,MAAM,SAAS,GAAG,cAAc,IAAI,cAAc,CAAC;IACnD,IAAI,SAAS;QAAE,GAAG,CAAC,YAAY,GAAG,SAAS,CAAC;IAC5C,IAAI,eAAe;QAAE,GAAG,CAAC,aAAa,GAAG,eAAe,CAAC;IACzD,IAAI,cAAc;QAAE,GAAG,CAAC,YAAY,GAAG,cAAc,CAAC;IACtD,OAAO,GAAG,CAAC;AACb,CAAC,CAAC;AAEF,MAAM,SAAS,GAAG,KAAK,IAAqB,EAAE;IAC5C,MAAM,MAAM,GAAa,EAAE,CAAC;IAC5B,IAAI,KAAK,EAAE,MAAM,KAAK,IAAI,OAAO,CAAC,KAAK,EAAE,CAAC;QACxC,MAAM,CAAC,IAAI,CAAC,KAAe,CAAC,CAAC;IAC/B,CAAC;IACD,OAAO,MAAM,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,QAAQ,CAAC,MAAM,CAAC,CAAC;AAChD,CAAC,CAAC;AAEF,MAAM,gBAAgB,GAAG,KAAK,EAAE,KAAwB,EAAmB,EAAE;IAC3E,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,EAAE,CAAC;QAC3C,OAAO,SAAS,EAAE,CAAC;IACrB,CAAC;IACD,MAAM,QAAQ,GAAG,MAAM,OAAO,CAAC,GAAG,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,CAAC;IAC1E,OAAO,QAAQ,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC7B,CAAC,CAAC;AAEF,MAAM,kBAAkB,GAAG,KAAK,EAC9B,KAAwB,EACqB,EAAE;IAC/C,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,EAAE,CAAC;QAC3C,MAAM,MAAM,GAAG,MAAM,SAAS,EAAE,CAAC;QACjC,OAAO,CAAC,EAAE,IAAI,EAAE,GAAG,EAAE,MAAM,EAAE,CAAC,CAAC;IACjC,CAAC;IACD,OAAO,OAAO,CAAC,GAAG,CAChB,KAAK,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,EAAE,EAAE;QACvB,MAAM,MAAM,GAAG,MAAM,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC;QAC5C,OAAO,EAAE,IAAI,EAAE,MAAM,EAAE,CAAC;IAC1B,CAAC,CAAC,CACH,CAAC;AACJ,CAAC,CAAC;AAYF;;;;GAIG;AACH,MAAM,kBAAkB,GAAG,KAAK,EAC9B,KAAuB,EACvB,MAAc,EACd,MAAkB,EAClB,OAAyB,EACV,EAAE;IACjB,sEAAsE;IACtE,qDAAqD;IACrD,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,IAAI,CAAC,OAAO,GAAG,MAAM,OAAO,CAAC;YAC3B,GAAG,EAAE,OAAO,EAAE;YACd,OAAO,EAAE,IAAI,CAAC,KAAK;YACnB,MAAM;YACN,MAAM,EAAE,MAAM,CAAC,aAAa;SAC7B,CAAC,CAAC;IACL,CAAC;AACH,CAAC,CAAC;AAEF,MAAM,mBAAmB,GAAG,KAAK,EAC/B,MAAkB,EAClB,YAAqB,EACrB,gBAA8C,EACZ,EAAE;IACpC,MAAM,MAAM,GAAG,MAAM,kBAAkB,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC;IAC3D,MAAM,GAAG,GAA4B,EAAE,CAAC;IACxC,KAAK,MAAM,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,MAAM,EAAE,CAAC;QACtC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE;YAAE,SAAS;QAC7B,MAAM,KAAK,GAAqB,YAAY;YAC1C,CAAC,CAAC,MAAM,uBAAuB,CAAC;gBAC5B,GAAG,EAAE,OAAO,EAAE;gBACd,OAAO,EAAE,MAAM,CAAC,OAAO;gBACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;gBACzB,MAAM;aACP,CAAC;YACJ,CAAC,CAAC,cAAc,CAAC;gBACb,OAAO,EAAE,MAAM,CAAC,OAAO;gBACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;gBACzB,MAAM;aACP,CAAC,CAAC;QACP,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;YACnB,MAAM,kBAAkB,CAAC,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,gBAAgB,IAAI,cAAc,CAAC,CAAC;QACtF,CAAC;QACD,GAAG,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC;IACrC,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC,CAAC;AAEF,MAAM,iBAAiB,GAAG,KAAK,EAC7B,MAAkB,EAClB,MAAuB,EACW,EAAE;IACpC,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IAC9C,MAAM,QAAQ,GAAG,MAAM,aAAa,CAAC,MAAM,CAAC,UAAU,EAAE,MAAM,CAAC,CAAC;IAChE,MAAM,OAAO,GAA4B,EAAE,CAAC;IAC5C,KAAK,MAAM,GAAG,IAAI,QAAQ,EAAE,CAAC;QAC3B,MAAM,KAAK,GAAqB,MAAM,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,EAAE;YAC9D,MAAM,MAAM,GAAG,mBAAmB,CAAC,OAAO,EAAE,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,IAAI,CAAC,CAAC;YAC/D,OAAO,cAAc,CAAC,OAAO,EAAE,MAAM,CAAC,CAAC;QACzC,CAAC,CAAC,CAAC;QACH,OAAO,CAAC,IAAI,CAAC,EAAE,IAAI,EAAE,GAAG,GAAG,CAAC,IAAI,WAAW,EAAE,OAAO,EAAE,KAAK,EAAE,CAAC,CAAC;IACjE,CAAC;IACD,OAAO,OAAO,CAAC;AACjB,CAAC,CAAC;AAEF,MAAM,cAAc,GAAG,CAAC,OAAe,EAAE,MAAc,EAAkB,EAAE;IACzE,2EAA2E;IAC3E,sDAAsD;IACtD,MAAM,IAAI,GAAG,OAAO,CAAC,OAAO,CAAC,CAAC;IAC9B,MAAM,QAAQ,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC,QAAQ,CAAC;IAC5C,OAAO;QACL,WAAW,EAAE,IAAI;QACjB,MAAM,EAAE,MAAgB;QACxB,SAAS,EAAE,CAAC,MAAM,GAAG,IAAI,CAAC,GAAG,IAAI,CAAC,UAAU;QAC5C,WAAW,EAAE,MAAM;QACnB,KAAK,EAAE,OAAO;QACd,QAAQ;QACR,SAAS,EAAE,WAAW;KACvB,CAAC;AACJ,CAAC,CAAC;AAEF,MAAM,CAAC,MAAM,IAAI,GAAG,KAAK,EAAE,IAAuB,EAAE,OAAgB,EAAE,EAAmB,EAAE;IACzF,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,IAAI,OAAO,CAAC,MAAM,CAAC;IAC7C,MAAM,MAAM,GAAG,IAAI,CAAC,MAAM,IAAI,OAAO,CAAC,MAAM,CAAC;IAC7C,MAAM,MAAM,GAAG,IAAI,CAAC,eAAe,IAAI,sBAAsB,CAAC;IAE9D,IAAI,MAAkB,CAAC;IACvB,IAAI,CAAC;QACH,MAAM,GAAG,SAAS,CAAC,IAAI,CAAC,CAAC;IAC3B,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,CAAC,KAAK,CAAC,GAAI,GAAa,CAAC,OAAO,OAAO,SAAS,EAAE,CAAC,CAAC;QAC1D,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,MAAM,CAAC,IAAI,EAAE,CAAC;QAChB,MAAM,CAAC,KAAK,CAAC,SAAS,CAAC,CAAC;QACxB,OAAO,CAAC,CAAC;IACX,CAAC;IACD,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;QACnB,MAAM,CAAC,KAAK,CAAC,eAAe,OAAO,IAAI,CAAC,CAAC;QACzC,OAAO,CAAC,CAAC;IACX,CAAC;IAED,8CAA8C;IAC9C,IAAI,CAAC,MAAM,CAAC,QAAQ,EAAE,CAAC;QACrB,IAAI,CAAC;YACH,IAAI,MAAM,CAAC,UAAU,EAAE,CAAC;gBACtB,MAAM,GAAG,GAAG,MAAM,kBAAkB,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC;gBACxD,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,CAAC,CAAC;YAChD,CAAC;iBAAM,CAAC;gBACN,MAAM,GAAG,GAAG,MAAM,UAAU,EAAE,CAAC;gBAC/B,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE,EAAE,MAAM,EAAE,GAAG,EAAE,CAAC,CAAC;YAChD,CAAC;QACH,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,CAAC,KAAK,CAAC,GAAI,GAAa,CAAC,OAAO,IAAI,CAAC,CAAC;YAC5C,OAAO,CAAC,CAAC;QACX,CAAC;IACH,CAAC;IAED,sEAAsE;IACtE,IAAI,CAAC,MAAM,CAAC,SAAS,EAAE,CAAC;QACtB,MAAM,QAAQ,GAAG,sBAAsB,EAAE,CAAC;QAC1C,MAAM,CAAC,QAAQ,GAAG,CAAC,QAAQ,CAAC,OAAO,CAAC,CAAC;QACrC,IAAI,QAAQ,CAAC,IAAI;YAAE,MAAM,CAAC,KAAK,CAAC,GAAG,QAAQ,CAAC,IAAI,IAAI,CAAC,CAAC;IACxD,CAAC;IAED,0EAA0E;IAC1E,2EAA2E;IAC3E,oCAAoC;IACpC,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACrE,MAAM,CAAC,KAAK,CAAC,iFAAiF,CAAC,CAAC;QAChG,OAAO,CAAC,CAAC;IACX,CAAC;IAED,MAAM,YAAY,GAAG,MAAM,CAAC,SAAS,IAAI,CAAC,MAAM,CAAC,OAAO,CAAC;IAEzD,wDAAwD;IACxD,yEAAyE;IACzE,qDAAqD;IACrD,IAAI,MAAM,CAAC,MAAM,KAAK,MAAM,IAAI,MAAM,CAAC,MAAM,KAAK,OAAO,EAAE,CAAC;QAC1D,MAAM,WAAW,GACf,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC;YAC1B,CAAC,CAAC,MAAM,mBAAmB,CAAC,MAAM,EAAE,YAAY,EAAE,IAAI,CAAC,gBAAgB,CAAC;YACxE,CAAC,CAAC,EAAE,CAAC;QACT,MAAM,YAAY,GAAG,MAAM,iBAAiB,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAC7D,MAAM,MAAM,GAAsB,EAAE,KAAK,EAAE,CAAC,GAAG,WAAW,EAAE,GAAG,YAAY,CAAC,EAAE,CAAC;QAC/E,IAAI,MAAM,CAAC,KAAK,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;YAC9B,MAAM,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;YACrD,OAAO,CAAC,CAAC;QACX,CAAC;QACD,IAAI,MAAM,CAAC,MAAM,KAAK,MAAM,EAAE,CAAC;YAC7B,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QACvD,CAAC;aAAM,CAAC;YACN,MAAM,KAAK,GAAG,OAAO,CAAC,MAAM,EAAE,EAAE,WAAW,EAAE,OAAO,EAAE,CAAC,CAAC;YACxD,MAAM,CAAC,KAAK,CAAC,GAAG,IAAI,CAAC,SAAS,CAAC,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC,IAAI,CAAC,CAAC;QACtD,CAAC;QACD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,qBAAqB;IACrB,IAAI,MAAM,GAAG,EAAE,CAAC;IAChB,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACjC,IAAI,CAAC;YACH,MAAM,GAAG,MAAM,gBAAgB,CAAC,MAAM,CAAC,UAAU,CAAC,CAAC;QACrD,CAAC;QAAC,OAAO,GAAG,EAAE,CAAC;YACb,MAAM,CAAC,KAAK,CAAC,yBAA0B,GAAa,CAAC,OAAO,IAAI,CAAC,CAAC;YAClE,OAAO,CAAC,CAAC;QACX,CAAC;IACH,CAAC;IAED,8DAA8D;IAC9D,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,MAAM,CAAC,IAAI,EAAE,EAAE,CAAC;QACnD,MAAM,CAAC,KAAK,CAAC,sCAAsC,CAAC,CAAC;QACrD,OAAO,CAAC,CAAC;IACX,CAAC;IAED,IAAI,WAAW,GAAqB,EAAE,CAAC;IACvC,IAAI,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACjC,WAAW,GAAG,YAAY;YACxB,CAAC,CAAC,MAAM,uBAAuB,CAAC;gBAC5B,GAAG,EAAE,OAAO,EAAE;gBACd,OAAO,EAAE,MAAM,CAAC,OAAO;gBACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;gBACzB,MAAM;aACP,CAAC;YACJ,CAAC,CAAC,cAAc,CAAC;gBACb,OAAO,EAAE,MAAM,CAAC,OAAO;gBACvB,QAAQ,EAAE,MAAM,CAAC,QAAQ;gBACzB,MAAM;aACP,CAAC,CAAC;QACP,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;YACnB,MAAM,kBAAkB,CACtB,WAAW,EACX,MAAM,EACN,MAAM,EACN,IAAI,CAAC,gBAAgB,IAAI,cAAc,CACxC,CAAC;QACJ,CAAC;IACH,CAAC;IAED,6EAA6E;IAC7E,MAAM,gBAAgB,GAAG,MAAM,iBAAiB,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjE,MAAM,UAAU,GAAG,gBAAgB,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC;IAC9D,MAAM,YAAY,GAAqB,CAAC,GAAG,WAAW,EAAE,GAAG,UAAU,CAAC,CAAC;IAEvE,MAAM,CAAC,KAAK,CAAC,GAAG,WAAW,CAAC,YAAY,CAAC,IAAI,CAAC,CAAC;IAC/C,MAAM,MAAM,GAAG,iBAAiB,CAAC,YAAY,CAAC,CAAC;IAC/C,IAAI,MAAM;QAAE,MAAM,CAAC,KAAK,CAAC,GAAG,MAAM,IAAI,CAAC,CAAC;IACxC,MAAM,OAAO,GAAG,aAAa,CAAC,YAAY,CAAC,CAAC;IAC5C,IAAI,OAAO;QAAE,MAAM,CAAC,KAAK,CAAC,GAAG,OAAO,IAAI,CAAC,CAAC;IAE1C,sFAAsF;IACtF,IAAI,MAAM,CAAC,MAAM,EAAE,CAAC;QAClB,MAAM,OAAO,GACX,MAAM,CAAC,UAAU,CAAC,MAAM,GAAG,CAAC;YAC1B,CAAC,CAAC,MAAM,mBAAmB,CAAC,MAAM,EAAE,YAAY,EAAE,IAAI,CAAC,gBAAgB,CAAC;YACxE,CAAC,CAAC,EAAE,CAAC;QACT,MAAM,QAAQ,GAAG,CAAC,GAAG,OAAO,EAAE,GAAG,gBAAgB,CAAC,CAAC;QACnD,MAAM,MAAM,GAAG,YAAY,CAAC,QAAQ,CAAC,CAAC;QACtC,IAAI,MAAM;YAAE,MAAM,CAAC,KAAK,CAAC,GAAG,MAAM,IAAI,CAAC,CAAC;IAC1C,CAAC;IAED,IAAI,YAAY,EAAE,CAAC;QACjB,MAAM,CAAC,KAAK,CACV,yGAAyG,CAC1G,CAAC;IACJ,CAAC;IACD,IAAI,MAAM,CAAC,OAAO,EAAE,CAAC;QACnB,MAAM,CAAC,KAAK,CACV,aAAa,MAAM,CAAC,aAAa,wBAAwB,MAAM,CAAC,aAAa,KAAK,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,wDAAwD,CACvJ,CAAC;IACJ,CAAC;IACD,OAAO,CAAC,CAAC;AACX,CAAC,CAAC;AAEF,MAAM,iBAAiB,GAAG,GAAY,EAAE;IACtC,MAAM,KAAK,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAC9B,IAAI,CAAC,KAAK;QAAE,OAAO,KAAK,CAAC;IACzB,IAAI,CAAC;QACH,OAAO,aAAa,CAAC,YAAY,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,KAAK,MAAM,CAAC,IAAI,CAAC,GAAG,CAAC;IACrE,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,KAAK,CAAC;IACf,CAAC;AACH,CAAC,CAAC;AAEF,IAAI,iBAAiB,EAAE,EAAE,CAAC;IACxB,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAC9B,CAAC,IAAI,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,IAAI,CAAC,EAC5B,CAAC,GAAY,EAAE,EAAE;QACf,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,qBAAsB,GAAa,CAAC,KAAK,IAAI,MAAM,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QACnF,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC,CACF,CAAC;AACJ,CAAC"}
|
package/dist/render.d.ts
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
|
-
import type { TokenizeResult } from '@tokenometer/core';
|
|
1
|
+
import type { TokenizeResult, TokenometerFileResult } from '@tokenometer/core';
|
|
2
2
|
export declare const renderTable: (results: readonly TokenizeResult[]) => string;
|
|
3
3
|
export declare const renderModelLimits: (results: readonly TokenizeResult[]) => string;
|
|
4
4
|
export declare const renderSummary: (results: readonly TokenizeResult[]) => string;
|
|
5
|
+
/**
|
|
6
|
+
* Per-file token + cost summary table. One row per input file (or virtual
|
|
7
|
+
* file for `--image` entries). No-op when there's only one file.
|
|
8
|
+
*/
|
|
9
|
+
export declare const renderByFile: (files: readonly TokenometerFileResult[]) => string;
|
|
5
10
|
//# sourceMappingURL=render.d.ts.map
|
package/dist/render.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"render.d.ts","sourceRoot":"","sources":["../src/render.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,cAAc,EAAE,MAAM,mBAAmB,CAAC;
|
|
1
|
+
{"version":3,"file":"render.d.ts","sourceRoot":"","sources":["../src/render.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,cAAc,EAAE,qBAAqB,EAAE,MAAM,mBAAmB,CAAC;AAmB/E,eAAO,MAAM,WAAW,GAAI,SAAS,SAAS,cAAc,EAAE,KAAG,MA+ChE,CAAC;AAQF,eAAO,MAAM,iBAAiB,GAAI,SAAS,SAAS,cAAc,EAAE,KAAG,MAgBtE,CAAC;AAEF,eAAO,MAAM,aAAa,GAAI,SAAS,SAAS,cAAc,EAAE,KAAG,MAOlE,CAAC;AAEF;;;GAGG;AACH,eAAO,MAAM,YAAY,GAAI,OAAO,SAAS,qBAAqB,EAAE,KAAG,MA2BtE,CAAC"}
|
package/dist/render.js
CHANGED
|
@@ -8,16 +8,35 @@ const formatCost = (usd) => {
|
|
|
8
8
|
return `$${usd.toFixed(6)}`;
|
|
9
9
|
return `$${usd.toExponential(2)}`;
|
|
10
10
|
};
|
|
11
|
+
const formatMs = (ms) => `${Math.round(ms)}`;
|
|
12
|
+
const formatTps = (tps) => tps >= 100 ? Math.round(tps).toString() : tps.toFixed(1);
|
|
11
13
|
export const renderTable = (results) => {
|
|
12
14
|
if (results.length === 0)
|
|
13
15
|
return '(no results)';
|
|
14
|
-
|
|
15
|
-
const
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
16
|
+
// Latency columns are only added when at least one cell has latency data.
|
|
17
|
+
const hasLatency = results.some((r) => r.latency !== undefined);
|
|
18
|
+
const headers = hasLatency
|
|
19
|
+
? ['model', 'format', 'tokens', 'est. cost', 'p50 ttft', 'p50 total', 'tokens/s']
|
|
20
|
+
: ['model', 'format', 'tokens', 'est. cost'];
|
|
21
|
+
const rows = results.map((r) => {
|
|
22
|
+
const base = [
|
|
23
|
+
r.model,
|
|
24
|
+
r.format,
|
|
25
|
+
`${r.approximate ? '~' : ' '}${r.inputTokens.toLocaleString()}`,
|
|
26
|
+
formatCost(r.inputCost),
|
|
27
|
+
];
|
|
28
|
+
if (!hasLatency)
|
|
29
|
+
return base;
|
|
30
|
+
if (r.latency) {
|
|
31
|
+
return [
|
|
32
|
+
...base,
|
|
33
|
+
`${formatMs(r.latency.p50.ttftMs)} ms`,
|
|
34
|
+
`${formatMs(r.latency.p50.totalMs)} ms`,
|
|
35
|
+
formatTps(r.latency.p50.tokensPerSec),
|
|
36
|
+
];
|
|
37
|
+
}
|
|
38
|
+
return [...base, '-', '-', '-'];
|
|
39
|
+
});
|
|
21
40
|
const widths = headers.map((h, colIdx) => {
|
|
22
41
|
const maxRowWidth = rows.reduce((acc, row) => Math.max(acc, row[colIdx]?.length ?? 0), 0);
|
|
23
42
|
return Math.max(h.length, maxRowWidth);
|
|
@@ -74,4 +93,33 @@ export const renderSummary = (results) => {
|
|
|
74
93
|
const ratio = priciest.inputCost / Math.max(cheapest.inputCost, Number.EPSILON);
|
|
75
94
|
return `\nCheapest: ${cheapest.model} as ${cheapest.format} (${formatCost(cheapest.inputCost)})\nPriciest: ${priciest.model} as ${priciest.format} (${formatCost(priciest.inputCost)}, ${ratio.toFixed(2)}x more)`;
|
|
76
95
|
};
|
|
96
|
+
/**
|
|
97
|
+
* Per-file token + cost summary table. One row per input file (or virtual
|
|
98
|
+
* file for `--image` entries). No-op when there's only one file.
|
|
99
|
+
*/
|
|
100
|
+
export const renderByFile = (files) => {
|
|
101
|
+
if (files.length <= 1)
|
|
102
|
+
return '';
|
|
103
|
+
const headers = ['File', 'Tokens', 'USD'];
|
|
104
|
+
const rows = files.map((f) => {
|
|
105
|
+
const tokens = f.results.reduce((acc, r) => acc + r.inputTokens, 0);
|
|
106
|
+
const cost = f.results.reduce((acc, r) => acc + r.inputCost, 0);
|
|
107
|
+
return [f.path, tokens.toLocaleString(), formatCost(cost)];
|
|
108
|
+
});
|
|
109
|
+
const widths = headers.map((h, colIdx) => {
|
|
110
|
+
const maxRowWidth = rows.reduce((acc, row) => Math.max(acc, row[colIdx]?.length ?? 0), 0);
|
|
111
|
+
return Math.max(h.length, maxRowWidth);
|
|
112
|
+
});
|
|
113
|
+
const sep = headers.map((_, i) => '─'.repeat(widths[i] ?? 0)).join(' ');
|
|
114
|
+
const headerLine = headers.map((h, i) => padRight(h, widths[i] ?? h.length)).join(' ');
|
|
115
|
+
const dataLines = rows.map((row) => row
|
|
116
|
+
.map((cell, i) => {
|
|
117
|
+
const isNumeric = i >= 1;
|
|
118
|
+
return isNumeric
|
|
119
|
+
? padLeft(cell, widths[i] ?? cell.length)
|
|
120
|
+
: padRight(cell, widths[i] ?? cell.length);
|
|
121
|
+
})
|
|
122
|
+
.join(' '));
|
|
123
|
+
return ['\nBy file:', ` ${headerLine}`, ` ${sep}`, ...dataLines.map((l) => ` ${l}`)].join('\n');
|
|
124
|
+
};
|
|
77
125
|
//# sourceMappingURL=render.js.map
|
package/dist/render.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"render.js","sourceRoot":"","sources":["../src/render.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,mBAAmB,CAAC;AAG7C,MAAM,QAAQ,GAAG,CAAC,KAAa,EAAE,KAAa,EAAU,EAAE,CACxD,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC;AAE3E,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,KAAa,EAAU,EAAE,CACvD,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;AAE3E,MAAM,UAAU,GAAG,CAAC,GAAW,EAAU,EAAE;IACzC,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;IAC7C,IAAI,GAAG,IAAI,QAAQ;QAAE,OAAO,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;IACjD,OAAO,IAAI,GAAG,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,CAAC;AACpC,CAAC,CAAC;AAEF,MAAM,CAAC,MAAM,WAAW,GAAG,CAAC,OAAkC,EAAU,EAAE;IACxE,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,cAAc,CAAC;IAEhD,MAAM,OAAO,GAAG,CAAC,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,WAAW,
|
|
1
|
+
{"version":3,"file":"render.js","sourceRoot":"","sources":["../src/render.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,mBAAmB,CAAC;AAG7C,MAAM,QAAQ,GAAG,CAAC,KAAa,EAAE,KAAa,EAAU,EAAE,CACxD,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,KAAK,GAAG,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,CAAC;AAE3E,MAAM,OAAO,GAAG,CAAC,KAAa,EAAE,KAAa,EAAU,EAAE,CACvD,KAAK,CAAC,MAAM,IAAI,KAAK,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK,GAAG,KAAK,CAAC,MAAM,CAAC,GAAG,KAAK,CAAC;AAE3E,MAAM,UAAU,GAAG,CAAC,GAAW,EAAU,EAAE;IACzC,IAAI,GAAG,IAAI,IAAI;QAAE,OAAO,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;IAC7C,IAAI,GAAG,IAAI,QAAQ;QAAE,OAAO,IAAI,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,EAAE,CAAC;IACjD,OAAO,IAAI,GAAG,CAAC,aAAa,CAAC,CAAC,CAAC,EAAE,CAAC;AACpC,CAAC,CAAC;AAEF,MAAM,QAAQ,GAAG,CAAC,EAAU,EAAU,EAAE,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;AAE7D,MAAM,SAAS,GAAG,CAAC,GAAW,EAAU,EAAE,CACxC,GAAG,IAAI,GAAG,CAAC,CAAC,CAAC,IAAI,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,QAAQ,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC;AAE3D,MAAM,CAAC,MAAM,WAAW,GAAG,CAAC,OAAkC,EAAU,EAAE;IACxE,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,cAAc,CAAC;IAEhD,0EAA0E;IAC1E,MAAM,UAAU,GAAG,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,OAAO,KAAK,SAAS,CAAC,CAAC;IAEhE,MAAM,OAAO,GAAG,UAAU;QACxB,CAAC,CAAE,CAAC,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,WAAW,EAAE,UAAU,EAAE,WAAW,EAAE,UAAU,CAAW;QAC5F,CAAC,CAAE,CAAC,OAAO,EAAE,QAAQ,EAAE,QAAQ,EAAE,WAAW,CAAW,CAAC;IAC1D,MAAM,IAAI,GAAe,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;QACzC,MAAM,IAAI,GAAG;YACX,CAAC,CAAC,KAAK;YACP,CAAC,CAAC,MAAM;YACR,GAAG,CAAC,CAAC,WAAW,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,GAAG,GAAG,CAAC,CAAC,WAAW,CAAC,cAAc,EAAE,EAAE;YAC/D,UAAU,CAAC,CAAC,CAAC,SAAS,CAAC;SACxB,CAAC;QACF,IAAI,CAAC,UAAU;YAAE,OAAO,IAAI,CAAC;QAC7B,IAAI,CAAC,CAAC,OAAO,EAAE,CAAC;YACd,OAAO;gBACL,GAAG,IAAI;gBACP,GAAG,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,MAAM,CAAC,KAAK;gBACtC,GAAG,QAAQ,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,OAAO,CAAC,KAAK;gBACvC,SAAS,CAAC,CAAC,CAAC,OAAO,CAAC,GAAG,CAAC,YAAY,CAAC;aACtC,CAAC;QACJ,CAAC;QACD,OAAO,CAAC,GAAG,IAAI,EAAE,GAAG,EAAE,GAAG,EAAE,GAAG,CAAC,CAAC;IAClC,CAAC,CAAC,CAAC;IAEH,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;QACvC,MAAM,WAAW,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,CAAC,EAAE,MAAM,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC1F,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,WAAW,CAAC,CAAC;IACzC,CAAC,CAAC,CAAC;IAEH,MAAM,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACxF,MAAM,SAAS,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IAC/E,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CACjC,GAAG;SACA,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,EAAE;QACf,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,CAAC;QACzB,OAAO,SAAS;YACd,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC;YACzC,CAAC,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,CAAC;IAC/C,CAAC,CAAC;SACD,IAAI,CAAC,IAAI,CAAC,CACd,CAAC;IAEF,OAAO,CAAC,UAAU,EAAE,SAAS,EAAE,GAAG,SAAS,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;AAC1D,CAAC,CAAC;AAEF,MAAM,gBAAgB,GAAG,CAAC,CAAS,EAAU,EAAE;IAC7C,IAAI,CAAC,IAAI,SAAS;QAAE,OAAO,GAAG,CAAC,CAAC,GAAG,SAAS,CAAC,CAAC,OAAO,CAAC,CAAC,GAAG,SAAS,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC;IACtF,IAAI,CAAC,IAAI,IAAI;QAAE,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,CAAC,GAAG,IAAI,CAAC,GAAG,CAAC;IACjD,OAAO,GAAG,CAAC,EAAE,CAAC;AAChB,CAAC,CAAC;AAEF,MAAM,CAAC,MAAM,iBAAiB,GAAG,CAAC,OAAkC,EAAU,EAAE;IAC9E,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IACpC,MAAM,IAAI,GAAG,IAAI,GAAG,EAAU,CAAC;IAC/B,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,KAAK,MAAM,CAAC,IAAI,OAAO,EAAE,CAAC;QACxB,IAAI,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,KAAK,CAAC;YAAE,SAAS;QAChC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QAClB,MAAM,CAAC,GAAG,QAAQ,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC;QAC5B,IAAI,CAAC,CAAC,CAAC,aAAa,IAAI,CAAC,CAAC,CAAC,eAAe;YAAE,SAAS;QACrD,MAAM,KAAK,GAAa,EAAE,CAAC;QAC3B,IAAI,CAAC,CAAC,aAAa;YAAE,KAAK,CAAC,IAAI,CAAC,OAAO,gBAAgB,CAAC,CAAC,CAAC,aAAa,CAAC,EAAE,CAAC,CAAC;QAC5E,IAAI,CAAC,CAAC,eAAe;YAAE,KAAK,CAAC,IAAI,CAAC,OAAO,gBAAgB,CAAC,CAAC,CAAC,eAAe,CAAC,EAAE,CAAC,CAAC;QAChF,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,CAAC,KAAK,CAAC,MAAM,CAAC,EAAE,CAAC,IAAI,KAAK,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;IAC7D,CAAC;IACD,IAAI,KAAK,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IAClC,OAAO,cAAc,KAAK,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,CAAC;AAC1C,CAAC,CAAC;AAEF,MAAM,CAAC,MAAM,aAAa,GAAG,CAAC,OAAkC,EAAU,EAAE;IAC1E,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC;QAAE,OAAO,EAAE,CAAC;IACpC,MAAM,QAAQ,GAAG,CAAC,GAAG,OAAO,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,GAAG,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3E,MAAM,QAAQ,GAAG,CAAC,GAAG,OAAO,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,SAAS,GAAG,CAAC,CAAC,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC;IAC3E,IAAI,CAAC,QAAQ,IAAI,CAAC,QAAQ,IAAI,QAAQ,KAAK,QAAQ;QAAE,OAAO,EAAE,CAAC;IAC/D,MAAM,KAAK,GAAG,QAAQ,CAAC,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,QAAQ,CAAC,SAAS,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC;IAChF,OAAO,eAAe,QAAQ,CAAC,KAAK,OAAO,QAAQ,CAAC,MAAM,KAAK,UAAU,CAAC,QAAQ,CAAC,SAAS,CAAC,gBAAgB,QAAQ,CAAC,KAAK,OAAO,QAAQ,CAAC,MAAM,KAAK,UAAU,CAAC,QAAQ,CAAC,SAAS,CAAC,KAAK,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,SAAS,CAAC;AACrN,CAAC,CAAC;AAEF;;;GAGG;AACH,MAAM,CAAC,MAAM,YAAY,GAAG,CAAC,KAAuC,EAAU,EAAE;IAC9E,IAAI,KAAK,CAAC,MAAM,IAAI,CAAC;QAAE,OAAO,EAAE,CAAC;IACjC,MAAM,OAAO,GAAG,CAAC,MAAM,EAAE,QAAQ,EAAE,KAAK,CAAU,CAAC;IACnD,MAAM,IAAI,GAAe,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE;QACvC,MAAM,MAAM,GAAG,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,CAAC,WAAW,EAAE,CAAC,CAAC,CAAC;QACpE,MAAM,IAAI,GAAG,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,GAAG,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,CAAC;QAChE,OAAO,CAAC,CAAC,CAAC,IAAI,EAAE,MAAM,CAAC,cAAc,EAAE,EAAE,UAAU,CAAC,IAAI,CAAC,CAAC,CAAC;IAC7D,CAAC,CAAC,CAAC;IACH,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,MAAM,EAAE,EAAE;QACvC,MAAM,WAAW,GAAG,IAAI,CAAC,MAAM,CAAC,CAAC,GAAG,EAAE,GAAG,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,EAAE,GAAG,CAAC,MAAM,CAAC,EAAE,MAAM,IAAI,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;QAC1F,OAAO,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,EAAE,WAAW,CAAC,CAAC;IACzC,CAAC,CAAC,CAAC;IACH,MAAM,GAAG,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,GAAG,CAAC,MAAM,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACzE,MAAM,UAAU,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,QAAQ,CAAC,CAAC,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;IACxF,MAAM,SAAS,GAAG,IAAI,CAAC,GAAG,CAAC,CAAC,GAAG,EAAE,EAAE,CACjC,GAAG;SACA,GAAG,CAAC,CAAC,IAAI,EAAE,CAAC,EAAE,EAAE;QACf,MAAM,SAAS,GAAG,CAAC,IAAI,CAAC,CAAC;QACzB,OAAO,SAAS;YACd,CAAC,CAAC,OAAO,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC;YACzC,CAAC,CAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,IAAI,IAAI,CAAC,MAAM,CAAC,CAAC;IAC/C,CAAC,CAAC;SACD,IAAI,CAAC,IAAI,CAAC,CACd,CAAC;IACF,OAAO,CAAC,YAAY,EAAE,KAAK,UAAU,EAAE,EAAE,KAAK,GAAG,EAAE,EAAE,GAAG,SAAS,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,IAAI,CAC1F,IAAI,CACL,CAAC;AACJ,CAAC,CAAC"}
|
package/dist/vision.d.ts
ADDED
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Reads an image from disk and returns its width/height. Wrapped so tests
|
|
3
|
+
* can mock by reading a deterministic dimension out of a synthetic file.
|
|
4
|
+
*/
|
|
5
|
+
export interface ImageDimensions {
|
|
6
|
+
width: number;
|
|
7
|
+
height: number;
|
|
8
|
+
}
|
|
9
|
+
export type ImageSizeReader = (path: string) => Promise<ImageDimensions>;
|
|
10
|
+
export declare const defaultImageSizeReader: ImageSizeReader;
|
|
11
|
+
/**
|
|
12
|
+
* Compute vision tokens for one image × one model. Dispatch by the model's
|
|
13
|
+
* registered provider. Throws a clear error when a provider doesn't support
|
|
14
|
+
* vision (we'll need this when Mistral / Cohere lands).
|
|
15
|
+
*/
|
|
16
|
+
export declare const computeVisionTokens: (modelId: string, dim: ImageDimensions, imagePath: string) => number;
|
|
17
|
+
export interface ResolvedImage {
|
|
18
|
+
path: string;
|
|
19
|
+
dim: ImageDimensions;
|
|
20
|
+
}
|
|
21
|
+
/**
|
|
22
|
+
* Resolve all `--image` paths to dimensions in parallel. Reuses the file IO
|
|
23
|
+
* once per image (we still re-dispatch per model since the formula differs).
|
|
24
|
+
*/
|
|
25
|
+
export declare const resolveImages: (paths: readonly string[], reader?: ImageSizeReader) => Promise<ResolvedImage[]>;
|
|
26
|
+
//# sourceMappingURL=vision.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"vision.d.ts","sourceRoot":"","sources":["../src/vision.ts"],"names":[],"mappings":"AAUA;;;GAGG;AACH,MAAM,WAAW,eAAe;IAC9B,KAAK,EAAE,MAAM,CAAC;IACd,MAAM,EAAE,MAAM,CAAC;CAChB;AAED,MAAM,MAAM,eAAe,GAAG,CAAC,IAAI,EAAE,MAAM,KAAK,OAAO,CAAC,eAAe,CAAC,CAAC;AAEzE,eAAO,MAAM,sBAAsB,EAAE,eAOpC,CAAC;AAEF;;;;GAIG;AACH,eAAO,MAAM,mBAAmB,GAC9B,SAAS,MAAM,EACf,KAAK,eAAe,EACpB,WAAW,MAAM,KAChB,MAsBF,CAAC;AAEF,MAAM,WAAW,aAAa;IAC5B,IAAI,EAAE,MAAM,CAAC;IACb,GAAG,EAAE,eAAe,CAAC;CACtB;AAED;;;GAGG;AACH,eAAO,MAAM,aAAa,GACxB,OAAO,SAAS,MAAM,EAAE,EACxB,SAAQ,eAAwC,KAC/C,OAAO,CAAC,aAAa,EAAE,CAMvB,CAAC"}
|
package/dist/vision.js
ADDED
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
import { readFile } from 'node:fs/promises';
|
|
2
|
+
import { anthropicVisionTokens, getModel, googleVisionTokens, openaiVisionTokens, } from '@tokenometer/core';
|
|
3
|
+
import { imageSize } from 'image-size';
|
|
4
|
+
export const defaultImageSizeReader = async (path) => {
|
|
5
|
+
const bytes = await readFile(path);
|
|
6
|
+
const dim = imageSize(new Uint8Array(bytes.buffer, bytes.byteOffset, bytes.byteLength));
|
|
7
|
+
if (!dim.width || !dim.height) {
|
|
8
|
+
throw new Error(`Could not read image dimensions from "${path}".`);
|
|
9
|
+
}
|
|
10
|
+
return { height: dim.height, width: dim.width };
|
|
11
|
+
};
|
|
12
|
+
/**
|
|
13
|
+
* Compute vision tokens for one image × one model. Dispatch by the model's
|
|
14
|
+
* registered provider. Throws a clear error when a provider doesn't support
|
|
15
|
+
* vision (we'll need this when Mistral / Cohere lands).
|
|
16
|
+
*/
|
|
17
|
+
export const computeVisionTokens = (modelId, dim, imagePath) => {
|
|
18
|
+
const model = getModel(modelId);
|
|
19
|
+
const provider = model.provider;
|
|
20
|
+
switch (provider) {
|
|
21
|
+
case 'anthropic':
|
|
22
|
+
return anthropicVisionTokens(dim);
|
|
23
|
+
case 'openai':
|
|
24
|
+
return openaiVisionTokens(dim);
|
|
25
|
+
case 'google':
|
|
26
|
+
return googleVisionTokens(dim);
|
|
27
|
+
case 'mistral':
|
|
28
|
+
case 'cohere':
|
|
29
|
+
throw new Error(`Vision tokens for provider "${provider}" are not yet supported (model "${modelId}", image "${imagePath}"). Use Claude, GPT-4o, or Gemini for image cost estimation.`);
|
|
30
|
+
default: {
|
|
31
|
+
const exhaustiveCheck = provider;
|
|
32
|
+
throw new Error(`Vision tokens are not supported for provider "${exhaustiveCheck}" (model "${modelId}", image "${imagePath}").`);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
};
|
|
36
|
+
/**
|
|
37
|
+
* Resolve all `--image` paths to dimensions in parallel. Reuses the file IO
|
|
38
|
+
* once per image (we still re-dispatch per model since the formula differs).
|
|
39
|
+
*/
|
|
40
|
+
export const resolveImages = async (paths, reader = defaultImageSizeReader) => Promise.all(paths.map(async (path) => {
|
|
41
|
+
const dim = await reader(path);
|
|
42
|
+
return { dim, path };
|
|
43
|
+
}));
|
|
44
|
+
//# sourceMappingURL=vision.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"vision.js","sourceRoot":"","sources":["../src/vision.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,kBAAkB,CAAC;AAC5C,OAAO,EACL,qBAAqB,EACrB,QAAQ,EACR,kBAAkB,EAClB,kBAAkB,GACnB,MAAM,mBAAmB,CAAC;AAE3B,OAAO,EAAE,SAAS,EAAE,MAAM,YAAY,CAAC;AAavC,MAAM,CAAC,MAAM,sBAAsB,GAAoB,KAAK,EAAE,IAAI,EAAE,EAAE;IACpE,MAAM,KAAK,GAAG,MAAM,QAAQ,CAAC,IAAI,CAAC,CAAC;IACnC,MAAM,GAAG,GAAG,SAAS,CAAC,IAAI,UAAU,CAAC,KAAK,CAAC,MAAM,EAAE,KAAK,CAAC,UAAU,EAAE,KAAK,CAAC,UAAU,CAAC,CAAC,CAAC;IACxF,IAAI,CAAC,GAAG,CAAC,KAAK,IAAI,CAAC,GAAG,CAAC,MAAM,EAAE,CAAC;QAC9B,MAAM,IAAI,KAAK,CAAC,yCAAyC,IAAI,IAAI,CAAC,CAAC;IACrE,CAAC;IACD,OAAO,EAAE,MAAM,EAAE,GAAG,CAAC,MAAM,EAAE,KAAK,EAAE,GAAG,CAAC,KAAK,EAAE,CAAC;AAClD,CAAC,CAAC;AAEF;;;;GAIG;AACH,MAAM,CAAC,MAAM,mBAAmB,GAAG,CACjC,OAAe,EACf,GAAoB,EACpB,SAAiB,EACT,EAAE;IACV,MAAM,KAAK,GAAG,QAAQ,CAAC,OAAO,CAAC,CAAC;IAChC,MAAM,QAAQ,GAAa,KAAK,CAAC,QAAQ,CAAC;IAC1C,QAAQ,QAAQ,EAAE,CAAC;QACjB,KAAK,WAAW;YACd,OAAO,qBAAqB,CAAC,GAAG,CAAC,CAAC;QACpC,KAAK,QAAQ;YACX,OAAO,kBAAkB,CAAC,GAAG,CAAC,CAAC;QACjC,KAAK,QAAQ;YACX,OAAO,kBAAkB,CAAC,GAAG,CAAC,CAAC;QACjC,KAAK,SAAS,CAAC;QACf,KAAK,QAAQ;YACX,MAAM,IAAI,KAAK,CACb,+BAA+B,QAAQ,mCAAmC,OAAO,aAAa,SAAS,8DAA8D,CACtK,CAAC;QACJ,OAAO,CAAC,CAAC,CAAC;YACR,MAAM,eAAe,GAAU,QAAQ,CAAC;YACxC,MAAM,IAAI,KAAK,CACb,iDAAiD,eAAe,aAAa,OAAO,aAAa,SAAS,KAAK,CAChH,CAAC;QACJ,CAAC;IACH,CAAC;AACH,CAAC,CAAC;AAOF;;;GAGG;AACH,MAAM,CAAC,MAAM,aAAa,GAAG,KAAK,EAChC,KAAwB,EACxB,SAA0B,sBAAsB,EACtB,EAAE,CAC5B,OAAO,CAAC,GAAG,CACT,KAAK,CAAC,GAAG,CAAC,KAAK,EAAE,IAAI,EAAE,EAAE;IACvB,MAAM,GAAG,GAAG,MAAM,MAAM,CAAC,IAAI,CAAC,CAAC;IAC/B,OAAO,EAAE,GAAG,EAAE,IAAI,EAAE,CAAC;AACvB,CAAC,CAAC,CACH,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tokenometer",
|
|
3
|
-
"version": "0.0
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "0.1.0",
|
|
4
|
+
"description": "Tokenometer CLI — LLM token cost + latency benchmarking across Claude, GPT-4o, Gemini, Mistral, and Cohere. Multi-format, empirical mode, vision tokens, SARIF output.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Faraazuddin Mohammed <mohdfaraaz1@gmail.com>",
|
|
7
7
|
"homepage": "https://tokenometer.vercel.app",
|
|
@@ -16,16 +16,46 @@
|
|
|
16
16
|
"keywords": [
|
|
17
17
|
"ai",
|
|
18
18
|
"anthropic",
|
|
19
|
+
"ci-cd",
|
|
19
20
|
"claude",
|
|
21
|
+
"claude-code",
|
|
22
|
+
"claude-code-skill",
|
|
20
23
|
"cli",
|
|
24
|
+
"code-scanning",
|
|
25
|
+
"codestral",
|
|
26
|
+
"cohere",
|
|
27
|
+
"command-r",
|
|
21
28
|
"cost",
|
|
29
|
+
"cost-calculator",
|
|
30
|
+
"cursor",
|
|
22
31
|
"gemini",
|
|
32
|
+
"github-action",
|
|
23
33
|
"gpt",
|
|
34
|
+
"gpt-4o",
|
|
35
|
+
"latency",
|
|
24
36
|
"llm",
|
|
37
|
+
"llm-cost",
|
|
38
|
+
"mistral",
|
|
39
|
+
"mistral-7b",
|
|
40
|
+
"mistral-large",
|
|
41
|
+
"mixtral",
|
|
42
|
+
"model-comparison",
|
|
43
|
+
"multimodal",
|
|
25
44
|
"openai",
|
|
45
|
+
"pixtral",
|
|
26
46
|
"prompt",
|
|
47
|
+
"prompt-cost",
|
|
48
|
+
"prompt-cost-regression",
|
|
49
|
+
"prompt-engineering",
|
|
50
|
+
"prompt-regression",
|
|
51
|
+
"sarif",
|
|
52
|
+
"tiktoken",
|
|
27
53
|
"token",
|
|
28
|
-
"
|
|
54
|
+
"token-budget",
|
|
55
|
+
"tokenizer",
|
|
56
|
+
"ttft",
|
|
57
|
+
"vision-tokens",
|
|
58
|
+
"vscode"
|
|
29
59
|
],
|
|
30
60
|
"type": "module",
|
|
31
61
|
"main": "./dist/index.js",
|
|
@@ -39,10 +69,7 @@
|
|
|
39
69
|
"import": "./dist/index.js"
|
|
40
70
|
}
|
|
41
71
|
},
|
|
42
|
-
"files": [
|
|
43
|
-
"dist",
|
|
44
|
-
"README.md"
|
|
45
|
-
],
|
|
72
|
+
"files": ["dist", "README.md"],
|
|
46
73
|
"publishConfig": {
|
|
47
74
|
"access": "public",
|
|
48
75
|
"registry": "https://registry.npmjs.org/"
|
|
@@ -52,7 +79,8 @@
|
|
|
52
79
|
"clean": "rm -rf dist"
|
|
53
80
|
},
|
|
54
81
|
"dependencies": {
|
|
55
|
-
"@tokenometer/core": "0.0
|
|
82
|
+
"@tokenometer/core": "0.1.0",
|
|
83
|
+
"image-size": "^2.0.2"
|
|
56
84
|
},
|
|
57
85
|
"devDependencies": {
|
|
58
86
|
"@types/node": "^22.10.5",
|