@nanogpt/private-mode 0.2.14 → 0.2.18
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +18 -9
- package/lib/documentConversion.js +90 -0
- package/lib/originPolicy.js +5 -0
- package/lib/requestTransforms.js +31 -11
- package/lib/server.js +253 -7
- package/lib/statusContract.js +2 -1
- package/lib/transportTimeouts.js +4 -1
- package/models/private-tee.json +10 -38
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -25,7 +25,7 @@ const client = new OpenAI({
|
|
|
25
25
|
});
|
|
26
26
|
|
|
27
27
|
const response = await client.chat.completions.create({
|
|
28
|
-
model: "private/glm-5-
|
|
28
|
+
model: "private/glm-5-3",
|
|
29
29
|
messages: [{ role: "user", content: "Hello" }],
|
|
30
30
|
});
|
|
31
31
|
```
|
|
@@ -46,7 +46,17 @@ NanoGPT can see account identity, selected private model, selected TEE target me
|
|
|
46
46
|
|
|
47
47
|
NanoGPT's web app can also use these models without running this local proxy. Select an eligible Private Mode model and use the Private Mode control in the model picker. This package is for API clients, CLIs, agents, and other OpenAI-compatible tools.
|
|
48
48
|
|
|
49
|
-
The web app supports local image attachments for Private Mode vision models (Kimi K3 and Gemma 4 31B). Image bytes are converted to base64 in the browser and included inside the EHBP-encrypted request; they are not uploaded to NanoGPT object storage. Other attachment types, web search, URL-scraped content, project tools, multi-model chat, Context Memory injection, quick replies, and automatic title generation remain disabled for private turns.
|
|
49
|
+
The web app supports local image attachments for Private Mode vision models (Kimi K3 and Gemma 4 31B). Image bytes are converted to base64 in the browser and included inside the EHBP-encrypted request; they are not uploaded to NanoGPT object storage. It also supports encrypted text extraction from one local PDF up to 3 MiB. Other attachment types, web search, URL-scraped content, project tools, multi-model chat, Context Memory injection, quick replies, and automatic title generation remain disabled for private turns.
|
|
50
|
+
|
|
51
|
+
The localhost proxy exposes the same encrypted PDF conversion path:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
curl --fail-with-body http://127.0.0.1:8787/v1/convert/file \
|
|
55
|
+
-H 'X-NanoGPT-Private-Model: private/glm-5-3' \
|
|
56
|
+
-F 'files=@document.pdf;type=application/pdf'
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Send exactly one PDF in the `files` field and select any model returned by the proxy's `/v1/models` endpoint with `X-NanoGPT-Private-Model`. The proxy accepts PDFs up to 3 MiB, replaces the local filename before encryption, and returns extracted Markdown in `document.md_content`. Conversion is metered separately from model token usage; the successful response's `X-NanoGPT-Document-Cost-USD` header reports the settled conversion charge. A model catalog value of `capabilities.pdf_upload: false` means the model does not accept a PDF natively in its prompt. It does not disable this separate encrypted text-extraction step.
|
|
50
60
|
|
|
51
61
|
In the hosted web app, decrypted Private Mode turns remain in local browser history. Cloud conversation sync is blocked for Private Mode chats unless password-based end-to-end sync is enabled; the recoverable default sync mode is not used for those chats.
|
|
52
62
|
|
|
@@ -56,7 +66,7 @@ Because the request body is encrypted before NanoGPT receives it, the service ca
|
|
|
56
66
|
|
|
57
67
|
Private Mode streaming has a 29-minute soft deadline so billing can settle before the route's 30-minute hard runtime limit. If the decrypted upstream stream ends without both a non-null `finish_reason` and the OpenAI `[DONE]` marker, the local proxy emits an SSE error with code `private_stream_incomplete`. Any text received before that error may be partial.
|
|
58
68
|
|
|
59
|
-
Private Mode
|
|
69
|
+
Private Mode supports Chat Completions for text generation and `/v1/convert/file` for encrypted PDF-to-Markdown conversion. The Responses API and Batch API are not available through the encrypted Private Mode endpoint. The standard Responses API outside Private Mode supports background requests within its configured 30-minute runtime limit; work that must reliably continue longer needs a durable job system.
|
|
60
70
|
|
|
61
71
|
Useful local checks:
|
|
62
72
|
|
|
@@ -64,22 +74,21 @@ Useful local checks:
|
|
|
64
74
|
GET http://127.0.0.1:8787/v1/models
|
|
65
75
|
GET http://127.0.0.1:8787/v1/private-mode/status
|
|
66
76
|
GET http://127.0.0.1:8787/v1/private-mode/attestation
|
|
77
|
+
POST http://127.0.0.1:8787/v1/convert/file
|
|
67
78
|
```
|
|
68
79
|
|
|
69
80
|
Supported private model IDs include:
|
|
70
81
|
|
|
71
|
-
- `private/deepseek-v4-flash` - DeepSeek V4 Flash
|
|
82
|
+
- `private/deepseek-v4-1-flash` - DeepSeek V4.1 Flash
|
|
72
83
|
- `private/kimi-k3`
|
|
73
84
|
- `private/gpt-oss-120b`
|
|
74
85
|
- `private/llama3-3-70b`
|
|
75
|
-
- `private/glm-5-2`
|
|
76
|
-
- `private/glm-5-2:thinking`
|
|
77
86
|
- `private/glm-5-3`
|
|
78
87
|
- `private/glm-5-3-flash`
|
|
79
88
|
- `private/gemma4-31b`
|
|
80
89
|
- `private/gemma4-31b:thinking`
|
|
81
90
|
|
|
82
|
-
|
|
91
|
+
GLM 5.3 Private and GLM 5.3 Flash Private each have a 1,048,576-token total context limit and a 131,072-token output ceiling. Kimi K3 Private has a 256,000-token context limit. DeepSeek V4.1 Flash Private has a 1,048,576-token total context limit and a 384,000-token output ceiling. The model-list extension `context_length` is the combined prompt-and-output window, while `max_output_tokens` is the output ceiling within that same window. These values are not additive.
|
|
83
92
|
|
|
84
93
|
### Thinking without leaving Private Mode
|
|
85
94
|
|
|
@@ -87,14 +96,14 @@ Keep the same private model ID when enabling thinking. For example:
|
|
|
87
96
|
|
|
88
97
|
```json
|
|
89
98
|
{
|
|
90
|
-
"model": "private/deepseek-v4-flash",
|
|
99
|
+
"model": "private/deepseek-v4-1-flash",
|
|
91
100
|
"messages": [{ "role": "user", "content": "What is 17 times 19?" }],
|
|
92
101
|
"reasoning_effort": "high",
|
|
93
102
|
"stream": true
|
|
94
103
|
}
|
|
95
104
|
```
|
|
96
105
|
|
|
97
|
-
DeepSeek V4 Flash
|
|
106
|
+
DeepSeek V4.1 Flash and Gemma 4 support optional thinking. Use `thinking: true` or `enable_thinking: true` to enable it; `thinking: false` disables it for these optional-thinking models. Kimi K3, GLM 5.3, and GLM 5.3 Flash always generate reasoning. GLM 5.3 models accept `low`, `high`, or `max` effort. `reasoning: { "exclude": true }` controls response visibility and does not disable thinking.
|
|
98
107
|
|
|
99
108
|
Non-streaming reasoning requests may take more than five minutes. The CLI allows the encrypted request to wait for the hosted route's response, with a 30-minute transport limit. The hosted route stops non-streaming generation after at most 29 minutes to leave time before its platform deadline. Use `stream: true` for earlier output, and configure your API client's timeout to allow long generations. Upgrade the local proxy as well as the hosted service to receive both timeout fixes.
|
|
100
109
|
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
export const MAX_PRIVATE_DOCUMENT_BYTES = 3 * 1024 * 1024;
|
|
2
|
+
export const MAX_PRIVATE_DOCUMENT_MULTIPART_BYTES = 4 * 1024 * 1024;
|
|
3
|
+
export const MAX_PRIVATE_DOCUMENT_RESPONSE_BYTES = 4 * 1024 * 1024;
|
|
4
|
+
export const PRIVATE_DOCUMENT_CONVERSION_PATH = '/v1/convert/file';
|
|
5
|
+
export const HOSTED_PRIVATE_DOCUMENT_CONVERSION_PATH = '/api/v1/private/tinfoil/v1/convert/file';
|
|
6
|
+
|
|
7
|
+
export function resolvePrivateDocumentMultipartLimit(maxBodyBytes) {
|
|
8
|
+
return Math.min(maxBodyBytes, MAX_PRIVATE_DOCUMENT_MULTIPART_BYTES);
|
|
9
|
+
}
|
|
10
|
+
|
|
11
|
+
export function createPrivateDocumentRequestLifecycle(res) {
|
|
12
|
+
const controller = new AbortController();
|
|
13
|
+
let complete = false;
|
|
14
|
+
res.once('close', () => {
|
|
15
|
+
if (!complete) controller.abort();
|
|
16
|
+
});
|
|
17
|
+
|
|
18
|
+
return {
|
|
19
|
+
signal: controller.signal,
|
|
20
|
+
isCancelled: () => Boolean(res.destroyed || controller.signal.aborted),
|
|
21
|
+
complete: () => {
|
|
22
|
+
complete = true;
|
|
23
|
+
},
|
|
24
|
+
};
|
|
25
|
+
}
|
|
26
|
+
|
|
27
|
+
export function isPrivateDocumentMultipartContentType(value) {
|
|
28
|
+
return typeof value === 'string'
|
|
29
|
+
&& /^multipart\/form-data\s*;[^\r\n]*\bboundary=(?:"[^"]+"|[^;\s]+)/i.test(value);
|
|
30
|
+
}
|
|
31
|
+
|
|
32
|
+
export function hasPrivateDocumentConversionResult(value) {
|
|
33
|
+
return Boolean(
|
|
34
|
+
value
|
|
35
|
+
&& typeof value === 'object'
|
|
36
|
+
&& value.document
|
|
37
|
+
&& typeof value.document === 'object'
|
|
38
|
+
&& typeof value.document.md_content === 'string'
|
|
39
|
+
&& value.document.md_content.trim(),
|
|
40
|
+
);
|
|
41
|
+
}
|
|
42
|
+
|
|
43
|
+
export async function preparePrivateDocumentMultipart(rawBody, contentType) {
|
|
44
|
+
let form;
|
|
45
|
+
try {
|
|
46
|
+
form = await new Request('http://127.0.0.1/', {
|
|
47
|
+
method: 'POST',
|
|
48
|
+
headers: { 'content-type': contentType },
|
|
49
|
+
body: rawBody,
|
|
50
|
+
}).formData();
|
|
51
|
+
} catch {
|
|
52
|
+
return {
|
|
53
|
+
ok: false,
|
|
54
|
+
status: 400,
|
|
55
|
+
code: 'invalid_document_multipart',
|
|
56
|
+
message: 'Private PDF conversion requires valid multipart form data.',
|
|
57
|
+
};
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
const entries = [...form.entries()];
|
|
61
|
+
const file = entries.length === 1 && entries[0][0] === 'files' ? entries[0][1] : null;
|
|
62
|
+
if (!file || typeof file === 'string' || typeof file.arrayBuffer !== 'function') {
|
|
63
|
+
return {
|
|
64
|
+
ok: false,
|
|
65
|
+
status: 400,
|
|
66
|
+
code: 'invalid_document_fields',
|
|
67
|
+
message: 'Private PDF conversion requires exactly one PDF in the files field.',
|
|
68
|
+
};
|
|
69
|
+
}
|
|
70
|
+
if (file.size > MAX_PRIVATE_DOCUMENT_BYTES) {
|
|
71
|
+
return {
|
|
72
|
+
ok: false,
|
|
73
|
+
status: 413,
|
|
74
|
+
code: 'document_too_large',
|
|
75
|
+
message: 'Private PDF conversion accepts a PDF up to 3 MiB.',
|
|
76
|
+
};
|
|
77
|
+
}
|
|
78
|
+
if (file.size === 0 || await file.slice(0, 5).text() !== '%PDF-') {
|
|
79
|
+
return {
|
|
80
|
+
ok: false,
|
|
81
|
+
status: 400,
|
|
82
|
+
code: 'invalid_document',
|
|
83
|
+
message: 'Private PDF conversion requires a non-empty file with a valid PDF signature.',
|
|
84
|
+
};
|
|
85
|
+
}
|
|
86
|
+
|
|
87
|
+
const forwardedForm = new FormData();
|
|
88
|
+
forwardedForm.append('files', new Blob([await file.arrayBuffer()], { type: 'application/pdf' }), 'document.pdf');
|
|
89
|
+
return { ok: true, body: forwardedForm, fileSize: file.size };
|
|
90
|
+
}
|
package/lib/originPolicy.js
CHANGED
|
@@ -111,6 +111,11 @@ export function getCorsHeadersForOrigin(originPolicy, origin) {
|
|
|
111
111
|
|
|
112
112
|
return {
|
|
113
113
|
'access-control-allow-origin': normalizedOrigin,
|
|
114
|
+
'access-control-expose-headers': [
|
|
115
|
+
'x-request-id',
|
|
116
|
+
'x-nanogpt-private-mode',
|
|
117
|
+
'x-nanogpt-document-cost-usd',
|
|
118
|
+
].join(', '),
|
|
114
119
|
vary: 'Origin',
|
|
115
120
|
};
|
|
116
121
|
}
|
package/lib/requestTransforms.js
CHANGED
|
@@ -236,8 +236,16 @@ function isReasoningEnabled(reasoning) {
|
|
|
236
236
|
return undefined;
|
|
237
237
|
}
|
|
238
238
|
|
|
239
|
-
function normalizeDeepSeekV4ReasoningEffort(value) {
|
|
239
|
+
function normalizeDeepSeekV4ReasoningEffort(value, model, nativeTemplateValue = false) {
|
|
240
240
|
const normalized = typeof value === 'string' ? value.trim().toLowerCase() : '';
|
|
241
|
+
if (model?.thinkingMode === 'deepseek-v4.1') {
|
|
242
|
+
if (nativeTemplateValue && ['low', 'high', 'xhigh'].includes(normalized)) {
|
|
243
|
+
return normalized;
|
|
244
|
+
}
|
|
245
|
+
if (normalized === 'low' || normalized === 'minimal') return 'low';
|
|
246
|
+
if (normalized === 'medium') return 'high';
|
|
247
|
+
return 'xhigh';
|
|
248
|
+
}
|
|
241
249
|
if (normalized === 'max' || normalized === 'xhigh') return 'max';
|
|
242
250
|
if (normalized === 'medium') return 'medium';
|
|
243
251
|
if (normalized === 'low' || normalized === 'minimal') return 'low';
|
|
@@ -606,14 +614,17 @@ function applyPrivateModeKimiK3RequestParams(body, model) {
|
|
|
606
614
|
);
|
|
607
615
|
}
|
|
608
616
|
|
|
609
|
-
function clampPrivateModeDeepSeekV4Output(body) {
|
|
617
|
+
function clampPrivateModeDeepSeekV4Output(body, model) {
|
|
618
|
+
const maxCompletionTokens = Number.isFinite(model?.maxOutputTokens)
|
|
619
|
+
? Math.max(1, Math.floor(model.maxOutputTokens))
|
|
620
|
+
: DEEPSEEK_V4_MAX_COMPLETION_TOKENS;
|
|
610
621
|
const requestedMaxTokens = typeof body.max_tokens === 'number' && Number.isFinite(body.max_tokens)
|
|
611
622
|
? body.max_tokens
|
|
612
623
|
: undefined;
|
|
613
624
|
if (requestedMaxTokens === undefined) return;
|
|
614
625
|
|
|
615
626
|
const normalizedMaxTokens = requestedMaxTokens < 0
|
|
616
|
-
?
|
|
627
|
+
? maxCompletionTokens
|
|
617
628
|
: requestedMaxTokens;
|
|
618
629
|
const promptTokenEstimate = estimatePrivateModeKimiK3PromptTokens(body);
|
|
619
630
|
const remainingContext = Math.max(
|
|
@@ -624,7 +635,7 @@ function clampPrivateModeDeepSeekV4Output(body) {
|
|
|
624
635
|
);
|
|
625
636
|
body.max_tokens = Math.min(
|
|
626
637
|
Math.max(1, Math.floor(normalizedMaxTokens)),
|
|
627
|
-
|
|
638
|
+
maxCompletionTokens,
|
|
628
639
|
remainingContext,
|
|
629
640
|
);
|
|
630
641
|
}
|
|
@@ -1051,8 +1062,11 @@ export function applyPrivateModelRequestMutations(body, model) {
|
|
|
1051
1062
|
applyTinfoilCompatibilityMutations(body, model);
|
|
1052
1063
|
if (isPrivateModeKimiK3Model(model)) {
|
|
1053
1064
|
applyPrivateModeKimiK3RequestParams(body, model);
|
|
1054
|
-
} else if (
|
|
1055
|
-
|
|
1065
|
+
} else if (
|
|
1066
|
+
model.thinkingMode === 'deepseek-v4' ||
|
|
1067
|
+
model.thinkingMode === 'deepseek-v4.1'
|
|
1068
|
+
) {
|
|
1069
|
+
clampPrivateModeDeepSeekV4Output(body, model);
|
|
1056
1070
|
} else if (model.thinkingMode === 'glm-5.3') {
|
|
1057
1071
|
clampPrivateModeGlm53Output(body);
|
|
1058
1072
|
}
|
|
@@ -1067,14 +1081,16 @@ export function applyPrivateModelRequestMutations(body, model) {
|
|
|
1067
1081
|
} else if (
|
|
1068
1082
|
model.thinkingMode === 'glm-5.2' ||
|
|
1069
1083
|
model.thinkingMode === 'glm-5.3' ||
|
|
1070
|
-
model.thinkingMode === 'deepseek-v4'
|
|
1084
|
+
model.thinkingMode === 'deepseek-v4' ||
|
|
1085
|
+
model.thinkingMode === 'deepseek-v4.1'
|
|
1071
1086
|
) {
|
|
1072
1087
|
const thinkingEnabled = model.thinkingMode === 'glm-5.3'
|
|
1073
1088
|
? true
|
|
1074
1089
|
: shouldEnableThinking(body, model);
|
|
1075
|
-
const
|
|
1076
|
-
?? (isPlainObject(body.reasoning) ? body.reasoning.effort : undefined)
|
|
1077
|
-
|
|
1090
|
+
const publicReasoningEffort = body.reasoning_effort
|
|
1091
|
+
?? (isPlainObject(body.reasoning) ? body.reasoning.effort : undefined);
|
|
1092
|
+
const nativeTemplateReasoningEffort = body.chat_template_kwargs?.reasoning_effort;
|
|
1093
|
+
const requestedReasoningEffort = publicReasoningEffort ?? nativeTemplateReasoningEffort;
|
|
1078
1094
|
body.chat_template_kwargs = {
|
|
1079
1095
|
...mergeChatTemplateKwargs(body),
|
|
1080
1096
|
thinking: thinkingEnabled,
|
|
@@ -1086,7 +1102,11 @@ export function applyPrivateModelRequestMutations(body, model) {
|
|
|
1086
1102
|
body.reasoning_effort = normalizeGlm53ReasoningEffort(requestedReasoningEffort);
|
|
1087
1103
|
delete body.chat_template_kwargs.reasoning_effort;
|
|
1088
1104
|
} else if (thinkingEnabled) {
|
|
1089
|
-
body.chat_template_kwargs.reasoning_effort = normalizeDeepSeekV4ReasoningEffort(
|
|
1105
|
+
body.chat_template_kwargs.reasoning_effort = normalizeDeepSeekV4ReasoningEffort(
|
|
1106
|
+
requestedReasoningEffort,
|
|
1107
|
+
model,
|
|
1108
|
+
publicReasoningEffort === undefined && nativeTemplateReasoningEffort !== undefined,
|
|
1109
|
+
);
|
|
1090
1110
|
} else {
|
|
1091
1111
|
delete body.chat_template_kwargs.reasoning_effort;
|
|
1092
1112
|
}
|
package/lib/server.js
CHANGED
|
@@ -12,6 +12,17 @@ import {
|
|
|
12
12
|
PRIVATE_MODE_DEFAULT_RESERVE_MAX_OUTPUT_TOKENS,
|
|
13
13
|
PRIVATE_MODE_STREAM_SOFT_DEADLINE_MS,
|
|
14
14
|
} from './constants.js';
|
|
15
|
+
import {
|
|
16
|
+
createPrivateDocumentRequestLifecycle,
|
|
17
|
+
hasPrivateDocumentConversionResult,
|
|
18
|
+
HOSTED_PRIVATE_DOCUMENT_CONVERSION_PATH,
|
|
19
|
+
isPrivateDocumentMultipartContentType,
|
|
20
|
+
MAX_PRIVATE_DOCUMENT_BYTES,
|
|
21
|
+
MAX_PRIVATE_DOCUMENT_RESPONSE_BYTES,
|
|
22
|
+
preparePrivateDocumentMultipart,
|
|
23
|
+
PRIVATE_DOCUMENT_CONVERSION_PATH,
|
|
24
|
+
resolvePrivateDocumentMultipartLimit,
|
|
25
|
+
} from './documentConversion.js';
|
|
15
26
|
import {
|
|
16
27
|
buildPrivateModeOriginPolicy,
|
|
17
28
|
getCorsHeadersForRequest,
|
|
@@ -53,6 +64,7 @@ function readMaxBodyBytes() {
|
|
|
53
64
|
}
|
|
54
65
|
|
|
55
66
|
const MAX_BODY_BYTES = readMaxBodyBytes();
|
|
67
|
+
const MAX_DOCUMENT_MULTIPART_BODY_BYTES = resolvePrivateDocumentMultipartLimit(MAX_BODY_BYTES);
|
|
56
68
|
|
|
57
69
|
function normalizeApiBase(apiBase) {
|
|
58
70
|
const parsed = new URL(apiBase);
|
|
@@ -111,6 +123,9 @@ export function privateModeStatus(apiBase, secureState, localBase, originPolicy,
|
|
|
111
123
|
status_path: '/v1/private-mode/status',
|
|
112
124
|
attestation_path: '/v1/private-mode/attestation',
|
|
113
125
|
chat_completions_path: '/v1/chat/completions',
|
|
126
|
+
document_conversion_path: PRIVATE_DOCUMENT_CONVERSION_PATH,
|
|
127
|
+
document_conversion_max_file_bytes: MAX_PRIVATE_DOCUMENT_BYTES,
|
|
128
|
+
document_conversion_max_multipart_bytes: MAX_DOCUMENT_MULTIPART_BODY_BYTES,
|
|
114
129
|
max_body_bytes: MAX_BODY_BYTES,
|
|
115
130
|
transport: 'ehbp',
|
|
116
131
|
request_body_encrypted: true,
|
|
@@ -163,14 +178,14 @@ function notFound(res, corsHeaders) {
|
|
|
163
178
|
}, corsHeaders);
|
|
164
179
|
}
|
|
165
180
|
|
|
166
|
-
function readBody(req) {
|
|
181
|
+
function readBody(req, maxBytes = MAX_BODY_BYTES) {
|
|
167
182
|
return new Promise((resolve, reject) => {
|
|
168
183
|
const chunks = [];
|
|
169
184
|
let size = 0;
|
|
170
185
|
req.on('data', (chunk) => {
|
|
171
|
-
if (size >
|
|
186
|
+
if (size > maxBytes) return;
|
|
172
187
|
size += chunk.length;
|
|
173
|
-
if (size >
|
|
188
|
+
if (size > maxBytes) {
|
|
174
189
|
reject(new Error('Request body is too large.'));
|
|
175
190
|
return;
|
|
176
191
|
}
|
|
@@ -190,8 +205,11 @@ function copyLocalHeaders(req) {
|
|
|
190
205
|
return headers;
|
|
191
206
|
}
|
|
192
207
|
|
|
193
|
-
async function runPreflight({ apiBase, apiKey, model, req, requestBodyBytes }) {
|
|
194
|
-
const preflightBody = {
|
|
208
|
+
async function runPreflight({ apiBase, apiKey, model, req, requestBodyBytes, operation, signal }) {
|
|
209
|
+
const preflightBody = {
|
|
210
|
+
model: model.id,
|
|
211
|
+
...(operation ? { operation } : {}),
|
|
212
|
+
};
|
|
195
213
|
if (Number.isSafeInteger(requestBodyBytes) && requestBodyBytes >= 0) {
|
|
196
214
|
preflightBody.requestBodyBytes = requestBodyBytes;
|
|
197
215
|
}
|
|
@@ -205,6 +223,7 @@ async function runPreflight({ apiBase, apiKey, model, req, requestBodyBytes }) {
|
|
|
205
223
|
...copyLocalHeaders(req),
|
|
206
224
|
},
|
|
207
225
|
body: JSON.stringify(preflightBody),
|
|
226
|
+
signal,
|
|
208
227
|
});
|
|
209
228
|
|
|
210
229
|
if (response.ok) {
|
|
@@ -244,6 +263,221 @@ async function runPreflight({ apiBase, apiKey, model, req, requestBodyBytes }) {
|
|
|
244
263
|
return { ok: false, status: response.status, body: errorBody };
|
|
245
264
|
}
|
|
246
265
|
|
|
266
|
+
async function readResponseBodyWithLimit(response, maxBytes) {
|
|
267
|
+
const reader = response.body?.getReader();
|
|
268
|
+
if (!reader) return Buffer.alloc(0);
|
|
269
|
+
const chunks = [];
|
|
270
|
+
let size = 0;
|
|
271
|
+
try {
|
|
272
|
+
while (true) {
|
|
273
|
+
const { done, value } = await reader.read();
|
|
274
|
+
if (done) break;
|
|
275
|
+
size += value.byteLength;
|
|
276
|
+
if (size > maxBytes) {
|
|
277
|
+
await reader.cancel();
|
|
278
|
+
throw new Error('Response body is too large.');
|
|
279
|
+
}
|
|
280
|
+
chunks.push(Buffer.from(value));
|
|
281
|
+
}
|
|
282
|
+
} finally {
|
|
283
|
+
reader.releaseLock();
|
|
284
|
+
}
|
|
285
|
+
return Buffer.concat(chunks);
|
|
286
|
+
}
|
|
287
|
+
|
|
288
|
+
async function handleDocumentConversion({ apiBase, apiKey, secureState, req, res, corsHeaders }) {
|
|
289
|
+
const contentType = typeof req.headers['content-type'] === 'string'
|
|
290
|
+
? req.headers['content-type']
|
|
291
|
+
: '';
|
|
292
|
+
if (!isPrivateDocumentMultipartContentType(contentType)) {
|
|
293
|
+
jsonResponse(res, 415, {
|
|
294
|
+
error: {
|
|
295
|
+
message: 'Private PDF conversion requires multipart/form-data with one files field.',
|
|
296
|
+
type: 'invalid_request_error',
|
|
297
|
+
code: 'invalid_document_content_type',
|
|
298
|
+
},
|
|
299
|
+
}, corsHeaders);
|
|
300
|
+
return;
|
|
301
|
+
}
|
|
302
|
+
|
|
303
|
+
const model = resolveModel(req.headers['x-nanogpt-private-model']);
|
|
304
|
+
if (!model) {
|
|
305
|
+
jsonResponse(res, 400, {
|
|
306
|
+
error: {
|
|
307
|
+
message: 'Set X-NanoGPT-Private-Model to a model returned by this proxy\'s /v1/models endpoint.',
|
|
308
|
+
type: 'invalid_request_error',
|
|
309
|
+
code: 'model_not_supported',
|
|
310
|
+
},
|
|
311
|
+
}, corsHeaders);
|
|
312
|
+
return;
|
|
313
|
+
}
|
|
314
|
+
|
|
315
|
+
let rawBody;
|
|
316
|
+
try {
|
|
317
|
+
rawBody = await readBody(req, MAX_DOCUMENT_MULTIPART_BODY_BYTES);
|
|
318
|
+
} catch {
|
|
319
|
+
jsonResponse(res, 413, {
|
|
320
|
+
error: {
|
|
321
|
+
message: 'Multipart request is too large. Private PDF conversion accepts a PDF up to 3 MiB.',
|
|
322
|
+
type: 'invalid_request_error',
|
|
323
|
+
code: 'body_too_large',
|
|
324
|
+
},
|
|
325
|
+
}, corsHeaders);
|
|
326
|
+
return;
|
|
327
|
+
}
|
|
328
|
+
if (rawBody.length === 0) {
|
|
329
|
+
jsonResponse(res, 400, {
|
|
330
|
+
error: {
|
|
331
|
+
message: 'Private PDF conversion requires one PDF in the files field.',
|
|
332
|
+
type: 'invalid_request_error',
|
|
333
|
+
code: 'missing_document',
|
|
334
|
+
},
|
|
335
|
+
}, corsHeaders);
|
|
336
|
+
return;
|
|
337
|
+
}
|
|
338
|
+
|
|
339
|
+
const prepared = await preparePrivateDocumentMultipart(rawBody, contentType);
|
|
340
|
+
if (!prepared.ok) {
|
|
341
|
+
jsonResponse(res, prepared.status, {
|
|
342
|
+
error: {
|
|
343
|
+
message: prepared.message,
|
|
344
|
+
type: 'invalid_request_error',
|
|
345
|
+
code: prepared.code,
|
|
346
|
+
},
|
|
347
|
+
}, corsHeaders);
|
|
348
|
+
return;
|
|
349
|
+
}
|
|
350
|
+
|
|
351
|
+
const requestLifecycle = createPrivateDocumentRequestLifecycle(res);
|
|
352
|
+
|
|
353
|
+
let preflight;
|
|
354
|
+
try {
|
|
355
|
+
preflight = await runPreflight({
|
|
356
|
+
apiBase,
|
|
357
|
+
apiKey,
|
|
358
|
+
model,
|
|
359
|
+
req,
|
|
360
|
+
requestBodyBytes: prepared.fileSize,
|
|
361
|
+
operation: 'document_conversion',
|
|
362
|
+
signal: requestLifecycle.signal,
|
|
363
|
+
});
|
|
364
|
+
} catch (error) {
|
|
365
|
+
requestLifecycle.complete();
|
|
366
|
+
if (requestLifecycle.isCancelled()) return;
|
|
367
|
+
throw error;
|
|
368
|
+
}
|
|
369
|
+
if (requestLifecycle.isCancelled()) {
|
|
370
|
+
requestLifecycle.complete();
|
|
371
|
+
return;
|
|
372
|
+
}
|
|
373
|
+
if (!preflight.ok) {
|
|
374
|
+
requestLifecycle.complete();
|
|
375
|
+
jsonResponse(res, preflight.status, preflight.body, corsHeaders);
|
|
376
|
+
return;
|
|
377
|
+
}
|
|
378
|
+
|
|
379
|
+
let responseClient;
|
|
380
|
+
|
|
381
|
+
let response;
|
|
382
|
+
try {
|
|
383
|
+
response = await fetchWithSecureClientRecovery({
|
|
384
|
+
secureState,
|
|
385
|
+
userCacheSecret: preflight.cacheScope,
|
|
386
|
+
shouldInvalidateError: () => !requestLifecycle.signal.aborted,
|
|
387
|
+
fetchWithClient: (client) => client.fetch(`${apiBase}${HOSTED_PRIVATE_DOCUMENT_CONVERSION_PATH}`, {
|
|
388
|
+
method: 'POST',
|
|
389
|
+
headers: {
|
|
390
|
+
authorization: `Bearer ${apiKey}`,
|
|
391
|
+
accept: 'application/json',
|
|
392
|
+
'x-nanogpt-private-model': model.id,
|
|
393
|
+
'x-nanogpt-private-cache-scope': buildPrivateModeCacheScopeProof(preflight.cacheScope),
|
|
394
|
+
[PRIVATE_MODE_PROXY_VERSION_HEADER]: PRIVATE_MODE_PROXY_VERSION,
|
|
395
|
+
'x-query-source': 'api',
|
|
396
|
+
...copyLocalHeaders(req),
|
|
397
|
+
},
|
|
398
|
+
body: prepared.body,
|
|
399
|
+
signal: requestLifecycle.signal,
|
|
400
|
+
}),
|
|
401
|
+
onResponseClient: (client) => {
|
|
402
|
+
responseClient = client;
|
|
403
|
+
},
|
|
404
|
+
});
|
|
405
|
+
} catch (error) {
|
|
406
|
+
requestLifecycle.complete();
|
|
407
|
+
if (requestLifecycle.isCancelled()) return;
|
|
408
|
+
jsonResponse(res, 502, {
|
|
409
|
+
error: {
|
|
410
|
+
message: normalizePrivateModeUpstreamErrorMessage(
|
|
411
|
+
error instanceof Error ? error.message : String(error),
|
|
412
|
+
'Private PDF conversion failed.',
|
|
413
|
+
),
|
|
414
|
+
type: 'api_error',
|
|
415
|
+
code: 'private_document_processing_failed',
|
|
416
|
+
},
|
|
417
|
+
}, corsHeaders);
|
|
418
|
+
return;
|
|
419
|
+
}
|
|
420
|
+
|
|
421
|
+
const responseHeaders = { ...corsHeaders };
|
|
422
|
+
for (const name of ['content-type', 'x-request-id', 'x-nanogpt-private-mode', 'x-nanogpt-document-cost-usd']) {
|
|
423
|
+
const value = response.headers.get(name);
|
|
424
|
+
if (value) responseHeaders[name] = value;
|
|
425
|
+
}
|
|
426
|
+
if (!response.ok) {
|
|
427
|
+
requestLifecycle.complete();
|
|
428
|
+
const errorHeaders = { ...responseHeaders };
|
|
429
|
+
delete errorHeaders['content-type'];
|
|
430
|
+
jsonResponse(res, response.status, {
|
|
431
|
+
error: {
|
|
432
|
+
message: await readErrorMessage(response, `Private PDF conversion failed with HTTP ${response.status}`),
|
|
433
|
+
type: 'api_error',
|
|
434
|
+
code: 'private_document_processing_failed',
|
|
435
|
+
},
|
|
436
|
+
}, errorHeaders);
|
|
437
|
+
secureState.markClientUsed(responseClient);
|
|
438
|
+
return;
|
|
439
|
+
}
|
|
440
|
+
|
|
441
|
+
if (response.headers.get('x-nanogpt-private-mode') !== 'tinfoil'
|
|
442
|
+
|| !response.headers.has('ehbp-response-nonce')) {
|
|
443
|
+
requestLifecycle.complete();
|
|
444
|
+
await response.body?.cancel().catch(() => undefined);
|
|
445
|
+
secureState.invalidateClient(responseClient);
|
|
446
|
+
jsonResponse(res, 502, {
|
|
447
|
+
error: {
|
|
448
|
+
message: 'Private PDF conversion returned an unverified response.',
|
|
449
|
+
type: 'api_error',
|
|
450
|
+
code: 'private_document_response_unverified',
|
|
451
|
+
},
|
|
452
|
+
}, corsHeaders);
|
|
453
|
+
return;
|
|
454
|
+
}
|
|
455
|
+
|
|
456
|
+
let responseBody;
|
|
457
|
+
try {
|
|
458
|
+
responseBody = await readResponseBodyWithLimit(response, MAX_PRIVATE_DOCUMENT_RESPONSE_BYTES);
|
|
459
|
+
const parsed = JSON.parse(responseBody.toString('utf8'));
|
|
460
|
+
if (!hasPrivateDocumentConversionResult(parsed)) throw new Error('missing_document_content');
|
|
461
|
+
} catch {
|
|
462
|
+
requestLifecycle.complete();
|
|
463
|
+
secureState.markClientUsed(responseClient);
|
|
464
|
+
jsonResponse(res, 502, {
|
|
465
|
+
error: {
|
|
466
|
+
message: 'Private PDF conversion returned an invalid response.',
|
|
467
|
+
type: 'api_error',
|
|
468
|
+
code: 'private_document_response_invalid',
|
|
469
|
+
},
|
|
470
|
+
}, corsHeaders);
|
|
471
|
+
return;
|
|
472
|
+
}
|
|
473
|
+
|
|
474
|
+
secureState.markClientUsed(responseClient);
|
|
475
|
+
responseHeaders['content-length'] = String(responseBody.length);
|
|
476
|
+
requestLifecycle.complete();
|
|
477
|
+
res.writeHead(response.status, responseHeaders);
|
|
478
|
+
res.end(responseBody);
|
|
479
|
+
}
|
|
480
|
+
|
|
247
481
|
async function getPreflightedClient({ apiBase, apiKey, secureState, req, model = MODELS[0] }) {
|
|
248
482
|
const preflight = await runPreflight({
|
|
249
483
|
apiBase,
|
|
@@ -454,7 +688,7 @@ function resolveAccessControlAllowHeaders(req) {
|
|
|
454
688
|
.join(', ');
|
|
455
689
|
}
|
|
456
690
|
|
|
457
|
-
return 'authorization, content-type, x-team-id';
|
|
691
|
+
return 'authorization, content-type, x-team-id, x-nanogpt-private-model';
|
|
458
692
|
}
|
|
459
693
|
|
|
460
694
|
function handleOptions(req, res, corsHeaders) {
|
|
@@ -552,8 +786,20 @@ export async function startPrivateModeProxy(options) {
|
|
|
552
786
|
return;
|
|
553
787
|
}
|
|
554
788
|
|
|
789
|
+
if (req.method === 'POST' && url.pathname === PRIVATE_DOCUMENT_CONVERSION_PATH) {
|
|
790
|
+
await handleDocumentConversion({
|
|
791
|
+
apiBase,
|
|
792
|
+
apiKey: options.apiKey,
|
|
793
|
+
secureState,
|
|
794
|
+
req,
|
|
795
|
+
res,
|
|
796
|
+
corsHeaders,
|
|
797
|
+
});
|
|
798
|
+
return;
|
|
799
|
+
}
|
|
800
|
+
|
|
555
801
|
if (req.method === 'GET' && url.pathname === '/') {
|
|
556
|
-
textResponse(res, 200, `NanoGPT Private Mode Proxy v${PRIVATE_MODE_PROXY_VERSION}\nOpenAI base URL: ${localBase}\nStatus: /v1/private-mode/status\n`, corsHeaders);
|
|
802
|
+
textResponse(res, 200, `NanoGPT Private Mode Proxy v${PRIVATE_MODE_PROXY_VERSION}\nOpenAI base URL: ${localBase}\nStatus: /v1/private-mode/status\nPDF conversion: ${PRIVATE_DOCUMENT_CONVERSION_PATH}\n`, corsHeaders);
|
|
557
803
|
return;
|
|
558
804
|
}
|
|
559
805
|
|
package/lib/statusContract.js
CHANGED
|
@@ -3,6 +3,7 @@ import { createHash } from 'node:crypto';
|
|
|
3
3
|
export const PRIVATE_MODE_FRONTEND_SUPPORTED_FEATURES = Object.freeze([
|
|
4
4
|
'text_chat',
|
|
5
5
|
'vision_image_inputs',
|
|
6
|
+
'encrypted_pdf_conversion',
|
|
6
7
|
'streaming',
|
|
7
8
|
'conversation_history',
|
|
8
9
|
'model_settings',
|
|
@@ -10,7 +11,7 @@ export const PRIVATE_MODE_FRONTEND_SUPPORTED_FEATURES = Object.freeze([
|
|
|
10
11
|
]);
|
|
11
12
|
|
|
12
13
|
export const PRIVATE_MODE_FRONTEND_DISABLED_FEATURES = Object.freeze([
|
|
13
|
-
'
|
|
14
|
+
'non_image_non_pdf_attachments',
|
|
14
15
|
'web_search',
|
|
15
16
|
'url_scraped_content',
|
|
16
17
|
'project_chats',
|
package/lib/transportTimeouts.js
CHANGED
|
@@ -5,6 +5,7 @@ export class PrivateModeDispatcher extends Agent {
|
|
|
5
5
|
constructor(apiBase) {
|
|
6
6
|
super();
|
|
7
7
|
this.chatUrl = new URL(`${apiBase.replace(/\/+$/, '')}/api/v1/private/tinfoil/v1/chat/completions`);
|
|
8
|
+
this.documentUrl = new URL(`${apiBase.replace(/\/+$/, '')}/api/v1/private/tinfoil/v1/convert/file`);
|
|
8
9
|
}
|
|
9
10
|
|
|
10
11
|
dispatch(options, handler) {
|
|
@@ -12,7 +13,9 @@ export class PrivateModeDispatcher extends Agent {
|
|
|
12
13
|
// dispatcher. Scope the CLI's override to its configured encrypted endpoint.
|
|
13
14
|
// Override dispatch options, not Agent defaults: fetch supplies its own
|
|
14
15
|
// five-minute timeout values on every dispatch.
|
|
15
|
-
|
|
16
|
+
const isPrivateModeRoute = String(options.origin) === this.chatUrl.origin
|
|
17
|
+
&& (options.path === this.chatUrl.pathname || options.path === this.documentUrl.pathname);
|
|
18
|
+
if (isPrivateModeRoute) {
|
|
16
19
|
options = {
|
|
17
20
|
...options,
|
|
18
21
|
headersTimeout: PRIVATE_MODE_STREAM_ROUTE_MAX_DURATION_MS,
|
package/models/private-tee.json
CHANGED
|
@@ -1,17 +1,17 @@
|
|
|
1
1
|
[
|
|
2
2
|
{
|
|
3
|
-
"id": "private/deepseek-v4-flash",
|
|
4
|
-
"name": "DeepSeek V4 Flash
|
|
5
|
-
"upstreamModel": "deepseek-v4-flash",
|
|
6
|
-
"billingModel": "
|
|
7
|
-
"providerPricingModel": "TEE/deepseek-v4-flash",
|
|
8
|
-
"teeTargetModel": "deepseek-v4-flash",
|
|
9
|
-
"thinkingMode": "deepseek-v4",
|
|
3
|
+
"id": "private/deepseek-v4-1-flash",
|
|
4
|
+
"name": "DeepSeek V4.1 Flash Private",
|
|
5
|
+
"upstreamModel": "deepseek-v4-1-flash",
|
|
6
|
+
"billingModel": "TEE/deepseek-v4.1-flash",
|
|
7
|
+
"providerPricingModel": "TEE/deepseek-v4.1-flash",
|
|
8
|
+
"teeTargetModel": "deepseek-v4-1-flash",
|
|
9
|
+
"thinkingMode": "deepseek-v4.1",
|
|
10
10
|
"maxInputTokens": 1048576,
|
|
11
|
-
"maxOutputTokens":
|
|
12
|
-
"created":
|
|
11
|
+
"maxOutputTokens": 384000,
|
|
12
|
+
"created": 1789084800,
|
|
13
13
|
"ownedBy": "nanogpt-private-mode",
|
|
14
|
-
"aliases": ["TEE/deepseek-v4-flash"]
|
|
14
|
+
"aliases": ["private/deepseek-v4.1-flash", "TEE/deepseek-v4.1-flash"]
|
|
15
15
|
},
|
|
16
16
|
{
|
|
17
17
|
"id": "private/kimi-k3",
|
|
@@ -48,34 +48,6 @@
|
|
|
48
48
|
"ownedBy": "nanogpt-private-mode",
|
|
49
49
|
"aliases": ["private/llama-3.3-70b", "TEE/llama3-3-70b"]
|
|
50
50
|
},
|
|
51
|
-
{
|
|
52
|
-
"id": "private/glm-5-2",
|
|
53
|
-
"name": "GLM 5.2 Private",
|
|
54
|
-
"upstreamModel": "glm-5-2",
|
|
55
|
-
"billingModel": "TEE/glm-5-2",
|
|
56
|
-
"providerPricingModel": "TEE/glm-5.2",
|
|
57
|
-
"teeTargetModel": "glm-5-2",
|
|
58
|
-
"thinkingMode": "glm-5.2",
|
|
59
|
-
"maxInputTokens": 393216,
|
|
60
|
-
"maxOutputTokens": 131072,
|
|
61
|
-
"created": 1781827200,
|
|
62
|
-
"ownedBy": "nanogpt-private-mode",
|
|
63
|
-
"aliases": ["private/glm-5.2", "TEE/glm-5-2", "TEE/glm-5.2"]
|
|
64
|
-
},
|
|
65
|
-
{
|
|
66
|
-
"id": "private/glm-5-2:thinking",
|
|
67
|
-
"name": "GLM 5.2 Thinking Private",
|
|
68
|
-
"upstreamModel": "glm-5-2",
|
|
69
|
-
"billingModel": "TEE/glm-5-2:thinking",
|
|
70
|
-
"providerPricingModel": "TEE/glm-5.2:thinking",
|
|
71
|
-
"teeTargetModel": "glm-5-2",
|
|
72
|
-
"thinkingMode": "glm-5.2",
|
|
73
|
-
"maxInputTokens": 393216,
|
|
74
|
-
"maxOutputTokens": 131072,
|
|
75
|
-
"created": 1781827200,
|
|
76
|
-
"ownedBy": "nanogpt-private-mode",
|
|
77
|
-
"aliases": ["private/glm-5.2:thinking", "TEE/glm-5-2:thinking", "TEE/glm-5.2:thinking"]
|
|
78
|
-
},
|
|
79
51
|
{
|
|
80
52
|
"id": "private/glm-5-3",
|
|
81
53
|
"name": "GLM 5.3 Private",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@nanogpt/private-mode",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.18",
|
|
4
4
|
"description": "OpenAI-compatible localhost proxy for NanoGPT Private Mode.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"publishConfig": {
|
|
@@ -15,6 +15,7 @@
|
|
|
15
15
|
"lib/cacheScope.js",
|
|
16
16
|
"lib/catalogCompatibility.js",
|
|
17
17
|
"lib/constants.js",
|
|
18
|
+
"lib/documentConversion.js",
|
|
18
19
|
"lib/originPolicy.js",
|
|
19
20
|
"lib/packageInfo.js",
|
|
20
21
|
"lib/requestTransforms.js",
|