pi-ko-sanitizer 0.0.0-stage → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +18 -0
- package/LICENSE +21 -0
- package/README-ko.md +54 -0
- package/README.md +53 -2
- package/package.json +49 -4
- package/src/index.ts +203 -0
- package/src/lexicon.ts +132 -0
- package/src/rewrite.ts +414 -0
- package/src/sanitize.ts +412 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.1.2
|
|
4
|
+
|
|
5
|
+
- Use synthetic test inputs rather than recorded prompts and model replies.
|
|
6
|
+
- Remove corpus-dependent development scripts and document the fixture policy.
|
|
7
|
+
- The sanitizer's runtime behavior is unchanged.
|
|
8
|
+
|
|
9
|
+
## 0.1.1
|
|
10
|
+
|
|
11
|
+
- Read short kana tails attached to Hangul words while preserving Japanese particles and protected text.
|
|
12
|
+
- Limit experimental model correction to the first eligible text part of a reply. Check substitutions at each occurrence.
|
|
13
|
+
- Publish English and Korean guides.
|
|
14
|
+
|
|
15
|
+
## 0.1.0
|
|
16
|
+
|
|
17
|
+
- Add a local dictionary pass and opt-in model correction for mixed-language Korean replies.
|
|
18
|
+
- Preserve protected text and report uncertain spans without deleting them.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 alice
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README-ko.md
ADDED
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# pi-ko-sanitizer
|
|
2
|
+
|
|
3
|
+
[English](README.md) | **한국어**
|
|
4
|
+
|
|
5
|
+
한국어 답변에 섞여 나온 중국어와 일본어 단어를 한국어로 바꾸는 [Pi](https://pi.dev) 확장입니다. 답변을 만든 뒤 일부 표현을 교정하며, 문장 전체를 번역하거나 특정 문자를 무조건 지우지 않습니다.
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
문서에 使用된 용어를 확인하세요 → 문서에 사용된 용어를 확인하세요
|
|
9
|
+
표의 页面을 열어 주세요 → 표의 페이지를 열어 주세요
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## 설치
|
|
13
|
+
|
|
14
|
+
Node.js 22.19.0 이상이 필요합니다.
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
pi install npm:pi-ko-sanitizer
|
|
18
|
+
pi -e npm:pi-ko-sanitizer
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
이 확장은 `message_end`에서 동작합니다. 한자 독음 변환기를 함께 쓰는 경우 원문을 인식할 수 있도록 이 확장을 먼저 불러오세요.
|
|
22
|
+
|
|
23
|
+
## 동작 방식
|
|
24
|
+
|
|
25
|
+
- **사전 교정은 기본으로 켜져 있습니다.** 모델을 호출하지 않고 로컬에서 처리합니다. 구간 전체를 해석할 수 있을 때만 바꾸고, 일부 한국어 조사도 고칩니다. 한글 단어 끝에 붙은 짧은 일본어 가나도 한글로 바꿀 수 있습니다.
|
|
26
|
+
- 사전 교정은 코드, URL, 일본어 문장, 한자 병기, 사용자가 질문한 용어를 건드리지 않습니다. 확신하기 어려운 표현은 지우지 않고 그대로 둡니다.
|
|
27
|
+
- **모델 교정은 실험 기능이며 기본으로 꺼져 있습니다.** 사전으로 해석할 수 없는 구간의 교정 표현을 현재 모델에 물어봅니다. 답변을 새로 만들지 않고, 허용된 표현만 원문에 반영합니다. 자연스러워 보이지만 뜻이 다른 표현을 고를 수 있습니다. 시간이 초과되거나 응답을 쓸 수 없으면 사전 교정 결과를 유지합니다.
|
|
28
|
+
|
|
29
|
+
## 설정
|
|
30
|
+
|
|
31
|
+
| 환경 변수 | 기본값 | 동작 |
|
|
32
|
+
|---|---|---|
|
|
33
|
+
| `PI_KO_SANITIZER_LLM` | 꺼짐 | `1`이면 교정 대상이 있는 첫 번째 텍스트 부분을 현재 모델에 보냅니다. |
|
|
34
|
+
| `PI_KO_SANITIZER_LLM_TIMEOUT` | `12000` | 모델 교정 제한 시간입니다. 단위는 밀리초입니다. |
|
|
35
|
+
| `PI_KO_SANITIZER_VERIFY` | 꺼짐 | `1`이면 교정 제안을 확인하기 위해 모델을 한 번 더 호출할 수 있습니다. 맞는 표현도 거절할 수 있습니다. |
|
|
36
|
+
| `PI_KO_SANITIZER_OFF` | 꺼짐 | `1`이면 교정과 로그 기록을 모두 끕니다. |
|
|
37
|
+
| `PI_KO_SANITIZER_DEBUG` | 꺼짐 | 추가 진단 로그를 켭니다. |
|
|
38
|
+
| `PI_KO_SANITIZER_LOG_TEXT` | 꺼짐 | `1`이면 감지된 표현의 원문을 로컬 진단 로그에 남깁니다. 기본값에서는 원문 대신 해시를 남깁니다. |
|
|
39
|
+
|
|
40
|
+
`/zh-ko`로 세션별 집계와 로컬 로그 경로를 볼 수 있습니다. 민감한 대화에서는 원문 기록을 끈 상태로 두세요. 원격 모델로 모델 교정을 켜면, 수정하지 않을 내용까지 포함해 선택된 텍스트 부분 전체가 모델 제공자에게 전송됩니다.
|
|
41
|
+
|
|
42
|
+
## 개발
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npm ci
|
|
46
|
+
npm run typecheck
|
|
47
|
+
npm test
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
테스트는 가상의 예시와 모의 모델 응답을 사용합니다. 대화 전문, 검색 기록, 실제 세션에서 수집한 프롬프트, 진단 로그는 커밋하지 마세요. 사전 처리 코어는 Pi에 의존하지 않습니다. `src/sanitize.ts`와 `src/rewrite.ts`를 참고하세요. 코어를 위한 별도 npm 진입점은 제공하지 않습니다.
|
|
51
|
+
|
|
52
|
+
## 라이선스
|
|
53
|
+
|
|
54
|
+
MIT.
|
package/README.md
CHANGED
|
@@ -1,3 +1,54 @@
|
|
|
1
|
-
#
|
|
1
|
+
# pi-ko-sanitizer
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
**English** | [한국어](README-ko.md)
|
|
4
|
+
|
|
5
|
+
A [Pi](https://pi.dev) extension that replaces stray Chinese and Japanese words in otherwise Korean replies. This is post-processing, not a translator or a character blacklist.
|
|
6
|
+
|
|
7
|
+
```
|
|
8
|
+
문서에 使用된 용어를 확인하세요 → 문서에 사용된 용어를 확인하세요
|
|
9
|
+
표의 页面을 열어 주세요 → 표의 페이지를 열어 주세요
|
|
10
|
+
```
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
Requires Node.js 22.19.0 or newer.
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
pi install npm:pi-ko-sanitizer
|
|
18
|
+
pi -e npm:pi-ko-sanitizer
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
The extension runs on `message_end`. If you also use a Hanja-to-Hangul converter, load this extension first so it can match the original text.
|
|
22
|
+
|
|
23
|
+
## How it works
|
|
24
|
+
|
|
25
|
+
- **Dictionary correction is on by default.** It runs locally without a model call. It replaces only spans it can read in full, adjusts some Korean particles, and can read short kana tails attached to Hangul words.
|
|
26
|
+
- The dictionary pass skips code, URLs, Japanese prose, Hanja glosses, and terms the user asked about. It leaves uncertain text in place instead of deleting it.
|
|
27
|
+
- **Model correction is experimental and off by default.** It asks the active model for substitutions for unreadable spans. It applies allowed substitutions to the original reply rather than regenerating the reply. A plausible but incorrect reading can still pass. A timeout or unusable result leaves the dictionary correction intact.
|
|
28
|
+
|
|
29
|
+
## Configuration
|
|
30
|
+
|
|
31
|
+
| Variable | Default | Behavior |
|
|
32
|
+
|---|---|---|
|
|
33
|
+
| `PI_KO_SANITIZER_LLM` | off | Set to `1` to ask the active model about the first text part with eligible spans. |
|
|
34
|
+
| `PI_KO_SANITIZER_LLM_TIMEOUT` | `12000` | Model correction budget in milliseconds. |
|
|
35
|
+
| `PI_KO_SANITIZER_VERIFY` | off | Set to `1` to allow a second model call to check the proposed readings. It can reject good readings too. |
|
|
36
|
+
| `PI_KO_SANITIZER_OFF` | off | Set to `1` to disable correction and logging. |
|
|
37
|
+
| `PI_KO_SANITIZER_DEBUG` | off | Enable extra diagnostic logging. |
|
|
38
|
+
| `PI_KO_SANITIZER_LOG_TEXT` | off | Set to `1` to include leak text in the local diagnostic log. By default, findings are represented by digests. |
|
|
39
|
+
|
|
40
|
+
Use `/zh-ko` to see session counters and the local log path. Keep raw logging off for sensitive conversations. If you enable model correction with a remote provider, the prompt sends the entire selected text part to that provider, including text that the extension will not edit.
|
|
41
|
+
|
|
42
|
+
## Development
|
|
43
|
+
|
|
44
|
+
```bash
|
|
45
|
+
npm ci
|
|
46
|
+
npm run typecheck
|
|
47
|
+
npm test
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Tests use synthetic examples and mock model responses. Do not commit conversation transcripts, search history, prompts captured from real sessions, or diagnostic logs. The dictionary core itself does not depend on Pi: see `src/sanitize.ts` and `src/rewrite.ts`. There is no separate npm entry point for the core.
|
|
51
|
+
|
|
52
|
+
## License
|
|
53
|
+
|
|
54
|
+
MIT.
|
package/package.json
CHANGED
|
@@ -1,6 +1,51 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "pi-ko-sanitizer",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"
|
|
5
|
-
"
|
|
6
|
-
|
|
3
|
+
"version": "0.1.2",
|
|
4
|
+
"description": "Replace leaked Chinese and Japanese in Korean replies with natural Korean. Deterministic dictionary by default, optional model rewrite.",
|
|
5
|
+
"license": "MIT",
|
|
6
|
+
"author": "stellarsailor",
|
|
7
|
+
"type": "module",
|
|
8
|
+
"repository": {
|
|
9
|
+
"type": "git",
|
|
10
|
+
"url": "git+https://github.com/stellarsailor/pi-ko-sanitizer.git"
|
|
11
|
+
},
|
|
12
|
+
"homepage": "https://github.com/stellarsailor/pi-ko-sanitizer#readme",
|
|
13
|
+
"bugs": {
|
|
14
|
+
"url": "https://github.com/stellarsailor/pi-ko-sanitizer/issues"
|
|
15
|
+
},
|
|
16
|
+
"engines": {
|
|
17
|
+
"node": ">=22.19.0"
|
|
18
|
+
},
|
|
19
|
+
"keywords": [
|
|
20
|
+
"pi-package",
|
|
21
|
+
"korean",
|
|
22
|
+
"chinese",
|
|
23
|
+
"sanitizer",
|
|
24
|
+
"code-switching",
|
|
25
|
+
"local-llm"
|
|
26
|
+
],
|
|
27
|
+
"files": [
|
|
28
|
+
"src",
|
|
29
|
+
"README.md",
|
|
30
|
+
"README-ko.md",
|
|
31
|
+
"CHANGELOG.md",
|
|
32
|
+
"LICENSE"
|
|
33
|
+
],
|
|
34
|
+
"pi": {
|
|
35
|
+
"extensions": [
|
|
36
|
+
"./src/index.ts"
|
|
37
|
+
]
|
|
38
|
+
},
|
|
39
|
+
"scripts": {
|
|
40
|
+
"test": "node --test 'test/*.test.mjs'",
|
|
41
|
+
"typecheck": "tsc --noEmit"
|
|
42
|
+
},
|
|
43
|
+
"peerDependencies": {
|
|
44
|
+
"@earendil-works/pi-coding-agent": "*"
|
|
45
|
+
},
|
|
46
|
+
"devDependencies": {
|
|
47
|
+
"@earendil-works/pi-coding-agent": "*",
|
|
48
|
+
"@types/node": "^26.6.6",
|
|
49
|
+
"typescript": "^5.9.3"
|
|
50
|
+
}
|
|
51
|
+
}
|
package/src/index.ts
ADDED
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
import { appendFileSync, mkdirSync, renameSync, statSync } from "node:fs";
|
|
2
|
+
import { createHash } from "node:crypto";
|
|
3
|
+
import { homedir } from "node:os";
|
|
4
|
+
import { dirname, join } from "node:path";
|
|
5
|
+
import type { ExtensionAPI, ExtensionContext } from "@earendil-works/pi-coding-agent";
|
|
6
|
+
import { processText, candidates, askedSpans, protectedMask, type Finding } from "./sanitize.ts";
|
|
7
|
+
import { rewriteReply } from "./rewrite.ts";
|
|
8
|
+
|
|
9
|
+
// Uncertain leaks go here, never into the reply. One JSON object per line.
|
|
10
|
+
const LOG = join(homedir(), ".pi", "agent", "logs", "ko-sanitizer-diag.jsonl");
|
|
11
|
+
const MAX_LOG = 512 * 1024;
|
|
12
|
+
const DEBUG = Boolean(process.env.PI_KO_SANITIZER_DEBUG);
|
|
13
|
+
// Raw leak text is off by default: the log is a counter, not a copy of the
|
|
14
|
+
// reply. Set PI_KO_SANITIZER_LOG_TEXT=1 when you need to read the spans.
|
|
15
|
+
const LOG_TEXT = process.env.PI_KO_SANITIZER_LOG_TEXT === "1";
|
|
16
|
+
// The sentence-level rewrite costs a model round trip per reply, so it is opt-in.
|
|
17
|
+
const LLM = process.env.PI_KO_SANITIZER_LLM === "1";
|
|
18
|
+
// PI_KO_SANITIZER_OFF=1 loads the package but does not edit or log replies.
|
|
19
|
+
const OFF = process.env.PI_KO_SANITIZER_OFF === "1";
|
|
20
|
+
const VERIFY = process.env.PI_KO_SANITIZER_VERIFY === "1";
|
|
21
|
+
const LLM_TIMEOUT_MS = Number(process.env.PI_KO_SANITIZER_LLM_TIMEOUT ?? 12000);
|
|
22
|
+
const LLM_MAX_SPANS = 8;
|
|
23
|
+
const LLM_MAX_CHARS = 6000;
|
|
24
|
+
|
|
25
|
+
// The log holds model text inside 40-character windows, so a key that rode
|
|
26
|
+
// along in a reply would be written to disk. Nothing here is worth that risk:
|
|
27
|
+
// anything shaped like a credential becomes a marker before it is logged.
|
|
28
|
+
const SECRET = new RegExp(
|
|
29
|
+
[
|
|
30
|
+
// vendor prefixes with a separator, anchored so task_manager stays text
|
|
31
|
+
"\\b(?:sk|ghp|gho|ghu|ghs|ghr|glpat|xox[abprs])[-_][A-Za-z0-9_-]{6,}",
|
|
32
|
+
"(?:AKIA|ASIA|AIza)[A-Za-z0-9_-]{8,}|ya29\\.[A-Za-z0-9_\\-]{8,}",
|
|
33
|
+
"eyJ[A-Za-z0-9_\\-.]{10,}",
|
|
34
|
+
"\\b(?:[A-Za-z0-9]{3,}[_-])?(?:api[_-]?key|apikey|access[_-]?token|access[_-]?key|secret[_-]?key|client[_-]?secret|password|passwd|pwd)\\b\\s*[=:]\\s*\\S+",
|
|
35
|
+
"\\bbearer\\s+[A-Za-z0-9._-]{8,}",
|
|
36
|
+
"\\b\\d{8,10}:AA[A-Za-z0-9_-]{30,}",
|
|
37
|
+
].join("|"),
|
|
38
|
+
"gi",
|
|
39
|
+
);
|
|
40
|
+
|
|
41
|
+
export function redact(value: unknown): string {
|
|
42
|
+
const text = String(value ?? "");
|
|
43
|
+
return text.replace(SECRET, (m) => `<redacted:${m.slice(0, 4)}…${m.length}>`);
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
function safe(value: unknown): unknown {
|
|
47
|
+
if (typeof value === "string") return redact(value);
|
|
48
|
+
if (Array.isArray(value)) return value.map(safe);
|
|
49
|
+
if (value && typeof value === "object") {
|
|
50
|
+
return Object.fromEntries(Object.entries(value).map(([k, v]) => [k, safe(v)]));
|
|
51
|
+
}
|
|
52
|
+
return value;
|
|
53
|
+
}
|
|
54
|
+
|
|
55
|
+
function log(record: Record<string, unknown>) {
|
|
56
|
+
const line = JSON.stringify(safe(record));
|
|
57
|
+
try {
|
|
58
|
+
if (statSync(LOG).size > MAX_LOG) renameSync(LOG, `${LOG}.1`);
|
|
59
|
+
appendFileSync(LOG, `${line}\n`);
|
|
60
|
+
} catch {
|
|
61
|
+
try {
|
|
62
|
+
mkdirSync(dirname(LOG), { recursive: true });
|
|
63
|
+
appendFileSync(LOG, `${line}\n`);
|
|
64
|
+
} catch {
|
|
65
|
+
// A missing diagnostic log must never cost the user their reply.
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
}
|
|
69
|
+
|
|
70
|
+
/** Store a fingerprint of the leak instead of the leak itself. */
|
|
71
|
+
export function findingRecord(finding: Finding): Record<string, unknown> {
|
|
72
|
+
const digest = createHash("sha1").update(finding.text).digest("hex").slice(0, 10);
|
|
73
|
+
return LOG_TEXT
|
|
74
|
+
? { kind: finding.kind, len: finding.text.length, digest, text: redact(finding.text) }
|
|
75
|
+
: { kind: finding.kind, len: finding.text.length, digest };
|
|
76
|
+
}
|
|
77
|
+
|
|
78
|
+
// The user asked about a term, so the answer must keep it: 造字 in a question about
|
|
79
|
+
// Chinese characters is content, not a leak. Read the last user message from the
|
|
80
|
+
// session and treat every span it mentions as untouchable.
|
|
81
|
+
function lastUserText(ctx: ExtensionContext): string {
|
|
82
|
+
try {
|
|
83
|
+
const entries = ctx.sessionManager?.getBranch?.() ?? [];
|
|
84
|
+
for (let i = entries.length - 1; i >= 0; i--) {
|
|
85
|
+
const message = (entries[i] as { message?: { role?: string; content?: unknown } }).message;
|
|
86
|
+
if (!message || message.role !== "user") continue;
|
|
87
|
+
const parts = Array.isArray(message.content) ? message.content : [];
|
|
88
|
+
return parts
|
|
89
|
+
.filter((p): p is { type: "text"; text: string } => (p as { type?: string }).type === "text")
|
|
90
|
+
.map((p) => p.text)
|
|
91
|
+
.join("\n");
|
|
92
|
+
}
|
|
93
|
+
} catch {
|
|
94
|
+
// A missing session must not stop the pass.
|
|
95
|
+
}
|
|
96
|
+
return "";
|
|
97
|
+
}
|
|
98
|
+
|
|
99
|
+
async function rewritePart(
|
|
100
|
+
text: string,
|
|
101
|
+
spans: string[],
|
|
102
|
+
ctx: ExtensionContext,
|
|
103
|
+
protect: (span: string) => boolean,
|
|
104
|
+
): Promise<{ text: string; applied: number; ms: number; timedOut: boolean }> {
|
|
105
|
+
const model = ctx.model;
|
|
106
|
+
if (!model || !ctx.modelRegistry.hasConfiguredAuth(model)) {
|
|
107
|
+
return { text, applied: 0, ms: 0, timedOut: false };
|
|
108
|
+
}
|
|
109
|
+
const base = protectedMask(text);
|
|
110
|
+
const guard = (offset: number, length: number) =>
|
|
111
|
+
base(offset, length) || protect(text.slice(offset, offset + length));
|
|
112
|
+
const outcome = await rewriteReply(
|
|
113
|
+
text,
|
|
114
|
+
spans,
|
|
115
|
+
async (prompt, signal) => {
|
|
116
|
+
// The signal must reach the provider: that is what cancels the HTTP request.
|
|
117
|
+
const response = await ctx.modelRegistry.complete(
|
|
118
|
+
model,
|
|
119
|
+
{ messages: [{ role: "user", content: [{ type: "text", text: prompt }], timestamp: Date.now() }] },
|
|
120
|
+
{ signal, sessionId: `ko-sanitizer-${Date.now()}` },
|
|
121
|
+
);
|
|
122
|
+
return response.content
|
|
123
|
+
.filter((c): c is { type: "text"; text: string } => c.type === "text")
|
|
124
|
+
.map((c) => c.text)
|
|
125
|
+
.join("\n");
|
|
126
|
+
},
|
|
127
|
+
{ timeoutMs: LLM_TIMEOUT_MS, verify: VERIFY },
|
|
128
|
+
guard,
|
|
129
|
+
);
|
|
130
|
+
return { text: outcome.text, applied: outcome.applied.length, ms: outcome.ms, timedOut: outcome.timedOut };
|
|
131
|
+
}
|
|
132
|
+
|
|
133
|
+
export default function (pi: ExtensionAPI) {
|
|
134
|
+
let replaced = 0;
|
|
135
|
+
let flagged = 0;
|
|
136
|
+
let rewritten = 0;
|
|
137
|
+
|
|
138
|
+
pi.on("message_end", async ({ message }, ctx) => {
|
|
139
|
+
if (OFF || message.role !== "assistant") return;
|
|
140
|
+
let changed = 0;
|
|
141
|
+
const found: Finding[] = [];
|
|
142
|
+
const protect = askedSpans(lastUserText(ctx));
|
|
143
|
+
const content = message.content.map((part) => {
|
|
144
|
+
if (part.type !== "text") return part;
|
|
145
|
+
const result = processText(part.text, { protect });
|
|
146
|
+
changed += result.count;
|
|
147
|
+
found.push(...result.findings);
|
|
148
|
+
return result.count ? { ...part, text: result.text } : part;
|
|
149
|
+
});
|
|
150
|
+
replaced += changed;
|
|
151
|
+
flagged += found.length;
|
|
152
|
+
|
|
153
|
+
// Sentence-level rewrite: only for leaks the dictionary could not read, one
|
|
154
|
+
// attempt, and the reply is kept if the model fails or answers off-contract.
|
|
155
|
+
let llm = { applied: 0, ms: 0, timedOut: 0 };
|
|
156
|
+
if (LLM && found.length) {
|
|
157
|
+
const perPart = content.filter((p) => p.type === "text");
|
|
158
|
+
const totalChars = perPart.reduce((n, p) => n + p.text.length, 0);
|
|
159
|
+
if (totalChars <= LLM_MAX_CHARS) {
|
|
160
|
+
try {
|
|
161
|
+
for (const [index, part] of content.entries()) {
|
|
162
|
+
if (part.type !== "text") continue;
|
|
163
|
+
const spans = [...new Set(candidates(part.text).filter((span) => !protect(span)))].slice(0, LLM_MAX_SPANS);
|
|
164
|
+
if (!spans.length) continue;
|
|
165
|
+
const out = await rewritePart(part.text, spans, ctx, protect);
|
|
166
|
+
llm = { applied: out.applied, ms: out.ms, timedOut: Number(out.timedOut) };
|
|
167
|
+
if (out.text !== part.text) content[index] = { ...part, text: out.text };
|
|
168
|
+
rewritten += out.applied;
|
|
169
|
+
break;
|
|
170
|
+
}
|
|
171
|
+
} catch (error) {
|
|
172
|
+
// Fail open: the dictionary result stands, the reply is never lost.
|
|
173
|
+
if (DEBUG) log({ t: new Date().toISOString(), pid: process.pid, msg: "llm-error", error: String(error).slice(0, 120) });
|
|
174
|
+
}
|
|
175
|
+
}
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
log({
|
|
179
|
+
t: new Date().toISOString(),
|
|
180
|
+
pid: process.pid,
|
|
181
|
+
msg: "reply",
|
|
182
|
+
chars: content.reduce((n, p) => n + (p.type === "text" ? p.text.length : 0), 0),
|
|
183
|
+
changed,
|
|
184
|
+
flagged: found.length,
|
|
185
|
+
llmApplied: llm.applied,
|
|
186
|
+
llmMs: llm.ms,
|
|
187
|
+
llmTimedOut: llm.timedOut,
|
|
188
|
+
});
|
|
189
|
+
if (DEBUG) log({ t: new Date().toISOString(), pid: process.pid, msg: "inspect", parts: content.length });
|
|
190
|
+
if (found.length) log({ t: new Date().toISOString(), pid: process.pid, msg: "findings", count: found.length, findings: found.map(findingRecord) });
|
|
191
|
+
const touched = changed > 0 || llm.applied > 0;
|
|
192
|
+
if (!touched) return;
|
|
193
|
+
return { message: { ...message, content } };
|
|
194
|
+
});
|
|
195
|
+
|
|
196
|
+
// The package is judged by how often it fires, so the counter ships with it.
|
|
197
|
+
pi.registerCommand("zh-ko", {
|
|
198
|
+
description: "Show replaced, flagged and rewritten counts for this session",
|
|
199
|
+
handler: async (_args, ctx) => {
|
|
200
|
+
ctx.ui.notify(`replaced ${replaced}, flagged ${flagged}, rewritten ${rewritten} (log: ${LOG})`, "info");
|
|
201
|
+
},
|
|
202
|
+
});
|
|
203
|
+
}
|
package/src/lexicon.ts
ADDED
|
@@ -0,0 +1,132 @@
|
|
|
1
|
+
// Hand-written closed vocabulary. No external dictionary (CC-CEDICT and
|
|
2
|
+
// friends carry share-alike terms and import a maintenance chain we do not
|
|
3
|
+
// want). Rule: two or more characters, and the Korean equivalent is what a
|
|
4
|
+
// Korean writer would actually type. Single-character function words (的, 了,
|
|
5
|
+
// 是) need deletion rules, not substitution — they land in 0.2.0.
|
|
6
|
+
//
|
|
7
|
+
// Ordered roughly by what local Chinese-origin models leak most: honorifics
|
|
8
|
+
// and polite formulas first, then pronouns, connectives, adverbs, and finally
|
|
9
|
+
// technical nouns where the Sino-Korean cognate is the natural Korean word.
|
|
10
|
+
export const LEXICON: ReadonlyArray<readonly [string, string]> = [
|
|
11
|
+
// honorifics and polite formulas
|
|
12
|
+
["本次会议", "이번 회의"], ["具体时间", "구체적인 시간"], ["贵公司", "귀사"],
|
|
13
|
+
["您的", "귀하의"], ["为您", "위해"], ["请问", "실례지만"], ["感谢您", "감사합니다"],
|
|
14
|
+
["谢谢", "감사합니다"], ["欢迎您", "환영합니다"], ["麻烦您", "번거롭지만"],
|
|
15
|
+
["辛苦", "수고"], ["尊敬", "존경"], ["请问您", "여쭙겠습니다"],
|
|
16
|
+
|
|
17
|
+
// pronouns and demonstratives
|
|
18
|
+
["这个", "이"], ["那个", "그"], ["这些", "이것들"], ["那些", "그것들"],
|
|
19
|
+
["我们", "우리"], ["你们", "여러분"], ["他们", "그들"], ["这里", "여기"],
|
|
20
|
+
["那里", "거기"], ["这样", "이렇게"], ["那样", "그렇게"], ["什么", "무엇"],
|
|
21
|
+
["怎么", "어떻게"], ["为什么", "왜"], ["哪个", "어느"],
|
|
22
|
+
|
|
23
|
+
// connectives and logical links
|
|
24
|
+
["因为", "왜냐하면"], ["所以", "그래서"], ["但是", "하지만"], ["如果", "만약"],
|
|
25
|
+
["虽然", "비록"], ["而且", "게다가"], ["然后", "그다음"], ["因此", "따라서"],
|
|
26
|
+
["不过", "하지만"], ["或者", "또는"], ["以及", "그리고"], ["除了", "제외하고"],
|
|
27
|
+
["由于", "때문에"], ["为了", "위해"], ["关于", "관해"], ["对于", "대해"],
|
|
28
|
+
["根据", "에 따르면"], ["通过", "통해"], ["例如", "예를 들어"], ["比如", "예를 들어"],
|
|
29
|
+
["其实", "사실은"], ["当然", "물론"], ["同时", "동시에"], ["另外", "또한"],
|
|
30
|
+
["此外", "게다가"], ["的时候", "때"], ["的话", "면"],
|
|
31
|
+
|
|
32
|
+
// adverbs, tense and modality
|
|
33
|
+
["已经", "이미"], ["还是", "여전히"], ["仍然", "여전히"], ["必须", "반드시"],
|
|
34
|
+
["应该", "해야"], ["可以", "가능"], ["能够", "할 수 있"], ["可能", "아마"],
|
|
35
|
+
["需要", "필요"], ["一定", "반드시"], ["非常", "매우"], ["比较", "비교적"],
|
|
36
|
+
["更加", "더"], ["特别", "특히"], ["尤其", "특히"], ["主要", "주로"],
|
|
37
|
+
["一般", "일반적으로"], ["目前", "현재"], ["现在", "지금"], ["以后", "이후"],
|
|
38
|
+
["之前", "이전"], ["之后", "이후"], ["刚才", "방금"], ["大概", "대략"],
|
|
39
|
+
["也许", "아마도"], ["几乎", "거의"], ["完全", "완전히"], ["直接", "직접"],
|
|
40
|
+
["立即", "즉시"], ["总是", "항상"], ["经常", "자주"], ["有时", "때때로"],
|
|
41
|
+
["再次", "다시"], ["一直", "계속"], ["一起", "함께"], ["并且", "그리고"],
|
|
42
|
+
|
|
43
|
+
// technical nouns and verbs where the Sino-Korean cognate is the natural word
|
|
44
|
+
["修改", "수정"], ["使用", "사용"], ["进行", "진행"], ["检查", "확인"],
|
|
45
|
+
["运行", "실행"], ["测试", "테스트"], ["编译", "컴파일"], ["报错", "오류"],
|
|
46
|
+
["错误", "오류"], ["文件", "파일"], ["代码", "코드"], ["函数", "함수"],
|
|
47
|
+
["变量", "변수"], ["类型", "타입"], ["接口", "인터페이스"], ["实现", "구현"],
|
|
48
|
+
["问题", "문제"], ["原因", "원인"], ["结果", "결과"], ["方法", "방법"],
|
|
49
|
+
["方式", "방식"], ["情况", "상황"], ["时间", "시간"], ["版本", "버전"],
|
|
50
|
+
["更新", "업데이트"], ["安装", "설치"], ["配置", "설정"], ["设置", "설정"],
|
|
51
|
+
["删除", "삭제"], ["添加", "추가"], ["创建", "생성"], ["支持", "지원"],
|
|
52
|
+
["包含", "포함"], ["建议", "제안"], ["注意", "주의"], ["确认", "확인"],
|
|
53
|
+
["完成", "완료"], ["开始", "시작"], ["结束", "종료"], ["失败", "실패"],
|
|
54
|
+
["成功", "성공"], ["性能", "성능"], ["内存", "메모리"], ["进程", "프로세스"],
|
|
55
|
+
["线程", "스레드"], ["网络", "네트워크"], ["服务", "서비스"], ["服务器", "서버"],
|
|
56
|
+
["数据库", "데이터베이스"], ["系统", "시스템"], ["参数", "매개변수"],
|
|
57
|
+
["返回值", "반환값"], ["默认", "기본값"], ["输入", "입력"], ["输出", "출력"],
|
|
58
|
+
["日志", "로그"], ["目录", "디렉터리"], ["路径", "경로"], ["命令", "명령"],
|
|
59
|
+
["环境", "환경"], ["依赖", "의존성"], ["模块", "모듈"], ["组件", "컴포넌트"],
|
|
60
|
+
["框架", "프레임워크"], ["文档", "문서"], ["示例", "예시"], ["方案", "방안"],
|
|
61
|
+
["步骤", "단계"], ["目标", "목표"], ["需求", "요구사항"], ["功能", "기능"],
|
|
62
|
+
["内容", "내용"], ["用户", "사용자"], ["项目", "프로젝트"], ["任务", "작업"],
|
|
63
|
+
["过程", "과정"], ["部分", "부분"], ["数量", "수량"], ["质量", "품질"],
|
|
64
|
+
["效率", "효율"], ["权限", "권한"], ["安全", "보안"], ["备份", "백업"],
|
|
65
|
+
["恢复", "복구"], ["缓存", "캐시"], ["解析", "파싱"], ["调用", "호출"],
|
|
66
|
+
["执行", "실행"], ["返回", "반환"], ["读取", "읽기"], ["写入", "쓰기"],
|
|
67
|
+
["保存", "저장"], ["加载", "로드"], ["提交", "커밋"], ["合并", "병합"],
|
|
68
|
+
["部署", "배포"], ["构建", "빌드"], ["调试", "디버깅"], ["输入框", "입력창"],
|
|
69
|
+
["下拉", "드롭다운"], ["按钮", "버튼"], ["页面", "페이지"], ["字段", "필드"],
|
|
70
|
+
["覆盖", "커버"],
|
|
71
|
+
// Entries have a fixed Korean reading; context-dependent spans stay untouched.
|
|
72
|
+
["展开", "전개"], ["同一个", "같은"], ["引擎", "엔진"], ["配额", "할당량"],
|
|
73
|
+
["管线", "파이프라인"], ["全文", "전문"], ["改变", "변경"], ["灾难", "재난"],
|
|
74
|
+
["基准线", "기준선"], ["零件", "부품"], ["代价", "비용"], ["分工", "분업"],
|
|
75
|
+
["补足", "보완"], ["集合", "집합"], ["中文", "중국어"], ["排队", "대기열"],
|
|
76
|
+
["疑心", "의심"],
|
|
77
|
+
["新功能", "새 기능"], ["之外", "이외"], ["永远", "항상"], ["追问", "추궁"],
|
|
78
|
+
["写的", "쓴"], ["定期对比", "정기 대조"], ["整形", "정형"],
|
|
79
|
+
["数字", "숫자"], ["正确地", "정확히"], ["非法", "불법"], ["查看", "조회"],
|
|
80
|
+
["决定", "결정"], ["短期", "단기"], ["清单", "목록"], ["别的", "다른"],
|
|
81
|
+
["鄰近", "인접"], ["环境变量", "환경 변수"],
|
|
82
|
+
["表格", "테이블"], ["菜单", "메뉴"], ["窗口", "창"], ["图标", "아이콘"],
|
|
83
|
+
["刚才的", "방금"], ["可以了", "됐습니다"], ["完成了", "완료했습니다"],
|
|
84
|
+
["没问题", "문제 없습니다"], ["是这样的", "이렇습니다"], ["请注意", "주의하세요"],
|
|
85
|
+
];
|
|
86
|
+
|
|
87
|
+
// Japanese leaks are function words too, and the same model family that leaks
|
|
88
|
+
// Chinese leaks these. Closed on purpose: a kana reading we cannot say out in
|
|
89
|
+
// Korean is a leak we report rather than rewrite, so this list stays short and
|
|
90
|
+
// every entry is a word a Korean writer actually types.
|
|
91
|
+
export const JAPANESE_LEAKS: ReadonlyArray<readonly [string, string]> = [
|
|
92
|
+
["新たに", "새롭게"], ["新しい", "새로운"], ["さらに", "추가로"], ["すでに", "이미"],
|
|
93
|
+
["また", "또"], ["トライ", "트라이"], ["具体的な", "구체적인"], ["やはり", "역시"], ["もちろん", "물론"], ["とても", "매우"],
|
|
94
|
+
];
|
|
95
|
+
|
|
96
|
+
// Hangul readings for short kana tails attached to Korean-rendered names.
|
|
97
|
+
export const KANA_READINGS: ReadonlyArray<readonly [string, string]> = [
|
|
98
|
+
["あ", "아"], ["い", "이"], ["う", "우"], ["え", "에"], ["お", "오"],
|
|
99
|
+
["か", "카"], ["き", "키"], ["く", "쿠"], ["け", "케"], ["こ", "코"],
|
|
100
|
+
["さ", "사"], ["し", "시"], ["す", "스"], ["せ", "세"], ["そ", "소"],
|
|
101
|
+
["た", "타"], ["ち", "치"], ["つ", "츠"], ["て", "테"], ["と", "토"],
|
|
102
|
+
["な", "나"], ["に", "니"], ["ぬ", "누"], ["ね", "네"], ["の", "노"],
|
|
103
|
+
["は", "하"], ["ひ", "히"], ["ふ", "후"], ["へ", "헤"], ["ほ", "호"],
|
|
104
|
+
["ま", "마"], ["み", "미"], ["む", "무"], ["め", "메"], ["も", "모"],
|
|
105
|
+
["や", "야"], ["ゆ", "유"], ["よ", "요"],
|
|
106
|
+
["ら", "라"], ["り", "리"], ["る", "루"], ["れ", "레"], ["ろ", "로"],
|
|
107
|
+
["わ", "와"], ["を", "오"], ["ん", "ㄴ"],
|
|
108
|
+
["が", "가"], ["ぎ", "기"], ["ぐ", "구"], ["げ", "게"], ["ご", "고"],
|
|
109
|
+
["ざ", "자"], ["じ", "지"], ["ず", "즈"], ["ぜ", "제"], ["ぞ", "조"],
|
|
110
|
+
["だ", "다"], ["ぢ", "지"], ["づ", "즈"], ["で", "데"], ["ど", "도"],
|
|
111
|
+
["ば", "바"], ["び", "비"], ["ぶ", "부"], ["べ", "베"], ["ぼ", "보"],
|
|
112
|
+
["ぱ", "파"], ["ぴ", "피"], ["ぷ", "푸"], ["ぺ", "페"], ["ぽ", "포"],
|
|
113
|
+
["ぁ", "아"], ["ぃ", "이"], ["ぅ", "우"], ["ぇ", "에"], ["ぉ", "오"],
|
|
114
|
+
["ゃ", "야"], ["ゅ", "유"], ["ょ", "요"],
|
|
115
|
+
["ア", "아"], ["イ", "이"], ["ウ", "우"], ["エ", "에"], ["オ", "오"],
|
|
116
|
+
["カ", "카"], ["キ", "키"], ["ク", "쿠"], ["ケ", "케"], ["コ", "코"],
|
|
117
|
+
["サ", "사"], ["シ", "시"], ["ス", "스"], ["セ", "세"], ["ソ", "소"],
|
|
118
|
+
["タ", "타"], ["チ", "치"], ["ツ", "츠"], ["テ", "테"], ["ト", "토"],
|
|
119
|
+
["ナ", "나"], ["ニ", "니"], ["ヌ", "누"], ["ネ", "네"], ["ノ", "노"],
|
|
120
|
+
["ハ", "하"], ["ヒ", "히"], ["フ", "후"], ["ヘ", "헤"], ["ホ", "호"],
|
|
121
|
+
["マ", "마"], ["ミ", "미"], ["ム", "무"], ["メ", "메"], ["モ", "모"],
|
|
122
|
+
["ヤ", "야"], ["ユ", "유"], ["ヨ", "요"],
|
|
123
|
+
["ラ", "라"], ["リ", "리"], ["ル", "루"], ["レ", "레"], ["ロ", "로"],
|
|
124
|
+
["ワ", "와"], ["ヲ", "오"], ["ン", "ㄴ"],
|
|
125
|
+
["ガ", "가"], ["ギ", "기"], ["グ", "구"], ["ゲ", "게"], ["ゴ", "고"],
|
|
126
|
+
["ザ", "자"], ["ジ", "지"], ["ズ", "즈"], ["ゼ", "제"], ["ゾ", "조"],
|
|
127
|
+
["ダ", "다"], ["ヂ", "지"], ["ヅ", "즈"], ["デ", "데"], ["ド", "도"],
|
|
128
|
+
["バ", "바"], ["ビ", "비"], ["ブ", "부"], ["ベ", "베"], ["ボ", "보"],
|
|
129
|
+
["パ", "파"], ["ピ", "피"], ["プ", "푸"], ["ペ", "페"], ["ポ", "포"],
|
|
130
|
+
["ァ", "아"], ["ィ", "이"], ["ゥ", "우"], ["ェ", "에"], ["ォ", "오"],
|
|
131
|
+
["ャ", "야"], ["ュ", "유"], ["ョ", "요"],
|
|
132
|
+
];
|