gpt-po 1.0.10 → 1.1.0

package/README.md

@@ -20,11 +20,11 @@ Set `OPENAI_API_KEY` before using this tool.
 ## Usage Scenarios
 
 - `gpt-po sync --po <file> --pot <file>` Update the po file based on the pot file, while preserving the original translations.
-- `gpt-po --po <file>` Translate specified po files to a designated target language. By default, the target language is Simplified Chinese.
-- `gpt-po --po <file> --lang <lang>` Translate specified po files to a designated target language.
+- `gpt-po --po <file>` Translate specified po files to a designated target language.
+- `gpt-po --po <file> --lang <lang>` Translate specified po files to a designated target language (overriding language specified in po file).
 - `gpt-po --dir .` Translate all po files in current directory to a designated target language.
 - `gpt-po userdict` Modify or view user dictionaries
-- `gpt-po userdict --explore` Explore user dictionaries, if you want add new dictionaries or modify existing dictionaries. dictionaries can be named as `dictionary-<lang>.json`, for example, `dictionary-simplified-chinese.json` is the dictionary for Simplified Chinese.
+- `gpt-po userdict --explore` Explore user dictionaries, if you want add new dictionaries or modify existing dictionaries. dictionaries can be named as `dictionary-<lang>.json`, for example, `dictionary-zh.json` is the dictionary for Simplified Chinese.
 - `gpt-po systemprompt` Modify or view system prompts, if you are not sure how to use it, you can leave it alone
 - `gpt-po systemprompt --reset` Reset system prompts
 
@@ -54,14 +54,15 @@ translate po file (default command)
 Options:
   -k, --key <key>        openai api key (env: OPENAI_API_KEY)
   --host <host>          openai api host (env: OPENAI_API_HOST)
-  --model <model>        openai model (choices: "gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314", "gpt-3.5-turbo", "gpt-3.5-turbo-0301",
-                         default: "gpt-3.5-turbo")
+  --model <model>        openai model (choices: "gpt-4o", "gpt-4-turbo", "gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314", "gpt-3.5-turbo", "gpt-3.5-turbo-0301",
+                         default: "gpt-4o")
   --po <file>            po file path
   --dir <dir>            po file directory
   -src, --source <lang>  source language (default: "english")
   --verbose              show verbose log
-  -l, --lang <lang>      target language (default: "simplified chinese")
+  -l, --lang <lang>      target language (ISO 639-1 code)
   -o, --output <file>    output file path, overwirte po file by default
+  --context              context file path (provides additional context to the bot)
   -h, --help             display help for command
 ```
 

package/README_zh-CN.md

@@ -15,69 +15,71 @@ npm install gpt-po
 
 使用此工具前先设置 `OPENAI_API_KEY`，Windows中使用 `set OPENAI_API_KEY=<key>`, Linux中 `export OPENAI_API_KEY=<key>`
 
-**建议使用付费的OpenAI API以提高翻译速度，免费的OpenAI API速度较慢（一分钟3条），且有使用限制。**
+**建议使用付费的OpenAI API以提高翻译速度，免费的OpenAI API速度较慢（一分钟3条一天200条），且有使用限制。**
+
+*国内用户要设置`HTTPS_PROXY`环境变量上梯子才能用*
 
 ## 常见用法
 
-- `gpt-po sync --po <file> --pot <file>` 根据pot文件更新po文件，保留原有翻译
-- `gpt-po --po <file>` 翻译指定的po文件到指定的目标语言，默认目标语言是简体中文
-- `gpt-po --po <file> --lang <lang>` Translate specified po files to a designated target language.
-- `gpt-po --dir .` 翻译当前目录下的所有po文件到指定的目标语言，默认目标语言是简体中文
-- `gpt-po userdict` 修改或查看用户字典
-- `gpt-po userdict --explore` 用于浏览字典目录, 如果你想要增加或修改现有的字典。字典可以按`dictionary-<lang>.json`格式命令, 例如, `dictionary-simplified-chinese.json` 是针对简体中文的字典.
-- `gpt-po systemprompt` 修改或查看系统提示词，如果你不确定如何使用，可以不用修改
-- `gpt-po systemprompt --reset` 重置系统提示词
+- `gpt-po sync --po <file> --pot <file>` 根据 pot 文件更新 po 文件，同时保留原有翻译。
+- `gpt-po --po <file>` 将指定的 po 文件翻译成目标语言。
+- `gpt-po --po <file> --lang <lang>` 将指定的 po 文件翻译成目标语言（覆盖 po 文件中指定的语言）。
+- `gpt-po --dir .` 将当前目录下的所有 po 文件翻译成目标语言。
+- `gpt-po userdict` 修改或查看用户词典。
+- `gpt-po userdict --explore` 浏览用户词典，如果您想添加新词典或修改现有词典，词典可以命名为 `dictionary-<lang>.json`，例如 `dictionary-zh.json` 是简体中文词典。
+- `gpt-po systemprompt` 修改或查看系统提示，如果您不确定如何使用它，可以忽略。
+- `gpt-po systemprompt --reset` 重置系统提示。
 
 ```
-Usage: gpt-po [options] [command]
+用法: gpt-po [options] [command]
 
-command tool for translate po files by gpt
+通过 gpt 翻译 po 文件的命令工具
 
-Options:
-  -V, --version           output the version number
-  -h, --help              display help for command
+选项:
+  -V, --version           输出版本号
+  -h, --help              显示命令帮助
 
-Commands:
-  translate [options]     translate po file (default command)
-  sync [options]          update po from pot file
-  systemprompt [options]  open/edit system prompt
-  userdict [options]      open/edit user dictionary
-  remove [options]        remove po entries by options
-  help [command]          display help for command
+命令:
+  translate [options]     翻译 po 文件（默认命令）
+  sync [options]          根据 pot 文件更新 po 文件
+  systemprompt [options]  打开/编辑系统提示
+  userdict [options]      打开/编辑用户词典
+  remove [options]        通过选项删除 po 条目
+  help [command]          显示命令帮助
 ```
 
 ```
-Usage: gpt-po [options]
-
-translate po file (default command)
-
-Options:
-  -k, --key <key>        openai api key (env: OPENAI_API_KEY)
-  --host <host>          openai api host (env: OPENAI_API_HOST)
-  --model <model>        openai model (choices: "gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314", "gpt-3.5-turbo", "gpt-3.5-turbo-0301",
-                         default: "gpt-3.5-turbo")
-  --po <file>            po file path
-  --dir <dir>            po file directory
-  -src, --source <lang>  source language (default: "english")
-  --verbose              show verbose log
-  -l, --lang <lang>      target language (default: "simplified chinese")
-  -o, --output <file>    output file path, overwirte po file by default
-  -h, --help             display help for command
+用法: gpt-po [options]
+
+翻译 po 文件（默认命令）
+
+选项:
+  -k, --key <key>        openai api key (环境变量: OPENAI_API_KEY)
+  --host <host>          openai api host (环境变量: OPENAI_API_HOST)
+  --model <model>        openai 模型 (选项: "gpt-4o", "gpt-4-turbo", "gpt-4", "gpt-4-0314", "gpt-4-32k", "gpt-4-32k-0314", "gpt-3.5-turbo", "gpt-3.5-turbo-0301", 默认: "gpt-4o")
+  --po <file>            po 文件路径
+  --dir <dir>            po 文件目录
+  -src, --source <lang>  源语言 (默认: "english")
+  --verbose              显示详细日志
+  -l, --lang <lang>      目标语言 (ISO 639-1 代码)
+  -o, --output <file>    输出文件路径，默认覆盖 po 文件
+  --context              上下文文件路径（为机器人提供额外的上下文）
+  -h, --help             显示命令帮助
 ```
 
 ```
-Usage: gpt-po remove [options]
-
-remove po entries by options
-
-Options:
-  --po <file>                       po file path
-  --fuzzy                           remove fuzzy entries
-  -obs, --obsolete                  remove obsolete entries
-  -ut, --untranslated               remove untranslated entries
-  -t, --translated                  remove translated entries
-  -tnf, --translated-not-fuzzy      remove translated not fuzzy entries
-  -ft, --fuzzy-translated           remove fuzzy translated entries
-  -rc, --reference-contains <text>  remove entries whose reference contains text, text can be a regular expression like /text/ig
-  -h, --help                        display help for command
+用法: gpt-po remove [options]
+
+通过选项删除 po 条目
+
+选项:
+  --po <file>                       po 文件路径
+  --fuzzy                           删除模糊条目
+  -obs, --obsolete                  删除过时条目
+  -ut, --untranslated               删除未翻译条目
+  -t, --translated                  删除已翻译条目
+  -tnf, --translated-not-fuzzy      删除已翻译但不是模糊的条目
+  -ft, --fuzzy-translated           删除模糊翻译条目
+  -rc, --reference-contains <text>  删除引用包含文本的条目，文本可以是正则表达式，例如 /text/ig
+  -h, --help                        显示命令帮助
 ```

package/lib/package.json

@@ -1,6 +1,6 @@
 {
     "name": "gpt-po",
-    "version": "1.0.10",
+    "version": "1.1.0",
     "description": "command tool for translate po files by gpt",
     "main": "lib/src/index.js",
     "bin": {

package/lib/src/index.js

@@ -24,8 +24,10 @@ program
     .addOption(new commander_1.Option("-k, --key <key>", "openai api key").env("OPENAI_API_KEY"))
     .addOption(new commander_1.Option("--host <host>", "openai api host").env("OPENAI_API_HOST"))
     .addOption(new commander_1.Option("--model <model>", "openai model")
-    .default("gpt-3.5-turbo")
+    .default("gpt-4o")
     .choices([
+    "gpt-4o",
+    "gpt-4-turbo",
     "gpt-4",
     "gpt-4-0314",
     "gpt-4-32k",
@@ -35,12 +37,13 @@ program
 ]))
     .addOption(new commander_1.Option("--po <file>", "po file path").conflicts("dir"))
     .addOption(new commander_1.Option("--dir <dir>", "po file directory").conflicts("po"))
-    .option("-src, --source <lang>", "source language", "English")
-    .option("-l, --lang <lang>", "target language", "Simplified Chinese")
+    .option("-src, --source <lang>", "source language (ISO 639-1)", "en")
+    .option("-l, --lang <lang>", "target language (ISO 639-1)")
     .option("--verbose", "print verbose log")
+    .option("--context <file>", "text file that provides the bot additional context")
     .addOption(new commander_1.Option("-o, --output <file>", "output file path, overwirte po file by default").conflicts("dir"))
     .action((args) => __awaiter(void 0, void 0, void 0, function* () {
-    const { key, host, model, po, dir, source, lang, verbose, output, checkRegx } = args;
+    const { key, host, model, po, dir, source, lang, verbose, output, context } = args;
     if (host) {
         process.env.OPENAI_API_HOST = host;
     }
@@ -54,10 +57,10 @@ program
     }
     (0, translate_1.init)();
     if (po) {
-        yield (0, translate_1.translatePo)(model, po, source, lang, verbose, output);
+        yield (0, translate_1.translatePo)(model, po, source, lang, verbose, output, context);
     }
     else if (dir) {
-        yield (0, translate_1.translatePoDir)(model, dir, source, lang, verbose);
+        yield (0, translate_1.translatePoDir)(model, dir, source, lang, verbose, context);
     }
     else {
         console.error("po file or directory is required");

package/lib/src/systemprompt.txt

@@ -1 +1,20 @@
-You are a translation expert, translate the text in a colloquial, professional and elegant manner without sounding like a machine translation. Remember to only translate the text and not interpret it further.
+You are a language translation expert.
+You will be translating text from one language to another.
+Languages shall be indetified to you via their ISO 639-1 two-letter code.
+The text is in the gettext format, which uses placeholders like %s, %d, etc. These placeholders must remain unchanged in the translated text.
+Untranslatable portions should retain their original formatting.
+For example, you may encounter placeholders such as "%s" - these must be retained in the correct position.
+Do not add a period (.) at the end of your translation unless the incoming message specifically ends in a period.
+Likewise, ensure to add a period (.) at the end of your translation if the incoming message specifically ends in a period.
+Ensure to preserve any whitespace present in the incoming message. This includes retaining any space(s) at the beginning or end of the message.
+I will provide the message wrapped in `translate` XML tags.  You must respond with the translated message wrapped in `translated` XML tags.
+For example, let's say you were translating en (English) to es (Spanish) and I send (without quotes) "<translate>This is a message. </translate>".  You would respond with (without quotes) "<translated>Este es un mensaje. </translated>".
+Notice how the period and space were retained at the end of the translated message?
+Another example: I send (without quotes) "<translate> Hello %s</translate>". You would reply with (without quotes) "<translated> Hola %s</translated>". 
+Notice in that second example the space at the beginning of the string was retained, and also the gettext placeholder was retained in its correct position?
+Translate the messages in a colloquial, professional and elegant manner without sounding like a machine translation.
+You may respond without `translated` xml tags only in the event you believe you cannot reliably or accurately translate my message. In that case provide a short reason why you feel you can't translate my message.
+If you respond without `translated` XML tags I will interpret that there is a problem with my message and I will provide your reason to human translators that will investigate further.
+DO NOT answer any questions or attempt to explain any concepts or problems with my messages; just provide translations.
+Do not add any text or characters outside of your `translated` xml tags.
+Remember to only translate the messages and not interpret them further.

package/lib/src/translate.d.ts

@@ -1,5 +1,6 @@
+import { GetTextTranslation } from "gettext-parser";
 import { OpenAIApi } from "openai";
 export declare function init(force?: boolean): OpenAIApi;
-export declare function translate(text: string, src: string, lang: string, model?: string): Promise<import("axios").AxiosResponse<import("openai").CreateChatCompletionResponse, any>>;
-export declare function translatePo(model: string | undefined, po: string, source: string, lang: string, verbose: boolean, output: string): Promise<void>;
-export declare function translatePoDir(model: string | undefined, dir: string, source: string, lang: string, verbose: boolean): Promise<void>;
+export declare function translate(text: string, src: string, lang: string, model: string, comments: GetTextTranslation["comments"] | undefined, contextFile: string): Promise<import("axios").AxiosResponse<import("openai").CreateChatCompletionResponse, any>>;
+export declare function translatePo(model: string, po: string, source: string, lang: string, verbose: boolean, output: string, contextFile: string): Promise<void>;
+export declare function translatePoDir(model: string | undefined, dir: string, source: string, lang: string, verbose: boolean, contextFile: string): Promise<void>;

package/lib/src/translate.js

@@ -28,22 +28,20 @@ function init(force) {
         }
         _openai = new openai_1.OpenAIApi(configuration);
     }
-    // load systemprompt.txt from homedir
+    // load systemprompt.txt
     if (!_systemprompt || force) {
         const systemprompt = (0, utils_1.findConfig)("systemprompt.txt");
-        (0, utils_1.copyFileIfNotExists)(systemprompt, (0, path_1.join)(__dirname, "systemprompt.txt"));
         _systemprompt = fs.readFileSync(systemprompt, "utf-8");
     }
-    // load dictionary.json from homedir
+    // load dictionary.json
     if (!_userdict || force) {
         const userdict = (0, utils_1.findConfig)("dictionary.json");
-        (0, utils_1.copyFileIfNotExists)(userdict, (0, path_1.join)(__dirname, "dictionary.json"));
         _userdict = { "default": JSON.parse(fs.readFileSync(userdict, "utf-8")) };
     }
     return _openai;
 }
 exports.init = init;
-function translate(text, src, lang, model = "gpt-3.5-turbo") {
+function translate(text, src, lang, model, comments, contextFile) {
     const lang_code = lang.toLowerCase().trim().replace(/[\W_]+/g, "-");
     const dicts = Object.entries(_userdict[lang_code] || _userdict["default"])
         .filter(([k, _]) => text.toLowerCase().includes(k.toLowerCase()))
@@ -52,26 +50,50 @@ function translate(text, src, lang, model = "gpt-3.5-turbo") {
         { role: "assistant", content: v },
     ])
         .flat();
+    var notes = "";
+    if (comments != undefined && comments.extracted != undefined)
+        notes = comments.extracted;
+    var context = "";
+    if (contextFile !== undefined)
+        context = "\n\n" + fs.readFileSync(contextFile, "utf-8");
     return _openai.createChatCompletion({
         model,
         temperature: 0.1,
         messages: [
             {
                 role: "system",
-                content: _systemprompt + ` Translate the ${src} content provided by the user into ${lang}. Please translate it as a text, not as a table. The parts that cannot be translated will retain their original format.`
+                content: _systemprompt + context
+            },
+            {
+                role: "user",
+                content: `Wait for my incoming message in "${src.toLowerCase()}" (an ISO 639-1 code) and translate it into "${lang.toLowerCase()}" (also an ISO 639-1 code), carefully following your system prompt. ` + notes
+            },
+            {
+                role: "assistant",
+                content: `Understood, I will translate your incoming "${src.toLowerCase()}" message into "${lang.toUpperCase()}", interpreting those as ISO 639-1 codes and carefully following my system prompt. Please go ahead and send your message for translation.`
             },
             // add userdict here
             ...dicts,
-            { role: "user", content: text },
+            {
+                role: "user",
+                content: "<translate>" + text + "</translate>"
+            },
         ],
     }, {
         timeout: 20000,
     });
 }
 exports.translate = translate;
-function translatePo(model = "gpt-3.5-turbo", po, source, lang, verbose, output) {
+function translatePo(model, po, source, lang, verbose, output, contextFile) {
     var _a;
     return __awaiter(this, void 0, void 0, function* () {
+        const potrans = yield (0, utils_1.parsePo)(po);
+        if (!lang)
+            lang = potrans.headers["Language"];
+        if (!lang) {
+            console.error("No language specified via po file or args");
+            return;
+        }
         // try to load dictionary by lang-code if it not loaded
         const lang_code = lang.toLowerCase().trim().replace(/[\W_]+/g, "-");
         if (!_userdict[lang_code]) {
@@ -81,7 +103,6 @@ function translatePo(model = "gpt-3.5-turbo", po, source, lang, verbose, output)
                 console.log(`dictionary-${lang_code}.json is loaded.`);
             }
         }
-        const potrans = yield (0, utils_1.parsePo)(po);
         const list = [];
         const trimRegx = /(?:^ )|(?: $)/;
         let trimed = false;
@@ -118,8 +139,15 @@ function translatePo(model = "gpt-3.5-turbo", po, source, lang, verbose, output)
             }
             const trans = list[i];
             try {
-                const res = yield translate(trans.msgid, source, lang, model);
-                trans.msgstr[0] = ((_a = res.data.choices[0].message) === null || _a === void 0 ? void 0 : _a.content) || trans.msgstr[0];
+                const res = yield translate(trans.msgid, source, lang, model, trans.comments, contextFile);
+                var translated = ((_a = res.data.choices[0].message) === null || _a === void 0 ? void 0 : _a.content) || trans.msgstr[0];
+                if (!translated.startsWith('<translated>') && !translated.endsWith('</translated>')) {
+                    // We got an error response
+                    console.log("Error: Unable to translate string [" + trans.msgid + "]. Bot says [" + translated + "]");
+                    continue;
+                }
+                // We got a valid translation response
+                trans.msgstr[0] = translated.replace('<translated>', '').replace('</translated>', '');
                 modified = true;
                 if (verbose) {
                     console.log(trans.msgid);
@@ -152,14 +180,14 @@ function translatePo(model = "gpt-3.5-turbo", po, source, lang, verbose, output)
     });
 }
 exports.translatePo = translatePo;
-function translatePoDir(model = "gpt-3.5-turbo", dir, source, lang, verbose) {
+function translatePoDir(model = "gpt-3.5-turbo", dir, source, lang, verbose, contextFile) {
     return __awaiter(this, void 0, void 0, function* () {
         const files = fs.readdirSync(dir);
         for (const file of files) {
             if (file.endsWith(".po")) {
                 const po = (0, path_1.join)(dir, file);
                 console.log(`translating ${po}`);
-                yield translatePo(model, po, source, lang, verbose, po);
+                yield translatePo(model, po, source, lang, verbose, po, contextFile);
             }
         }
     });

package/lib/src/utils.js

@@ -101,7 +101,7 @@ function findConfig(fileName) {
     const gitDir = gitRootDir() || currentDir;
     const homeDir = (0, os_1.homedir)();
     const filePaths = [
-        path.join(currentDir, ".gpt-po", fileName),
+        path.join(currentDir, "src", fileName),
         path.join(gitDir, ".gpt-po", fileName),
         path.join(homeDir, ".config", ".gpt-po", fileName)
     ];

package/package.json

@@ -1,55 +1,55 @@
-{
-  "name": "gpt-po",
-  "version": "1.0.10",
-  "description": "command tool for translate po files by gpt",
-  "main": "lib/src/index.js",
-  "bin": {
-    "gpt-po": "lib/src/index.js"
-  },
-  "scripts": {
-    "start": "node lib/src/index.js",
-    "test": "echo \"Error: no test specified\" && exit 1",
-    "prebuild": "rimraf ./lib",
-    "build": "tsc",
-    "postbuild": "ncp ./src/ ./lib/src/ \"--filter=^(?!.*\\.ts$).*$\"",
-    "format": "prettier --write \"src/**/*.ts\" \"src/**/*.js\"",
-    "lint": "tslint -p tsconfig.json"
-  },
-  "repository": {
-    "type": "git",
-    "url": "git+https://github.com/ryanhex53/gpt-po.git"
-  },
-  "author": "Ryan",
-  "license": "MIT",
-  "bugs": {
-    "url": "https://github.com/ryanhex53/gpt-po/issues"
-  },
-  "homepage": "https://github.com/ryanhex53/gpt-po#readme",
-  "keywords": [
-    "gettext",
-    "po",
-    "pot",
-    "chatgpt",
-    "openai"
-  ],
-  "devDependencies": {
-    "@types/gettext-parser": "^4.0.2",
-    "@types/jest": "^29.5.1",
-    "jest": "^29.5.0",
-    "ncp": "^2.0.0",
-    "prettier": "^3.0.3",
-    "rimraf": "^5.0.0",
-    "ts-jest": "^29.1.0",
-    "tslint": "^6.1.3",
-    "tslint-config-prettier": "^1.18.0",
-    "typescript": "^5.0.4"
-  },
-  "dependencies": {
-    "commander": "^10.0.1",
-    "gettext-parser": "^6.0.0",
-    "openai": "^3.2.1"
-  },
-  "engines": {
-    "node": ">=12.0.0"
-  }
-}
+{
+  "name": "gpt-po",
+  "version": "1.1.0",
+  "description": "command tool for translate po files by gpt",
+  "main": "lib/src/index.js",
+  "bin": {
+    "gpt-po": "lib/src/index.js"
+  },
+  "scripts": {
+    "start": "node lib/src/index.js",
+    "test": "echo \"Error: no test specified\" && exit 1",
+    "prebuild": "rimraf ./lib",
+    "build": "tsc",
+    "postbuild": "ncp ./src/ ./lib/src/ \"--filter=^(?!.*\\.ts$).*$\"",
+    "format": "prettier --write \"src/**/*.ts\" \"src/**/*.js\"",
+    "lint": "tslint -p tsconfig.json"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/ryanhex53/gpt-po.git"
+  },
+  "author": "Ryan",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/ryanhex53/gpt-po/issues"
+  },
+  "homepage": "https://github.com/ryanhex53/gpt-po#readme",
+  "keywords": [
+    "gettext",
+    "po",
+    "pot",
+    "chatgpt",
+    "openai"
+  ],
+  "devDependencies": {
+    "@types/gettext-parser": "^4.0.2",
+    "@types/jest": "^29.5.1",
+    "jest": "^29.5.0",
+    "ncp": "^2.0.0",
+    "prettier": "^3.0.3",
+    "rimraf": "^5.0.0",
+    "ts-jest": "^29.1.0",
+    "tslint": "^6.1.3",
+    "tslint-config-prettier": "^1.18.0",
+    "typescript": "^5.0.4"
+  },
+  "dependencies": {
+    "commander": "^10.0.1",
+    "gettext-parser": "^6.0.0",
+    "openai": "^3.2.1"
+  },
+  "engines": {
+    "node": ">=12.0.0"
+  }
+}