picturereader 1.0.3 → 2.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +76 -43
- package/cordis.patch.yml +4 -4
- package/package.json +6 -3
- package/scripts/preview.mjs +40 -40
- package/scripts/setup-ocr.mjs +96 -96
- package/skills/image-reading.md +119 -96
- package/skills/vision-analyze.md +256 -0
- package/src/guard.js +101 -0
- package/src/index.js +32 -30
- package/src/tool.js +548 -548
- package/src/vision-analyze.js +260 -0
- package/src/vlm.js +269 -0
package/LICENSE
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2026 picturereader
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
-
furnished to do so, subject to the following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
-
SOFTWARE.
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 picturereader
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,24 +1,36 @@
|
|
|
1
1
|
# picturereader
|
|
2
2
|
|
|
3
|
-
> 给纯文本模型(如 deepseek-v4-flash
|
|
4
|
-
>
|
|
5
|
-
> - **ZCode 版**:ZCode 桌面端插件(`zcode-plugin`,经 MCP 暴露工具)
|
|
3
|
+
> **v2.0.0** —— 给纯文本模型(如 deepseek-v4-flash)的**全能"读图"能力**。
|
|
4
|
+
> 融合 **独立伪多模态识图** 与 **外部视觉 API 接口**,一个插件搞定全部,**无需另装任何插件**。
|
|
6
5
|
|
|
7
|
-
>
|
|
8
|
-
>
|
|
9
|
-
> **纯本地、零外部模型依赖、零 API key、零 Python(PaddleOCR 为可选增强)**。
|
|
6
|
+
> - **DSH 版**:DeepSeek Harness 插件(`dsh-plugin`,含 DSH EAC 桌面端)—— 本分支(main)
|
|
7
|
+
> - **ZCode 版**:ZCode 桌面端插件(`zcode-plugin`,经 MCP 暴露工具)—— [zcode 分支](https://github.com/jing-hy/picturereader/tree/zcode)
|
|
10
8
|
|
|
11
9
|
[](https://github.com/awesome-dsh-plugin/awesome-dsh-plugin)
|
|
12
10
|
|
|
11
|
+
## 本轮工作(v2.0.0 新增)
|
|
12
|
+
|
|
13
|
+
在原有的**本地伪多模态识图**(像素扫描 + OCR + 取样,零外部依赖)之上,融合了社区 `dsh-universal-vision` 的优势,形成一套**统一、可交叉验证的完整识图栈**:
|
|
14
|
+
|
|
15
|
+
1. **新增外部视觉 API 接口**(`vlm.js`):桥接任意 OpenAI 兼容的视觉端点(本地 llama-server / LM Studio / vLLM / 云端网关),让模型获得真正的**语义理解**能力(场景、角色、界面、风格)。
|
|
16
|
+
2. **新增统一分析工具 `vision_analyze`**(`vision-analyze.js`):一次调用即可取回「低信息拦截 + 像素扫描 + OCR + VLM」多路证据。
|
|
17
|
+
3. **新增低信息量拦截**(`guard.js`):自动识别空白/未渲染/简单图片,避免小 VLM 在空图上幻觉,也节省调用成本。
|
|
18
|
+
4. **证据交叉验证**:VLM 描述与像素/OCR 实测冲突时,以实测为准——伪多模态与真 VLM 互为印证,抑制幻觉。
|
|
19
|
+
5. **多次提问**:可对同一张图用 `vision_analyze` 以不同 `prompt` 反复提问,从多角度复核同一内容。
|
|
20
|
+
|
|
21
|
+
> **核心优势一句话**:**独立伪多模态识图(零依赖、可离线) + 外部 API 语义接口(可选、即插即用)** —— 简单图用像素就够,复杂图一键接 VLM,一个插件全包含,不需要再装 `dsh-universal-vision` 或任何其他读图插件。
|
|
22
|
+
|
|
13
23
|
## 版本总览
|
|
14
24
|
|
|
15
25
|
| 版本 | 平台 | 形态 | 源码 | 安装 |
|
|
16
26
|
|---|---|---|---|---|
|
|
17
|
-
| **DSH 版** | DeepSeek Harness(含 EAC 桌面端) | npm 插件(`dsh.bundle`) |
|
|
18
|
-
| **ZCode 版** | ZCode 桌面端 | 本地 marketplace 插件(MCP server + skill) | [
|
|
27
|
+
| **DSH 版** | DeepSeek Harness(含 EAC 桌面端) | npm 插件(`dsh.bundle`) | 本仓库(main) | `dsh plugin --profile web add picturereader` |
|
|
28
|
+
| **ZCode 版** | ZCode 桌面端 | 本地 marketplace 插件(MCP server + skill) | [zcode 分支](https://github.com/jing-hy/picturereader/tree/zcode) | `npm install picturereader-zcode` |
|
|
19
29
|
|
|
20
30
|
两个版本共用同一套业务核心(`src/core.js`)与读图方法论 skill(`image-reading`),
|
|
21
|
-
|
|
31
|
+
四个工具行为完全一致:`image_scan` / `image_ocr` / `image_sample` / `vision_analyze`。
|
|
32
|
+
|
|
33
|
+
> **兼容性**:DSH 版已验证兼容 **DeepSeek Harness EAC 4.2.0** 及 `@deepseek-ai/dsh-client-ui-workspace` **rc.7**(4.2.0 配套的官方工作区插件版本)。
|
|
22
34
|
|
|
23
35
|
> **ZCode 版性能说明**:ZCode 版通过 MCP(stdio 子进程)暴露工具,每次调用都要
|
|
24
36
|
> 经历进程通信与序列化开销,**速度明显慢于 DSH 版**(DSH 版为插件内直接调用)。
|
|
@@ -32,7 +44,8 @@ DeepSeek 等纯文本模型没有视觉编码器,无法直接看图。picturer
|
|
|
32
44
|
2. **主动找主体**:px_per_cell 定向放大(深色/低对比/小色块不会漏)
|
|
33
45
|
3. **文字验证**:PaddleOCR 实读(防多模态幻觉)
|
|
34
46
|
4. **材质判断**:image_sample 像素取样
|
|
35
|
-
5.
|
|
47
|
+
5. **(可选)VLM 语义理解**:外部视觉 API 提供场景/角色/界面/风格的自然语言描述
|
|
48
|
+
6. **综合描述**:带证据等级的连贯画面描述,伪多模态与 VLM 交叉验证
|
|
36
49
|
|
|
37
50
|
## 工具
|
|
38
51
|
|
|
@@ -41,6 +54,7 @@ DeepSeek 等纯文本模型没有视觉编码器,无法直接看图。picturer
|
|
|
41
54
|
| `image_scan` | 全局/区域扫描:亮度/颜色网格 + regions 色块 + shade diversity + texture mix + structure(条纹/对称) + **像素级 colors** + **hue families 纯色指纹**;支持 `focus`/`region` 局部放大、`px_per_cell` 像素密度定向放大 |
|
|
42
55
|
| `image_ocr` | 文字识别双引擎:`windows`(内置,默认)/ `paddle`(选装,发光/弯曲/游戏字更强),失败自动降级不崩溃 |
|
|
43
56
|
| `image_sample` | 8×8 精确像素取样,判断材质/纹理(金属/木纹/织物/皮肤/噪点) |
|
|
57
|
+
| `vision_analyze` | **统一入口(v2.0.0 新增)**:低信息拦截 + 可选像素扫描/OCR/VLM,组合证据返回;VLM 可选配置 |
|
|
44
58
|
|
|
45
59
|
### 读图方法论 skill(image-reading)
|
|
46
60
|
|
|
@@ -50,6 +64,7 @@ DeepSeek 等纯文本模型没有视觉编码器,无法直接看图。picturer
|
|
|
50
64
|
- **hue 场景指纹**:cyan 高=水/雾/湖泊,green 高=森林,orange/red 高=暖色人物/火光,blue 高=夜空科幻,achromatic+rough=废墟,green+yellow=翠绿能量/浮空仙境
|
|
51
65
|
- **多模态模型校验规则**:游戏名/品牌等文字必须 OCR 实读(多模态模型会猜错);发光元素颜色以 hue 实测为准(多模态模型对发光色的描述系统性不可靠);低对比主体(暗色人物/小色块)必须放大确认
|
|
52
66
|
- **主动验证**:低对比主体(暗色人物/小色块)必须放大确认
|
|
67
|
+
- **vision_analyze 使用**(v2.0.0):先 image_scan 自己看,简单图用像素,复杂图再调 VLM;描述与实测冲突时以实测为准
|
|
53
68
|
|
|
54
69
|
## 安装
|
|
55
70
|
|
|
@@ -67,25 +82,44 @@ copy skills\image-reading.md %USERPROFILE%\.dsh\skills\ # Windows
|
|
|
67
82
|
# 3.(可选)PaddleOCR 增强引擎:node scripts/setup-ocr.mjs
|
|
68
83
|
```
|
|
69
84
|
|
|
70
|
-
重启 DSH Desktop 后,模型工具列表出现 `image_scan` / `image_ocr` / `image_sample`,
|
|
85
|
+
重启 DSH Desktop 后,模型工具列表出现 `image_scan` / `image_ocr` / `image_sample` / `vision_analyze`,
|
|
71
86
|
技能目录出现 `image-reading`。
|
|
72
87
|
|
|
73
88
|
### ZCode 版
|
|
74
89
|
|
|
75
|
-
ZCode
|
|
90
|
+
ZCode 版位于本仓库的 **[zcode 分支](https://github.com/jing-hy/picturereader/tree/zcode)**。
|
|
76
91
|
|
|
77
|
-
ZCode 版通过 **MCP server**(`mcp/server.js`,stdio
|
|
92
|
+
ZCode 版通过 **MCP server**(`mcp/server.js`,stdio)把四个工具暴露给 ZCode,
|
|
78
93
|
读图方法论作为 **skill**(`skills/image-reading/`)随插件分发,业务逻辑 `src/core.js`
|
|
79
|
-
|
|
94
|
+
与 DSH 版完全一致。安装:`npm install picturereader-zcode`。
|
|
80
95
|
|
|
81
96
|
## 使用
|
|
82
97
|
|
|
83
98
|
直接对模型说:
|
|
84
99
|
|
|
85
100
|
> 用 image_scan 看一下 <路径> 这张图,细看感兴趣的部分
|
|
101
|
+
> (复杂场景可接着用 vision_analyze 获取语义描述并交叉验证)
|
|
86
102
|
|
|
87
103
|
模型会加载 `image-reading` 方法论自动执行完整流程(定调 → 找主体 → 验证 → 描述)。
|
|
88
104
|
|
|
105
|
+
### vision_analyze 用法(v2.0.0)
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
vision_analyze(
|
|
109
|
+
file_path="C:/shot.png",
|
|
110
|
+
prompt="描述这个界面,有哪些元素?布局是否正常?",
|
|
111
|
+
include_scan=true, # 像素扫描证据(默认 true)
|
|
112
|
+
include_ocr=true, # OCR 文字证据(默认 false)
|
|
113
|
+
include_vlm=true, # 外部 VLM 语义描述(默认 true,但未配置 SEE_BASE 时自动跳过)
|
|
114
|
+
allow_low_info=false, # 空白/简单图是否强制调 VLM(默认 false)
|
|
115
|
+
stop_after=false # 调用后是否关闭本插件启动的本地服务器
|
|
116
|
+
)
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
- **先自己看,再决定**:建议先用 `image_scan` 了解图片,简单图用像素就够;复杂/精密场景再开 VLM。
|
|
120
|
+
- **多次提问**:对同一张图换不同 `prompt` 反复调用,从多角度复核。
|
|
121
|
+
- **交叉验证**:VLM 描述与像素/OCR 冲突时,以实测为准。
|
|
122
|
+
|
|
89
123
|
## 输出示例
|
|
90
124
|
|
|
91
125
|
```
|
|
@@ -98,25 +132,38 @@ colors by area: ...(像素级真实占比)
|
|
|
98
132
|
luminance grid / color grid
|
|
99
133
|
```
|
|
100
134
|
|
|
101
|
-
##
|
|
135
|
+
## 环境变量
|
|
136
|
+
|
|
137
|
+
### PaddleOCR(可选)
|
|
102
138
|
|
|
103
139
|
| 变量 | 默认值 | 作用 |
|
|
104
140
|
|---|---|---|
|
|
105
141
|
| `DSH_PADDLE_PYTHON` | `C:\Users\Administrator\paddle_venv\Scripts\python.exe` | PaddleOCR 解释器路径(与原插件同名,便于直接迁移) |
|
|
106
142
|
| `DSH_PADDLE_CACHE` | `<插件目录>\.paddlex-cache` | PaddleX 模型缓存目录 |
|
|
107
143
|
|
|
144
|
+
### 外部视觉 API / VLM(可选,**默认不配置**)
|
|
145
|
+
|
|
146
|
+
| 变量 | 默认值 | 作用 |
|
|
147
|
+
|---|---|---|
|
|
148
|
+
| `SEE_BASE` | `(空)` | OpenAI 兼容视觉端点(留空 = VLM 禁用;本地 llama-server / LM Studio / vLLM / 云端网关) |
|
|
149
|
+
| `SEE_MODEL` | `(空)` | 视觉模型名(如 `google/gemma-4-12b-qat`) |
|
|
150
|
+
| `SEE_API_KEY` | `(空)` | API key(本地端点可随便填,云端需要真实 key) |
|
|
151
|
+
| `SEE_SERVER_EXE` / `SEE_SERVER_MODEL` / `SEE_SERVER_MMPROJ` | `(空)` | 本地 llama-server 自启路径(可选,配置后插件可自动拉起本地视觉服务器) |
|
|
152
|
+
| `SEE_SERVER_PORT` / `SEE_SERVER_NGL` / `SEE_SERVER_CTX` | `8080` / `20` / `16384` | 本地服务器参数 |
|
|
153
|
+
|
|
154
|
+
> **VLM 配置说明**:默认不配置 VLM,`vision_analyze` 会跳过 VLM 调用,只返回像素扫描和 OCR 证据(保持零外部依赖)。需要语义理解时,设置 `SEE_BASE` + `SEE_MODEL` 即可,例如指向本地 LM Studio(`http://127.0.0.1:1234/v1`)。
|
|
155
|
+
|
|
108
156
|
## 开发
|
|
109
157
|
|
|
110
158
|
```sh
|
|
111
159
|
# DSH 版(本仓库)
|
|
112
160
|
npm install
|
|
113
|
-
npm test # node:test
|
|
161
|
+
npm test # node:test
|
|
114
162
|
node scripts/setup-ocr.mjs # 可选:装 PaddleOCR
|
|
115
163
|
node scripts/preview.mjs # 生成 fixtures 并预览渲染
|
|
116
164
|
|
|
117
|
-
# ZCode
|
|
118
|
-
git
|
|
119
|
-
cd picturereader-zcode
|
|
165
|
+
# ZCode 版(本仓库 zcode 分支)
|
|
166
|
+
git checkout zcode
|
|
120
167
|
npm install
|
|
121
168
|
npm test # node:test
|
|
122
169
|
node scripts/setup-ocr.mjs # 可选
|
|
@@ -127,34 +174,20 @@ node scripts/setup-ocr.mjs # 可选
|
|
|
127
174
|
下次调用即生效**;工具定义(schema/描述)改动需重启桌面端。
|
|
128
175
|
|
|
129
176
|
**热插拔(ZCode 版)**:MCP server 从所选目录运行,改 `src/core.js` 下次调用即生效
|
|
130
|
-
(详见
|
|
177
|
+
(详见 zcode 分支 README)。
|
|
131
178
|
|
|
132
179
|
## 优势
|
|
133
180
|
|
|
134
|
-
-
|
|
135
|
-
|
|
136
|
-
-
|
|
137
|
-
|
|
138
|
-
-
|
|
181
|
+
- **一个插件全包含,无需另装**:独立的伪多模态识图 + 可选外部视觉 API,融合在一个包里,不需要再装 `dsh-universal-vision` 或其他任何读图插件
|
|
182
|
+
- **核心链路零外部依赖**:扫描/取样/解码纯本地纯 JS,不调任何视觉 API;语义理解要么交给主模型,要么按需桥接你自己的 VLM
|
|
183
|
+
- **双版本独立分发,互不污染**:DSH 用 `picturereader`(本页),ZCode 用 `picturereader-zcode`(zcode 分支);DSH 版带 `dsh.bundle`、ZCode 版带 `mcp` + `.zcode-plugin`,各装各的宿主,**不会把 ZCode 版误装进 DSH,也不会把 DSH 版误装进 ZCode**
|
|
184
|
+
- **外部 API 可选、即插即用**:默认不配置 `SEE_BASE` 保持纯本地;配了即接入语义理解,本地 llama-server / LM Studio / vLLM / 云端 OpenAI 兼容端点通吃
|
|
185
|
+
- **低信息量拦截**(v2.0.0):自动识别空白/未渲染/简单图,避免小 VLM 幻觉、省调用成本
|
|
186
|
+
- **证据交叉验证**(v2.0.0):VLM 描述与像素/OCR 实测冲突时以实测为准——伪多模态与真 VLM 互为印证,抑制幻觉
|
|
187
|
+
- **多次提问**(v2.0.0):同一张图可换不同 `prompt` 反复 `vision_analyze`,从多角度复核
|
|
188
|
+
- **可追溯、可验证**:每个结论都有数据支撑(hue 占比、色块坐标、OCR 文本+置信度)
|
|
139
189
|
- **成本低**:一次扫描 ≈0.6–2.2K tokens;PaddleOCR 本地跑,无 API 费用
|
|
140
|
-
-
|
|
141
|
-
- **方法论沉淀**:附带的 image-reading skill 把读图经验固化(场景指纹/校验规则),
|
|
142
|
-
模型每次看图都带着经过大量图片验证的经验
|
|
143
|
-
|
|
144
|
-
## 局限性(重要)
|
|
145
|
-
|
|
146
|
-
- **不是真正的视觉模型**:文本网格信息量有限,**人脸/表情/花纹等像素级细节读不出**;
|
|
147
|
-
这是文本模态的硬上限,放大(px_per_cell)只能缩小差距,不能消除
|
|
148
|
-
- **语义推断依赖主模型能力**:物体识别("这是树/空间站")是 LLM 基于结构证据的推测,
|
|
149
|
-
不是视觉模型的确证——复杂/罕见物体可能推断错误
|
|
150
|
-
- **OCR 引擎边界**:Windows OCR 对发光/弯曲/艺术字失效;PaddleOCR 强很多但需选装,
|
|
151
|
-
且对极小文字/极端艺术字仍可能失败(可配合放大)
|
|
152
|
-
- **性能**:4K 图解码 ~230ms;PaddleOCR 每次调用需 ~2s 加载模型;大图网格渲染
|
|
153
|
-
token 随 size 增长(64×64 color ≈ 3–5K tokens)
|
|
154
|
-
- **ZCode 版 MCP 开销**:ZCode 版工具经 MCP stdio 子进程通信,单次调用比 DSH 版慢
|
|
155
|
-
(进程启动 + JSON-RPC 序列化);高频/批量看图建议用 DSH 版
|
|
156
|
-
- **WebP 不支持**(提示转 PNG/JPEG);GIF 只读首帧
|
|
157
|
-
- **多模态模型的描述不可全信**(本插件可交叉验证,但最终语义仍需人工判断关键场景)
|
|
190
|
+
- **方法论沉淀**:附带的 image-reading skill 把读图经验固化(场景指纹/校验规则),模型每次看图都带着经过大量图片验证的经验
|
|
158
191
|
|
|
159
192
|
## License
|
|
160
193
|
|
package/cordis.patch.yml
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# picturereader — bundle patch: inserts the image_scan plugin row.
|
|
2
|
-
- insert:
|
|
3
|
-
- id: picturereader
|
|
4
|
-
name: 'picturereader'
|
|
1
|
+
# picturereader — bundle patch: inserts the image_scan plugin row.
|
|
2
|
+
- insert:
|
|
3
|
+
- id: picturereader
|
|
4
|
+
name: 'picturereader'
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "picturereader",
|
|
3
|
-
"version": "
|
|
4
|
-
"description": "
|
|
3
|
+
"version": "2.0.1",
|
|
4
|
+
"description": "Unified image understanding plugin for DeepSeek Harness (DSH). Fuses frame-free pseudo-multimodal reading (pixel scan + OCR + sample) with an optional external vision API bridge (local llama-server / LM Studio / any OpenAI-compatible endpoint) for full semantic understanding. No extra plugin needed.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"main": "src/index.js",
|
|
7
7
|
"exports": {
|
|
@@ -31,7 +31,10 @@
|
|
|
31
31
|
"ocr",
|
|
32
32
|
"paddleocr",
|
|
33
33
|
"hue",
|
|
34
|
-
"multimodal"
|
|
34
|
+
"multimodal",
|
|
35
|
+
"vlm",
|
|
36
|
+
"vision-analyze",
|
|
37
|
+
"llm-vision"
|
|
35
38
|
],
|
|
36
39
|
"license": "MIT",
|
|
37
40
|
"author": "picturereader",
|
package/scripts/preview.mjs
CHANGED
|
@@ -1,40 +1,40 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Generate fixture images on disk and print the image_scan render for a
|
|
3
|
-
* quick visual sanity check without booting a harness.
|
|
4
|
-
* Usage: node scripts/preview.mjs
|
|
5
|
-
* @module picturereader/scripts/preview
|
|
6
|
-
*/
|
|
7
|
-
|
|
8
|
-
import { mkdirSync, writeFileSync } from 'node:fs';
|
|
9
|
-
import { join, dirname } from 'node:path';
|
|
10
|
-
import { fileURLToPath } from 'node:url';
|
|
11
|
-
import { analyzeImage, renderImageScan } from '../src/core.js';
|
|
12
|
-
import { makeChartRgba, makeQuadrantRgba, pngFromRgba, jpegFromRgba, gifFromRgba, bmpFromRgba } from '../tests/fixtures.mjs';
|
|
13
|
-
|
|
14
|
-
const outDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'tests', 'fixtures-out');
|
|
15
|
-
mkdirSync(outDir, { recursive: true });
|
|
16
|
-
|
|
17
|
-
const images = [
|
|
18
|
-
{ name: 'chart', width: 600, height: 400, rgba: makeChartRgba(), formats: ['png', 'jpeg', 'gif', 'bmp'] },
|
|
19
|
-
{ name: 'quadrant', width: 100, height: 100, rgba: makeQuadrantRgba(), formats: ['png'] }
|
|
20
|
-
];
|
|
21
|
-
|
|
22
|
-
for (const image of images) {
|
|
23
|
-
for (const format of image.formats) {
|
|
24
|
-
const buffer =
|
|
25
|
-
format === 'png' ? pngFromRgba(image.width, image.height, image.rgba)
|
|
26
|
-
: format === 'jpeg' ? jpegFromRgba(image.width, image.height, image.rgba)
|
|
27
|
-
: format === 'gif' ? gifFromRgba(image.width, image.height, image.rgba)
|
|
28
|
-
: bmpFromRgba(image.width, image.height, image.rgba, 24);
|
|
29
|
-
writeFileSync(join(outDir, `${image.name}.${format}`), buffer);
|
|
30
|
-
}
|
|
31
|
-
const analysis = analyzeImage(image.rgba, image.width, image.height, { size: 32, mode: 'auto', region: undefined });
|
|
32
|
-
console.log(renderImageScan({
|
|
33
|
-
path: join(outDir, `${image.name}.png`),
|
|
34
|
-
width: image.width,
|
|
35
|
-
height: image.height,
|
|
36
|
-
region: 'full',
|
|
37
|
-
...analysis
|
|
38
|
-
}));
|
|
39
|
-
console.log();
|
|
40
|
-
}
|
|
1
|
+
/**
|
|
2
|
+
* Generate fixture images on disk and print the image_scan render for a
|
|
3
|
+
* quick visual sanity check without booting a harness.
|
|
4
|
+
* Usage: node scripts/preview.mjs
|
|
5
|
+
* @module picturereader/scripts/preview
|
|
6
|
+
*/
|
|
7
|
+
|
|
8
|
+
import { mkdirSync, writeFileSync } from 'node:fs';
|
|
9
|
+
import { join, dirname } from 'node:path';
|
|
10
|
+
import { fileURLToPath } from 'node:url';
|
|
11
|
+
import { analyzeImage, renderImageScan } from '../src/core.js';
|
|
12
|
+
import { makeChartRgba, makeQuadrantRgba, pngFromRgba, jpegFromRgba, gifFromRgba, bmpFromRgba } from '../tests/fixtures.mjs';
|
|
13
|
+
|
|
14
|
+
const outDir = join(dirname(fileURLToPath(import.meta.url)), '..', 'tests', 'fixtures-out');
|
|
15
|
+
mkdirSync(outDir, { recursive: true });
|
|
16
|
+
|
|
17
|
+
const images = [
|
|
18
|
+
{ name: 'chart', width: 600, height: 400, rgba: makeChartRgba(), formats: ['png', 'jpeg', 'gif', 'bmp'] },
|
|
19
|
+
{ name: 'quadrant', width: 100, height: 100, rgba: makeQuadrantRgba(), formats: ['png'] }
|
|
20
|
+
];
|
|
21
|
+
|
|
22
|
+
for (const image of images) {
|
|
23
|
+
for (const format of image.formats) {
|
|
24
|
+
const buffer =
|
|
25
|
+
format === 'png' ? pngFromRgba(image.width, image.height, image.rgba)
|
|
26
|
+
: format === 'jpeg' ? jpegFromRgba(image.width, image.height, image.rgba)
|
|
27
|
+
: format === 'gif' ? gifFromRgba(image.width, image.height, image.rgba)
|
|
28
|
+
: bmpFromRgba(image.width, image.height, image.rgba, 24);
|
|
29
|
+
writeFileSync(join(outDir, `${image.name}.${format}`), buffer);
|
|
30
|
+
}
|
|
31
|
+
const analysis = analyzeImage(image.rgba, image.width, image.height, { size: 32, mode: 'auto', region: undefined });
|
|
32
|
+
console.log(renderImageScan({
|
|
33
|
+
path: join(outDir, `${image.name}.png`),
|
|
34
|
+
width: image.width,
|
|
35
|
+
height: image.height,
|
|
36
|
+
region: 'full',
|
|
37
|
+
...analysis
|
|
38
|
+
}));
|
|
39
|
+
console.log();
|
|
40
|
+
}
|
package/scripts/setup-ocr.mjs
CHANGED
|
@@ -1,96 +1,96 @@
|
|
|
1
|
-
/**
|
|
2
|
-
* Optional install helper for the PaddleOCR engine (image_ocr engine="paddle").
|
|
3
|
-
* PaddleOCR is RECOMMENDED (far better at glowing/curved/game text) but
|
|
4
|
-
* OPTIONAL — image_ocr degrades to the Windows engine when it is missing.
|
|
5
|
-
*
|
|
6
|
-
* What this does:
|
|
7
|
-
* 1. Ensures a Python 3.12+ interpreter exists (downloads the official
|
|
8
|
-
* 3.12.10 installer from the npmmirror mirror if missing).
|
|
9
|
-
* 2. Creates/repairs the paddle_venv.
|
|
10
|
-
* 3. Installs paddlepaddle + paddleocr from the Tsinghua PyPI mirror.
|
|
11
|
-
* 4. Warms the model cache by running one recognition on a test image.
|
|
12
|
-
*
|
|
13
|
-
* Usage: node scripts/setup-ocr.mjs
|
|
14
|
-
*/
|
|
15
|
-
import { existsSync, mkdirSync } from 'node:fs';
|
|
16
|
-
import { spawnSync } from 'node:child_process';
|
|
17
|
-
import { join, dirname } from 'node:path';
|
|
18
|
-
import { fileURLToPath } from 'node:url';
|
|
19
|
-
|
|
20
|
-
const PY312 = 'C:\\Users\\Administrator\\Python312\\python.exe';
|
|
21
|
-
const VENV = 'C:\\Users\\Administrator\\paddle_venv\\Scripts\\python.exe';
|
|
22
|
-
const INSTALLER = 'C:\\Users\\Administrator\\Downloads\\python-3.12.10-amd64.exe';
|
|
23
|
-
const INSTALLER_URL = 'https://registry.npmmirror.com/-/binary/python/3.12.10/python-3.12.10-amd64.exe';
|
|
24
|
-
const PYPI = 'https://pypi.tuna.tsinghua.edu.cn/simple';
|
|
25
|
-
const CACHE = join(dirname(fileURLToPath(import.meta.url)), '..', '.paddlex-cache');
|
|
26
|
-
|
|
27
|
-
function run(cmd, args, opts = {}) {
|
|
28
|
-
console.log(`> ${cmd} ${args.join(' ')}`);
|
|
29
|
-
const result = spawnSync(cmd, args, { stdio: 'inherit', ...opts });
|
|
30
|
-
if (result.status !== 0) {
|
|
31
|
-
console.error(`!! command failed (exit ${result.status})`);
|
|
32
|
-
process.exit(1);
|
|
33
|
-
}
|
|
34
|
-
}
|
|
35
|
-
|
|
36
|
-
// 1. base Python 3.12
|
|
37
|
-
if (!existsSync(PY312)) {
|
|
38
|
-
console.log('[1/4] Python 3.12 missing — downloading installer (npmmirror mirror)...');
|
|
39
|
-
run('curl.exe', ['-L', '-o', INSTALLER, INSTALLER_URL]);
|
|
40
|
-
console.log('[1/4] Installing Python 3.12 to C:\\Users\\Administrator\\Python312 (user-level, silent)...');
|
|
41
|
-
spawnSync(INSTALLER, [
|
|
42
|
-
'/quiet', 'InstallAllUsers=0', 'TargetDir=C:\\Users\\Administrator\\Python312',
|
|
43
|
-
'Include_pip=1', 'PrependPath=0', 'Include_test=0', 'Include_launcher=0'
|
|
44
|
-
], { stdio: 'inherit' });
|
|
45
|
-
if (!existsSync(PY312)) {
|
|
46
|
-
console.error('!! Python install did not produce ' + PY312);
|
|
47
|
-
process.exit(1);
|
|
48
|
-
}
|
|
49
|
-
} else {
|
|
50
|
-
console.log('[1/4] Python 3.12 found at ' + PY312);
|
|
51
|
-
}
|
|
52
|
-
|
|
53
|
-
// 2. venv
|
|
54
|
-
if (!existsSync(VENV)) {
|
|
55
|
-
console.log('[2/4] Creating paddle_venv...');
|
|
56
|
-
mkdirSync(dirname(VENV), { recursive: true });
|
|
57
|
-
run(PY312, ['-m', 'venv', 'C:\\Users\\Administrator\\paddle_venv']);
|
|
58
|
-
} else {
|
|
59
|
-
console.log('[2/4] paddle_venv found');
|
|
60
|
-
}
|
|
61
|
-
|
|
62
|
-
// 3. paddlepaddle + paddleocr
|
|
63
|
-
const probe = spawnSync(VENV, ['-c', 'import paddleocr; print(paddleocr.__version__)'], { encoding: 'utf8' });
|
|
64
|
-
if (probe.status !== 0) {
|
|
65
|
-
console.log('[3/4] Installing paddlepaddle + paddleocr (Tsinghua mirror, ~1-3 min)...');
|
|
66
|
-
run(VENV, ['-m', 'pip', 'install', '-i', PYPI, '--upgrade', 'pip']);
|
|
67
|
-
run(VENV, ['-m', 'pip', 'install', '-i', PYPI, 'paddlepaddle==3.3.1', 'paddleocr']);
|
|
68
|
-
} else {
|
|
69
|
-
console.log(`[3/4] paddleocr already installed (${probe.stdout.trim()})`);
|
|
70
|
-
}
|
|
71
|
-
|
|
72
|
-
// 4. warm the model cache with one recognition
|
|
73
|
-
console.log('[4/4] Warming the model cache (first run downloads detection/recognition models)...');
|
|
74
|
-
const testImage = join(dirname(fileURLToPath(import.meta.url)), '..', 'tests', 'fixtures-out', 'ocr-test.png');
|
|
75
|
-
mkdirSync(join(CACHE), { recursive: true });
|
|
76
|
-
if (existsSync(testImage)) {
|
|
77
|
-
const warm = spawnSync(VENV, ['-c', [
|
|
78
|
-
'from paddleocr import PaddleOCR',
|
|
79
|
-
"ocr = PaddleOCR(lang='ch', use_doc_orientation_classify=False, use_doc_unwarping=False, use_textline_orientation=False, enable_mkldnn=False)",
|
|
80
|
-
`result = ocr.predict(r'${testImage.replaceAll("'", "''")}')`,
|
|
81
|
-
'print("warm-up OCR ok, lines:", sum(len(r.get("rec_texts") or []) for r in result))'
|
|
82
|
-
].join('; ')], {
|
|
83
|
-
env: { ...process.env, PADDLE_PDX_CACHE_HOME: CACHE, PYTHONIOENCODING: 'utf-8' },
|
|
84
|
-
encoding: 'utf8'
|
|
85
|
-
});
|
|
86
|
-
if (warm.status !== 0) {
|
|
87
|
-
console.error('!! warm-up failed — see output above; the engine may still work once models download');
|
|
88
|
-
process.exit(1);
|
|
89
|
-
}
|
|
90
|
-
console.log(warm.stdout.trim());
|
|
91
|
-
} else {
|
|
92
|
-
console.log('[4/4] test image missing — skip warm-up (first image_ocr paddle call will download models)');
|
|
93
|
-
}
|
|
94
|
-
|
|
95
|
-
console.log('\nDone. image_ocr engine="paddle" is now available.');
|
|
96
|
-
console.log('Verify: ask the model to read an image with image_ocr(engine="paddle").');
|
|
1
|
+
/**
|
|
2
|
+
* Optional install helper for the PaddleOCR engine (image_ocr engine="paddle").
|
|
3
|
+
* PaddleOCR is RECOMMENDED (far better at glowing/curved/game text) but
|
|
4
|
+
* OPTIONAL — image_ocr degrades to the Windows engine when it is missing.
|
|
5
|
+
*
|
|
6
|
+
* What this does:
|
|
7
|
+
* 1. Ensures a Python 3.12+ interpreter exists (downloads the official
|
|
8
|
+
* 3.12.10 installer from the npmmirror mirror if missing).
|
|
9
|
+
* 2. Creates/repairs the paddle_venv.
|
|
10
|
+
* 3. Installs paddlepaddle + paddleocr from the Tsinghua PyPI mirror.
|
|
11
|
+
* 4. Warms the model cache by running one recognition on a test image.
|
|
12
|
+
*
|
|
13
|
+
* Usage: node scripts/setup-ocr.mjs
|
|
14
|
+
*/
|
|
15
|
+
import { existsSync, mkdirSync } from 'node:fs';
|
|
16
|
+
import { spawnSync } from 'node:child_process';
|
|
17
|
+
import { join, dirname } from 'node:path';
|
|
18
|
+
import { fileURLToPath } from 'node:url';
|
|
19
|
+
|
|
20
|
+
const PY312 = 'C:\\Users\\Administrator\\Python312\\python.exe';
|
|
21
|
+
const VENV = 'C:\\Users\\Administrator\\paddle_venv\\Scripts\\python.exe';
|
|
22
|
+
const INSTALLER = 'C:\\Users\\Administrator\\Downloads\\python-3.12.10-amd64.exe';
|
|
23
|
+
const INSTALLER_URL = 'https://registry.npmmirror.com/-/binary/python/3.12.10/python-3.12.10-amd64.exe';
|
|
24
|
+
const PYPI = 'https://pypi.tuna.tsinghua.edu.cn/simple';
|
|
25
|
+
const CACHE = join(dirname(fileURLToPath(import.meta.url)), '..', '.paddlex-cache');
|
|
26
|
+
|
|
27
|
+
function run(cmd, args, opts = {}) {
|
|
28
|
+
console.log(`> ${cmd} ${args.join(' ')}`);
|
|
29
|
+
const result = spawnSync(cmd, args, { stdio: 'inherit', ...opts });
|
|
30
|
+
if (result.status !== 0) {
|
|
31
|
+
console.error(`!! command failed (exit ${result.status})`);
|
|
32
|
+
process.exit(1);
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
|
|
36
|
+
// 1. base Python 3.12
|
|
37
|
+
if (!existsSync(PY312)) {
|
|
38
|
+
console.log('[1/4] Python 3.12 missing — downloading installer (npmmirror mirror)...');
|
|
39
|
+
run('curl.exe', ['-L', '-o', INSTALLER, INSTALLER_URL]);
|
|
40
|
+
console.log('[1/4] Installing Python 3.12 to C:\\Users\\Administrator\\Python312 (user-level, silent)...');
|
|
41
|
+
spawnSync(INSTALLER, [
|
|
42
|
+
'/quiet', 'InstallAllUsers=0', 'TargetDir=C:\\Users\\Administrator\\Python312',
|
|
43
|
+
'Include_pip=1', 'PrependPath=0', 'Include_test=0', 'Include_launcher=0'
|
|
44
|
+
], { stdio: 'inherit' });
|
|
45
|
+
if (!existsSync(PY312)) {
|
|
46
|
+
console.error('!! Python install did not produce ' + PY312);
|
|
47
|
+
process.exit(1);
|
|
48
|
+
}
|
|
49
|
+
} else {
|
|
50
|
+
console.log('[1/4] Python 3.12 found at ' + PY312);
|
|
51
|
+
}
|
|
52
|
+
|
|
53
|
+
// 2. venv
|
|
54
|
+
if (!existsSync(VENV)) {
|
|
55
|
+
console.log('[2/4] Creating paddle_venv...');
|
|
56
|
+
mkdirSync(dirname(VENV), { recursive: true });
|
|
57
|
+
run(PY312, ['-m', 'venv', 'C:\\Users\\Administrator\\paddle_venv']);
|
|
58
|
+
} else {
|
|
59
|
+
console.log('[2/4] paddle_venv found');
|
|
60
|
+
}
|
|
61
|
+
|
|
62
|
+
// 3. paddlepaddle + paddleocr
|
|
63
|
+
const probe = spawnSync(VENV, ['-c', 'import paddleocr; print(paddleocr.__version__)'], { encoding: 'utf8' });
|
|
64
|
+
if (probe.status !== 0) {
|
|
65
|
+
console.log('[3/4] Installing paddlepaddle + paddleocr (Tsinghua mirror, ~1-3 min)...');
|
|
66
|
+
run(VENV, ['-m', 'pip', 'install', '-i', PYPI, '--upgrade', 'pip']);
|
|
67
|
+
run(VENV, ['-m', 'pip', 'install', '-i', PYPI, 'paddlepaddle==3.3.1', 'paddleocr']);
|
|
68
|
+
} else {
|
|
69
|
+
console.log(`[3/4] paddleocr already installed (${probe.stdout.trim()})`);
|
|
70
|
+
}
|
|
71
|
+
|
|
72
|
+
// 4. warm the model cache with one recognition
|
|
73
|
+
console.log('[4/4] Warming the model cache (first run downloads detection/recognition models)...');
|
|
74
|
+
const testImage = join(dirname(fileURLToPath(import.meta.url)), '..', 'tests', 'fixtures-out', 'ocr-test.png');
|
|
75
|
+
mkdirSync(join(CACHE), { recursive: true });
|
|
76
|
+
if (existsSync(testImage)) {
|
|
77
|
+
const warm = spawnSync(VENV, ['-c', [
|
|
78
|
+
'from paddleocr import PaddleOCR',
|
|
79
|
+
"ocr = PaddleOCR(lang='ch', use_doc_orientation_classify=False, use_doc_unwarping=False, use_textline_orientation=False, enable_mkldnn=False)",
|
|
80
|
+
`result = ocr.predict(r'${testImage.replaceAll("'", "''")}')`,
|
|
81
|
+
'print("warm-up OCR ok, lines:", sum(len(r.get("rec_texts") or []) for r in result))'
|
|
82
|
+
].join('; ')], {
|
|
83
|
+
env: { ...process.env, PADDLE_PDX_CACHE_HOME: CACHE, PYTHONIOENCODING: 'utf-8' },
|
|
84
|
+
encoding: 'utf8'
|
|
85
|
+
});
|
|
86
|
+
if (warm.status !== 0) {
|
|
87
|
+
console.error('!! warm-up failed — see output above; the engine may still work once models download');
|
|
88
|
+
process.exit(1);
|
|
89
|
+
}
|
|
90
|
+
console.log(warm.stdout.trim());
|
|
91
|
+
} else {
|
|
92
|
+
console.log('[4/4] test image missing — skip warm-up (first image_ocr paddle call will download models)');
|
|
93
|
+
}
|
|
94
|
+
|
|
95
|
+
console.log('\nDone. image_ocr engine="paddle" is now available.');
|
|
96
|
+
console.log('Verify: ask the model to read an image with image_ocr(engine="paddle").');
|