fletchshot-mcp 0.1.2 → 0.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -67,6 +67,10 @@ No other permission, entitlement, or FletchShot setting is required — and
67
67
  nothing is configured inside FletchShot's own Settings window for this (see
68
68
  FletchShot's help page for why).
69
69
 
70
+ If the Screen Recording permission is missing, `capture` fails with a message
71
+ telling you to grant it (System Settings → Privacy & Security → Screen
72
+ Recording) to the process that launched this server and relaunch it.
73
+
70
74
  ## Tools
71
75
 
72
76
  1. **`capture`** — take a screenshot. `mode: "window"` matches a window by
@@ -75,19 +79,63 @@ FletchShot's help page for why).
75
79
  the whole screen; `mode: "region"` captures a screen-coordinate rectangle
76
80
  after a delay (for menus/popovers, which can't be captured as a standalone
77
81
  window — ask the human to open the menu within the delay). Returns the
78
- file path, pixel size, scale (Retina = 2x, etc.), and the image itself
79
- (unless it's too large to inline).
82
+ file path and pixel size of the **original** image (that's the coordinate
83
+ space `open_draft` always uses), plus a downscaled preview (long edge
84
+ capped at 1568px — see "Preview images" below) embedded as the tool's
85
+ image content.
80
86
  2. **`open_draft`** — take the path `capture` returned plus a list of simple
81
- annotations (`arrow`, `text`, `rect`, `ellipse`, `mosaic`, `number`, all in
82
- image-pixel, top-left-origin coordinates) and open it in FletchShot as an
83
- editable draft. The human adjusts it, then presses **Return to agent**
84
- (⌘-Return).
87
+ annotations (`arrow`, `text`, `rect`, `ellipse`, `mosaic`, `number`) and
88
+ open it in FletchShot as an editable draft. The human adjusts it, then
89
+ presses **Return to agent** (⌘-Return). Coordinates can be given either as
90
+ numeric image pixels (`{x,y}` or `[x,y]`, top-left origin) or as a **text
91
+ anchor** `{text: "some label"}` — see "Text anchors" below. If a text
92
+ anchor doesn't resolve to exactly one match, the call fails with the list
93
+ of strings it did recognize (with `occurrence` numbers) so the agent can
94
+ retry.
85
95
  3. **`wait_for_return`** — wait until the human returns the draft (or a
86
- timeout), then return the finished image.
96
+ timeout), then return the finished image (same preview-downscaling as
97
+ `capture`, unless `includeImage: false` is passed, in which case only the
98
+ path and pixel size are returned).
87
99
 
88
100
  Typical flow: `capture` → `open_draft` → (human edits, presses ⌘-Return) →
89
101
  `wait_for_return`.
90
102
 
103
+ ### Text anchors
104
+
105
+ Any point (`arrow.from`/`arrow.to`, `text.at`, `number.at`) or region
106
+ (`rect`/`ellipse`/`mosaic`) can be given as `{text: "label", occurrence?: n}`
107
+ instead of numeric coordinates. `open_draft` runs on-device OCR (macOS
108
+ Vision, `VNRecognizeTextRequest`, Japanese + English) over the image to find
109
+ that string and computes the coordinates itself — so if the target has
110
+ visible text, there's no need to zoom into the screenshot, measure pixels, or
111
+ write a script to find its position; just pass the string you can read.
112
+ Targets without text (icons, shapes, …) still need an approximate numeric
113
+ position — that's fine, the human fine-tunes it in FletchShot anyway.
114
+
115
+ - `arrow.to` as a text anchor docks the arrowhead just outside the nearest
116
+ edge of the text (not centered on it, so it doesn't cover the label); if
117
+ `arrow.from` is omitted, it's placed automatically outside the text in
118
+ whichever diagonal direction stays on-screen.
119
+ - `text.at` / `number.at` as a text anchor are placed just next to the target
120
+ (to the right by default) instead of on top of it.
121
+ - `rect` / `ellipse` / `mosaic` as a text anchor become the text's bounding
122
+ box plus a small margin (`margin`, default 12px).
123
+ - If the string matches more than once, the call fails and returns every
124
+ match found with its `occurrence` number; pass that number to disambiguate.
125
+ - For scale: at the default annotation size, text is drawn about 55px tall
126
+ and an arrowhead is about 120px long × 100px wide on a 2x (Retina) capture.
127
+
128
+ ### Preview images
129
+
130
+ The images embedded in `capture` and `wait_for_return` are downscaled copies
131
+ (long edge capped at 1568px by default, via macOS `sips`; the original file
132
+ on disk is untouched) — this is what keeps a single screenshot from costing
133
+ tens of thousands of tokens. `open_draft` coordinates are always in the
134
+ **original** image's pixel space (`capture`'s `widthPx`/`heightPx`), never
135
+ the preview's; the tool results also return `previewScaleFactor` (preview
136
+ pixels × `previewScaleFactor` = original pixels) in case you need to reason
137
+ about the preview image itself.
138
+
91
139
  ## Privacy
92
140
 
93
141
  This server does not use the network at all (stdio transport only). Captured
@@ -165,21 +213,60 @@ args = ["-y", "fletchshot-mcp"]
165
213
  これ以外の許可・entitlement・FletchShot側の設定は不要です。FletchShotの設定画面にも、この
166
214
  機能のための項目は置いていません(理由はFletchShotのヘルプページを参照してください)。
167
215
 
216
+ 「画面収録」の許可が無い場合、`capture` は許可の付与場所(システム設定 > プライバシーと
217
+ セキュリティ > 画面収録)と、起動元プロセスを再起動する必要がある旨を案内するメッセージで
218
+ 失敗します。
219
+
168
220
  ## ツール
169
221
 
170
222
  1. **`capture`** — 画面を撮る。`mode: "window"` はアプリ名・ウィンドウタイトルの部分一致で
171
223
  ウィンドウ単体を撮る(一致なし・複数一致なら撮らずに候補一覧を返す)。`mode: "screen"` は
172
224
  画面全体。`mode: "region"` は画面座標の矩形を遅延つきで撮る(メニュー・ポップオーバー用。
173
- ウィンドウ単体では撮れないため、遅延の間に人へメニューを開いてもらう)。戻り値はファイルパス・
174
- ピクセルの幅高さ・倍率(Retinaなら2倍等)と、大きすぎない限り画像そのもの。
225
+ ウィンドウ単体では撮れないため、遅延の間に人へメニューを開いてもらう)。戻り値のファイルパス・
226
+ 幅高さは**元画像**のもの(open_draftの座標は常にこちらが基準)。埋め込む画像は長辺最大1568px
227
+ に縮小したプレビュー(後述「プレビュー画像」)。
175
228
  2. **`open_draft`** — `capture` が返したパスと、簡易な注釈(`arrow`・`text`・`rect`・
176
- `ellipse`・`mosaic`・`number`。すべて画像ピクセル・左上原点で指定)を受け取り、FletchShotで
177
- 掴める下書きとして開く。人が直して「エージェントに返す」(⌘↩)を押す。
229
+ `ellipse`・`mosaic`・`number`)を受け取り、FletchShotで掴める下書きとして開く。人が直して
230
+ 「エージェントに返す」(⌘↩)を押す。座標は数値({x,y}・[x,y]・画像ピクセル・左上原点)か、
231
+ **文字アンカー** `{text: "対象の文字列"}`(後述)のいずれかで指定する。文字アンカーが一意に
232
+ 解決できないときは、認識できた候補一覧(occurrence番号つき)とともに失敗するので、
233
+ occurrenceを指定して呼び直せる。
178
234
  3. **`wait_for_return`** — 人が下書きを返す(または時間切れになる)まで待ち、仕上がり画像を
179
- 返す。
235
+ 返す(captureと同様、埋め込みは縮小プレビュー。`includeImage: false` ならパスと幅高さだけ
236
+ 返す)。
180
237
 
181
238
  典型的な流れ: `capture` → `open_draft` →(人が直して⌘↩)→ `wait_for_return`。
182
239
 
240
+ ### 文字アンカー
241
+
242
+ 点座標(`arrow.from`/`arrow.to`・`text.at`・`number.at`)や範囲(`rect`・`ellipse`・`mosaic`)は、
243
+ 数値座標の代わりに `{text: "対象の文字列", occurrence?: n}` で指定できる。`open_draft` は
244
+ 画像に対してmacOS Vision(`VNRecognizeTextRequest`、日本語・英語)でオンデバイスの文字認識を
245
+ 行い、その文字列を探して座標を自動算出する——つまり対象に文字が写っていれば、画像を拡大して
246
+ 画素を確認したり座標を測るスクリプトを書いたりせずに、見えている文字列をそのまま渡せばよい。
247
+ 文字の無い対象(アイコンや図形など)はおおまかな数値座標で構わない(人がFletchShot上で微調整
248
+ する前提)。
249
+
250
+ - `arrow.to` を文字アンカーにすると、矢頭は文字の中心ではなく、文字の縁から少し外側(文字を
251
+ 隠さない位置)に自動配置する。`arrow.from` を省略すると、画像の端に出ない範囲で対象の外側・
252
+ 斜め方向から自動で引く。
253
+ - `text.at`・`number.at` を文字アンカーにすると、対象に重ねず近く(既定で右側)に自動で置く。
254
+ - `rect`・`ellipse`・`mosaic` を文字アンカーにすると、文字の枠に余白(`margin`、既定12px)を
255
+ 足した範囲になる。
256
+ - 同じ文字列が複数見つかった場合は失敗し、見つかった候補すべてをoccurrence番号つきで返す。
257
+ その番号を指定して呼び直せばよい。
258
+ - 目安: 既定の注釈サイズでは、文字の高さは約55px、矢頭は長さ約120px・幅約100px(2倍=Retina
259
+ 撮影時)。
260
+
261
+ ### プレビュー画像
262
+
263
+ `capture`・`wait_for_return` が埋め込む画像は、長辺最大1568px(既定値)に縮小したコピー
264
+ (macOS標準の`sips`を使用。ディスク上の元ファイルは書き換えない)——1枚のスクリーンショットだけで
265
+ 数万トークンを消費してしまうのを防ぐための縮小。`open_draft`の座標は常に**元画像**のピクセル
266
+ 座標(`capture`が返す`widthPx`/`heightPx`基準)であり、プレビューの座標ではない。戻り値の
267
+ `previewScaleFactor`(プレビュー座標×`previewScaleFactor`=元画像ピクセル)は、プレビュー画像
268
+ そのものについて考える必要があるときのために返している。
269
+
183
270
  ## プライバシー
184
271
 
185
272
  本サーバーは通信を一切行いません(stdioのみ)。撮影した画像・下書きはディスクの一時ディレクトリへ
@@ -0,0 +1,278 @@
1
+ // エージェント向けの注釈入力スキーマと、その中の「文字アンカー」({text:"...", occurrence?}での
2
+ // 座標・範囲指定)を画像ピクセル座標へ解決する処理。
3
+ //
4
+ // 契約: ここでの型(AgentAnnotationInput)はopen_draftツールの入力そのもの。座標は
5
+ // - {x,y} / [x,y](従来どおり。数値・画像ピクセル・左上原点)
6
+ // - {text, occurrence?}(文字アンカー。Vision文字認識で対象文字列の位置を調べて自動算出する)
7
+ // のいずれかで指定できる。resolveAnnotationAnchors() で全ての文字アンカーを解決し、
8
+ // 従来の数値座標だけのAnnotationInput(src/annotations.ts)へ変換してからconvertAnnotationへ渡す
9
+ // ——つまりannotations.ts側(座標変換・AnnotationArchive化)は一切変更していない。
10
+ import { z } from "zod";
11
+ import { swiftTextRecognizer } from "./vision.js";
12
+ import { pickTextAnchor, boxCenter, dockPointOnBox, defaultFromPoint, defaultLabelPoint, expandBoxWithMargin, arrowFootprintBox, estimateTextFootprintSize, numberStampFootprintBox, NUMBER_STAMP_DIAMETER_PX, DEFAULT_REGION_MARGIN_PX, } from "./textAnchor.js";
13
+ // --- 入力スキーマ -----------------------------------------------------------------------
14
+ const NumericPointSchema = z
15
+ .union([z.object({ x: z.number(), y: z.number() }), z.tuple([z.number(), z.number()])])
16
+ .transform((v) => (Array.isArray(v) ? { x: v[0], y: v[1] } : v));
17
+ const TextAnchorPointSchema = z.object({
18
+ text: z.string().min(1).describe("対象の文字列(部分一致可)。Vision文字認識で画像内の位置を調べ、座標を自動算出する"),
19
+ occurrence: z
20
+ .number()
21
+ .int()
22
+ .positive()
23
+ .optional()
24
+ .describe("同じ文字列が画像内に複数あるときの何番目か(1始まり)。省略時に複数一致したらisErrorで候補を返す"),
25
+ });
26
+ /** 点座標: {x,y} / [x,y] / {text,occurrence?} のいずれか。 */
27
+ const AgentPointSchema = z.union([NumericPointSchema, TextAnchorPointSchema]);
28
+ function isTextAnchorPoint(p) {
29
+ return typeof p.text === "string";
30
+ }
31
+ const COLOR_DESC = '色名(red/blue/green/yellow/orange/purple/pink/black/white/gray)か "#rrggbb"。省略時は赤';
32
+ export const AgentArrowInputSchema = z.object({
33
+ type: z.literal("arrow"),
34
+ from: AgentPointSchema.optional().describe("矢印の付け根。省略するとtoの外側の空いていそうな位置に自動で置く(画像の端には出さない)。" +
35
+ "{x,y}・[x,y]・{text,occurrence?}のいずれか"),
36
+ to: AgentPointSchema.describe("矢印の先(矢頭)。{text,...}で指定すると、その文字を矢頭が隠さないよう、文字の矢印の根元側に" +
37
+ "近い端を少し外側から指す位置へ自動配置する。{x,y}・[x,y]・{text,occurrence?}のいずれか"),
38
+ color: z.string().optional().describe(COLOR_DESC),
39
+ });
40
+ export const AgentTextInputSchema = z.object({
41
+ type: z.literal("text"),
42
+ at: AgentPointSchema.describe("文字ブロック左上。{text,...}で指定すると対象に重ねず、対象の近く(既定で右側)の空いていそうな" +
43
+ "位置に自動で置く。{x,y}・[x,y]・{text,occurrence?}のいずれか"),
44
+ text: z.string().min(1),
45
+ width: z.number().positive().optional().describe("折り返し幅(画像ピクセル)。省略時は折り返さない"),
46
+ color: z.string().optional(),
47
+ });
48
+ export const AgentNumberInputSchema = z.object({
49
+ type: z.literal("number"),
50
+ at: AgentPointSchema.describe("番号スタンプの中心。{text,...}で指定すると対象の近く(既定で右側)の空いていそうな位置に自動で置く。" +
51
+ "{x,y}・[x,y]・{text,occurrence?}のいずれか"),
52
+ n: z.number().int().positive(),
53
+ color: z.string().optional(),
54
+ });
55
+ function regionFields(kindLabel) {
56
+ return {
57
+ x: z.number().optional().describe(`${kindLabel}の左上X(画像ピクセル)。text指定時は省略する`),
58
+ y: z.number().optional().describe(`${kindLabel}の左上Y(画像ピクセル)。text指定時は省略する`),
59
+ w: z.number().positive().optional().describe(`${kindLabel}の幅(画像ピクセル)。text指定時は省略する`),
60
+ h: z.number().positive().optional().describe(`${kindLabel}の高さ(画像ピクセル)。text指定時は省略する`),
61
+ text: z
62
+ .string()
63
+ .optional()
64
+ .describe(`x/y/w/hの代わりに対象の文字列を指定する(部分一致可)。Vision文字認識で枠を求め、marginの余白を` +
65
+ `足した範囲を${kindLabel}にする。x/y/w/hとは同時に指定しない`),
66
+ occurrence: z.number().int().positive().optional().describe("同じ文字列が複数あるときの何番目か(1始まり)"),
67
+ margin: z
68
+ .number()
69
+ .nonnegative()
70
+ .optional()
71
+ .describe(`text指定時の余白(画像ピクセル)。省略時は既定値(${DEFAULT_REGION_MARGIN_PX}px)`),
72
+ };
73
+ }
74
+ export const AgentRectInputSchema = z.object({ type: z.literal("rect"), ...regionFields("矩形"), color: z.string().optional() });
75
+ export const AgentEllipseInputSchema = z.object({ type: z.literal("ellipse"), ...regionFields("楕円"), color: z.string().optional() });
76
+ export const AgentMosaicInputSchema = z.object({ type: z.literal("mosaic"), ...regionFields("モザイク範囲") });
77
+ export const AgentAnnotationInputSchema = z.discriminatedUnion("type", [
78
+ AgentArrowInputSchema,
79
+ AgentTextInputSchema,
80
+ AgentRectInputSchema,
81
+ AgentEllipseInputSchema,
82
+ AgentMosaicInputSchema,
83
+ AgentNumberInputSchema,
84
+ ]);
85
+ function resolveTextAnchorBox(anchor, lines, imagePath, recognizer) {
86
+ const pick = pickTextAnchor(lines, anchor.text, anchor.occurrence);
87
+ if (pick.status !== "resolved") {
88
+ return { issue: { status: pick.status, candidates: pick.candidates } };
89
+ }
90
+ const m = pick.match;
91
+ const box = m.exact ? m.line.box : recognizer.boxForTextRange(imagePath, m.lineIndex, m.utf16Start, m.utf16Length);
92
+ return { box };
93
+ }
94
+ /**
95
+ * 自動配置(arrow.fromの省略時・text.at/number.atの文字アンカー時)が重なりを避ける対象の領域。
96
+ * 「座標で明示された注釈 → 文字アンカーの範囲もの(rect等) → 矢印 → text/numberの自動配置」の順に
97
+ * 解決し、先に確定したものの領域をここへ積んで、後から解決するものがそれを避けられるようにする。
98
+ */
99
+ function classifyBucket(input) {
100
+ switch (input.type) {
101
+ case "rect":
102
+ case "ellipse":
103
+ case "mosaic":
104
+ return input.text !== undefined ? 2 : 1;
105
+ case "arrow": {
106
+ const toIsAnchor = isTextAnchorPoint(input.to);
107
+ const fromIsAnchor = input.from !== undefined && isTextAnchorPoint(input.from);
108
+ const fromOmitted = input.from === undefined;
109
+ return toIsAnchor || fromIsAnchor || fromOmitted ? 3 : 1;
110
+ }
111
+ case "text":
112
+ case "number":
113
+ return isTextAnchorPoint(input.at) ? 4 : 1;
114
+ }
115
+ }
116
+ /**
117
+ * annotations中の文字アンカーをすべて画像ピクセル座標へ解決し、従来のAnnotationInput[]へ変換する。
118
+ * 1件でも解決できなければ(no_match/ambiguous)下書きは作らず、issuesにすべての問題を集めて返す
119
+ * (エージェントが一度の呼び直しで複数の指定ミスに対応できるように)。
120
+ *
121
+ * 自動配置(arrow.from省略時・text.at/number.atの文字アンカー時)は、先に解決した他の注釈の領域と
122
+ * (Vision文字認識を使った場合は)認識できた他の文字列の枠を避けるよう、bucket 1→2→3→4の順に
123
+ * 処理する(classifyBucket参照)。出力順は入力の順序を保つ。
124
+ */
125
+ export function resolveAnnotationAnchors(imagePath, inputs, bounds, recognizer = swiftTextRecognizer, scale = 1) {
126
+ const issues = [];
127
+ let lines;
128
+ const getLines = () => {
129
+ if (!lines)
130
+ lines = recognizer.listTextLines(imagePath);
131
+ return lines;
132
+ };
133
+ // 自動配置の重なり回避に使う「認識できた他の文字列の枠」。Visionを一度も呼んでいなければ
134
+ // (=この下書きに文字アンカーが1つも無ければ)空のまま——自動配置のためだけに新たにOCRを
135
+ // 走らせて速度NFRを損なうことはしない(他の理由で既に呼んでいればその結果を再利用する)。
136
+ const ocrLineBoxes = () => (lines ?? []).map((l) => l.box);
137
+ const resolvedByIndex = new Array(inputs.length);
138
+ const occupied = [];
139
+ const reportFailure = (annotationIndex, field, anchor, result) => {
140
+ issues.push({
141
+ annotationIndex,
142
+ field,
143
+ query: anchor.text,
144
+ occurrence: anchor.occurrence,
145
+ status: result.status,
146
+ candidates: result.candidates,
147
+ });
148
+ };
149
+ for (const bucket of [1, 2, 3, 4]) {
150
+ inputs.forEach((input, annotationIndex) => {
151
+ if (classifyBucket(input) !== bucket)
152
+ return;
153
+ switch (input.type) {
154
+ case "arrow": {
155
+ let toBox;
156
+ let toPoint;
157
+ if (isTextAnchorPoint(input.to)) {
158
+ const r = resolveTextAnchorBox(input.to, getLines(), imagePath, recognizer);
159
+ if ("issue" in r) {
160
+ reportFailure(annotationIndex, "arrow.to", input.to, r.issue);
161
+ return;
162
+ }
163
+ toBox = r.box;
164
+ }
165
+ else {
166
+ toPoint = input.to;
167
+ }
168
+ let fromPoint;
169
+ if (input.from !== undefined) {
170
+ if (isTextAnchorPoint(input.from)) {
171
+ const r = resolveTextAnchorBox(input.from, getLines(), imagePath, recognizer);
172
+ if ("issue" in r) {
173
+ reportFailure(annotationIndex, "arrow.from", input.from, r.issue);
174
+ return;
175
+ }
176
+ fromPoint = boxCenter(r.box);
177
+ }
178
+ else {
179
+ fromPoint = input.from;
180
+ }
181
+ }
182
+ const avoid = [...occupied, ...ocrLineBoxes()];
183
+ if (toBox) {
184
+ const reference = fromPoint ?? defaultFromPoint(toBox, bounds, avoid, scale);
185
+ toPoint = dockPointOnBox(toBox, reference, scale);
186
+ if (fromPoint === undefined)
187
+ fromPoint = reference;
188
+ }
189
+ if (fromPoint === undefined) {
190
+ // to・fromとも数値指定で、fromだけ省略: toの位置を基準にした小さな仮想の箱で自動配置する
191
+ const tp = toPoint;
192
+ fromPoint = defaultFromPoint({ x: tp.x - 1, y: tp.y - 1, w: 2, h: 2 }, bounds, avoid, scale);
193
+ }
194
+ resolvedByIndex[annotationIndex] = { type: "arrow", from: fromPoint, to: toPoint, color: input.color };
195
+ occupied.push(arrowFootprintBox(fromPoint, toPoint, scale));
196
+ return;
197
+ }
198
+ case "text": {
199
+ let at;
200
+ const footprintSize = estimateTextFootprintSize(input.text, input.width, scale);
201
+ if (isTextAnchorPoint(input.at)) {
202
+ const r = resolveTextAnchorBox(input.at, getLines(), imagePath, recognizer);
203
+ if ("issue" in r) {
204
+ reportFailure(annotationIndex, "text.at", input.at, r.issue);
205
+ return;
206
+ }
207
+ at = defaultLabelPoint(r.box, bounds, footprintSize, "topLeft", [...occupied, ...ocrLineBoxes()], scale);
208
+ }
209
+ else {
210
+ at = input.at;
211
+ }
212
+ resolvedByIndex[annotationIndex] = { type: "text", at, text: input.text, width: input.width, color: input.color };
213
+ occupied.push({ x: at.x, y: at.y, w: footprintSize.w, h: footprintSize.h });
214
+ return;
215
+ }
216
+ case "number": {
217
+ let at;
218
+ const stampSize = NUMBER_STAMP_DIAMETER_PX * scale;
219
+ if (isTextAnchorPoint(input.at)) {
220
+ const r = resolveTextAnchorBox(input.at, getLines(), imagePath, recognizer);
221
+ if ("issue" in r) {
222
+ reportFailure(annotationIndex, "number.at", input.at, r.issue);
223
+ return;
224
+ }
225
+ at = defaultLabelPoint(r.box, bounds, { w: stampSize, h: stampSize }, "center", [...occupied, ...ocrLineBoxes()], scale);
226
+ }
227
+ else {
228
+ at = input.at;
229
+ }
230
+ resolvedByIndex[annotationIndex] = { type: "number", at, n: input.n, color: input.color };
231
+ occupied.push(numberStampFootprintBox(at, scale));
232
+ return;
233
+ }
234
+ case "rect":
235
+ case "ellipse":
236
+ case "mosaic": {
237
+ const hasText = input.text !== undefined;
238
+ const hasCoords = input.x !== undefined && input.y !== undefined && input.w !== undefined && input.h !== undefined;
239
+ if (hasText === hasCoords) {
240
+ issues.push({
241
+ annotationIndex,
242
+ field: `${input.type}.region`,
243
+ query: input.text ?? "",
244
+ status: "no_match",
245
+ candidates: [],
246
+ });
247
+ return;
248
+ }
249
+ let region;
250
+ if (hasText) {
251
+ const anchor = { text: input.text, occurrence: input.occurrence };
252
+ const r = resolveTextAnchorBox(anchor, getLines(), imagePath, recognizer);
253
+ if ("issue" in r) {
254
+ reportFailure(annotationIndex, `${input.type}.region`, anchor, r.issue);
255
+ return;
256
+ }
257
+ region = expandBoxWithMargin(r.box, input.margin ?? DEFAULT_REGION_MARGIN_PX);
258
+ }
259
+ else {
260
+ region = { x: input.x, y: input.y, w: input.w, h: input.h };
261
+ }
262
+ if (input.type === "mosaic") {
263
+ resolvedByIndex[annotationIndex] = { type: "mosaic", ...region };
264
+ }
265
+ else {
266
+ resolvedByIndex[annotationIndex] = { type: input.type, ...region, color: input.color };
267
+ }
268
+ occupied.push(region);
269
+ return;
270
+ }
271
+ }
272
+ });
273
+ }
274
+ if (issues.length > 0)
275
+ return { ok: false, issues };
276
+ return { ok: true, annotations: resolvedByIndex };
277
+ }
278
+ //# sourceMappingURL=agentAnnotations.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"agentAnnotations.js","sourceRoot":"","sources":["../../src/agentAnnotations.ts"],"names":[],"mappings":"AAAA,8DAA8D;AAC9D,4BAA4B;AAC5B,EAAE;AACF,2DAA2D;AAC3D,yCAAyC;AACzC,+DAA+D;AAC/D,yDAAyD;AACzD,2EAA2E;AAC3E,2DAA2D;AAE3D,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAC;AAGxB,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EACL,cAAc,EACd,SAAS,EACT,cAAc,EACd,gBAAgB,EAChB,iBAAiB,EACjB,mBAAmB,EACnB,iBAAiB,EACjB,yBAAyB,EACzB,uBAAuB,EACvB,wBAAwB,EACxB,wBAAwB,GAIzB,MAAM,iBAAiB,CAAC;AAEzB,qFAAqF;AAErF,MAAM,kBAAkB,GAAG,CAAC;KACzB,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,CAAC,CAAC,CAAC;KACtF,SAAS,CAAC,CAAC,CAAC,EAA4B,EAAE,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;AAE7F,MAAM,qBAAqB,GAAG,CAAC,CAAC,MAAM,CAAC;IACrC,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,QAAQ,CAAC,8CAA8C,CAAC;IAChF,UAAU,EAAE,CAAC;SACV,MAAM,EAAE;SACR,GAAG,EAAE;SACL,QAAQ,EAAE;SACV,QAAQ,EAAE;SACV,QAAQ,CAAC,sDAAsD,CAAC;CACpE,CAAC,CAAC;AAEH,qDAAqD;AACrD,MAAM,gBAAgB,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,kBAAkB,EAAE,qBAAqB,CAAC,CAAC,CAAC;AAG9E,SAAS,iBAAiB,CAAC,CAAa;IACtC,OAAO,OAAQ,CAAwB,CAAC,IAAI,KAAK,QAAQ,CAAC;AAC5D,CAAC;AAED,MAAM,UAAU,GAAG,gFAAgF,CAAC;AAEpG,MAAM,CAAC,MAAM,qBAAqB,GAAG,CAAC,CAAC,MAAM,CAAC;IAC5C,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC;IACxB,IAAI,EAAE,gBAAgB,CAAC,QAAQ,EAAE,CAAC,QAAQ,CACxC,gDAAgD;QAC9C,qCAAqC,CACxC;IACD,EAAE,EAAE,gBAAgB,CAAC,QAAQ,CAC3B,qDAAqD;QACnD,2DAA2D,CAC9D;IACD,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,UAAU,CAAC;CAClD,CAAC,CAAC;AAEH,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,MAAM,CAAC;IAC3C,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC;IACvB,EAAE,EAAE,gBAAgB,CAAC,QAAQ,CAC3B,sDAAsD;QACpD,8CAA8C,CACjD;IACD,IAAI,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,CAAC,CAAC,CAAC;IACvB,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,0BAA0B,CAAC;IAC5E,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;CAC7B,CAAC,CAAC;AAEH,MAAM,CAAC,MAAM,sBAAsB,GAAG,CAAC,CAAC,MAAM,CAAC;IAC7C,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,QAAQ,CAAC;IACzB,EAAE,EAAE,gBAAgB,CAAC,QAAQ,CAC3B,yDAAyD;QACvD,qCAAqC,CACxC;IACD,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE;IAC9B,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE;CAC7B,CAAC,CAAC;AAEH,SAAS,YAAY,CAAC,SAAiB;IACrC,OAAO;QACL,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,GAAG,SAAS,2BAA2B,CAAC;QAC1E,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,GAAG,SAAS,2BAA2B,CAAC;QAC1E,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,GAAG,SAAS,yBAAyB,CAAC;QACnF,CAAC,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,GAAG,SAAS,0BAA0B,CAAC;QACpF,IAAI,EAAE,CAAC;aACJ,MAAM,EAAE;aACR,QAAQ,EAAE;aACV,QAAQ,CACP,2DAA2D;YACzD,SAAS,SAAS,uBAAuB,CAC5C;QACH,UAAU,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,GAAG,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,EAAE,CAAC,QAAQ,CAAC,yBAAyB,CAAC;QACtF,MAAM,EAAE,CAAC;aACN,MAAM,EAAE;aACR,WAAW,EAAE;aACb,QAAQ,EAAE;aACV,QAAQ,CAAC,8BAA8B,wBAAwB,KAAK,CAAC;KACzE,CAAC;AACJ,CAAC;AAED,MAAM,CAAC,MAAM,oBAAoB,GAAG,CAAC,CAAC,MAAM,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,MAAM,CAAC,EAAE,GAAG,YAAY,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;AAC/H,MAAM,CAAC,MAAM,uBAAuB,GAAG,CAAC,CAAC,MAAM,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,SAAS,CAAC,EAAE,GAAG,YAAY,CAAC,IAAI,CAAC,EAAE,KAAK,EAAE,CAAC,CAAC,MAAM,EAAE,CAAC,QAAQ,EAAE,EAAE,CAAC,CAAC;AACrI,MAAM,CAAC,MAAM,sBAAsB,GAAG,CAAC,CAAC,MAAM,CAAC,EAAE,IAAI,EAAE,CAAC,CAAC,OAAO,CAAC,QAAQ,CAAC,EAAE,GAAG,YAAY,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC;AAEzG,MAAM,CAAC,MAAM,0BAA0B,GAAG,CAAC,CAAC,kBAAkB,CAAC,MAAM,EAAE;IACrE,qBAAqB;IACrB,oBAAoB;IACpB,oBAAoB;IACpB,uBAAuB;IACvB,sBAAsB;IACtB,sBAAsB;CACvB,CAAC,CAAC;AAuBH,SAAS,oBAAoB,CAC3B,MAA6C,EAC7C,KAAiB,EACjB,SAAiB,EACjB,UAA0B;IAE1B,MAAM,IAAI,GAAG,cAAc,CAAC,KAAK,EAAE,MAAM,CAAC,IAAI,EAAE,MAAM,CAAC,UAAU,CAAC,CAAC;IACnE,IAAI,IAAI,CAAC,MAAM,KAAK,UAAU,EAAE,CAAC;QAC/B,OAAO,EAAE,KAAK,EAAE,EAAE,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,UAAU,EAAE,IAAI,CAAC,UAAU,EAAE,EAAE,CAAC;IACzE,CAAC;IACD,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC;IACrB,MAAM,GAAG,GAAG,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,CAAC,UAAU,CAAC,eAAe,CAAC,SAAS,EAAE,CAAC,CAAC,SAAS,EAAE,CAAC,CAAC,UAAU,EAAE,CAAC,CAAC,WAAW,CAAC,CAAC;IACnH,OAAO,EAAE,GAAG,EAAE,CAAC;AACjB,CAAC;AAED;;;;GAIG;AACH,SAAS,cAAc,CAAC,KAA2B;IACjD,QAAQ,KAAK,CAAC,IAAI,EAAE,CAAC;QACnB,KAAK,MAAM,CAAC;QACZ,KAAK,SAAS,CAAC;QACf,KAAK,QAAQ;YACX,OAAO,KAAK,CAAC,IAAI,KAAK,SAAS,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC1C,KAAK,OAAO,CAAC,CAAC,CAAC;YACb,MAAM,UAAU,GAAG,iBAAiB,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC;YAC/C,MAAM,YAAY,GAAG,KAAK,CAAC,IAAI,KAAK,SAAS,IAAI,iBAAiB,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;YAC/E,MAAM,WAAW,GAAG,KAAK,CAAC,IAAI,KAAK,SAAS,CAAC;YAC7C,OAAO,UAAU,IAAI,YAAY,IAAI,WAAW,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QAC3D,CAAC;QACD,KAAK,MAAM,CAAC;QACZ,KAAK,QAAQ;YACX,OAAO,iBAAiB,CAAC,KAAK,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IAC/C,CAAC;AACH,CAAC;AAED;;;;;;;;GAQG;AACH,MAAM,UAAU,wBAAwB,CACtC,SAAiB,EACjB,MAA8B,EAC9B,MAAmB,EACnB,aAA6B,mBAAmB,EAChD,KAAK,GAAG,CAAC;IAET,MAAM,MAAM,GAAkB,EAAE,CAAC;IACjC,IAAI,KAA6B,CAAC;IAClC,MAAM,QAAQ,GAAG,GAAe,EAAE;QAChC,IAAI,CAAC,KAAK;YAAE,KAAK,GAAG,UAAU,CAAC,aAAa,CAAC,SAAS,CAAC,CAAC;QACxD,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;IACF,iDAAiD;IACjD,iDAAiD;IACjD,gDAAgD;IAChD,MAAM,YAAY,GAAG,GAAe,EAAE,CAAC,CAAC,KAAK,IAAI,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;IAEvE,MAAM,eAAe,GAAG,IAAI,KAAK,CAA8B,MAAM,CAAC,MAAM,CAAC,CAAC;IAC9E,MAAM,QAAQ,GAAe,EAAE,CAAC;IAEhC,MAAM,aAAa,GAAG,CACpB,eAAuB,EACvB,KAAa,EACb,MAA6C,EAC7C,MAA+E,EAC/E,EAAE;QACF,MAAM,CAAC,IAAI,CAAC;YACV,eAAe;YACf,KAAK;YACL,KAAK,EAAE,MAAM,CAAC,IAAI;YAClB,UAAU,EAAE,MAAM,CAAC,UAAU;YAC7B,MAAM,EAAE,MAAM,CAAC,MAAM;YACrB,UAAU,EAAE,MAAM,CAAC,UAAU;SAC9B,CAAC,CAAC;IACL,CAAC,CAAC;IAEF,KAAK,MAAM,MAAM,IAAI,CAAC,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,CAAU,EAAE,CAAC;QAC3C,MAAM,CAAC,OAAO,CAAC,CAAC,KAAK,EAAE,eAAe,EAAE,EAAE;YACxC,IAAI,cAAc,CAAC,KAAK,CAAC,KAAK,MAAM;gBAAE,OAAO;YAE7C,QAAQ,KAAK,CAAC,IAAI,EAAE,CAAC;gBACnB,KAAK,OAAO,CAAC,CAAC,CAAC;oBACb,IAAI,KAA2B,CAAC;oBAChC,IAAI,OAA0B,CAAC;oBAC/B,IAAI,iBAAiB,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;wBAChC,MAAM,CAAC,GAAG,oBAAoB,CAAC,KAAK,CAAC,EAAE,EAAE,QAAQ,EAAE,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;wBAC5E,IAAI,OAAO,IAAI,CAAC,EAAE,CAAC;4BACjB,aAAa,CAAC,eAAe,EAAE,UAAU,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;4BAC9D,OAAO;wBACT,CAAC;wBACD,KAAK,GAAG,CAAC,CAAC,GAAG,CAAC;oBAChB,CAAC;yBAAM,CAAC;wBACN,OAAO,GAAG,KAAK,CAAC,EAAE,CAAC;oBACrB,CAAC;oBAED,IAAI,SAA4B,CAAC;oBACjC,IAAI,KAAK,CAAC,IAAI,KAAK,SAAS,EAAE,CAAC;wBAC7B,IAAI,iBAAiB,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;4BAClC,MAAM,CAAC,GAAG,oBAAoB,CAAC,KAAK,CAAC,IAAI,EAAE,QAAQ,EAAE,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;4BAC9E,IAAI,OAAO,IAAI,CAAC,EAAE,CAAC;gCACjB,aAAa,CAAC,eAAe,EAAE,YAAY,EAAE,KAAK,CAAC,IAAI,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;gCAClE,OAAO;4BACT,CAAC;4BACD,SAAS,GAAG,SAAS,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC;wBAC/B,CAAC;6BAAM,CAAC;4BACN,SAAS,GAAG,KAAK,CAAC,IAAI,CAAC;wBACzB,CAAC;oBACH,CAAC;oBAED,MAAM,KAAK,GAAG,CAAC,GAAG,QAAQ,EAAE,GAAG,YAAY,EAAE,CAAC,CAAC;oBAC/C,IAAI,KAAK,EAAE,CAAC;wBACV,MAAM,SAAS,GAAG,SAAS,IAAI,gBAAgB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;wBAC7E,OAAO,GAAG,cAAc,CAAC,KAAK,EAAE,SAAS,EAAE,KAAK,CAAC,CAAC;wBAClD,IAAI,SAAS,KAAK,SAAS;4BAAE,SAAS,GAAG,SAAS,CAAC;oBACrD,CAAC;oBACD,IAAI,SAAS,KAAK,SAAS,EAAE,CAAC;wBAC5B,qDAAqD;wBACrD,MAAM,EAAE,GAAG,OAAgB,CAAC;wBAC5B,SAAS,GAAG,gBAAgB,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,GAAG,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,CAAC,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,CAAC;oBAC/F,CAAC;oBACD,eAAe,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,EAAE,OAAO,EAAE,IAAI,EAAE,SAAS,EAAE,EAAE,EAAE,OAAgB,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC;oBAChH,QAAQ,CAAC,IAAI,CAAC,iBAAiB,CAAC,SAAS,EAAE,OAAgB,EAAE,KAAK,CAAC,CAAC,CAAC;oBACrE,OAAO;gBACT,CAAC;gBAED,KAAK,MAAM,CAAC,CAAC,CAAC;oBACZ,IAAI,EAAS,CAAC;oBACd,MAAM,aAAa,GAAG,yBAAyB,CAAC,KAAK,CAAC,IAAI,EAAE,KAAK,CAAC,KAAK,EAAE,KAAK,CAAC,CAAC;oBAChF,IAAI,iBAAiB,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;wBAChC,MAAM,CAAC,GAAG,oBAAoB,CAAC,KAAK,CAAC,EAAE,EAAE,QAAQ,EAAE,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;wBAC5E,IAAI,OAAO,IAAI,CAAC,EAAE,CAAC;4BACjB,aAAa,CAAC,eAAe,EAAE,SAAS,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;4BAC7D,OAAO;wBACT,CAAC;wBACD,EAAE,GAAG,iBAAiB,CAAC,CAAC,CAAC,GAAG,EAAE,MAAM,EAAE,aAAa,EAAE,SAAS,EAAE,CAAC,GAAG,QAAQ,EAAE,GAAG,YAAY,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC;oBAC3G,CAAC;yBAAM,CAAC;wBACN,EAAE,GAAG,KAAK,CAAC,EAAE,CAAC;oBAChB,CAAC;oBACD,eAAe,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC;oBAClH,QAAQ,CAAC,IAAI,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,EAAE,aAAa,CAAC,CAAC,EAAE,CAAC,EAAE,aAAa,CAAC,CAAC,EAAE,CAAC,CAAC;oBAC5E,OAAO;gBACT,CAAC;gBAED,KAAK,QAAQ,CAAC,CAAC,CAAC;oBACd,IAAI,EAAS,CAAC;oBACd,MAAM,SAAS,GAAG,wBAAwB,GAAG,KAAK,CAAC;oBACnD,IAAI,iBAAiB,CAAC,KAAK,CAAC,EAAE,CAAC,EAAE,CAAC;wBAChC,MAAM,CAAC,GAAG,oBAAoB,CAAC,KAAK,CAAC,EAAE,EAAE,QAAQ,EAAE,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;wBAC5E,IAAI,OAAO,IAAI,CAAC,EAAE,CAAC;4BACjB,aAAa,CAAC,eAAe,EAAE,WAAW,EAAE,KAAK,CAAC,EAAE,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;4BAC/D,OAAO;wBACT,CAAC;wBACD,EAAE,GAAG,iBAAiB,CACpB,CAAC,CAAC,GAAG,EACL,MAAM,EACN,EAAE,CAAC,EAAE,SAAS,EAAE,CAAC,EAAE,SAAS,EAAE,EAC9B,QAAQ,EACR,CAAC,GAAG,QAAQ,EAAE,GAAG,YAAY,EAAE,CAAC,EAChC,KAAK,CACN,CAAC;oBACJ,CAAC;yBAAM,CAAC;wBACN,EAAE,GAAG,KAAK,CAAC,EAAE,CAAC;oBAChB,CAAC;oBACD,eAAe,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,EAAE,KAAK,CAAC,CAAC,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC;oBAC1F,QAAQ,CAAC,IAAI,CAAC,uBAAuB,CAAC,EAAE,EAAE,KAAK,CAAC,CAAC,CAAC;oBAClD,OAAO;gBACT,CAAC;gBAED,KAAK,MAAM,CAAC;gBACZ,KAAK,SAAS,CAAC;gBACf,KAAK,QAAQ,CAAC,CAAC,CAAC;oBACd,MAAM,OAAO,GAAG,KAAK,CAAC,IAAI,KAAK,SAAS,CAAC;oBACzC,MAAM,SAAS,GAAG,KAAK,CAAC,CAAC,KAAK,SAAS,IAAI,KAAK,CAAC,CAAC,KAAK,SAAS,IAAI,KAAK,CAAC,CAAC,KAAK,SAAS,IAAI,KAAK,CAAC,CAAC,KAAK,SAAS,CAAC;oBACnH,IAAI,OAAO,KAAK,SAAS,EAAE,CAAC;wBAC1B,MAAM,CAAC,IAAI,CAAC;4BACV,eAAe;4BACf,KAAK,EAAE,GAAG,KAAK,CAAC,IAAI,SAAS;4BAC7B,KAAK,EAAE,KAAK,CAAC,IAAI,IAAI,EAAE;4BACvB,MAAM,EAAE,UAAU;4BAClB,UAAU,EAAE,EAAE;yBACf,CAAC,CAAC;wBACH,OAAO;oBACT,CAAC;oBACD,IAAI,MAAsD,CAAC;oBAC3D,IAAI,OAAO,EAAE,CAAC;wBACZ,MAAM,MAAM,GAAG,EAAE,IAAI,EAAE,KAAK,CAAC,IAAc,EAAE,UAAU,EAAE,KAAK,CAAC,UAAU,EAAE,CAAC;wBAC5E,MAAM,CAAC,GAAG,oBAAoB,CAAC,MAAM,EAAE,QAAQ,EAAE,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;wBAC1E,IAAI,OAAO,IAAI,CAAC,EAAE,CAAC;4BACjB,aAAa,CAAC,eAAe,EAAE,GAAG,KAAK,CAAC,IAAI,SAAS,EAAE,MAAM,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC;4BACxE,OAAO;wBACT,CAAC;wBACD,MAAM,GAAG,mBAAmB,CAAC,CAAC,CAAC,GAAG,EAAE,KAAK,CAAC,MAAM,IAAI,wBAAwB,CAAC,CAAC;oBAChF,CAAC;yBAAM,CAAC;wBACN,MAAM,GAAG,EAAE,CAAC,EAAE,KAAK,CAAC,CAAW,EAAE,CAAC,EAAE,KAAK,CAAC,CAAW,EAAE,CAAC,EAAE,KAAK,CAAC,CAAW,EAAE,CAAC,EAAE,KAAK,CAAC,CAAW,EAAE,CAAC;oBACtG,CAAC;oBACD,IAAI,KAAK,CAAC,IAAI,KAAK,QAAQ,EAAE,CAAC;wBAC5B,eAAe,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,EAAE,QAAQ,EAAE,GAAG,MAAM,EAAE,CAAC;oBACnE,CAAC;yBAAM,CAAC;wBACN,eAAe,CAAC,eAAe,CAAC,GAAG,EAAE,IAAI,EAAE,KAAK,CAAC,IAAI,EAAE,GAAG,MAAM,EAAE,KAAK,EAAE,KAAK,CAAC,KAAK,EAAE,CAAC;oBACzF,CAAC;oBACD,QAAQ,CAAC,IAAI,CAAC,MAAM,CAAC,CAAC;oBACtB,OAAO;gBACT,CAAC;YACH,CAAC;QACH,CAAC,CAAC,CAAC;IACL,CAAC;IAED,IAAI,MAAM,CAAC,MAAM,GAAG,CAAC;QAAE,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,CAAC;IACpD,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,WAAW,EAAE,eAAoC,EAAE,CAAC;AACzE,CAAC"}
@@ -4,12 +4,47 @@
4
4
  // 参照: docs/plans/2026-11-12-FR-39-エージェント下書き.md 手順3の決め事「MCPのツール」1.capture、
5
5
  // および手順1「途中で分かったこと」(ウィンドウ単体で撮る・矢頭は文字側の端・メニューは遅延撮影+
6
6
  // 関係ない部分にモザイク)。
7
+ //
8
+ // 画面収録の許可エラー: `screencapture`は「画面収録」の許可(システム設定 > プライバシーと
9
+ // セキュリティ > 画面収録)が起動元プロセスに無いと、終了コード0以外で失敗する(macOS 10.15+の
10
+ // 既知の挙動。具体的なstderr文言はOS版により変わりうるため未検証・断定しない)。失敗時には
11
+ // CGPreflightScreenCaptureAccess()(プロンプトを出さず現在の許可状態だけを確認するAPI。
12
+ // windowList.tsのCGWindowListCopyWindowInfo同様、実機でswift経由の呼び出しが動くことを確認済み)
13
+ // で許可状態を追加確認し、許可が無いと分かっている場合だけ専用の案内文を返す。常時この確認を
14
+ // 行うと撮影の速度NFRを損なうため、失敗時(既にエラー状態で速度が問題にならない)だけ行う。
7
15
  import { spawnSync } from "node:child_process";
8
- import { mkdirSync, readFileSync, statSync } from "node:fs";
16
+ import { mkdirSync, mkdtempSync, readFileSync, statSync, writeFileSync } from "node:fs";
9
17
  import { tmpdir } from "node:os";
10
18
  import { join } from "node:path";
11
19
  import { parseChunks, readIhdr, readScale } from "./png.js";
12
20
  import { findWindows, listWindows } from "./windowList.js";
21
+ const SCREEN_RECORDING_CHECK_SWIFT = `
22
+ import CoreGraphics
23
+ print(CGPreflightScreenCaptureAccess() ? "granted" : "denied")
24
+ `;
25
+ /**
26
+ * CGPreflightScreenCaptureAccess()で画面収録の許可状態を確認する(プロンプトは出さない)。
27
+ * swiftが無い・判定に失敗した場合はundefined(判定不能)を返す。
28
+ */
29
+ function checkScreenRecordingAccess() {
30
+ try {
31
+ const dir = mkdtempSync(join(tmpdir(), "fletchshot-mcp-perm-"));
32
+ const scriptPath = join(dir, "check.swift");
33
+ writeFileSync(scriptPath, SCREEN_RECORDING_CHECK_SWIFT, "utf8");
34
+ const result = spawnSync("swift", [scriptPath], { encoding: "utf8", timeout: 10_000 });
35
+ if (result.status !== 0)
36
+ return undefined;
37
+ return result.stdout.trim() === "granted";
38
+ }
39
+ catch {
40
+ return undefined;
41
+ }
42
+ }
43
+ const SCREEN_RECORDING_HINT_CONFIRMED = "「画面収録」の許可(システム設定 > プライバシーとセキュリティ > 画面収録)が、このMCPサーバーを" +
44
+ "起動したプロセス(ターミナル・Claude Code等)に付与されていません。許可を追加し、起動元を" +
45
+ "再起動(ターミナルを終了して開き直す等)してから、もう一度実行してください。";
46
+ const SCREEN_RECORDING_HINT_UNKNOWN = "「画面収録」の許可(システム設定 > プライバシーとセキュリティ > 画面収録)がこのターミナル/" +
47
+ "プロセスに付与されているか確認してください。";
13
48
  export function captureDir() {
14
49
  const dir = join(tmpdir(), "fletchshot-mcp-captures");
15
50
  mkdirSync(dir, { recursive: true });
@@ -23,9 +58,9 @@ function newCapturePath() {
23
58
  function runScreencapture(args) {
24
59
  const result = spawnSync("screencapture", args, { encoding: "utf8", timeout: 30_000 });
25
60
  if (result.status !== 0) {
26
- throw new Error(`screencapture に失敗しました(終了コード ${result.status}): ${result.stderr || "(詳細なし)"}\n` +
27
- "「画面収録」の許可(システム設定 > プライバシーとセキュリティ)がこのターミナル/プロセスに" +
28
- "付与されているか確認してください。");
61
+ const granted = checkScreenRecordingAccess();
62
+ const hint = granted === false ? SCREEN_RECORDING_HINT_CONFIRMED : SCREEN_RECORDING_HINT_UNKNOWN;
63
+ throw new Error(`screencapture に失敗しました(終了コード ${result.status}): ${result.stderr || "(詳細なし)"}\n${hint}`);
29
64
  }
30
65
  }
31
66
  export function readCaptureMeta(path) {
@@ -1 +1 @@
1
- {"version":3,"file":"capture.js","sourceRoot":"","sources":["../../src/capture.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,2CAA2C;AAC3C,EAAE;AACF,0EAA0E;AAC1E,mDAAmD;AACnD,gBAAgB;AAEhB,OAAO,EAAE,SAAS,EAAE,MAAM,oBAAoB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,QAAQ,EAAE,MAAM,SAAS,CAAC;AAC5D,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AACjC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC5D,OAAO,EAAE,WAAW,EAAE,WAAW,EAAmB,MAAM,iBAAiB,CAAC;AAE5E,MAAM,UAAU,UAAU;IACxB,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,EAAE,EAAE,yBAAyB,CAAC,CAAC;IACtD,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACpC,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,cAAc;IACrB,MAAM,KAAK,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;IAC7D,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACpD,OAAO,IAAI,CAAC,UAAU,EAAE,EAAE,WAAW,KAAK,IAAI,IAAI,MAAM,CAAC,CAAC;AAC5D,CAAC;AAED,SAAS,gBAAgB,CAAC,IAAc;IACtC,MAAM,MAAM,GAAG,SAAS,CAAC,eAAe,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC,CAAC;IACvF,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,MAAM,IAAI,KAAK,CACb,+BAA+B,MAAM,CAAC,MAAM,MAAM,MAAM,CAAC,MAAM,IAAI,QAAQ,IAAI;YAC7E,iDAAiD;YACjD,mBAAmB,CACtB,CAAC;IACJ,CAAC;AACH,CAAC;AASD,MAAM,UAAU,eAAe,CAAC,IAAY;IAC1C,MAAM,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC/B,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;IAChC,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC;IAC3C,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;IAChC,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AAC3D,CAAC;AAOD,qBAAqB;AACrB,MAAM,UAAU,aAAa;IAC3B,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;IAC/B,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,kDAAkD;AAClD,MAAM,UAAU,oBAAoB,CAAC,GAAY,EAAE,KAAc;IAC/D,MAAM,GAAG,GAAG,WAAW,EAAE,CAAC;IAC1B,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,EAAE,GAAG,EAAE,KAAK,CAAC,CAAC;IAC7C,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,GAAG,EAAE,CAAC;IACjD,CAAC;IACD,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvB,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,OAAO,EAAE,CAAC;IACtD,CAAC;IACD,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC;IAClE,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,0DAA0D;AAC1D,MAAM,UAAU,aAAa,CAC3B,CAAS,EACT,CAAS,EACT,CAAS,EACT,CAAS,EACT,YAAoB;IAEpB,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,YAAY,CAAC,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC,CAAC;IACxF,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,4DAA4D;AAC5D,MAAM,UAAU,aAAa,CAAC,IAAY;IACxC,OAAO,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC;AAC7B,CAAC"}
1
+ {"version":3,"file":"capture.js","sourceRoot":"","sources":["../../src/capture.ts"],"names":[],"mappings":"AAAA,qDAAqD;AACrD,2CAA2C;AAC3C,EAAE;AACF,0EAA0E;AAC1E,mDAAmD;AACnD,gBAAgB;AAChB,EAAE;AACF,yDAAyD;AACzD,yDAAyD;AACzD,mDAAmD;AACnD,+DAA+D;AAC/D,wEAAwE;AACxE,gDAAgD;AAChD,iDAAiD;AAEjD,OAAO,EAAE,SAAS,EAAE,MAAM,oBAAoB,CAAC;AAC/C,OAAO,EAAE,SAAS,EAAE,WAAW,EAAE,YAAY,EAAE,QAAQ,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACxF,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AACjC,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,MAAM,UAAU,CAAC;AAC5D,OAAO,EAAE,WAAW,EAAE,WAAW,EAAmB,MAAM,iBAAiB,CAAC;AAE5E,MAAM,4BAA4B,GAAG;;;CAGpC,CAAC;AAEF;;;GAGG;AACH,SAAS,0BAA0B;IACjC,IAAI,CAAC;QACH,MAAM,GAAG,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,sBAAsB,CAAC,CAAC,CAAC;QAChE,MAAM,UAAU,GAAG,IAAI,CAAC,GAAG,EAAE,aAAa,CAAC,CAAC;QAC5C,aAAa,CAAC,UAAU,EAAE,4BAA4B,EAAE,MAAM,CAAC,CAAC;QAChE,MAAM,MAAM,GAAG,SAAS,CAAC,OAAO,EAAE,CAAC,UAAU,CAAC,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC,CAAC;QACvF,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,SAAS,CAAC;QAC1C,OAAO,MAAM,CAAC,MAAM,CAAC,IAAI,EAAE,KAAK,SAAS,CAAC;IAC5C,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,SAAS,CAAC;IACnB,CAAC;AACH,CAAC;AAED,MAAM,+BAA+B,GACnC,sDAAsD;IACtD,oDAAoD;IACpD,wCAAwC,CAAC;AAE3C,MAAM,6BAA6B,GACjC,mDAAmD;IACnD,wBAAwB,CAAC;AAE3B,MAAM,UAAU,UAAU;IACxB,MAAM,GAAG,GAAG,IAAI,CAAC,MAAM,EAAE,EAAE,yBAAyB,CAAC,CAAC;IACtD,SAAS,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,CAAC,CAAC;IACpC,OAAO,GAAG,CAAC;AACb,CAAC;AAED,SAAS,cAAc;IACrB,MAAM,KAAK,GAAG,IAAI,IAAI,EAAE,CAAC,WAAW,EAAE,CAAC,OAAO,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC;IAC7D,MAAM,IAAI,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC,QAAQ,CAAC,EAAE,CAAC,CAAC,KAAK,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC;IACpD,OAAO,IAAI,CAAC,UAAU,EAAE,EAAE,WAAW,KAAK,IAAI,IAAI,MAAM,CAAC,CAAC;AAC5D,CAAC;AAED,SAAS,gBAAgB,CAAC,IAAc;IACtC,MAAM,MAAM,GAAG,SAAS,CAAC,eAAe,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,CAAC,CAAC;IACvF,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,MAAM,OAAO,GAAG,0BAA0B,EAAE,CAAC;QAC7C,MAAM,IAAI,GAAG,OAAO,KAAK,KAAK,CAAC,CAAC,CAAC,+BAA+B,CAAC,CAAC,CAAC,6BAA6B,CAAC;QACjG,MAAM,IAAI,KAAK,CAAC,+BAA+B,MAAM,CAAC,MAAM,MAAM,MAAM,CAAC,MAAM,IAAI,QAAQ,KAAK,IAAI,EAAE,CAAC,CAAC;IAC1G,CAAC;AACH,CAAC;AASD,MAAM,UAAU,eAAe,CAAC,IAAY;IAC1C,MAAM,GAAG,GAAG,YAAY,CAAC,IAAI,CAAC,CAAC;IAC/B,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;IAChC,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC;IAC3C,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;IAChC,OAAO,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,CAAC;AAC3D,CAAC;AAOD,qBAAqB;AACrB,MAAM,UAAU,aAAa;IAC3B,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC,CAAC;IAC/B,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,kDAAkD;AAClD,MAAM,UAAU,oBAAoB,CAAC,GAAY,EAAE,KAAc;IAC/D,MAAM,GAAG,GAAG,WAAW,EAAE,CAAC;IAC1B,MAAM,OAAO,GAAG,WAAW,CAAC,GAAG,EAAE,GAAG,EAAE,KAAK,CAAC,CAAC;IAC7C,IAAI,OAAO,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACzB,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,UAAU,EAAE,GAAG,EAAE,CAAC;IACjD,CAAC;IACD,IAAI,OAAO,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;QACvB,OAAO,EAAE,MAAM,EAAE,WAAW,EAAE,UAAU,EAAE,OAAO,EAAE,CAAC;IACtD,CAAC;IACD,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,IAAI,CAAC,CAAC,CAAC;IAClE,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,0DAA0D;AAC1D,MAAM,UAAU,aAAa,CAC3B,CAAS,EACT,CAAS,EACT,CAAS,EACT,CAAS,EACT,YAAoB;IAEpB,MAAM,IAAI,GAAG,cAAc,EAAE,CAAC;IAC9B,gBAAgB,CAAC,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,CAAC,YAAY,CAAC,EAAE,IAAI,EAAE,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,EAAE,EAAE,IAAI,CAAC,CAAC,CAAC;IACxF,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,IAAI,EAAE,eAAe,CAAC,IAAI,CAAC,EAAE,CAAC;AAC7D,CAAC;AAED,4DAA4D;AAC5D,MAAM,UAAU,aAAa,CAAC,IAAY;IACxC,OAAO,QAAQ,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC;AAC7B,CAAC"}
package/dist/src/draft.js CHANGED
@@ -1,16 +1,23 @@
1
1
  // `open_draft` ツールの実装本体。
2
2
  //
3
- // 流れ(docs/plans/2026-11-12-FR-39-エージェント下書き.md「下書きの形式v1」「MCPのツール」2.open_draft):
4
- // 1. imagePathのPNGからpHYs(倍率)とIHDR(高さ)を読む
5
- // 2. エージェントの簡易な注釈記述を doc 座標へ変換し、AnnotationArchive 封筒 + 目印キーのJSONを作る
6
- // 3. ベース画像(注釈は焼かない)のバイト列へ、そのJSONをiTXt "Comment"として埋め込み、outputPathへ書く
7
- // 4. `open -b app.fletchshot.FletchShot <outputPath>` でFletchShotを開く(バンドルIDで呼ぶ。
3
+ // 流れ(docs/plans/2026-11-12-FR-39-エージェント下書き.md「下書きの形式v1」「MCPのツール」2.open_draft、
4
+ // および文字アンカー・プレビュー縮小 0.1.3での拡張):
5
+ // 1. imagePathのPNGからpHYs(倍率)とIHDR(高さ・幅)を読む
6
+ // 2. エージェントの注釈記述(文字アンカー{text,...}を含みうる)から、まず文字アンカーをすべて
7
+ // 画像ピクセル座標へ解決する(resolveAnnotationAnchors)。1件でも解決できなければ下書きは
8
+ // 作らず、問題一覧(issues)を返す。
9
+ // 3. 解決済みの注釈(従来のAnnotationInput)を doc 座標へ変換し、AnnotationArchive 封筒 +
10
+ // 目印キーのJSONを作る
11
+ // 4. ベース画像(注釈は焼かない)のバイト列へ、そのJSONをiTXt "Comment"として埋め込み、outputPathへ書く
12
+ // 5. `open -b app.fletchshot.FletchShot <outputPath>` でFletchShotを開く(バンドルIDで呼ぶ。
8
13
  // 名前解決だと古いビルドに届くことがあるため——手順1で実機確認済みの不具合)
9
14
  import { spawnSync } from "node:child_process";
10
15
  import { dirname, extname, join, basename } from "node:path";
11
16
  import { readFileSync, writeFileSync } from "node:fs";
12
17
  import { parseChunks, readIhdr, readScale, embedComment } from "./png.js";
13
18
  import { buildDraftCommentJson } from "./annotations.js";
19
+ import { resolveAnnotationAnchors } from "./agentAnnotations.js";
20
+ import { swiftTextRecognizer } from "./vision.js";
14
21
  export const FLETCHSHOT_BUNDLE_ID = "app.fletchshot.FletchShot";
15
22
  export function defaultDraftPath(imagePath) {
16
23
  const dir = dirname(imagePath);
@@ -25,14 +32,22 @@ export function readImageGeometry(imagePath) {
25
32
  const scale = readScale(chunks);
26
33
  return { geo: { scale, heightPx: height }, widthPx: width, buf };
27
34
  }
28
- /** 下書きPNGを作って書き出す(openは呼ばない。テストしやすいよう分離)。 */
29
- export function writeDraft(imagePath, annotations, outputPath) {
35
+ /**
36
+ * 下書きPNGを作って書き出す(openは呼ばない。テストしやすいよう分離)。
37
+ * recognizerは文字アンカー解決に使うVision呼び出しの差し替え口(テスト用。既定はswiftTextRecognizer)。
38
+ */
39
+ export function writeDraft(imagePath, annotations, outputPath, recognizer = swiftTextRecognizer) {
30
40
  const { geo, widthPx, buf } = readImageGeometry(imagePath);
31
- const commentJson = buildDraftCommentJson(annotations, geo);
41
+ const resolution = resolveAnnotationAnchors(imagePath, annotations, { width: widthPx, height: geo.heightPx }, recognizer, geo.scale);
42
+ if (!resolution.ok) {
43
+ return { status: "anchor_error", issues: resolution.issues };
44
+ }
45
+ const commentJson = buildDraftCommentJson(resolution.annotations, geo);
32
46
  const draftBytes = embedComment(buf, commentJson);
33
47
  const draftPath = outputPath ?? defaultDraftPath(imagePath);
34
48
  writeFileSync(draftPath, draftBytes);
35
49
  return {
50
+ status: "ok",
36
51
  draftPath,
37
52
  widthPx,
38
53
  heightPx: geo.heightPx,
@@ -1 +1 @@
1
- {"version":3,"file":"draft.js","sourceRoot":"","sources":["../../src/draft.ts"],"names":[],"mappings":"AAAA,yBAAyB;AACzB,EAAE;AACF,+EAA+E;AAC/E,0CAA0C;AAC1C,qEAAqE;AACrE,sEAAsE;AACtE,iFAAiF;AACjF,6CAA6C;AAE7C,OAAO,EAAE,SAAS,EAAE,MAAM,oBAAoB,CAAC;AAC/C,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAC7D,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACtD,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAC1E,OAAO,EAAE,qBAAqB,EAA4C,MAAM,kBAAkB,CAAC;AAEnG,MAAM,CAAC,MAAM,oBAAoB,GAAG,2BAA2B,CAAC;AAEhE,MAAM,UAAU,gBAAgB,CAAC,SAAiB;IAChD,MAAM,GAAG,GAAG,OAAO,CAAC,SAAS,CAAC,CAAC;IAC/B,MAAM,GAAG,GAAG,OAAO,CAAC,SAAS,CAAC,IAAI,MAAM,CAAC;IACzC,MAAM,IAAI,GAAG,QAAQ,CAAC,SAAS,EAAE,GAAG,CAAC,CAAC;IACtC,OAAO,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,SAAS,GAAG,EAAE,CAAC,CAAC;AAC1C,CAAC;AAUD,MAAM,UAAU,iBAAiB,CAC/B,SAAiB;IAEjB,MAAM,GAAG,GAAG,YAAY,CAAC,SAAS,CAAC,CAAC;IACpC,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;IAChC,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC;IAC3C,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;IAChC,OAAO,EAAE,GAAG,EAAE,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,EAAE,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC;AACnE,CAAC;AAED,6CAA6C;AAC7C,MAAM,UAAU,UAAU,CACxB,SAAiB,EACjB,WAA8B,EAC9B,UAAmB;IAEnB,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,GAAG,iBAAiB,CAAC,SAAS,CAAC,CAAC;IAC3D,MAAM,WAAW,GAAG,qBAAqB,CAAC,WAAW,EAAE,GAAG,CAAC,CAAC;IAC5D,MAAM,UAAU,GAAG,YAAY,CAAC,GAAG,EAAE,WAAW,CAAC,CAAC;IAClD,MAAM,SAAS,GAAG,UAAU,IAAI,gBAAgB,CAAC,SAAS,CAAC,CAAC;IAC5D,aAAa,CAAC,SAAS,EAAE,UAAU,CAAC,CAAC;IACrC,OAAO;QACL,SAAS;QACT,OAAO;QACP,QAAQ,EAAE,GAAG,CAAC,QAAQ;QACtB,KAAK,EAAE,GAAG,CAAC,KAAK;QAChB,eAAe,EAAE,WAAW,CAAC,MAAM;KACpC,CAAC;AACJ,CAAC;AAED,sEAAsE;AACtE,MAAM,UAAU,gBAAgB,CAAC,IAAY;IAC3C,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,oBAAoB,EAAE,IAAI,CAAC,EAAE;QACnE,QAAQ,EAAE,MAAM;QAChB,OAAO,EAAE,MAAM;KAChB,CAAC,CAAC;IACH,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,aAAa,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC;IAC9E,CAAC;IACD,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,CAAC;AACtB,CAAC"}
1
+ {"version":3,"file":"draft.js","sourceRoot":"","sources":["../../src/draft.ts"],"names":[],"mappings":"AAAA,yBAAyB;AACzB,EAAE;AACF,8EAA8E;AAC9E,gCAAgC;AAChC,4CAA4C;AAC5C,wDAAwD;AACxD,+DAA+D;AAC/D,2BAA2B;AAC3B,qEAAqE;AACrE,mBAAmB;AACnB,sEAAsE;AACtE,iFAAiF;AACjF,6CAA6C;AAE7C,OAAO,EAAE,SAAS,EAAE,MAAM,oBAAoB,CAAC;AAC/C,OAAO,EAAE,OAAO,EAAE,OAAO,EAAE,IAAI,EAAE,QAAQ,EAAE,MAAM,WAAW,CAAC;AAC7D,OAAO,EAAE,YAAY,EAAE,aAAa,EAAE,MAAM,SAAS,CAAC;AACtD,OAAO,EAAE,WAAW,EAAE,QAAQ,EAAE,SAAS,EAAE,YAAY,EAAE,MAAM,UAAU,CAAC;AAC1E,OAAO,EAAE,qBAAqB,EAAsB,MAAM,kBAAkB,CAAC;AAC7E,OAAO,EAAE,wBAAwB,EAA+C,MAAM,uBAAuB,CAAC;AAC9G,OAAO,EAAE,mBAAmB,EAAuB,MAAM,aAAa,CAAC;AAEvE,MAAM,CAAC,MAAM,oBAAoB,GAAG,2BAA2B,CAAC;AAEhE,MAAM,UAAU,gBAAgB,CAAC,SAAiB;IAChD,MAAM,GAAG,GAAG,OAAO,CAAC,SAAS,CAAC,CAAC;IAC/B,MAAM,GAAG,GAAG,OAAO,CAAC,SAAS,CAAC,IAAI,MAAM,CAAC;IACzC,MAAM,IAAI,GAAG,QAAQ,CAAC,SAAS,EAAE,GAAG,CAAC,CAAC;IACtC,OAAO,IAAI,CAAC,GAAG,EAAE,GAAG,IAAI,SAAS,GAAG,EAAE,CAAC,CAAC;AAC1C,CAAC;AAaD,MAAM,UAAU,iBAAiB,CAC/B,SAAiB;IAEjB,MAAM,GAAG,GAAG,YAAY,CAAC,SAAS,CAAC,CAAC;IACpC,MAAM,MAAM,GAAG,WAAW,CAAC,GAAG,CAAC,CAAC;IAChC,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,GAAG,QAAQ,CAAC,MAAM,CAAC,CAAC;IAC3C,MAAM,KAAK,GAAG,SAAS,CAAC,MAAM,CAAC,CAAC;IAChC,OAAO,EAAE,GAAG,EAAE,EAAE,KAAK,EAAE,QAAQ,EAAE,MAAM,EAAE,EAAE,OAAO,EAAE,KAAK,EAAE,GAAG,EAAE,CAAC;AACnE,CAAC;AAED;;;GAGG;AACH,MAAM,UAAU,UAAU,CACxB,SAAiB,EACjB,WAAmC,EACnC,UAAmB,EACnB,aAA6B,mBAAmB;IAEhD,MAAM,EAAE,GAAG,EAAE,OAAO,EAAE,GAAG,EAAE,GAAG,iBAAiB,CAAC,SAAS,CAAC,CAAC;IAC3D,MAAM,UAAU,GAAG,wBAAwB,CACzC,SAAS,EACT,WAAW,EACX,EAAE,KAAK,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,CAAC,QAAQ,EAAE,EACxC,UAAU,EACV,GAAG,CAAC,KAAK,CACV,CAAC;IACF,IAAI,CAAC,UAAU,CAAC,EAAE,EAAE,CAAC;QACnB,OAAO,EAAE,MAAM,EAAE,cAAc,EAAE,MAAM,EAAE,UAAU,CAAC,MAAM,EAAE,CAAC;IAC/D,CAAC;IACD,MAAM,WAAW,GAAG,qBAAqB,CAAC,UAAU,CAAC,WAAW,EAAE,GAAG,CAAC,CAAC;IACvE,MAAM,UAAU,GAAG,YAAY,CAAC,GAAG,EAAE,WAAW,CAAC,CAAC;IAClD,MAAM,SAAS,GAAG,UAAU,IAAI,gBAAgB,CAAC,SAAS,CAAC,CAAC;IAC5D,aAAa,CAAC,SAAS,EAAE,UAAU,CAAC,CAAC;IACrC,OAAO;QACL,MAAM,EAAE,IAAI;QACZ,SAAS;QACT,OAAO;QACP,QAAQ,EAAE,GAAG,CAAC,QAAQ;QACtB,KAAK,EAAE,GAAG,CAAC,KAAK;QAChB,eAAe,EAAE,WAAW,CAAC,MAAM;KACpC,CAAC;AACJ,CAAC;AAED,sEAAsE;AACtE,MAAM,UAAU,gBAAgB,CAAC,IAAY;IAC3C,MAAM,MAAM,GAAG,SAAS,CAAC,MAAM,EAAE,CAAC,IAAI,EAAE,oBAAoB,EAAE,IAAI,CAAC,EAAE;QACnE,QAAQ,EAAE,MAAM;QAChB,OAAO,EAAE,MAAM;KAChB,CAAC,CAAC;IACH,IAAI,MAAM,CAAC,MAAM,KAAK,CAAC,EAAE,CAAC;QACxB,OAAO,EAAE,EAAE,EAAE,KAAK,EAAE,MAAM,EAAE,MAAM,CAAC,MAAM,IAAI,aAAa,MAAM,CAAC,MAAM,EAAE,EAAE,CAAC;IAC9E,CAAC;IACD,OAAO,EAAE,EAAE,EAAE,IAAI,EAAE,CAAC;AACtB,CAAC"}