@xpert-ai/plugin-anydoc 0.0.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.xpertai-plugin/plugin.json +5 -0
- package/README.md +47 -0
- package/dist/_assets/LICENSE.anydoc +21 -0
- package/dist/_assets/README.md +8 -0
- package/dist/_assets/icon.svg +3 -0
- package/dist/index.d.ts +4 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +29 -0
- package/dist/lib/convert.d.ts +36 -0
- package/dist/lib/convert.d.ts.map +1 -0
- package/dist/lib/convert.js +241 -0
- package/dist/lib/plugin-anydoc.d.ts +2 -0
- package/dist/lib/plugin-anydoc.d.ts.map +1 -0
- package/dist/lib/plugin-anydoc.js +3 -0
- package/dist/lib/plugin.module.d.ts +3 -0
- package/dist/lib/plugin.module.d.ts.map +1 -0
- package/dist/lib/plugin.module.js +10 -0
- package/dist/lib/transformer.strategy.d.ts +38 -0
- package/dist/lib/transformer.strategy.d.ts.map +1 -0
- package/dist/lib/transformer.strategy.js +110 -0
- package/dist/lib/types.d.ts +15 -0
- package/dist/lib/types.d.ts.map +1 -0
- package/dist/lib/types.js +50 -0
- package/dist/sandbox-actions/convert/action.json +9 -0
- package/dist/sandbox-actions/convert/bundle/convert.mjs +80 -0
- package/dist/sandbox-actions/convert/bundle/pdf.mjs +66 -0
- package/dist/sandbox-actions/convert/bundle/result.mjs +71 -0
- package/dist/sandbox-actions/convert/bundle/runner.mjs +51 -0
- package/package.json +42 -0
package/README.md
ADDED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# AnyDoc
|
|
2
|
+
|
|
3
|
+
Knowledge-base document parser `@xpert-ai/plugin-anydoc`. Install as a **system plugin**; it registers the `anydoc` transformer and `anydoc.convert` Sandbox Action (1.0.3).
|
|
4
|
+
|
|
5
|
+
## Execution
|
|
6
|
+
|
|
7
|
+
- Uses platform Sandbox Jobs and scoped Workspace Files. API processes do not load native modules or start Java themselves.
|
|
8
|
+
- Runtime profile: `document/node-20/v1`. Conversion dependencies are installed, pinned and verified by the host Runtime Suite.
|
|
9
|
+
- No URL, token, executable-path or dependency-installation settings in the parser form.
|
|
10
|
+
- Input limit 100 MiB; output/decoded assets limit 128 MiB; 1,000 assets maximum; job timeout 300 seconds.
|
|
11
|
+
- Each conversion uses tenant/document/content/runtime-scoped identity and unique persisted asset paths. Cancelling processing cancels the same Job, including retries.
|
|
12
|
+
|
|
13
|
+
## Formats and limits
|
|
14
|
+
|
|
15
|
+
Supports `doc`, `docx`, `ppt`, `pptx`, `xls`, `xlsx`, `csv`, `odt`, `ods`, `odp`, `rtf`, `epub`, and PDF using `@firecrawl/anydoc@0.2.4`.
|
|
16
|
+
|
|
17
|
+
XLS/XLSX/CSV use **document/form interpretation**; row-record ingestion remains builtin. CSV must be UTF-8. Embedded raster images are exported separately for the existing image-understanding stage; unsupported embedded objects are retained as file assets. Their page/position is not invented.
|
|
18
|
+
|
|
19
|
+
Native PDF text is extracted locally. When AnyDoc reports scanned pages, the managed Runtime isolates pages and renders scans as PNG assets, preserving page order and coverage diagnostics. Enable knowledge-base image understanding and select a vision model to transcribe those pages. Without successful image understanding, scanned pages remain explicitly unrecognized. The renderer limits PDFs to 500 pages and page images to a 2,200-pixel longest edge. Firecrawl hosted OCR is never enabled; whether vision inference is local depends on the configured model.
|
|
20
|
+
|
|
21
|
+
Conversion failures are failed Jobs with bounded parser error codes. Retrying after recovery executes again; successful Job reuse includes the Runtime dependency fingerprint.
|
|
22
|
+
|
|
23
|
+
## Local development
|
|
24
|
+
|
|
25
|
+
The host must include this Runtime profile and support knowledge-document `fileScope`. Use the platform-managed installer from the host checkout with Node 20.20.2:
|
|
26
|
+
|
|
27
|
+
```sh
|
|
28
|
+
corepack pnpm --filter @xpert-ai/sandbox-runtime install:document-node
|
|
29
|
+
corepack pnpm --filter @xpert-ai/sandbox-runtime verify:local-document-node
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Local process execution is development/test only. Production needs a published `document-node` OCI image bound through the existing Runtime infrastructure; jobs use non-root, read-only filesystem and network isolation. Installing this plugin does not publish/bind a Runtime image.
|
|
33
|
+
|
|
34
|
+
From `xpertai/`:
|
|
35
|
+
|
|
36
|
+
```sh
|
|
37
|
+
corepack pnpm exec nx run @xpert-ai/plugin-anydoc:build
|
|
38
|
+
corepack pnpm exec nx run @xpert-ai/plugin-anydoc:test
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
From the repository root:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
node plugin-dev-harness/dist/index.js --workspace ./xpertai --plugin @xpert-ai/plugin-anydoc
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
The installable package includes `.xpertai-plugin/plugin.json`, which declares `dist/sandbox-actions/convert/action.json`, and its hashed bundle. Official artwork is bundled under `dist/_assets` and shared by the plugin card and parser metadata; displaying it does not fetch external URLs. After installing from this workspace into the local host, restart the API explicitly, then select AnyDoc under the relevant file type. Existing documents need reprocessing to use a new parser. Browser/knowledge-base indexing acceptance is separate from converter and lifecycle tests.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Sideguide Technologies Inc.
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Official branding asset
|
|
2
|
+
|
|
3
|
+
- Source: https://raw.githubusercontent.com/firecrawl/anydoc/261fc257d17c3eab0f673be31c408fd9fdc2171a/wasm/www/assets/logo.svg
|
|
4
|
+
- Retrieved: 2026-09-15
|
|
5
|
+
- SHA-256: `eaae76b7227c4b6a694c7575c0c665bd627e373b408b9d1806e0dc876e664289`
|
|
6
|
+
- Kept unmodified and bundled locally for plugin and parser identification.
|
|
7
|
+
|
|
8
|
+
The original project retains its branding rights.
|
|
@@ -0,0 +1,3 @@
|
|
|
1
|
+
<svg width="200" height="284" viewBox="0 0 200 284" fill="none" xmlns="http://www.w3.org/2000/svg">
|
|
2
|
+
<path d="M166.862 90.7716C155.812 94.0514 147.483 101.471 141.383 109.53C140.073 111.26 137.343 109.96 137.863 107.841C149.543 59.8136 134.113 19.896 86.0157 0.247269C83.5758 -0.752669 81.0359 1.43719 81.6759 3.99704C103.555 91.8416 11.5294 84.432 23.1588 184.016C23.3588 185.726 21.4389 186.896 20.039 185.896C15.6792 182.766 10.8095 176.236 7.46963 171.647C6.48968 170.297 4.36978 170.677 3.9198 172.287C1.25994 181.906 0 190.965 0 199.965C0 234.963 17.9891 265.771 45.2177 283.63C46.7777 284.65 48.7776 283.19 48.2476 281.4C46.8477 276.7 46.0577 271.74 45.9977 266.611C45.9977 263.461 46.1977 260.241 46.6877 257.241C47.8276 249.702 50.4475 242.522 54.8473 235.983C69.9365 213.334 100.185 191.455 95.3552 161.747C95.0453 159.867 97.2651 158.627 98.6651 159.917C119.974 179.386 124.194 205.575 120.694 229.063C120.394 231.103 122.954 232.193 124.244 230.593C127.504 226.513 131.483 222.933 135.813 220.244C136.893 219.574 138.333 220.084 138.743 221.284C141.153 228.293 144.733 234.873 148.113 241.452C152.152 249.362 154.302 258.391 153.962 267.951C153.792 272.6 153.022 277.1 151.732 281.38C151.182 283.19 153.162 284.7 154.752 283.66C182.001 265.801 200 234.993 200 199.975C200 187.806 197.87 175.876 193.84 164.697C185.391 141.248 163.952 123.64 169.372 93.0815C169.632 91.6216 168.282 90.3517 166.862 90.7716Z" fill="#FA5D19" style="fill:#FA5D19;fill:color(display-p3 0.9816 0.3634 0.0984);fill-opacity:1;"/>
|
|
3
|
+
</svg>
|
package/dist/index.d.ts
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,sBAAsB,CAAA;AAIvD,QAAA,MAAM,MAAM,EAAE,WAuBb,CAAA;AACD,eAAe,MAAM,CAAA"}
|
package/dist/index.js
ADDED
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
import { readFileSync } from 'node:fs';
|
|
2
|
+
import { AnyDocPluginModule } from './lib/plugin.module.js';
|
|
3
|
+
import { ConfigSchema, Icon } from './lib/types.js';
|
|
4
|
+
const pkg = JSON.parse(readFileSync(new URL('../package.json', import.meta.url), 'utf8'));
|
|
5
|
+
const plugin = {
|
|
6
|
+
meta: {
|
|
7
|
+
name: pkg.name,
|
|
8
|
+
version: pkg.version,
|
|
9
|
+
level: 'system',
|
|
10
|
+
artifactNamespace: 'anydoc',
|
|
11
|
+
category: 'integration',
|
|
12
|
+
displayName: 'AnyDoc',
|
|
13
|
+
description: 'Offline knowledge document parsing through platform-managed Sandbox Jobs.',
|
|
14
|
+
icon: Icon,
|
|
15
|
+
keywords: ['document', 'parser', 'pdf', 'anydoc'],
|
|
16
|
+
author: 'XpertAI Team'
|
|
17
|
+
},
|
|
18
|
+
config: { schema: ConfigSchema, formSchema: { type: 'object', properties: {} } },
|
|
19
|
+
register() {
|
|
20
|
+
return { module: AnyDocPluginModule, global: true };
|
|
21
|
+
},
|
|
22
|
+
onStart(ctx) {
|
|
23
|
+
ctx.logger.log('AnyDoc parser started');
|
|
24
|
+
},
|
|
25
|
+
onStop(ctx) {
|
|
26
|
+
ctx.logger.log('AnyDoc parser stopped');
|
|
27
|
+
}
|
|
28
|
+
};
|
|
29
|
+
export default plugin;
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
import { type AgentMiddlewareRuntimeCapabilityRegistry, type WorkspaceFileScope } from '@xpert-ai/plugin-sdk';
|
|
2
|
+
export type AnyDocFileScope = WorkspaceFileScope & {
|
|
3
|
+
organizationId?: string | null;
|
|
4
|
+
};
|
|
5
|
+
export declare class AnyDocSandboxConverter {
|
|
6
|
+
private readonly capabilities?;
|
|
7
|
+
constructor(capabilities?: Pick<AgentMiddlewareRuntimeCapabilityRegistry, 'get'>);
|
|
8
|
+
checkHealth(): Promise<import("@xpert-ai/plugin-sdk").SandboxJobActionHealth>;
|
|
9
|
+
convert(filePath: string, extension: string, options: {
|
|
10
|
+
fileScope?: AnyDocFileScope;
|
|
11
|
+
documentId?: string;
|
|
12
|
+
stage: 'test' | 'prod';
|
|
13
|
+
signal?: AbortSignal;
|
|
14
|
+
}): Promise<{
|
|
15
|
+
sandboxJobId: string;
|
|
16
|
+
runtimeProfile: string;
|
|
17
|
+
ok?: true;
|
|
18
|
+
pages?: {
|
|
19
|
+
status?: "text" | "needs-ocr";
|
|
20
|
+
markdown?: string;
|
|
21
|
+
page?: number;
|
|
22
|
+
}[];
|
|
23
|
+
markdown?: string;
|
|
24
|
+
assets?: {
|
|
25
|
+
page?: number;
|
|
26
|
+
name?: string;
|
|
27
|
+
mimeType?: string;
|
|
28
|
+
data?: string;
|
|
29
|
+
size?: number;
|
|
30
|
+
sha256?: string;
|
|
31
|
+
sourceType?: "pdf_page";
|
|
32
|
+
}[];
|
|
33
|
+
}>;
|
|
34
|
+
private runtime;
|
|
35
|
+
}
|
|
36
|
+
//# sourceMappingURL=convert.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"convert.d.ts","sourceRoot":"","sources":["../../src/lib/convert.ts"],"names":[],"mappings":"AAEA,OAAO,EAKL,KAAK,wCAAwC,EAC7C,KAAK,kBAAkB,EACxB,MAAM,sBAAsB,CAAA;AAI7B,MAAM,MAAM,eAAe,GAAG,kBAAkB,GAAG;IAAE,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAA;CAAE,CAAA;AAgDrF,qBACa,sBAAsB;IAI/B,OAAO,CAAC,QAAQ,CAAC,YAAY,CAAC;gBAAb,YAAY,CAAC,EAAE,IAAI,CAAC,wCAAwC,EAAE,KAAK,CAAC;IAGjF,WAAW;IAkBX,OAAO,CACX,QAAQ,EAAE,MAAM,EAChB,SAAS,EAAE,MAAM,EACjB,OAAO,EAAE;QACP,SAAS,CAAC,EAAE,eAAe,CAAA;QAC3B,UAAU,CAAC,EAAE,MAAM,CAAA;QACnB,KAAK,EAAE,MAAM,GAAG,MAAM,CAAA;QACtB,MAAM,CAAC,EAAE,WAAW,CAAA;KACrB;;;;;;;;;;;;;;;;;;;;IAiIH,OAAO,CAAC,OAAO;CAShB"}
|
|
@@ -0,0 +1,241 @@
|
|
|
1
|
+
import { __decorate, __metadata, __param } from "tslib";
|
|
2
|
+
import { Inject, Injectable, Optional, ServiceUnavailableException } from '@nestjs/common';
|
|
3
|
+
import { createHash } from 'node:crypto';
|
|
4
|
+
import { isSandboxJobRuntimeError, SandboxJobsRuntimeCapability, WorkspaceFilesRuntimeCapability, XPERT_RUNTIME_CAPABILITIES_TOKEN } from '@xpert-ai/plugin-sdk';
|
|
5
|
+
import { z } from 'zod';
|
|
6
|
+
import { PACKAGE_NAME, ACTION, ACTION_VERSION, FILE_TYPES } from './types.js';
|
|
7
|
+
const failureSchema = z.object({
|
|
8
|
+
ok: z.literal(false),
|
|
9
|
+
code: z.enum([
|
|
10
|
+
'EMPTY_FILE',
|
|
11
|
+
'EMPTY_TEXT',
|
|
12
|
+
'INPUT_TOO_LARGE',
|
|
13
|
+
'OUTPUT_TOO_LARGE',
|
|
14
|
+
'INVALID_DOCUMENT',
|
|
15
|
+
'ENCRYPTED',
|
|
16
|
+
'UNSUPPORTED_FORMAT',
|
|
17
|
+
'UNSUPPORTED_ENCODING',
|
|
18
|
+
'NEEDS_OCR',
|
|
19
|
+
'INCOMPLETE_PAGES',
|
|
20
|
+
'RESOURCE_LIMIT',
|
|
21
|
+
'RUNTIME_INVALID'
|
|
22
|
+
]),
|
|
23
|
+
pages: z.array(z.number().int().positive().max(10000)).max(100).optional()
|
|
24
|
+
});
|
|
25
|
+
const resultSchema = z.discriminatedUnion('ok', [
|
|
26
|
+
failureSchema,
|
|
27
|
+
z.object({
|
|
28
|
+
ok: z.literal(true),
|
|
29
|
+
markdown: z.string().min(1),
|
|
30
|
+
pages: z
|
|
31
|
+
.array(z.object({ page: z.number().int().positive(), markdown: z.string(), status: z.enum(['text', 'needs-ocr']) }))
|
|
32
|
+
.max(10000)
|
|
33
|
+
.optional(),
|
|
34
|
+
assets: z
|
|
35
|
+
.array(z.object({
|
|
36
|
+
name: z.string().regex(/^[a-zA-Z0-9_-]+\.[a-zA-Z0-9]+$/),
|
|
37
|
+
mimeType: z.string().max(100),
|
|
38
|
+
data: z.string(),
|
|
39
|
+
size: z
|
|
40
|
+
.number()
|
|
41
|
+
.int()
|
|
42
|
+
.nonnegative()
|
|
43
|
+
.max(128 * 1024 * 1024),
|
|
44
|
+
sha256: z.string().regex(/^[a-f0-9]{64}$/),
|
|
45
|
+
page: z.number().int().positive().optional(),
|
|
46
|
+
sourceType: z.literal('pdf_page').optional()
|
|
47
|
+
}))
|
|
48
|
+
.max(1000)
|
|
49
|
+
})
|
|
50
|
+
]);
|
|
51
|
+
let AnyDocSandboxConverter = class AnyDocSandboxConverter {
|
|
52
|
+
constructor(capabilities) {
|
|
53
|
+
this.capabilities = capabilities;
|
|
54
|
+
}
|
|
55
|
+
async checkHealth() {
|
|
56
|
+
const { jobs } = this.runtime();
|
|
57
|
+
const health = await jobs.getActionHealth({
|
|
58
|
+
pluginName: PACKAGE_NAME,
|
|
59
|
+
action: ACTION,
|
|
60
|
+
actionVersion: ACTION_VERSION
|
|
61
|
+
});
|
|
62
|
+
if (!health.available)
|
|
63
|
+
throw new ServiceUnavailableException(health.reason === 'ACTION_MISSING'
|
|
64
|
+
? 'AnyDoc Sandbox Action is missing (ACTION_MISSING). Install AnyDoc as a system plugin, then restart the API.'
|
|
65
|
+
: `AnyDoc Sandbox Runtime is unavailable (${health.reason ?? 'RUNTIME_UNBOUND'}). ${health.message ?? 'Check the platform Sandbox Runtime configuration.'}`);
|
|
66
|
+
return health;
|
|
67
|
+
}
|
|
68
|
+
async convert(filePath, extension, options) {
|
|
69
|
+
if (!FILE_TYPES.includes(extension))
|
|
70
|
+
throw new Error(`AnyDoc does not support this file type: ${extension}`);
|
|
71
|
+
const scope = options.fileScope;
|
|
72
|
+
if (!scope?.tenantId || scope.catalog !== 'knowledges' || !scope.scopeId) {
|
|
73
|
+
throw new Error('AnyDoc requires the host knowledge-base file scope');
|
|
74
|
+
}
|
|
75
|
+
options.signal?.throwIfAborted();
|
|
76
|
+
const health = await this.checkHealth();
|
|
77
|
+
const { files, jobs } = this.runtime();
|
|
78
|
+
const reference = await files.resolveRuntimeReference({
|
|
79
|
+
...scope,
|
|
80
|
+
source: 'platform.workspace.files',
|
|
81
|
+
filePath,
|
|
82
|
+
workspacePath: filePath
|
|
83
|
+
});
|
|
84
|
+
const source = await files.readBuffer(reference);
|
|
85
|
+
if (!source.buffer.length)
|
|
86
|
+
throw new Error('ANYDOC_EMPTY_FILE');
|
|
87
|
+
if (source.buffer.length > 100 * 1024 * 1024)
|
|
88
|
+
throw new Error('ANYDOC_INPUT_TOO_LARGE');
|
|
89
|
+
const checksum = sha256(source.buffer);
|
|
90
|
+
const identity = sha256(JSON.stringify([
|
|
91
|
+
scope.tenantId,
|
|
92
|
+
scope.organizationId,
|
|
93
|
+
scope.userId,
|
|
94
|
+
scope.scopeId,
|
|
95
|
+
options.documentId,
|
|
96
|
+
reference.filePath,
|
|
97
|
+
extension,
|
|
98
|
+
checksum,
|
|
99
|
+
options.stage,
|
|
100
|
+
ACTION_VERSION,
|
|
101
|
+
health.sandboxRuntimeVersion,
|
|
102
|
+
health.artifactDigest,
|
|
103
|
+
health.manifest?.dependenciesSha256
|
|
104
|
+
]));
|
|
105
|
+
// A retry reattaches Core's existing Job; cancellation must target that same ID.
|
|
106
|
+
const jobId = jobIdFromIdentity(identity);
|
|
107
|
+
const folder = `anydoc/jobs/${identity}`;
|
|
108
|
+
options.signal?.throwIfAborted();
|
|
109
|
+
let cancelRetry;
|
|
110
|
+
const cancel = () => {
|
|
111
|
+
// Abort can arrive before Core persists the Job. Retry until it exists or run() settles.
|
|
112
|
+
const attempt = () => void jobs
|
|
113
|
+
.cancel({ jobId })
|
|
114
|
+
.then(() => clearInterval(cancelRetry))
|
|
115
|
+
.catch(() => undefined);
|
|
116
|
+
cancelRetry = setInterval(attempt, 100);
|
|
117
|
+
cancelRetry.unref();
|
|
118
|
+
attempt();
|
|
119
|
+
};
|
|
120
|
+
options.signal?.addEventListener('abort', cancel, { once: true });
|
|
121
|
+
try {
|
|
122
|
+
const result = await jobs
|
|
123
|
+
.run({
|
|
124
|
+
jobId,
|
|
125
|
+
action: ACTION,
|
|
126
|
+
actionVersion: ACTION_VERSION,
|
|
127
|
+
idempotencyKey: `anydoc:${identity}`,
|
|
128
|
+
scope: {
|
|
129
|
+
tenantId: scope.tenantId,
|
|
130
|
+
organizationId: scope.organizationId,
|
|
131
|
+
userId: scope.userId,
|
|
132
|
+
pluginName: PACKAGE_NAME,
|
|
133
|
+
businessResourceType: 'knowledge-document',
|
|
134
|
+
businessResourceId: options.documentId ?? identity
|
|
135
|
+
},
|
|
136
|
+
payload: { extension },
|
|
137
|
+
files: [{ reference, targetPath: 'source.bin', size: source.buffer.length, sha256: checksum }],
|
|
138
|
+
outputs: [
|
|
139
|
+
{
|
|
140
|
+
path: 'result.json',
|
|
141
|
+
originalName: 'result.json',
|
|
142
|
+
mimeType: 'application/json',
|
|
143
|
+
destination: { ...scope, folder }
|
|
144
|
+
}
|
|
145
|
+
],
|
|
146
|
+
timeoutMs: 300000
|
|
147
|
+
})
|
|
148
|
+
.catch(rethrowConversionFailure);
|
|
149
|
+
options.signal?.throwIfAborted();
|
|
150
|
+
const output = result.outputs.find((item) => item.path === 'result.json');
|
|
151
|
+
if (!output || output.size > 128 * 1024 * 1024)
|
|
152
|
+
throw new Error('AnyDoc Sandbox output is missing or exceeds its limit');
|
|
153
|
+
const content = await files.readBuffer(output.reference);
|
|
154
|
+
if (content.buffer.length > 128 * 1024 * 1024)
|
|
155
|
+
throw new Error('ANYDOC_OUTPUT_TOO_LARGE');
|
|
156
|
+
if (content.buffer.length !== output.size || sha256(content.buffer) !== output.sha256) {
|
|
157
|
+
throw new Error('AnyDoc Sandbox output integrity check failed');
|
|
158
|
+
}
|
|
159
|
+
const converted = resultSchema.parse(JSON.parse(content.buffer.toString('utf8')));
|
|
160
|
+
if (converted.ok === false)
|
|
161
|
+
throw Object.assign(new Error(`ANYDOC_${converted.code}`), { pages: converted.pages });
|
|
162
|
+
let decodedSize = Buffer.byteLength(converted.markdown);
|
|
163
|
+
const names = new Set();
|
|
164
|
+
for (const asset of converted.assets) {
|
|
165
|
+
const data = Buffer.from(asset.data, 'base64');
|
|
166
|
+
if (names.has(asset.name) ||
|
|
167
|
+
data.toString('base64') !== asset.data ||
|
|
168
|
+
data.length !== asset.size ||
|
|
169
|
+
sha256(data) !== asset.sha256)
|
|
170
|
+
throw new Error('ANYDOC_INVALID_DOCUMENT');
|
|
171
|
+
names.add(asset.name);
|
|
172
|
+
decodedSize += data.length;
|
|
173
|
+
if (decodedSize > 128 * 1024 * 1024)
|
|
174
|
+
throw new Error('ANYDOC_OUTPUT_TOO_LARGE');
|
|
175
|
+
}
|
|
176
|
+
if (converted.pages) {
|
|
177
|
+
for (const [index, page] of converted.pages.entries()) {
|
|
178
|
+
const images = converted.assets.filter((asset) => asset.sourceType === 'pdf_page' && asset.page === page.page);
|
|
179
|
+
if (page.page !== index + 1 ||
|
|
180
|
+
!page.markdown.trim() ||
|
|
181
|
+
(page.status === 'needs-ocr' &&
|
|
182
|
+
(images.length !== 1 || !page.markdown.includes(`xpert-asset://${images[0].name}`)))) {
|
|
183
|
+
throw new Error('ANYDOC_INCOMPLETE_PAGES');
|
|
184
|
+
}
|
|
185
|
+
}
|
|
186
|
+
}
|
|
187
|
+
if (!converted.markdown.trim())
|
|
188
|
+
throw new Error('ANYDOC_EMPTY_TEXT');
|
|
189
|
+
return { ...converted, sandboxJobId: result.id, runtimeProfile: result.runtimeProfile };
|
|
190
|
+
}
|
|
191
|
+
finally {
|
|
192
|
+
options.signal?.removeEventListener('abort', cancel);
|
|
193
|
+
clearInterval(cancelRetry);
|
|
194
|
+
}
|
|
195
|
+
}
|
|
196
|
+
runtime() {
|
|
197
|
+
const jobs = this.capabilities?.get(SandboxJobsRuntimeCapability);
|
|
198
|
+
const files = this.capabilities?.get(WorkspaceFilesRuntimeCapability);
|
|
199
|
+
if (!jobs || !files)
|
|
200
|
+
throw new ServiceUnavailableException('AnyDoc requires platform Sandbox Jobs and Workspace Files. Update and restart the API.');
|
|
201
|
+
return { jobs, files };
|
|
202
|
+
}
|
|
203
|
+
};
|
|
204
|
+
AnyDocSandboxConverter = __decorate([
|
|
205
|
+
Injectable(),
|
|
206
|
+
__param(0, Optional()),
|
|
207
|
+
__param(0, Inject(XPERT_RUNTIME_CAPABILITIES_TOKEN)),
|
|
208
|
+
__metadata("design:paramtypes", [Object])
|
|
209
|
+
], AnyDocSandboxConverter);
|
|
210
|
+
export { AnyDocSandboxConverter };
|
|
211
|
+
/** Restore only the bounded error envelope emitted by this Action, retaining other Runtime failures. */
|
|
212
|
+
function rethrowConversionFailure(error) {
|
|
213
|
+
if (isSandboxJobRuntimeError(error)) {
|
|
214
|
+
const match = error.message.match(/^ANYDOC_CONVERSION_ERROR: (\{[^\r\n]{1,2048}\})$/m);
|
|
215
|
+
let failure;
|
|
216
|
+
if (match) {
|
|
217
|
+
try {
|
|
218
|
+
const parsed = failureSchema.safeParse(JSON.parse(match[1]));
|
|
219
|
+
if (parsed.success)
|
|
220
|
+
failure = parsed.data;
|
|
221
|
+
}
|
|
222
|
+
catch {
|
|
223
|
+
// Malformed or truncated output remains the original Runtime error.
|
|
224
|
+
}
|
|
225
|
+
}
|
|
226
|
+
if (failure)
|
|
227
|
+
throw Object.assign(new Error(`ANYDOC_${failure.code}`), { pages: failure.pages });
|
|
228
|
+
}
|
|
229
|
+
throw error;
|
|
230
|
+
}
|
|
231
|
+
function sha256(value) {
|
|
232
|
+
return createHash('sha256').update(value).digest('hex');
|
|
233
|
+
}
|
|
234
|
+
/** UUIDv8 derived from the tenant-scoped conversion identity. */
|
|
235
|
+
function jobIdFromIdentity(identity) {
|
|
236
|
+
const bytes = Buffer.from(identity.slice(0, 32), 'hex');
|
|
237
|
+
bytes[6] = (bytes[6] & 0x0f) | 0x80;
|
|
238
|
+
bytes[8] = (bytes[8] & 0x3f) | 0x80;
|
|
239
|
+
const hex = bytes.toString('hex');
|
|
240
|
+
return `${hex.slice(0, 8)}-${hex.slice(8, 12)}-${hex.slice(12, 16)}-${hex.slice(16, 20)}-${hex.slice(20)}`;
|
|
241
|
+
}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"plugin-anydoc.d.ts","sourceRoot":"","sources":["../../src/lib/plugin-anydoc.ts"],"names":[],"mappings":"AAAA,wBAAgB,YAAY,IAAI,MAAM,CAErC"}
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"plugin.module.d.ts","sourceRoot":"","sources":["../../src/lib/plugin.module.ts"],"names":[],"mappings":"AAGA,qBACa,kBAAkB;CAAG"}
|
|
@@ -0,0 +1,10 @@
|
|
|
1
|
+
import { __decorate } from "tslib";
|
|
2
|
+
import { XpertServerPlugin } from '@xpert-ai/plugin-sdk';
|
|
3
|
+
import { AnyDocSandboxConverter } from './convert.js';
|
|
4
|
+
import { AnyDocTransformerStrategy } from './transformer.strategy.js';
|
|
5
|
+
let AnyDocPluginModule = class AnyDocPluginModule {
|
|
6
|
+
};
|
|
7
|
+
AnyDocPluginModule = __decorate([
|
|
8
|
+
XpertServerPlugin({ providers: [AnyDocSandboxConverter, AnyDocTransformerStrategy] })
|
|
9
|
+
], AnyDocPluginModule);
|
|
10
|
+
export { AnyDocPluginModule };
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
import { type IDocumentTransformerStrategy, type Permissions, type TDocumentTransformerConfig, type WorkspaceFileScope } from '@xpert-ai/plugin-sdk';
|
|
2
|
+
import { AnyDocSandboxConverter } from './convert.js';
|
|
3
|
+
type Config = TDocumentTransformerConfig & {
|
|
4
|
+
signal?: AbortSignal;
|
|
5
|
+
fileScope?: WorkspaceFileScope;
|
|
6
|
+
};
|
|
7
|
+
type Documents = Parameters<IDocumentTransformerStrategy['transformDocuments']>[0];
|
|
8
|
+
type Results = Awaited<ReturnType<IDocumentTransformerStrategy['transformDocuments']>>;
|
|
9
|
+
export declare class AnyDocTransformerStrategy implements IDocumentTransformerStrategy<Config> {
|
|
10
|
+
private readonly converter;
|
|
11
|
+
constructor(converter: AnyDocSandboxConverter);
|
|
12
|
+
readonly permissions: Permissions;
|
|
13
|
+
readonly meta: {
|
|
14
|
+
name: string;
|
|
15
|
+
label: {
|
|
16
|
+
en_US: string;
|
|
17
|
+
zh_Hans: string;
|
|
18
|
+
};
|
|
19
|
+
description: {
|
|
20
|
+
en_US: string;
|
|
21
|
+
zh_Hans: string;
|
|
22
|
+
};
|
|
23
|
+
icon: {
|
|
24
|
+
type: "svg";
|
|
25
|
+
value: string;
|
|
26
|
+
};
|
|
27
|
+
supportedFileTypes: string[];
|
|
28
|
+
providesImageText: boolean;
|
|
29
|
+
configSchema: {
|
|
30
|
+
type: string;
|
|
31
|
+
properties: {};
|
|
32
|
+
};
|
|
33
|
+
};
|
|
34
|
+
validateConfig(config: Config): Promise<void>;
|
|
35
|
+
transformDocuments(documents: Documents, config: Config): Promise<Results>;
|
|
36
|
+
}
|
|
37
|
+
export {};
|
|
38
|
+
//# sourceMappingURL=transformer.strategy.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"transformer.strategy.d.ts","sourceRoot":"","sources":["../../src/lib/transformer.strategy.ts"],"names":[],"mappings":"AACA,OAAO,EAEL,KAAK,4BAA4B,EACjC,KAAK,WAAW,EAEhB,KAAK,0BAA0B,EAC/B,KAAK,kBAAkB,EACxB,MAAM,sBAAsB,CAAA;AAE7B,OAAO,EAAE,sBAAsB,EAAE,MAAM,cAAc,CAAA;AAGrD,KAAK,MAAM,GAAG,0BAA0B,GAAG;IAAE,MAAM,CAAC,EAAE,WAAW,CAAC;IAAC,SAAS,CAAC,EAAE,kBAAkB,CAAA;CAAE,CAAA;AACnG,KAAK,SAAS,GAAG,UAAU,CAAC,4BAA4B,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAC,CAAC,CAAA;AAClF,KAAK,OAAO,GAAG,OAAO,CAAC,UAAU,CAAC,4BAA4B,CAAC,oBAAoB,CAAC,CAAC,CAAC,CAAA;AACtF,qBAEa,yBAA0B,YAAW,4BAA4B,CAAC,MAAM,CAAC;IACxE,OAAO,CAAC,QAAQ,CAAC,SAAS;gBAAT,SAAS,EAAE,sBAAsB;IAC9D,QAAQ,CAAC,WAAW,EAAE,WAAW,CAAqE;IACtG,QAAQ,CAAC,IAAI;;;;;;;;;;;;;;;;;;;;MAWZ;IACK,cAAc,CAAC,MAAM,EAAE,MAAM;IAI7B,kBAAkB,CAAC,SAAS,EAAE,SAAS,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,OAAO,CAAC;CA2EjF"}
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
import { __decorate, __metadata } from "tslib";
|
|
2
|
+
import { Injectable } from '@nestjs/common';
|
|
3
|
+
import { DocumentTransformerStrategy } from '@xpert-ai/plugin-sdk';
|
|
4
|
+
import { randomUUID } from 'node:crypto';
|
|
5
|
+
import { AnyDocSandboxConverter } from './convert.js';
|
|
6
|
+
import { FILE_TYPES, Icon, documentExtension } from './types.js';
|
|
7
|
+
let AnyDocTransformerStrategy = class AnyDocTransformerStrategy {
|
|
8
|
+
constructor(converter) {
|
|
9
|
+
this.converter = converter;
|
|
10
|
+
this.permissions = [{ type: 'filesystem', operations: ['read', 'write'], scope: [] }];
|
|
11
|
+
this.meta = {
|
|
12
|
+
name: 'anydoc',
|
|
13
|
+
label: { en_US: 'AnyDoc', zh_Hans: 'AnyDoc' },
|
|
14
|
+
description: {
|
|
15
|
+
en_US: 'Local document conversion using Sandbox Jobs. Scanned PDF pages use the configured image-understanding model.',
|
|
16
|
+
zh_Hans: '通过平台沙箱本地转换文档;PDF 扫描页需开启图像理解并配置视觉模型。'
|
|
17
|
+
},
|
|
18
|
+
icon: Icon,
|
|
19
|
+
supportedFileTypes: FILE_TYPES,
|
|
20
|
+
providesImageText: false,
|
|
21
|
+
configSchema: { type: 'object', properties: {} }
|
|
22
|
+
};
|
|
23
|
+
}
|
|
24
|
+
async validateConfig(config) {
|
|
25
|
+
config.signal?.throwIfAborted();
|
|
26
|
+
await this.converter.checkHealth();
|
|
27
|
+
}
|
|
28
|
+
async transformDocuments(documents, config) {
|
|
29
|
+
const fs = config.permissions?.fileSystem;
|
|
30
|
+
if (!fs)
|
|
31
|
+
throw new Error('AnyDoc requires the scoped knowledge-base filesystem');
|
|
32
|
+
const output = [];
|
|
33
|
+
for (const document of documents) {
|
|
34
|
+
config.signal?.throwIfAborted();
|
|
35
|
+
const extension = documentExtension(document.type, document.mimeType);
|
|
36
|
+
if (!document.filePath || !FILE_TYPES.includes(extension))
|
|
37
|
+
throw new Error('ANYDOC_UNSUPPORTED_FORMAT');
|
|
38
|
+
const result = await this.converter.convert(document.filePath, extension, {
|
|
39
|
+
signal: config.signal,
|
|
40
|
+
fileScope: config.fileScope,
|
|
41
|
+
documentId: document.id,
|
|
42
|
+
stage: config.stage
|
|
43
|
+
});
|
|
44
|
+
const folder = 'anydoc/' + randomUUID();
|
|
45
|
+
const assets = [];
|
|
46
|
+
const references = new Map();
|
|
47
|
+
for (const asset of result.assets) {
|
|
48
|
+
config.signal?.throwIfAborted();
|
|
49
|
+
const filePath = `${folder}/${asset.name}`;
|
|
50
|
+
const url = await fs.writeFile(filePath, Buffer.from(asset.data, 'base64'));
|
|
51
|
+
references.set(`xpert-asset://${asset.name}`, url);
|
|
52
|
+
assets.push({
|
|
53
|
+
type: /^(image\/(png|jpeg|gif|webp))$/.test(asset.mimeType) ? 'image' : 'file',
|
|
54
|
+
filePath,
|
|
55
|
+
url,
|
|
56
|
+
order: assets.length,
|
|
57
|
+
...(asset.sourceType ? { sourceType: asset.sourceType } : {}),
|
|
58
|
+
...(asset.page ? { page: asset.page } : {})
|
|
59
|
+
});
|
|
60
|
+
}
|
|
61
|
+
const replaceReferences = (text) => text.replace(/xpert-asset:\/\/[a-zA-Z0-9_-]+\.[a-zA-Z0-9]+/g, (match) => references.get(match) ?? match);
|
|
62
|
+
const markdown = replaceReferences(result.markdown);
|
|
63
|
+
const originalPath = `${folder}/result.md`;
|
|
64
|
+
assets.push({ type: 'file', filePath: originalPath, url: await fs.writeFile(originalPath, markdown) });
|
|
65
|
+
const pages = result.pages ?? [{ page: undefined, markdown: result.markdown, status: 'text' }];
|
|
66
|
+
const chunks = pages
|
|
67
|
+
.filter((page) => page.markdown.trim())
|
|
68
|
+
.map((page, index) => ({
|
|
69
|
+
pageContent: replaceReferences(page.markdown),
|
|
70
|
+
metadata: {
|
|
71
|
+
chunkId: randomUUID(),
|
|
72
|
+
chunkIndex: index,
|
|
73
|
+
parser: 'anydoc',
|
|
74
|
+
mediaType: page.status === 'needs-ocr' ? 'image' : 'text',
|
|
75
|
+
contentFormat: 'markdown',
|
|
76
|
+
...(page.page ? { page: page.page } : {}),
|
|
77
|
+
assets: assets.filter((asset) => asset.type === 'image' && (!page.page || asset.page === page.page))
|
|
78
|
+
}
|
|
79
|
+
}));
|
|
80
|
+
output.push({
|
|
81
|
+
id: document.id,
|
|
82
|
+
chunks,
|
|
83
|
+
metadata: {
|
|
84
|
+
chunkId: randomUUID(),
|
|
85
|
+
parser: 'anydoc',
|
|
86
|
+
sandboxJobId: result.sandboxJobId,
|
|
87
|
+
runtimeProfile: result.runtimeProfile,
|
|
88
|
+
...(result.pages ? {
|
|
89
|
+
parserDiagnostics: {
|
|
90
|
+
schemaVersion: 1,
|
|
91
|
+
pages: result.pages.map((page) => ({
|
|
92
|
+
page: page.page, status: page.status,
|
|
93
|
+
imagePaths: assets.filter((asset) => asset.sourceType === 'pdf_page' && asset.page === page.page)
|
|
94
|
+
.map((asset) => asset.filePath)
|
|
95
|
+
}))
|
|
96
|
+
}
|
|
97
|
+
} : {}),
|
|
98
|
+
assets
|
|
99
|
+
}
|
|
100
|
+
});
|
|
101
|
+
}
|
|
102
|
+
return output;
|
|
103
|
+
}
|
|
104
|
+
};
|
|
105
|
+
AnyDocTransformerStrategy = __decorate([
|
|
106
|
+
Injectable(),
|
|
107
|
+
DocumentTransformerStrategy('anydoc'),
|
|
108
|
+
__metadata("design:paramtypes", [AnyDocSandboxConverter])
|
|
109
|
+
], AnyDocTransformerStrategy);
|
|
110
|
+
export { AnyDocTransformerStrategy };
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
import { z } from 'zod';
|
|
2
|
+
export declare const PACKAGE_NAME = "@xpert-ai/plugin-anydoc";
|
|
3
|
+
export declare const PARSER_NAME = "anydoc";
|
|
4
|
+
export declare const FILE_TYPES: string[];
|
|
5
|
+
export declare const ACTION = "anydoc.convert";
|
|
6
|
+
export declare const ACTION_VERSION = "1.0.3";
|
|
7
|
+
export declare const PROFILE = "document/node-20/v1";
|
|
8
|
+
export declare const ConfigSchema: z.ZodObject<{}, "strict", z.ZodTypeAny, {}, {}>;
|
|
9
|
+
export declare const Icon: {
|
|
10
|
+
type: "svg";
|
|
11
|
+
value: string;
|
|
12
|
+
};
|
|
13
|
+
/** Host uploads usually supply an extension; MIME-only callers use the same upstream format. */
|
|
14
|
+
export declare function documentExtension(type?: string, mimeType?: string): string;
|
|
15
|
+
//# sourceMappingURL=types.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/lib/types.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,CAAC,EAAE,MAAM,KAAK,CAAA;AAEvB,eAAO,MAAM,YAAY,4BAA4B,CAAA;AACrD,eAAO,MAAM,WAAW,WAAW,CAAA;AACnC,eAAO,MAAM,UAAU,UActB,CAAA;AACD,eAAO,MAAM,MAAM,mBAAmB,CAAA;AACtC,eAAO,MAAM,cAAc,UAAU,CAAA;AACrC,eAAO,MAAM,OAAO,wBAAwB,CAAA;AAC5C,eAAO,MAAM,YAAY,iDAAwB,CAAA;AACjD,eAAO,MAAM,IAAI;;;CAGhB,CAAA;AAED,gGAAgG;AAChG,wBAAgB,iBAAiB,CAAC,IAAI,CAAC,EAAE,MAAM,EAAE,QAAQ,CAAC,EAAE,MAAM,GAAG,MAAM,CAsB1E"}
|
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
import { z } from 'zod';
|
|
2
|
+
import { readFileSync } from 'node:fs';
|
|
3
|
+
export const PACKAGE_NAME = '@xpert-ai/plugin-anydoc';
|
|
4
|
+
export const PARSER_NAME = 'anydoc';
|
|
5
|
+
export const FILE_TYPES = [
|
|
6
|
+
'doc',
|
|
7
|
+
'docx',
|
|
8
|
+
'ppt',
|
|
9
|
+
'pptx',
|
|
10
|
+
'xls',
|
|
11
|
+
'xlsx',
|
|
12
|
+
'csv',
|
|
13
|
+
'odt',
|
|
14
|
+
'ods',
|
|
15
|
+
'odp',
|
|
16
|
+
'rtf',
|
|
17
|
+
'epub',
|
|
18
|
+
'pdf'
|
|
19
|
+
];
|
|
20
|
+
export const ACTION = 'anydoc.convert';
|
|
21
|
+
export const ACTION_VERSION = '1.0.3';
|
|
22
|
+
export const PROFILE = 'document/node-20/v1';
|
|
23
|
+
export const ConfigSchema = z.object({}).strict();
|
|
24
|
+
export const Icon = {
|
|
25
|
+
type: 'svg',
|
|
26
|
+
value: readFileSync(new URL('../_assets/icon.svg', import.meta.url), 'utf8')
|
|
27
|
+
};
|
|
28
|
+
/** Host uploads usually supply an extension; MIME-only callers use the same upstream format. */
|
|
29
|
+
export function documentExtension(type, mimeType) {
|
|
30
|
+
const value = (type && type !== 'unknown' ? type : mimeType)
|
|
31
|
+
?.split(';')[0]
|
|
32
|
+
.trim()
|
|
33
|
+
.toLowerCase()
|
|
34
|
+
.replace(/^.*\//, '')
|
|
35
|
+
.replace(/^\./, '') ?? '';
|
|
36
|
+
const aliases = {
|
|
37
|
+
msword: 'doc',
|
|
38
|
+
'vnd.ms-powerpoint': 'ppt',
|
|
39
|
+
'vnd.ms-excel': 'xls',
|
|
40
|
+
'vnd.openxmlformats-officedocument.wordprocessingml.document': 'docx',
|
|
41
|
+
'vnd.openxmlformats-officedocument.presentationml.presentation': 'pptx',
|
|
42
|
+
'vnd.openxmlformats-officedocument.spreadsheetml.sheet': 'xlsx',
|
|
43
|
+
'vnd.oasis.opendocument.text': 'odt',
|
|
44
|
+
'vnd.oasis.opendocument.spreadsheet': 'ods',
|
|
45
|
+
'vnd.oasis.opendocument.presentation': 'odp',
|
|
46
|
+
'epub+zip': 'epub',
|
|
47
|
+
'x-rtf': 'rtf'
|
|
48
|
+
};
|
|
49
|
+
return Object.prototype.hasOwnProperty.call(aliases, value) ? aliases[value] : value;
|
|
50
|
+
}
|
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "anydoc.convert",
|
|
3
|
+
"version": "1.0.3",
|
|
4
|
+
"runtimeProfile": "document/node-20/v1",
|
|
5
|
+
"runtimeContractVersion": "1",
|
|
6
|
+
"bundle": "./bundle",
|
|
7
|
+
"entrypoint": "runner.mjs",
|
|
8
|
+
"bundleSha256": "3c9193c6992c0a22ae42bb26ccf96be5c54df6014126069c5e1beac99e64afe1"
|
|
9
|
+
}
|
|
@@ -0,0 +1,80 @@
|
|
|
1
|
+
import { createRequire } from 'node:module'
|
|
2
|
+
import path from 'node:path'
|
|
3
|
+
import { INPUT_LIMIT, ASSET_LIMIT, asset, fail, finish, readBounded } from './result.mjs'
|
|
4
|
+
import { convertPdfPages } from './pdf.mjs'
|
|
5
|
+
|
|
6
|
+
export const FORMATS = ['doc', 'docx', 'ppt', 'pptx', 'xls', 'xlsx', 'csv', 'odt', 'ods', 'odp', 'rtf', 'epub', 'pdf']
|
|
7
|
+
|
|
8
|
+
export async function convert(file, extension, output, suppliedApi) {
|
|
9
|
+
if (!FORMATS.includes(extension)) fail('UNSUPPORTED_FORMAT')
|
|
10
|
+
let bytes
|
|
11
|
+
try {
|
|
12
|
+
bytes = await readBounded(file, INPUT_LIMIT)
|
|
13
|
+
} catch (error) {
|
|
14
|
+
if (error.code === 'OUTPUT_TOO_LARGE') fail('INPUT_TOO_LARGE')
|
|
15
|
+
throw error
|
|
16
|
+
}
|
|
17
|
+
if (!bytes.length) fail('EMPTY_FILE')
|
|
18
|
+
let api = suppliedApi
|
|
19
|
+
if (!api) {
|
|
20
|
+
const root = process.env.XPERT_SANDBOX_DOCUMENT_DEPENDENCY_ROOT
|
|
21
|
+
if (!root || !path.isAbsolute(root)) fail('RUNTIME_INVALID')
|
|
22
|
+
try {
|
|
23
|
+
api = createRequire(path.join(root, 'package.json'))('@firecrawl/anydoc')
|
|
24
|
+
} catch {
|
|
25
|
+
fail('RUNTIME_INVALID')
|
|
26
|
+
}
|
|
27
|
+
}
|
|
28
|
+
try {
|
|
29
|
+
const format = api.formatFromExtension(extension)
|
|
30
|
+
const detected = api.formatFromBytes(bytes)
|
|
31
|
+
if (!format || (detected && detected !== format)) fail('INVALID_DOCUMENT')
|
|
32
|
+
if (!detected && extension !== 'csv') fail('INVALID_DOCUMENT')
|
|
33
|
+
if (extension === 'csv') {
|
|
34
|
+
if (bytes.includes(0)) fail('INVALID_DOCUMENT')
|
|
35
|
+
let text
|
|
36
|
+
try {
|
|
37
|
+
text = new TextDecoder('utf-8', { fatal: true }).decode(bytes)
|
|
38
|
+
} catch {
|
|
39
|
+
fail('UNSUPPORTED_ENCODING')
|
|
40
|
+
}
|
|
41
|
+
if (!text.trim()) fail('EMPTY_TEXT')
|
|
42
|
+
}
|
|
43
|
+
// Never delegate to Firecrawl hosted OCR, even when the host has a Firecrawl API key.
|
|
44
|
+
let markdown
|
|
45
|
+
try {
|
|
46
|
+
markdown = await api.toMarkdownBytes(bytes, format, { ocr: 'reject' })
|
|
47
|
+
} catch (error) {
|
|
48
|
+
if (extension !== 'pdf' || error?.code !== 'needsOcr') throw error
|
|
49
|
+
return await convertPdfPages(bytes, api)
|
|
50
|
+
}
|
|
51
|
+
const assets = []
|
|
52
|
+
if (extension !== 'pdf') {
|
|
53
|
+
const document = await api.toDocument(bytes, format)
|
|
54
|
+
if (document.assets.length > ASSET_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
55
|
+
for (const source of document.assets) {
|
|
56
|
+
const suffix =
|
|
57
|
+
{ 'image/png': 'png', 'image/jpeg': 'jpg', 'image/gif': 'gif', 'image/webp': 'webp', 'image/svg+xml': 'svg' }[
|
|
58
|
+
source.mediaType
|
|
59
|
+
] ?? 'bin'
|
|
60
|
+
const name = `asset-${assets.length + 1}.${suffix}`
|
|
61
|
+
assets.push(asset(name, source.mediaType, Buffer.from(source.data)))
|
|
62
|
+
// Native Markdown omits embedded images. Export them explicitly without inventing page/position data.
|
|
63
|
+
if (['png', 'jpg', 'gif', 'webp'].includes(suffix))
|
|
64
|
+
markdown += `\n\n\n`
|
|
65
|
+
}
|
|
66
|
+
}
|
|
67
|
+
return finish({ markdown, assets })
|
|
68
|
+
} catch (error) {
|
|
69
|
+
const code = {
|
|
70
|
+
needsOcr: 'NEEDS_OCR',
|
|
71
|
+
encrypted: 'ENCRYPTED',
|
|
72
|
+
unsupported: 'UNSUPPORTED_FORMAT',
|
|
73
|
+
malformed: 'INVALID_DOCUMENT',
|
|
74
|
+
missingPart: 'INVALID_DOCUMENT',
|
|
75
|
+
resourceLimit: 'RESOURCE_LIMIT'
|
|
76
|
+
}[error?.code]
|
|
77
|
+
if (code) fail(code, error.pages)
|
|
78
|
+
throw error
|
|
79
|
+
}
|
|
80
|
+
}
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
import { createRequire } from 'node:module'
|
|
2
|
+
import path from 'node:path'
|
|
3
|
+
import { asset, fail, finish, OUTPUT_LIMIT } from './result.mjs'
|
|
4
|
+
|
|
5
|
+
/** AnyDoc rejects a whole mixed PDF. Isolate pages so its native text extraction is retained. */
|
|
6
|
+
export async function convertPdfPages(bytes, api) {
|
|
7
|
+
const root = process.env.XPERT_SANDBOX_DOCUMENT_DEPENDENCY_ROOT
|
|
8
|
+
if (!root || !path.isAbsolute(root)) fail('RUNTIME_INVALID')
|
|
9
|
+
let PDFDocument, PDFiumLibrary, PNG
|
|
10
|
+
try {
|
|
11
|
+
const require = createRequire(path.join(root, 'package.json'))
|
|
12
|
+
;({ PDFDocument } = require('pdf-lib'))
|
|
13
|
+
;({ PDFiumLibrary } = require('@hyzyla/pdfium'))
|
|
14
|
+
;({ PNG } = require('pngjs'))
|
|
15
|
+
} catch {
|
|
16
|
+
fail('RUNTIME_INVALID')
|
|
17
|
+
}
|
|
18
|
+
const source = await PDFDocument.load(bytes)
|
|
19
|
+
const count = source.getPageCount()
|
|
20
|
+
if (count < 1 || count > 500) fail('RESOURCE_LIMIT')
|
|
21
|
+
const pdfium = await PDFiumLibrary.init()
|
|
22
|
+
try {
|
|
23
|
+
const pdf = await pdfium.loadDocument(Uint8Array.from(bytes))
|
|
24
|
+
try {
|
|
25
|
+
if (pdf.getPageCount() !== count) fail('INCOMPLETE_PAGES')
|
|
26
|
+
const pages = []
|
|
27
|
+
const assets = []
|
|
28
|
+
let size = 0
|
|
29
|
+
for (let index = 0; index < count; index++) {
|
|
30
|
+
const pageNumber = index + 1
|
|
31
|
+
const single = await PDFDocument.create()
|
|
32
|
+
const [copied] = await single.copyPages(source, [index])
|
|
33
|
+
single.addPage(copied)
|
|
34
|
+
let markdown
|
|
35
|
+
try {
|
|
36
|
+
markdown = await api.toMarkdownBytes(await single.save(), api.formatFromExtension('pdf'), { ocr: 'reject' })
|
|
37
|
+
} catch (error) {
|
|
38
|
+
if (error?.code !== 'needsOcr') throw error
|
|
39
|
+
}
|
|
40
|
+
if (markdown?.trim()) {
|
|
41
|
+
pages.push({ page: pageNumber, markdown, status: 'text' })
|
|
42
|
+
size += Buffer.byteLength(markdown)
|
|
43
|
+
} else {
|
|
44
|
+
const page = pdf.getPage(index)
|
|
45
|
+
const dimensions = page.getSize()
|
|
46
|
+
if (![dimensions.width, dimensions.height].every((n) => Number.isFinite(n) && n > 0))
|
|
47
|
+
fail('INVALID_DOCUMENT')
|
|
48
|
+
const scale = Math.min(2, 2200 / Math.max(dimensions.width, dimensions.height))
|
|
49
|
+
const bitmap = await page.render({ scale, render: 'bitmap' })
|
|
50
|
+
const png = new PNG({ width: bitmap.width, height: bitmap.height })
|
|
51
|
+
png.data = Buffer.from(bitmap.data)
|
|
52
|
+
const image = asset(`page-${pageNumber}.png`, 'image/png', PNG.sync.write(png), pageNumber)
|
|
53
|
+
assets.push({ ...image, sourceType: 'pdf_page' })
|
|
54
|
+
size += image.size
|
|
55
|
+
pages.push({ page: pageNumber, status: 'needs-ocr', markdown: `` })
|
|
56
|
+
}
|
|
57
|
+
if (size > OUTPUT_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
58
|
+
}
|
|
59
|
+
return finish({ markdown: pages.map((page) => page.markdown).join('\n\n'), pages, assets })
|
|
60
|
+
} finally {
|
|
61
|
+
pdf.destroy()
|
|
62
|
+
}
|
|
63
|
+
} finally {
|
|
64
|
+
pdfium.destroy()
|
|
65
|
+
}
|
|
66
|
+
}
|
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
import { createHash } from 'node:crypto'
|
|
2
|
+
import { readFile, lstat, realpath, readdir } from 'node:fs/promises'
|
|
3
|
+
import path from 'node:path'
|
|
4
|
+
export const INPUT_LIMIT = 100 * 1024 * 1024
|
|
5
|
+
export const OUTPUT_LIMIT = 128 * 1024 * 1024
|
|
6
|
+
export const ASSET_LIMIT = 1000
|
|
7
|
+
export function fail(code, pages) {
|
|
8
|
+
throw Object.assign(new Error(code), { code, pages })
|
|
9
|
+
}
|
|
10
|
+
export function asset(name, mimeType, data, page) {
|
|
11
|
+
if (data.length > OUTPUT_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
12
|
+
return {
|
|
13
|
+
name,
|
|
14
|
+
mimeType,
|
|
15
|
+
size: data.length,
|
|
16
|
+
sha256: createHash('sha256').update(data).digest('hex'),
|
|
17
|
+
data: data.toString('base64'),
|
|
18
|
+
...(page ? { page } : {})
|
|
19
|
+
}
|
|
20
|
+
}
|
|
21
|
+
export async function readBounded(file, limit = OUTPUT_LIMIT) {
|
|
22
|
+
const stat = await lstat(file)
|
|
23
|
+
if (!stat.isFile() || stat.isSymbolicLink()) fail('INVALID_DOCUMENT')
|
|
24
|
+
if (stat.size > limit) fail('OUTPUT_TOO_LARGE')
|
|
25
|
+
const data = await readFile(file)
|
|
26
|
+
if (data.length > limit) fail('OUTPUT_TOO_LARGE')
|
|
27
|
+
return data
|
|
28
|
+
}
|
|
29
|
+
export async function safeOutput(root, relative) {
|
|
30
|
+
if (
|
|
31
|
+
!relative ||
|
|
32
|
+
path.isAbsolute(relative) ||
|
|
33
|
+
/[\\\0]/.test(relative) ||
|
|
34
|
+
relative.split('/').some((p) => !p || p === '..' || p === '.')
|
|
35
|
+
)
|
|
36
|
+
fail('INVALID_DOCUMENT')
|
|
37
|
+
const target = path.join(root, relative)
|
|
38
|
+
let current = root
|
|
39
|
+
for (const part of relative.split('/')) {
|
|
40
|
+
current = path.join(current, part)
|
|
41
|
+
if ((await lstat(current)).isSymbolicLink()) fail('INVALID_DOCUMENT')
|
|
42
|
+
}
|
|
43
|
+
const resolved = await realpath(target)
|
|
44
|
+
if (!resolved.startsWith((await realpath(root)) + path.sep)) fail('INVALID_DOCUMENT')
|
|
45
|
+
return target
|
|
46
|
+
}
|
|
47
|
+
export async function checkOutputTree(root) {
|
|
48
|
+
let total = 0,
|
|
49
|
+
count = 0
|
|
50
|
+
async function visit(dir) {
|
|
51
|
+
for (const entry of await readdir(dir, { withFileTypes: true })) {
|
|
52
|
+
if (++count > 5000 || entry.isSymbolicLink()) fail('OUTPUT_TOO_LARGE')
|
|
53
|
+
const file = path.join(dir, entry.name)
|
|
54
|
+
if (entry.isDirectory()) await visit(file)
|
|
55
|
+
else {
|
|
56
|
+
const stat = await lstat(file)
|
|
57
|
+
if (!stat.isFile()) fail('INVALID_DOCUMENT')
|
|
58
|
+
total += stat.size
|
|
59
|
+
if (total > OUTPUT_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
60
|
+
}
|
|
61
|
+
}
|
|
62
|
+
}
|
|
63
|
+
await visit(root)
|
|
64
|
+
}
|
|
65
|
+
export function finish(result) {
|
|
66
|
+
if (!result.markdown.trim()) fail('EMPTY_TEXT')
|
|
67
|
+
if (result.assets.length > ASSET_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
68
|
+
const decoded = Buffer.byteLength(result.markdown) + result.assets.reduce((n, a) => n + a.size, 0)
|
|
69
|
+
if (decoded > OUTPUT_LIMIT || Buffer.byteLength(JSON.stringify(result)) > OUTPUT_LIMIT) fail('OUTPUT_TOO_LARGE')
|
|
70
|
+
return { ok: true, ...result }
|
|
71
|
+
}
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
import { readFile, writeFile, mkdir } from 'node:fs/promises'
|
|
2
|
+
import path from 'node:path'
|
|
3
|
+
import { convert } from './convert.mjs'
|
|
4
|
+
const argument = (name) => {
|
|
5
|
+
const index = process.argv.indexOf(name)
|
|
6
|
+
if (index < 0 || !process.argv[index + 1]) throw new Error('Invalid Action arguments')
|
|
7
|
+
return path.resolve(process.argv[index + 1])
|
|
8
|
+
}
|
|
9
|
+
try {
|
|
10
|
+
const requestPath = argument('--request'),
|
|
11
|
+
output = argument('--output')
|
|
12
|
+
const request = JSON.parse(await readFile(requestPath, 'utf8'))
|
|
13
|
+
if (request.contractVersion !== '1' || request.action !== 'anydoc.convert' || request.actionVersion !== '1.0.3')
|
|
14
|
+
throw new Error('Invalid Action contract')
|
|
15
|
+
await mkdir(output, { recursive: true })
|
|
16
|
+
let result
|
|
17
|
+
try {
|
|
18
|
+
result = await convert(path.join(path.dirname(requestPath), 'source.bin'), request.payload?.extension, output)
|
|
19
|
+
} catch (error) {
|
|
20
|
+
const codes = [
|
|
21
|
+
'EMPTY_FILE',
|
|
22
|
+
'EMPTY_TEXT',
|
|
23
|
+
'INPUT_TOO_LARGE',
|
|
24
|
+
'OUTPUT_TOO_LARGE',
|
|
25
|
+
'INVALID_DOCUMENT',
|
|
26
|
+
'ENCRYPTED',
|
|
27
|
+
'UNSUPPORTED_FORMAT',
|
|
28
|
+
'UNSUPPORTED_ENCODING',
|
|
29
|
+
'NEEDS_OCR',
|
|
30
|
+
'INCOMPLETE_PAGES',
|
|
31
|
+
'RESOURCE_LIMIT',
|
|
32
|
+
'RUNTIME_INVALID'
|
|
33
|
+
]
|
|
34
|
+
result = {
|
|
35
|
+
ok: false,
|
|
36
|
+
code: codes.includes(error?.code) ? error.code : 'INVALID_DOCUMENT',
|
|
37
|
+
...(Array.isArray(error?.pages)
|
|
38
|
+
? { pages: error.pages.filter((n) => Number.isSafeInteger(n) && n > 0 && n <= 10000).slice(0, 100) }
|
|
39
|
+
: {})
|
|
40
|
+
}
|
|
41
|
+
}
|
|
42
|
+
await writeFile(path.join(output, 'result.json'), JSON.stringify(result))
|
|
43
|
+
if (!result.ok) {
|
|
44
|
+
// Core must persist a failed Job so a later attempt can run after recovery.
|
|
45
|
+
process.stderr.write(`ANYDOC_CONVERSION_ERROR: ${JSON.stringify(result)}\n`)
|
|
46
|
+
process.exitCode = 1
|
|
47
|
+
}
|
|
48
|
+
} catch {
|
|
49
|
+
process.stderr.write('EXPORT_OUTPUT_INVALID: Document conversion Action failed.\n')
|
|
50
|
+
process.exitCode = 1
|
|
51
|
+
}
|
package/package.json
ADDED
|
@@ -0,0 +1,42 @@
|
|
|
1
|
+
{
|
|
2
|
+
"name": "@xpert-ai/plugin-anydoc",
|
|
3
|
+
"version": "0.0.1",
|
|
4
|
+
"type": "module",
|
|
5
|
+
"main": "./dist/index.js",
|
|
6
|
+
"module": "./dist/index.js",
|
|
7
|
+
"types": "./dist/index.d.ts",
|
|
8
|
+
"exports": {
|
|
9
|
+
"./package.json": "./package.json",
|
|
10
|
+
".": {
|
|
11
|
+
"@xpert-plugins-starter/source": "./src/index.ts",
|
|
12
|
+
"types": "./dist/index.d.ts",
|
|
13
|
+
"import": "./dist/index.js",
|
|
14
|
+
"default": "./dist/index.js"
|
|
15
|
+
}
|
|
16
|
+
},
|
|
17
|
+
"files": [
|
|
18
|
+
"dist",
|
|
19
|
+
".xpertai-plugin",
|
|
20
|
+
"!**/*.tsbuildinfo"
|
|
21
|
+
],
|
|
22
|
+
"dependencies": {
|
|
23
|
+
"tslib": "^2.3.0"
|
|
24
|
+
},
|
|
25
|
+
"license": "AGPL-3.0",
|
|
26
|
+
"author": "XpertAI",
|
|
27
|
+
"description": "AnyDoc knowledge document parser using platform-managed Sandbox Jobs.",
|
|
28
|
+
"peerDependencies": {
|
|
29
|
+
"@nestjs/common": "^11.1.6",
|
|
30
|
+
"@xpert-ai/plugin-sdk": "^3.17.3",
|
|
31
|
+
"zod": "3.25.67"
|
|
32
|
+
},
|
|
33
|
+
"xpert": {
|
|
34
|
+
"plugin": {
|
|
35
|
+
"level": "system",
|
|
36
|
+
"artifactNamespace": "anydoc"
|
|
37
|
+
}
|
|
38
|
+
},
|
|
39
|
+
"scripts": {
|
|
40
|
+
"test:runtime": "node --test tests/*.test.mjs"
|
|
41
|
+
}
|
|
42
|
+
}
|