filens-ocr-linux-arm64 0.1.0 → 0.3.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 FiLens contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,21 +1,52 @@
 # filens Agent CLI
 
+<!-- agent-cli-size:npm:start -->
+**v0.3.0 · MIT · CLI 2.10 MiB · Lite 7.85 MiB · Full 31.05 MiB** (macOS arm64, installed; Full includes OCR models).
+
+| Platform | CLI executable | Lite installed / download | Full installed / download |
+| --- | ---: | ---: | ---: |
+| darwin-arm64 | 2.10 MiB | 7.85 / 6.02 MiB | 31.05 / 20.22 MiB |
+| linux-x64 | 1.88 MiB | 7.62 / 6.74 MiB | 36.57 / 22.72 MiB |
+| linux-arm64 | 1.77 MiB | 7.51 / 6.63 MiB | 30.94 / 20.42 MiB |
+| win32-x64 | 1.80 MiB | 7.54 / 6.66 MiB | 36.32 / 22.89 MiB |
+
+Totals add the launcher and selected platform packages from npm pack. Node.js, npm caches and filesystem overhead are excluded. Bundled dependencies retain their own licenses; see THIRD-PARTY-NOTICES.md in the platform packages.
+<!-- agent-cli-size:npm:end -->
+
 Read local documents and provide text and cited evidence for agents. Supports
-Chinese/English reading intents, plain text and JSON. Semantic reasoning is
-performed by the calling agent. All document parsing uses the shared WASM core.
+Chinese/English reading intents, text, JSON, Markdown, HTML, YAML, XML, CSV and
+TSV. Source layouts include PDF/OCR coordinates and logical document structure.
+Version 0.3.0 also provides model-free extractive summaries through
+`summarize` / `prune`, with optional lexical focus and original-text provenance.
+Check `capacity.operations` when using an older installation.
+Paraphrasing, translation and question
+answering belong to the calling agent. Parsing and pruning use the shared WASM core.
 
 ```sh
 # Full: native RapidOCR NCNN and models included; no Python
-npm install -g filens
+npm install -g filens-cli
 filens '读取 "report.pdf"，以 JSON 输出'
 filens 'what can you do as json'
 
 # Lite: CLI + WASM; add native OCR when needed
-npm install -g filens-lite
+npm install -g filens-cli-lite
 filens setup-ocr
 filens 'read "scan.pdf" as json'
+filens convert report.doc report.docx
+filens convert data.csv --to xlsx
+
+# Temporary invocation; no global installation
+npx --yes filens-cli-lite read report.pdf --json
 ```
 
+The CLI is Rust, built with stripped symbols, LTO and size optimization.
+Windows/Linux executables use UPX `--best --lzma`, followed by `--test`.
+Mach-O arm64 is unsupported by UPX and remains uncompressed.
+WASM resources use fast gzip compression and are decoded locally as needed.
+The package includes `skills/filens/SKILL.md`, with standard YAML front matter
+and agent guidance for evidence, partial results, OCR and coordinate units.
+Copy `skills/filens/` to your agent's skills directory to enable discovery.
+
 ## More examples
 
 ```sh
@@ -25,25 +56,123 @@ filens '帮我理解“会议纪要.docx”，用 JSON 返回'
 filens 'Please extract all text from "report.pdf" in JSON'
 filens '识别 "invoice.png" 中的文字，以 JSON 输出'
 filens '理解 "embedded-images.docx"，以 JSON 输出'
+filens 'read "report.docx" as markdown'
+filens '获取 "scan.pdf" 的布局，以 YAML 输出'
+filens layout invoice.png --format json
+filens read data.xlsx --format csv
 echo '{"operation":"understand","input":"report.pdf"}' | filens
 ```
 
-Natural language recognizes a bounded set of reading intents. `understand`
+Natural language recognizes a bounded set of document intents. `understand`
 returns source text and evidence for the calling agent; it does not generate
-an answer. One local file per request, text/JSON output. Summarization,
-translation, questions, field extraction, page selection, URL fetching and file
-conversion are not implemented by this CLI. Some unsupported phrasings fall
+an answer. Reading and pruning accept one local file per request. Builds advertising `summarize` select
+key original excerpts without an LLM, using density/TF-IDF, optional BM25 focus
+and cosine-diversity pruning. For example:
+
+```sh
+filens summarize report.pdf --query 'budget risk' --summary-chars 1000 --json
+filens '提取 "report.pdf" 的关键内容，关注预算和风险，以 JSON 输出'
+```
+
+Inspect `content.summary` for original offsets, `sourceComplete`, `omitted`,
+`sampled` and per-excerpt `clipped` markers. The default budget is 30% of source
+characters, rounded up and capped at 2000 characters. Scores are ranking weights,
+not confidence, and an empty lexical match does not prove the topic is absent.
+
+Version 0.3.0 includes directory-wide `search` with BM25, case-sensitive literal
+and filename modes, original excerpts and optional layout. Check
+`capacity.operations` for `search` when using an older version.
+
+```sh
+filens '在 "./documents" 中搜索 "预算 风险"，以 JSON 输出'
+filens 'search "./documents" for "budget risk" as markdown'
+filens search ./documents --query FILENS2026 --search-mode exact --json
+filens 'search "documents.zip" for "budget risk" as json'
+filens read 'documents.zip!/attachments.tar.gz!/report.pdf' --layout --json
+```
+
+macOS uses native FTS and does not embed hdrw. Windows/Linux embed compiled
+private hdrw WASI for mounted-directory enumeration; no private source or raw
+device access is needed. Search keeps its catalog and extracted text in request-local
+memory, with no persistent index/body cache or source hashing. Common text decoding and
+ranking reuse the frontend Rust core directly; document extraction uses WASM.
+Embedding-based semantic retrieval is not implemented. See the
+[search guide](https://github.com/IceWhaleTech/filens/blob/main/docs/agent-cli-search.zh-CN.md)
+for scope, limits, provenance and partial results.
+
+Builds advertising `capacity.search.archives` automatically traverse ZIP,
+TAR and GZIP containers, including nested archives up to eight layers. Search
+results retain virtual member paths and structured container chains; the same
+paths support reading, layouts and summaries, including OCR. Expansion uses
+bounded request-local temporary files. RAR/7z and archive password decryption
+are not currently supported.
+
+## Duplicate files and similar images
+
+Version 0.3.0 provides Czkawka-style directory scans. Check `capacity.operations`
+for `similar` when using an older installation.
+
+```sh
+filens duplicates ./files --json
+filens similar-images ./photos --threshold 8 --json
+filens similar ./photos --similarity-mode images --format csv
+echo '{"operation":"similar","input":"./photos","similarityMode":"images","threshold":8}' | filens
+```
+
+Duplicate scans filter by size and prefix hash, then compare full-file SHA-256.
+Image scans use perceptual hashes for JPEG, PNG, WebP, GIF, BMP and TIFF, including
+resized and recompressed variants. No OCR or model is required. Threshold is a
+Hamming distance from 0 to 24 (default 8), not a confidence percentage.
+
+Defaults are 2000 candidate files, depth 32 and 100 returned groups; adjust
+`--max-files`, `--max-depth`, `--max-results` and `--timeout` as needed. Inspect
+`similarity.complete`, `truncated`, `scan` and `processing`; partial scans retain
+groups and return exit 3. Paths are relative to `source.path`. Scans report groups
+for review; perceptual matches can miss altered images or match different images.
+Similarity fingerprints are recomputed each time. The document search cache and
+`--no-cache` flag do not apply. See the
+[similarity guide](https://github.com/IceWhaleTech/filens/blob/main/docs/agent-cli-similarity.md)
+for source builds, limits, grouping semantics and the measured resource-size delta.
+
+## Limits and conversion
+
+Paraphrasing, translation, questions, field extraction, page selection and URL fetching
+are not implemented by this CLI. Some unsupported phrasings fall
 back to reading the whole file: “only the first page” still reads all pages,
-“only list amounts” still returns all text, and “as yaml” falls back to text.
+“only list amounts” still returns all text.
 Use the JSON request schema or explicit `--format` when strict validation is
 required. Quote file paths inside natural-language sentences.
 
+`convert` uses the same WASM conversion engines as the frontend. Inspect
+`capacity.conversionPairs` before choosing a target; text extraction support
+does not imply conversion support. Examples include DOC/RTF/ODF/text to
+OpenXML, CSV/JSON to XLSX, camera RAW to JPEG and ICNS to PNG. DOCX to PDF is
+not a supported pair. Target extensions are inferred from the output path or
+selected with `--to`; `--format` selects response serialization. Conversion
+responses default to JSON and include `output.path`, `conversion.fidelity`
+and warnings. Existing outputs require `--force`. Failed and canceled
+conversions leave destinations intact.
+
+JSON/YAML/XML preserve the full response protocol. Markdown/HTML export document
+content with headings and tables; `--layout` adds a metadata appendix. CSV/TSV
+export a single table as rows, or layout blocks as records when `--layout` is
+requested. `layout` defaults to JSON. PDF coordinates use points and a top-left
+origin; image OCR uses orientation-corrected pixels. Embedded-image coordinates
+are local to each image. Flow documents expose logical structure, without
+invented page positions. Detection scores do not measure transcription accuracy.
+
 OCR output may contain errors, including stray text detected in landscape
 photos. `ok=true` means extraction completed; it does not measure OCR accuracy.
+
+Supported PDF ToUnicode errors are repaired per font in the shared WASM core;
+unusable text mappings can require OCR even with a nonempty text layer. Missing
+fonts and incorrect rendering may still make numeric tables unreliable.
+JSON responses contain local source paths and document text; redact them before
+sharing. Private evaluation inputs and raw reports are excluded from packages.
 EXIF orientation is applied, while a physically inverted image may still have
 its text lines returned in reversed order.
 
-Packages support macOS 12+ arm64 and Linux glibc 2.17+ x64/arm64. Only the current
+Version 0.3.0 provides Windows 10/11 x64, macOS 12+ arm64 and Linux glibc 2.17+ x64/arm64 packages. Windows/Linux native execution remains pending validation. Only the current
 platform is installed. Node.js 20.11+ is required for the npm launcher; native
 binaries also work directly. No Python, Homebrew or compiler is needed at runtime.
 RapidOCR defaults: max side 960, padding 50, orientation classification enabled,
@@ -61,4 +190,4 @@ one profile at a time. Platform packages are implementation components.
 [Protocol, limitations and source build](https://github.com/IceWhaleTech/filens/blob/main/docs/agent-cli.md)
 
 Locally built release tarballs can also be installed with
-`npm install -g /path/to/filens-0.1.0.tgz /path/to/filens-cli-PLATFORM-0.1.0.tgz /path/to/filens-ocr-PLATFORM-0.1.0.tgz`.
+`npm install -g /path/to/filens-cli-0.3.0.tgz /path/to/filens-cli-PLATFORM-0.3.0.tgz /path/to/filens-ocr-PLATFORM-0.3.0.tgz`.

package/THIRD-PARTY-NOTICES.md

@@ -7,7 +7,7 @@
   invokes it as an isolated optional capability; document parsing stays in WASM.
 - Tencent NCNN, source d0d50631179e380e587fb188338f5683d2c93275, BSD 3-Clause.
   License: native/licenses/NCNN.txt. OpenMP uses bundled SimpleOMP.
-- Linux companions include Zig, LLVM libc++, libc++abi and libunwind runtime
+- Linux/Windows companions include Zig and LLVM libc++ runtime components; Linux also includes libc++abi and libunwind runtime
   components. Their notices are in native/licenses/{Zig,libcxx,libcxxabi,libunwind}.txt.
   The macOS companion uses the operating system C++ library.
 - RapidAI RapidOcrNcnn 1.2.0 OCR pipeline; PaddleOCR models: PP-OCRv6 small
@@ -18,10 +18,50 @@
   https://github.com/RapidAI/RapidOcrNcnn
   https://github.com/PaddlePaddle/PaddleOCR
 
-The Go host and native wrapper also contain Go, purego, wazero, imaging,
+The Rust CLI uses Wasmi / wasmi_wasi 0.46.0 (MIT OR Apache-2.0),
+wasi-common (Apache-2.0 WITH LLVM-exception), libloading, serde / serde_json /
+serde_yaml, regex, flate2, png, tempfile and ctrlc. The locked dependencies'
+license texts and version metadata are collected into licenses/rust-cli.txt.
+PDFium remains the WASM artifact from go-pdfium v1.17.3, verified by SHA-256;
+its upstream license notice is included as go-pdfium@v1.17.3-LICENSE.txt.
+The optional image WASM engine uses jpeg-encoder and the repository's modified
+rawloader 0.37.2 (LGPL-2.1). RAW source and local changes are maintained at
+wasm/converter-image-rust/vendor/rawloader in the filens source repository;
+the package includes its LGPL and X3F notices in licenses/rust-cli.txt.
+
+The optional native OCR wrapper also contains Go, purego, imaging,
 gotokenizers, regexp2, protobuf and Go x/image/x/text components. Their notices
-are included in the package license directory. macOS wrapper builds use the
-pinned published source snapshot with deployment target 12.0.
+are included in the package license directory. All wrappers are rebuilt from the
+pinned published source snapshot with Go -trimpath. The macOS target is 12.0;
+Linux wrappers use Zig's glibc 2.17 target. wrapper-build.json records the
+source and rebuilt artifact hashes without workstation paths.
+Windows x64 uses Zig's GNU target, system-only DLL dependencies and bundled
+SimpleOMP. The NCNN source snapshot uses std::call_once for its Windows
+initialization; the wrapper uses LoadLibrary and NCNN's UTF-16 model-path APIs.
+These compatibility patches are maintained in scripts/build-agent-ncnn-windows.mjs
+and packaging/ncnn-windows/native_loader_windows.go.
+
+UPX 5.1.0 compresses Windows/Linux CLI executables. Its license explicitly
+permits distribution of packed programs without imposing its license on them:
+https://github.com/upx/upx/blob/v5.1.0/LICENSE
+The external UPX tool is needed only for source builds, not at runtime.
+
+The shared text WASM core uses lopdf 0.46.0 (MIT), ttf-parser 0.25.1
+(MIT OR Apache-2.0), and unicode-normalization 0.1.26
+((MIT OR Apache-2.0) AND Unicode-3.0) for PDF font mapping recovery.
+Their notices and those of the added transitive Rust components are included
+in the package license directory as pdf-unicode-rust.txt.
+
+Legacy Go library hosts retain their existing dependency notices, including
+wazero and go.yaml.in/yaml/v3; the distributed CLI itself is Rust.
+
+FiLens first-party code is licensed under MIT; see the package's LICENSE file.
+Bundled dependencies and the Photos application code retain their original
+licenses. This notice does not relicense those components.
+
+## Private directory discovery component
+
+Windows and Linux CLI binaries embed compiled hdrw WASI, a proprietary component used through the FiLens directory adapter. hdrw source is not included or licensed as part of FiLens. macOS release binaries use native system FTS and do not embed this component. Maintainer provenance and artifact hashes are recorded in `cli/assets/hdrw.json`; the adapter exposes mounted-directory enumeration, without raw device access.
 
-This notice describes third-party components and does not grant a new license
-for filens or the Photos application code.
+The embedded WASI module also includes Go standard library/runtime components;
+their license is included in the package license directory as Go.txt.