@woladi/sortai 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Adrian Wołczuk
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,193 @@
+# sortai
+
+> macOS CLI that walks a folder, OCRs every file with Apple Vision, and writes inferred Finder tags + comments — using a local Ollama model by default, or a cloud LLM with optional PII pseudonymisation.
+
+This is the TypeScript successor to the legacy Python `tagger_*.py` scripts. Native Swift OCR is now delegated to the [`macos-vision`](https://www.npmjs.com/package/macos-vision) package, so there is **no Python, no `swiftc`, no manual setup** — just `npx sortai`.
+
+## Requirements
+
+- macOS 12+
+- Node.js 20+
+- Xcode Command Line Tools (`xcode-select --install`) — needed by `macos-vision` to build its Swift binary at install time
+- One of:
+  - [Ollama](https://ollama.com) running locally (default) — keeps everything offline
+  - Anthropic or OpenAI API key — for cloud LLM with optional `--mask`
+
+## Quick start
+
+```bash
+# First run creates ~/.config/sortai/config.json with the default taxonomy
+npx sortai
+
+# Dry-run on the Desktop with local Ollama (default mistral-nemo)
+npx sortai ~/Desktop --dry-run
+
+# Actually write tags & comments
+npx sortai ~/Desktop
+```
+
+> The first invocation only writes the config and exits. Edit the file to fit your taxonomy, then re-run.
+
+### Cloud mode (optional)
+
+```bash
+# Anthropic Claude, with PII masked locally via pseudonym-mcp before the upstream call
+npx sortai ~/Desktop --cloud anthropic --mask --api-key sk-ant-...
+
+# OpenAI without masking (everything in the OCR'd text goes to the provider)
+ANTHROPIC_API_KEY=sk-ant-... npx sortai ~/Desktop --cloud openai
+```
+
+When `--mask` is set, `sortai` spawns [`pseudonym-mcp`](https://www.npmjs.com/package/pseudonym-mcp) over stdio, runs `mask_text` on the OCR'd text, sends the masked version to the cloud LLM, then `unmask_text` on the returned comment. Tags are taxonomy-bound and never round-trip through the cloud as user values.
+
+> **Pseudonymisation is a defence-in-depth control, not a compliance silver bullet.** Pseudonymised data is still personal data under GDPR Art. 4(5). Read the `pseudonym-mcp` README for the honest limitations.
+
+## How it works
+
+```
+folder (recursive walk, .dotfiles + excluded dirs skipped)
+   │
+   ▼
+dedup.ts: SHA256 over file bytes ← byte-identical groups → #Duplikat preTag
+   │
+   ▼  for each file
+macos-vision: ocr(path, { startPage, maxPages })
+                                ← Apple Vision OCR (PDF auto-rasterised, page-bounded)
+   │
+   ▼
+pretag.ts: PATH_TAG_RULES        ← regex rules from config
+   │
+   ▼  ≥4 pre-tags AND no OCR text → skip LLM (fast path)
+LLM tag/comment inference:
+   ├── default: local Ollama (mistral-nemo) — fully offline
+   └── --cloud anthropic|openai:
+         ├── --mask → pseudonym-mcp.mask_text(ocr)
+         ├── cloud LLM gets masked OCR text
+         └── --mask → pseudonym-mcp.unmask_text(comment)
+   │
+   ▼  strict-evidence validation, contextual guards (#CV vs financial, noOcr → no strict)
+   │  per-file 180s watchdog → fallback if a single call hangs
+macos.ts: xattr -wx + binary plist
+   ├── com.apple.metadata:_kMDItemUserTags    (Finder tags)
+   └── com.apple.metadata:kMDItemFinderComment (Finder comment)
+   ├── mdimport <file>                         (Spotlight reindex, fire-and-forget)
+```
+
+> Why not `osascript` + Finder `set tags`? It returns `-10006` on macOS 26+ (Tahoe). `xattr` + a binary plist is the same path the Python tagger used and works on every macOS version.
+
+## CLI flags
+
+| Flag | Default | Description |
+|------|---------|-------------|
+| `<folder>` | from config | Folder to scan recursively |
+| `--config <path>` | `~/.config/sortai/config.json` | Alternative config file |
+| `--dry-run` | off | Print results without writing tags/comments |
+| `--model <name>` | `mistral-nemo` (Ollama) | LLM model name |
+| `--ollama-url <url>` | `http://localhost:11434` | Ollama server |
+| `--cloud anthropic\|openai` | — | Switch to a cloud LLM |
+| `--api-key <key>` | env | API key for the cloud provider |
+| `--mask` | off | Pseudonymise OCR via pseudonym-mcp (only with `--cloud`) |
+| `--lang en\|pl` | `pl` | Language for pseudonym-mcp regex rules |
+| `--exclude <names>` | from config | Comma-separated folder names to skip |
+| `--limit <n>` | — | Process at most N files |
+| `--skip-tagged` | off | Skip files that already carry `cfg.tags.autoTag` (`#AI_Sorted` by default) |
+| `--no-dedup` | off | Skip SHA256 hashing pre-pass (no hash-based `#Duplikat`) |
+| `--verbose` | off | Extra logs |
+
+Environment variables: `SORTAI_API_KEY`, `ANTHROPIC_API_KEY`, `OPENAI_API_KEY`.
+
+## Configuration
+
+The default taxonomy that ships in `defaults.ts` is intentionally generic (`#Bank`, `#Faktura`, `#Umowa`, `#CV`, `#Wniosek`, `#Screenshot`, …) and is meant as a starting point. **Edit `~/.config/sortai/config.json` after the first run** to match your own categories — vendors, projects, clients, recurring matters.
+
+The config file is plain JSON. Sections:
+
+```json
+{
+  "scan": {
+    "folder": "~/Desktop",
+    "excludeFolders": ["node_modules", ".git", ".cache"],
+    "skipExtensions": [".ds_store", ".sig", ".localized", ".tmp", ".lock", ".pyc"],
+    "ocrExtensions": [".pdf", ".png", ".jpg", ".jpeg", ".webp", ".heic"],
+    "videoExtensions": [".mov", ".mp4", ".m4v"]
+  },
+  "ocr": { "maxChars": 4000, "llmMaxChars": 1500, "startPage": 1, "maxPages": 2 },
+  "llm": {
+    "provider": "ollama",
+    "model": "mistral-nemo",
+    "temperature": 0.15,
+    "numPredict": 300,
+    "ollamaUrl": "http://localhost:11434"
+  },
+  "mask":  { "enabled": false, "lang": "pl" },
+  "dedup": { "enabled": true,  "maxFileSizeMB": 200 },
+  "tags": {
+    "allowed":  ["#Bank", "#Faktura", "#Umowa", "#CV", "#Wniosek", "#AI_Sorted"],
+    "strict":   ["#Bank", "#Faktura", "#KartaKredytowa", "#Kredyt", "#RODO"],
+    "aliases":  { "#Invoice": "#Faktura", "#Mortgage": "#Kredyt", "#GDPR": "#RODO" },
+    "strictEvidence": {
+      "#Bank":    ["bank", "iban", "rachunek bankowy"],
+      "#Faktura": ["faktura", "invoice", " vat "]
+    },
+    "pathRules": [
+      { "pattern": "\\bbank\\b|iban|rachunek", "flags": "i", "tags": ["#Bank"] },
+      { "pattern": "faktura|invoice",          "flags": "i", "tags": ["#Faktura"] }
+    ],
+    "autoTag": "#AI_Sorted"
+  },
+  "context": "1-2 sentence description of yourself and ongoing matters — used by the LLM as background. Example: 'Self-employed designer in Warsaw, clients AcmeCorp + BetaInc.'"
+}
+```
+
+- `scan.folder` / `scan.excludeFolders` / `scan.skipExtensions` — what to walk, skip, and ignore by extension.
+- `ocr.maxChars` / `ocr.llmMaxChars` — cap on OCR text fed to the post-filter and to the LLM prompt.
+- `ocr.startPage` / `ocr.maxPages` — PDF page range (1-based). Default `1` / `2` only OCRs the first two pages; raise it for content-heavy documents.
+- `mask` — pseudonymisation toggle for `--cloud` (no-op without `--cloud`).
+- `dedup` — SHA256 duplicate detection (see below).
+- `tags.allowed` — set of tags the LLM is allowed to return; anything else is dropped.
+- `tags.strict` — subset of `allowed`. Strict tags only land on the file if at least one `strictEvidence` keyword appears verbatim in the OCR or filename.
+- `tags.aliases` — model-friendly normalisation (`#Invoice` → `#Faktura`).
+- `tags.pathRules` — regex patterns over `path.replace(/[\\/_-]/g, " ") + " " + ocrText`. Multiple rules can match; results merge into `preTags`.
+- `tags.autoTag` — appended to every successfully tagged file (sentinel so you can find "already processed" items in Finder and `--skip-tagged` works).
+- `context` — pinned to the system prompt as background knowledge. **Edit this** — the default is a placeholder.
+
+## Duplicate detection
+
+`sortai` ships two independent duplicate signals:
+
+- **`#Duplikat`** — SHA256 over file bytes, computed for every file before the main pipeline. Files in a group of ≥2 identical hashes all get this tag. Catches `cp foo bar`, sync conflicts, etc. — anything bit-identical regardless of name. Skipped for files > `cfg.dedup.maxFileSizeMB` (200 by default) and 0-byte files.
+- **`#PrawdopodobnaKopia`** — heuristic over filename + OCR: matches `copy`, `kopia`, `duplikat`, `(N)` in parentheses. Catches macOS Finder "Duplicate", Preview "Save As" copies, manual versioning — where the bytes differ (different `mtime`, repacked PDF, embedded timestamp) but the file is logically a copy.
+
+A file can carry both, one, or neither. Skip the hash pre-pass with `--no-dedup` if it's too slow on huge media libraries.
+
+## What about Markdown export?
+
+`sortai` is the *tagger*. If you want image/PDF → Markdown, use `macos-vision` directly:
+
+```bash
+npx macos-vision --markdown invoice.pdf -o invoice.md
+```
+
+That's the same Apple Vision + Ollama pipeline (VisionScribe), without the file-tagging layer.
+
+## Privacy
+
+- **Default (Ollama)**: nothing leaves your machine.
+- **`--cloud` without `--mask`**: the *full* OCR text of every scanned file is sent to your chosen provider. Use only when you trust the provider with the documents.
+- **`--cloud --mask`**: the OCR text is masked locally first; tokens like `[PERSON:1]`, `[PESEL:1]` flow to the cloud instead of literals. Structure, dates, amounts, and any PII the regex/LLM detector misses still travel. See [`pseudonym-mcp`](https://www.npmjs.com/package/pseudonym-mcp) for the full caveats.
+- File metadata is written via `osascript` (Apple Events). `sortai` makes no other network calls beyond your chosen LLM provider.
+
+## Development
+
+```bash
+git clone https://github.com/woladi/sortai.git
+cd sortai
+npm install            # macOS only; Linux/Windows needs --ignore-scripts to skip the native build
+npm run typecheck
+npm run build
+node dist/cli.js --help
+```
+
+## License
+
+MIT — Adrian Wołczuk

package/dist/cli.js

@@ -0,0 +1,261 @@
+#!/usr/bin/env node
+import { Command } from 'commander';
+import chalk from 'chalk';
+import ora from 'ora';
+import path from 'node:path';
+import { existsSync } from 'node:fs';
+import { expandHome, loadConfig } from './config.js';
+import { walkFiles } from './walker.js';
+import { extractOcrText } from './ocr.js';
+import { preTagFromPath } from './pretag.js';
+import { mergeTags } from './tags.js';
+import { writeFileMetadata } from './macos.js';
+import { Masker } from './mask.js';
+import { inferTagsAndComment } from './llm/index.js';
+import { findDuplicates } from './dedup.js';
+function applyOverrides(cfg, opts) {
+    const apiKey = opts.apiKey
+        ?? process.env.SORTAI_API_KEY
+        ?? (opts.cloud === 'anthropic' ? process.env.ANTHROPIC_API_KEY : undefined)
+        ?? (opts.cloud === 'openai' ? process.env.OPENAI_API_KEY : undefined);
+    const provider = opts.cloud ?? 'ollama';
+    const defaultCloudModels = {
+        anthropic: 'claude-sonnet-4-6',
+        openai: 'gpt-4o-mini',
+    };
+    return {
+        ...cfg,
+        scan: {
+            ...cfg.scan,
+            excludeFolders: opts.exclude
+                ? opts.exclude.split(',').map(s => s.trim()).filter(Boolean)
+                : cfg.scan.excludeFolders,
+        },
+        llm: {
+            ...cfg.llm,
+            provider,
+            model: opts.model ?? (opts.cloud ? defaultCloudModels[opts.cloud] ?? cfg.llm.model : cfg.llm.model),
+            ollamaUrl: opts.ollamaUrl ?? cfg.llm.ollamaUrl,
+            apiKey,
+        },
+        mask: {
+            ...cfg.mask,
+            enabled: opts.mask,
+            lang: opts.lang ?? cfg.mask.lang,
+        },
+        dedup: {
+            ...cfg.dedup,
+            enabled: opts.dedup,
+        },
+    };
+}
+async function main() {
+    const program = new Command();
+    program
+        .name('sortai')
+        .description('macOS CLI that tags files based on OCR + LLM-inferred Finder tags & comments')
+        .version('0.1.0')
+        .argument('[folder]', 'folder to scan (overrides config.scan.folder)')
+        .option('--config <path>', 'path to config JSON (default: ~/.config/sortai/config.json)')
+        .option('--dry-run', 'do not write tags/comments; just log', false)
+        .option('--model <name>', 'LLM model name (default depends on provider)')
+        .option('--ollama-url <url>', 'Ollama base URL (default: http://localhost:11434)')
+        .option('--cloud <provider>', "use a cloud LLM: 'anthropic' or 'openai'")
+        .option('--api-key <key>', 'API key for cloud provider (or env SORTAI_API_KEY / ANTHROPIC_API_KEY / OPENAI_API_KEY)')
+        .option('--mask', 'pseudonymise OCR via pseudonym-mcp before sending to cloud LLM', false)
+        .option('--lang <code>', "pseudonym-mcp language: 'en' | 'pl' (default: pl)")
+        .option('--exclude <patterns>', 'comma-separated folder names to skip (overrides config)')
+        .option('--limit <n>', 'process at most N files', v => parseInt(v, 10))
+        .option('--skip-tagged', 'skip files that already have the auto-tag (cfg.tags.autoTag)', false)
+        .option('--no-dedup', 'skip SHA256 duplicate detection (hash-based #Duplikat tag)')
+        .option('--verbose', 'extra logs', false)
+        .parse(process.argv);
+    const opts = program.opts();
+    const folderArg = program.args[0];
+    if (opts.cloud && !['anthropic', 'openai'].includes(opts.cloud)) {
+        process.stderr.write(chalk.red(`Unknown --cloud provider: ${opts.cloud}\n`));
+        process.exit(1);
+    }
+    if (opts.mask && !opts.cloud) {
+        process.stderr.write(chalk.yellow('⚠️  --mask without --cloud is a no-op (local Ollama already keeps data offline).\n'));
+    }
+    let configResult;
+    try {
+        configResult = await loadConfig(opts.config);
+    }
+    catch (err) {
+        const msg = err instanceof Error ? err.message : String(err);
+        process.stderr.write(chalk.red(`Config error: ${msg}\n`));
+        process.exit(1);
+    }
+    if (configResult.created) {
+        process.stdout.write(chalk.green(`✨ Created default config at ${configResult.path}\n`));
+        process.stdout.write('   Edit it to customise tags, then re-run.\n');
+        return;
+    }
+    const cfg = applyOverrides(configResult.config, opts);
+    if (cfg.llm.provider !== 'ollama' && !cfg.llm.apiKey) {
+        process.stderr.write(chalk.red(`Missing API key for ${cfg.llm.provider}. Pass --api-key or set the env var.\n`));
+        process.exit(1);
+    }
+    const rawFolder = folderArg ?? cfg.scan.folder;
+    const root = path.resolve(expandHome(rawFolder));
+    if (!existsSync(root)) {
+        process.stderr.write(chalk.red(`Folder does not exist: ${root}\n`));
+        process.exit(1);
+    }
+    let masker;
+    if (cfg.mask.enabled && cfg.llm.provider !== 'ollama') {
+        masker = new Masker(cfg);
+        const spin = ora('Starting pseudonym-mcp…').start();
+        try {
+            await masker.connect();
+            spin.succeed('pseudonym-mcp ready');
+        }
+        catch (err) {
+            spin.fail(err instanceof Error ? err.message : String(err));
+            masker = undefined;
+        }
+    }
+    process.stdout.write(chalk.cyan(`🚀 Scanning ${root}\n`));
+    process.stdout.write(`   Provider: ${cfg.llm.provider} (${cfg.llm.model})`);
+    if (cfg.mask.enabled && masker)
+        process.stdout.write(chalk.gray(' [masked]'));
+    if (opts.dryRun)
+        process.stdout.write(chalk.yellow(' [dry-run]'));
+    process.stdout.write('\n');
+    if (cfg.scan.excludeFolders.length) {
+        process.stdout.write(chalk.gray(`   Excluded: ${cfg.scan.excludeFolders.join(', ')}\n`));
+    }
+    process.stdout.write('\n');
+    let allFiles = await walkFiles(root, cfg);
+    process.stdout.write(`📁 Files: ${allFiles.length}\n`);
+    if (opts.skipTagged) {
+        const { execFile } = await import('node:child_process');
+        const { promisify } = await import('node:util');
+        const exec = promisify(execFile);
+        const before = allFiles.length;
+        const filtered = [];
+        for (const f of allFiles) {
+            try {
+                const { stdout: md } = await exec('mdls', ['-name', 'kMDItemUserTags', '-raw', f], { timeout: 3_000 });
+                if (!md.includes(cfg.tags.autoTag))
+                    filtered.push(f);
+            }
+            catch {
+                filtered.push(f);
+            }
+        }
+        allFiles = filtered;
+        process.stdout.write(chalk.gray(`   Skip-tagged: ${before - allFiles.length} pominięte, ${allFiles.length} do przetworzenia\n`));
+    }
+    if (opts.limit && opts.limit > 0 && allFiles.length > opts.limit) {
+        allFiles = allFiles.slice(0, opts.limit);
+        process.stdout.write(chalk.gray(`   Limit: ${opts.limit} plików\n`));
+    }
+    let dedup;
+    if (cfg.dedup.enabled && allFiles.length > 1) {
+        process.stdout.write(chalk.gray(`🔢 Hashing ${allFiles.length} files for dedup…\n`));
+        dedup = await findDuplicates(allFiles, cfg);
+        process.stdout.write(chalk.gray(`   Hashed: ${dedup.hashedFiles}, skipped >${cfg.dedup.maxFileSizeMB}MB: ${dedup.skippedLarge}, ` +
+            `duplicate groups: ${dedup.totalGroups}, files in groups: ${dedup.totalDuplicates}\n`));
+    }
+    process.stdout.write('\n');
+    const stats = { ok: 0, preOnly: 0, skipped: 0, errors: 0, total: allFiles.length };
+    const skipExt = new Set(cfg.scan.skipExtensions);
+    const ocrExt = new Set(cfg.scan.ocrExtensions);
+    const videoExt = new Set(cfg.scan.videoExtensions);
+    for (const filePath of allFiles) {
+        const rel = path.relative(root, filePath);
+        const name = path.basename(filePath);
+        const ext = path.extname(filePath).toLowerCase();
+        if (skipExt.has(ext)) {
+            stats.skipped++;
+            continue;
+        }
+        process.stdout.write(chalk.bold(`🔍 ${rel}\n`));
+        let ocrText = '';
+        if (ocrExt.has(ext)) {
+            process.stdout.write('  📖 OCR…');
+            ocrText = await extractOcrText(filePath, cfg);
+            const words = ocrText.split(/\s+/).filter(Boolean).length;
+            process.stdout.write(` ${words} words\n`);
+        }
+        else if (videoExt.has(ext)) {
+            process.stdout.write('  🎬 Video\n');
+        }
+        else {
+            process.stdout.write(`  📄 ${ext}\n`);
+        }
+        const preTagsBase = preTagFromPath(filePath, ocrText, cfg);
+        const dupGroup = dedup?.groupByFile.get(filePath);
+        const preTags = dupGroup ? mergeTags(cfg, preTagsBase, ['#Duplikat']) : preTagsBase;
+        if (dupGroup) {
+            const others = dupGroup.files.filter(f => f !== filePath).map(f => path.basename(f));
+            process.stdout.write(chalk.magenta(`  🧬 Duplicate of: ${others.join(', ')}\n`));
+        }
+        let finalTags;
+        let finalComment;
+        if (preTags.length >= 4 && !ocrText.trim()) {
+            finalTags = mergeTags(cfg, preTags, [cfg.tags.autoTag]);
+            finalComment = `Auto z nazwy/ścieżki: ${name}.`;
+            process.stdout.write(chalk.gray(`  ⚡ Pre-only: ${preTags.join(' ')}\n`));
+            stats.preOnly++;
+        }
+        else {
+            process.stdout.write(chalk.gray(`  🧠 ${cfg.llm.provider}…\n`));
+            const PER_FILE_TIMEOUT_MS = 180_000;
+            let timer;
+            const timeout = new Promise((_, reject) => {
+                timer = setTimeout(() => reject(new Error(`per-file timeout after ${PER_FILE_TIMEOUT_MS}ms`)), PER_FILE_TIMEOUT_MS);
+            });
+            try {
+                const result = await Promise.race([
+                    inferTagsAndComment({ fileName: name, ext, preTags, ocrText }, cfg, masker),
+                    timeout,
+                ]);
+                finalTags = mergeTags(cfg, result.tags, [cfg.tags.autoTag]);
+                finalComment = result.comment || `Plik: ${name}.`;
+            }
+            catch (err) {
+                const msg = err instanceof Error ? err.message : String(err);
+                process.stdout.write(chalk.yellow(`  ⏱  ${msg} — fallback\n`));
+                finalTags = mergeTags(cfg, preTags, [cfg.tags.autoTag]).slice(0, 6);
+                finalComment = `Plik: ${name}.`;
+            }
+            finally {
+                if (timer)
+                    clearTimeout(timer);
+            }
+        }
+        if (opts.dryRun) {
+            process.stdout.write(chalk.green(`  ✅ ${finalTags.join(' ')}\n`));
+            process.stdout.write(chalk.gray(`  📝 ${finalComment}\n\n`));
+            stats.ok++;
+            continue;
+        }
+        try {
+            await writeFileMetadata(filePath, finalTags, finalComment);
+            process.stdout.write(chalk.green(`  ✅ ${finalTags.join(' ')}\n`));
+            process.stdout.write(chalk.gray(`  📝 ${finalComment}\n\n`));
+            stats.ok++;
+        }
+        catch (err) {
+            const msg = err instanceof Error ? err.message : String(err);
+            process.stdout.write(chalk.red(`  ❌ Write failed: ${msg}\n\n`));
+            stats.errors++;
+        }
+    }
+    if (masker)
+        await masker.close();
+    process.stdout.write('═══════════════════════════════════════════════════════\n');
+    process.stdout.write(chalk.bold('✨ Done\n'));
+    process.stdout.write(chalk.green(`   ✅ Success:       ${stats.ok}\n`));
+    process.stdout.write(chalk.gray(`   ⚡ Pre-only:       ${stats.preOnly}\n`));
+    process.stdout.write(chalk.gray(`   ⏭  Skipped:       ${stats.skipped}\n`));
+    process.stdout.write(chalk.red(`   ❌ Errors:        ${stats.errors}\n`));
+}
+main().catch(err => {
+    process.stderr.write(chalk.red(`Fatal: ${err instanceof Error ? err.message : String(err)}\n`));
+    process.exit(1);
+});

package/dist/config.js

@@ -0,0 +1,70 @@
+import { promises as fs } from 'node:fs';
+import { existsSync } from 'node:fs';
+import path from 'node:path';
+import os from 'node:os';
+import { z } from 'zod';
+import { DEFAULT_CONFIG } from './defaults.js';
+const PathRuleSchema = z.object({
+    pattern: z.string(),
+    flags: z.string().optional(),
+    tags: z.array(z.string()),
+});
+const ConfigSchema = z.object({
+    scan: z.object({
+        folder: z.string(),
+        excludeFolders: z.array(z.string()),
+        skipExtensions: z.array(z.string()),
+        ocrExtensions: z.array(z.string()),
+        videoExtensions: z.array(z.string()),
+    }),
+    ocr: z.object({
+        maxChars: z.number().int().positive(),
+        llmMaxChars: z.number().int().positive(),
+        startPage: z.number().int().positive().default(1),
+        maxPages: z.number().int().positive().default(2),
+    }),
+    llm: z.object({
+        provider: z.enum(['ollama', 'anthropic', 'openai']),
+        model: z.string(),
+        temperature: z.number(),
+        numPredict: z.number().int().positive(),
+        ollamaUrl: z.string().url(),
+        apiKey: z.string().optional(),
+    }),
+    mask: z.object({
+        enabled: z.boolean(),
+        lang: z.enum(['en', 'pl']),
+    }),
+    dedup: z.object({
+        enabled: z.boolean().default(true),
+        maxFileSizeMB: z.number().int().positive().default(200),
+    }).default({ enabled: true, maxFileSizeMB: 200 }),
+    tags: z.object({
+        allowed: z.array(z.string()),
+        strict: z.array(z.string()),
+        aliases: z.record(z.string(), z.string()),
+        strictEvidence: z.record(z.string(), z.array(z.string())),
+        pathRules: z.array(PathRuleSchema),
+        autoTag: z.string(),
+    }),
+    context: z.string(),
+});
+export const DEFAULT_CONFIG_PATH = path.join(process.env.XDG_CONFIG_HOME ?? path.join(os.homedir(), '.config'), 'sortai', 'config.json');
+export function expandHome(p) {
+    if (p.startsWith('~')) {
+        return path.join(os.homedir(), p.slice(1));
+    }
+    return p;
+}
+export async function loadConfig(customPath) {
+    const cfgPath = customPath ? path.resolve(expandHome(customPath)) : DEFAULT_CONFIG_PATH;
+    if (!existsSync(cfgPath)) {
+        await fs.mkdir(path.dirname(cfgPath), { recursive: true });
+        await fs.writeFile(cfgPath, JSON.stringify(DEFAULT_CONFIG, null, 2), 'utf8');
+        return { config: DEFAULT_CONFIG, path: cfgPath, created: true };
+    }
+    const raw = await fs.readFile(cfgPath, 'utf8');
+    const parsed = JSON.parse(raw);
+    const config = ConfigSchema.parse(parsed);
+    return { config, path: cfgPath, created: false };
+}

package/dist/dedup.js

@@ -0,0 +1,64 @@
+import { promises as fs, createReadStream } from 'node:fs';
+import { createHash } from 'node:crypto';
+async function hashFile(filePath) {
+    return new Promise((resolve, reject) => {
+        const hash = createHash('sha256');
+        const stream = createReadStream(filePath);
+        stream.on('data', chunk => hash.update(chunk));
+        stream.on('end', () => resolve(hash.digest('hex')));
+        stream.on('error', reject);
+    });
+}
+export async function findDuplicates(files, cfg) {
+    const empty = {
+        groupByFile: new Map(),
+        totalGroups: 0,
+        totalDuplicates: 0,
+        skippedLarge: 0,
+        hashedFiles: 0,
+    };
+    if (!cfg.dedup.enabled || files.length < 2)
+        return empty;
+    const maxBytes = cfg.dedup.maxFileSizeMB * 1024 * 1024;
+    const hashes = new Map();
+    let skippedLarge = 0;
+    let hashedFiles = 0;
+    const BATCH = 8;
+    for (let i = 0; i < files.length; i += BATCH) {
+        const batch = files.slice(i, i + BATCH);
+        await Promise.all(batch.map(async (f) => {
+            try {
+                const st = await fs.stat(f);
+                if (!st.isFile())
+                    return;
+                if (st.size > maxBytes) {
+                    skippedLarge++;
+                    return;
+                }
+                if (st.size === 0)
+                    return;
+                const h = await hashFile(f);
+                if (!hashes.has(h))
+                    hashes.set(h, []);
+                hashes.get(h).push(f);
+                hashedFiles++;
+            }
+            catch {
+                // unreadable or transient; skip silently
+            }
+        }));
+    }
+    const groupByFile = new Map();
+    let totalGroups = 0;
+    let totalDuplicates = 0;
+    for (const [hash, paths] of hashes) {
+        if (paths.length > 1) {
+            totalGroups++;
+            totalDuplicates += paths.length;
+            const group = { hash, files: paths };
+            for (const p of paths)
+                groupByFile.set(p, group);
+        }
+    }
+    return { groupByFile, totalGroups, totalDuplicates, skippedLarge, hashedFiles };
+}