hazo_files 1.4.6 → 1.5.1

package/README.md

@@ -20,6 +20,7 @@ A powerful, modular file management package for Node.js and React applications w
 - **File Change Detection**: xxHash-based content hashing for efficient change detection
 - **Content Tagging**: Optional LLM-based content classification at upload time or on-demand via `content_tag` field
 - **Schema Migrations**: Built-in V2/V3 migration utilities for adding reference tracking and content tagging to existing databases
+- **Background Upload Pipelines**: Framework-agnostic `UploadManager` + React `HazoFileUploadProvider` for multi-step upload pipelines that survive component unmount, with optional sonner toast bridge
 - **TypeScript**: Full type safety and IntelliSense support
 - **OAuth Integration**: Built-in Google Drive and Dropbox OAuth authentication
 - **Prompt Cache Invalidation**: Passthrough for hazo_llm_api prompt cache management via server instance
@@ -61,6 +62,12 @@ npm install server-only       # Server-side safety (recommended)
 npm install xxhash-wasm       # File change detection (optional)
 ```
 
+For the background-upload sonner toast bridge (optional):
+
+```bash
+npm install sonner            # Toast notifications for background upload pipelines
+```
+
 ### Tailwind CSS v4 Setup (Required for UI Components)
 
 If you're using Tailwind CSS v4 with the UI components, you must add a `@source` directive to your CSS file to ensure Tailwind scans the package's files for utility classes.
@@ -510,6 +517,167 @@ const fileManager = await createInitializedFileManager({
 });
 ```
 
+## Database Schema
+
+Database tables are only required if you use `TrackedFileManager`, `FileMetadataService`, `NamingConventionService`, or `UploadExtractService`. Plain `FileManager` (filesystem only) needs no tables.
+
+There are two tables:
+
+- **`hazo_files`** — file metadata, hashes, references, content tags
+- **`hazo_files_naming`** — saved naming conventions
+
+The DDL below is also exposed programmatically via `HAZO_FILES_TABLE_SCHEMA` and `HAZO_FILES_NAMING_TABLE_SCHEMA` (see [Programmatic Setup](#programmatic-setup) below). Run the raw SQL if you prefer to manage migrations with your existing tooling (psql, sqlite3, Flyway, Knex, etc.).
+
+### `hazo_files` Table
+
+#### PostgreSQL
+
+```sql
+CREATE TABLE IF NOT EXISTS hazo_files (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  filename TEXT NOT NULL,
+  file_type TEXT NOT NULL,
+  file_data TEXT DEFAULT '{}',
+  created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
+  changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
+  file_path TEXT NOT NULL,
+  storage_type TEXT NOT NULL,
+  file_hash TEXT,
+  file_size BIGINT,
+  file_changed_at TIMESTAMP WITH TIME ZONE,
+  file_refs TEXT DEFAULT '[]',
+  ref_count INTEGER DEFAULT 0,
+  status TEXT DEFAULT 'active',
+  scope_id UUID,
+  uploaded_by UUID,
+  storage_verified_at TIMESTAMP WITH TIME ZONE,
+  deleted_at TIMESTAMP WITH TIME ZONE,
+  original_filename TEXT,
+  content_tag TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_hazo_files_path ON hazo_files (file_path);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_storage ON hazo_files (storage_type);
+CREATE UNIQUE INDEX IF NOT EXISTS idx_hazo_files_path_storage ON hazo_files (file_path, storage_type);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_hash ON hazo_files (file_hash);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_status ON hazo_files (status);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_scope ON hazo_files (scope_id);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_ref_count ON hazo_files (ref_count);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_deleted ON hazo_files (deleted_at);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_content_tag ON hazo_files (content_tag);
+```
+
+#### SQLite
+
+```sql
+CREATE TABLE IF NOT EXISTS hazo_files (
+  id TEXT PRIMARY KEY,
+  filename TEXT NOT NULL,
+  file_type TEXT NOT NULL,
+  file_data TEXT DEFAULT '{}',
+  created_at TEXT NOT NULL,
+  changed_at TEXT NOT NULL,
+  file_path TEXT NOT NULL,
+  storage_type TEXT NOT NULL,
+  file_hash TEXT,
+  file_size INTEGER,
+  file_changed_at TEXT,
+  file_refs TEXT DEFAULT '[]',
+  ref_count INTEGER DEFAULT 0,
+  status TEXT DEFAULT 'active',
+  scope_id TEXT,
+  uploaded_by TEXT,
+  storage_verified_at TEXT,
+  deleted_at TEXT,
+  original_filename TEXT,
+  content_tag TEXT
+);
+
+CREATE INDEX IF NOT EXISTS idx_hazo_files_path ON hazo_files (file_path);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_storage ON hazo_files (storage_type);
+CREATE UNIQUE INDEX IF NOT EXISTS idx_hazo_files_path_storage ON hazo_files (file_path, storage_type);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_hash ON hazo_files (file_hash);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_status ON hazo_files (status);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_scope ON hazo_files (scope_id);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_ref_count ON hazo_files (ref_count);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_deleted ON hazo_files (deleted_at);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_content_tag ON hazo_files (content_tag);
+```
+
+### `hazo_files_naming` Table
+
+Required only if you use `NamingConventionService` to persist saved naming rules.
+
+#### PostgreSQL
+
+```sql
+CREATE TABLE IF NOT EXISTS hazo_files_naming (
+  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
+  scope_id UUID,
+  naming_title TEXT NOT NULL,
+  naming_type TEXT NOT NULL CHECK(naming_type IN ('file', 'folder', 'both')),
+  naming_value TEXT NOT NULL,
+  created_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
+  changed_at TIMESTAMP WITH TIME ZONE NOT NULL DEFAULT NOW(),
+  variables TEXT DEFAULT '[]'
+);
+
+CREATE INDEX IF NOT EXISTS idx_hazo_files_naming_scope ON hazo_files_naming (scope_id);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_naming_type ON hazo_files_naming (naming_type);
+```
+
+#### SQLite
+
+```sql
+CREATE TABLE IF NOT EXISTS hazo_files_naming (
+  id TEXT PRIMARY KEY,
+  scope_id TEXT,
+  naming_title TEXT NOT NULL,
+  naming_type TEXT NOT NULL CHECK(naming_type IN ('file', 'folder', 'both')),
+  naming_value TEXT NOT NULL,
+  created_at TEXT NOT NULL,
+  changed_at TEXT NOT NULL,
+  variables TEXT DEFAULT '[]'
+);
+
+CREATE INDEX IF NOT EXISTS idx_hazo_files_naming_scope ON hazo_files_naming (scope_id);
+CREATE INDEX IF NOT EXISTS idx_hazo_files_naming_type ON hazo_files_naming (naming_type);
+```
+
+### Programmatic Setup
+
+The same DDL is available as exported constants so you can run it from your app's startup or migration script without hand-copying SQL:
+
+```typescript
+import {
+  HAZO_FILES_TABLE_SCHEMA,
+  HAZO_FILES_NAMING_TABLE_SCHEMA,
+} from 'hazo_files';
+
+// Pick 'sqlite' or 'postgres'
+const dbType: 'sqlite' | 'postgres' = 'sqlite';
+
+// Create hazo_files table
+await db.run(HAZO_FILES_TABLE_SCHEMA[dbType].ddl);
+for (const idx of HAZO_FILES_TABLE_SCHEMA[dbType].indexes) {
+  await db.run(idx);
+}
+
+// Create hazo_files_naming table (only if using NamingConventionService)
+await db.run(HAZO_FILES_NAMING_TABLE_SCHEMA[dbType].ddl);
+for (const idx of HAZO_FILES_NAMING_TABLE_SCHEMA[dbType].indexes) {
+  await db.run(idx);
+}
+```
+
+For PostgreSQL, swap `db.run(...)` for `client.query(...)`.
+
+To use a custom table name, see `getSchemaForTable(name, dbType)` and `getNamingSchemaForTable(name, dbType)`.
+
+### Upgrading Existing Tables
+
+If you already have a pre-V2 or pre-V3 `hazo_files` table, see [Database Migration (Existing Databases)](#database-migration-existing-databases) and [V3 Database Migration](#v3-database-migration) for the `ALTER TABLE` scripts and migration helpers.
+
 ## UI Components
 
 ### FileBrowser Component
@@ -1440,6 +1608,157 @@ const { fileManager, metadataService, namingService, extractionService, uploadEx
 invalidatePromptCache?.('classification', 'classify_document');
 ```
 
+## Background Upload Pipelines
+
+A framework-agnostic upload pipeline engine that survives React component unmount. Useful when uploads include multi-step server work (upload → LLM extract → user confirmation → DB commit) and the user may navigate away mid-flight.
+
+Two subpath exports:
+
+- `hazo_files/background-upload` — core, no React dependency (`UploadManager`, `Job`, `PipelineExecutor`, `TypedEventEmitter`, all types)
+- `hazo_files/background-upload/react` — React bindings (`HazoFileUploadProvider`, `useFileUpload`, `useJobStatus`, `useFileUploadToasts`)
+
+### Core API (framework-agnostic)
+
+```typescript
+import { UploadManager } from 'hazo_files/background-upload';
+import type { PipelineStep, PipelineContext, JobHandle } from 'hazo_files/background-upload';
+
+const uploadStep: PipelineStep = {
+  name: 'upload',
+  async execute(ctx: PipelineContext, handle: JobHandle) {
+    handle.set_status('uploading');
+    for (let i = 0; i < ctx.files.length; i++) {
+      // ... POST file to your /api/files/upload route
+      handle.set_progress(i + 1, ctx.files.length);
+    }
+  },
+};
+
+const extractStep: PipelineStep = {
+  name: 'extract',
+  async execute(ctx, handle) {
+    handle.set_status('processing');
+    const extracted = await fetch('/api/extract', { /* ... */ }).then(r => r.json());
+    ctx.extracted_data = extracted;
+  },
+};
+
+const manager = new UploadManager({
+  max_concurrent: 2,
+  default_pipeline_steps: [uploadStep, extractStep],
+});
+
+manager.on('job:completed', ({ job }) => console.log('done', job.job_id));
+
+const batch_id = manager.submit_batch({
+  files: [file1, file2],
+  group_id: 'project-123',
+  group_label: 'Q4 Tax Documents',
+});
+```
+
+### React Provider + Hooks
+
+```tsx
+// app/layout.tsx (Next.js) or your app root
+'use client';
+import { HazoFileUploadProvider } from 'hazo_files/background-upload/react';
+import { Toaster } from 'sonner';
+
+export default function RootLayout({ children }: { children: React.ReactNode }) {
+  return (
+    <HazoFileUploadProvider config={{ max_concurrent: 2 }}>
+      <Toaster richColors />
+      {children}
+    </HazoFileUploadProvider>
+  );
+}
+```
+
+```tsx
+'use client';
+import { useFileUpload, useJobStatus } from 'hazo_files/background-upload/react';
+
+export function UploadButton() {
+  const { submit_batch, active_jobs } = useFileUpload();
+
+  function onPick(files: FileList) {
+    submit_batch({
+      files: Array.from(files),
+      group_id: 'project-123',
+      group_label: 'Project 123',
+      pipeline_steps: [/* your PipelineStep[] */],
+    });
+  }
+
+  return (
+    <>
+      <input type="file" multiple onChange={(e) => onPick(e.target.files!)} />
+      <ul>
+        {active_jobs.map((j) => (
+          <li key={j.job_id}>{j.group_label}: {j.status}</li>
+        ))}
+      </ul>
+    </>
+  );
+}
+
+// Track a single job
+export function JobBadge({ job_id }: { job_id: string }) {
+  const job = useJobStatus(job_id);
+  if (!job) return null;
+  return <span>{job.status} {job.progress && `${job.progress.current}/${job.progress.total}`}</span>;
+}
+```
+
+### Confirmation Steps (user-in-the-loop)
+
+```typescript
+const confirmStep: PipelineStep = {
+  name: 'confirm',
+  async execute(ctx, handle) {
+    handle.set_status('awaiting_confirmation');
+    const result = await handle.request_confirmation({
+      conflicts: ctx.extracted_data.conflicts,
+    });
+    if (!result.confirmed) throw new Error('User cancelled');
+    // ctx.extracted_data now reflects user choices via result.data
+  },
+};
+```
+
+In the UI, subscribe to `job:confirmation_needed` (or read jobs in `awaiting_confirmation` status), render a dialog, then:
+
+```typescript
+const { resolve_confirmation } = useFileUpload();
+resolve_confirmation(job_id, { confirmed: true, data: userChoices });
+```
+
+### Sonner Toast Bridge
+
+The provider mounts a `ToastBridge` by default (`enable_toasts={true}`) that uses sonner to notify on `job:completed`, `job:error`, `job:confirmation_needed`, and `batch:completed`. Sonner is a soft optional peer dependency — if it isn't installed, the bridge is a no-op. Set `enable_toasts={false}` on the provider to opt out, or wire `useFileUploadToasts(manager)` yourself for custom toast behavior.
+
+### Events
+
+| Event | Payload | Fired when |
+|-------|---------|------------|
+| `job:created` | `{ job }` | Job enters the queue |
+| `job:status_changed` | `{ job, previous_status }` | Status transitions (queued → uploading → processing → ...) |
+| `job:progress` | `{ job }` | `handle.set_progress` called inside a pipeline step |
+| `job:completed` | `{ job }` | All pipeline steps finished successfully |
+| `job:error` | `{ job, error }` | A pipeline step threw |
+| `job:confirmation_needed` | `{ job, payload }` | `handle.request_confirmation` called |
+| `job:confirmation_resolved` | `{ job, result }` | `resolve_confirmation` called |
+| `batch:progress` | `{ batch }` | Any job in the batch settles |
+| `batch:completed` | `{ batch }` | All jobs in the batch are `done` or `error` |
+
+### Design Notes
+
+- **Survives unmount**: `UploadManager` lives on a `useRef`; pipelines run on the manager, not on React state. Navigating away does not abort uploads.
+- **Single source of truth**: `useFileUpload` / `useJobStatus` subscribe via `useSyncExternalStore` against the manager's event emitter, so multiple components stay consistent.
+- **Concurrency**: `max_concurrent` controls how many jobs the executor runs in parallel; the rest wait in the FIFO queue.
+- **No DB writes**: This module is purely an in-memory pipeline runner — your pipeline steps own all server I/O.
+
 ## API Reference
 
 ### FileManager