hazo_files 1.4.1 → 1.4.3

package/README.md

@@ -16,7 +16,10 @@ A powerful, modular file management package for Node.js and React applications w
 - **Extraction Data Management**: Track and manage LLM-extracted metadata with merge strategies
 - **LLM Integration**: Built-in support for hazo_llm_api document/image extraction
 - **Upload + Extract Workflow**: Combined service for uploading files with automatic LLM extraction and naming
+- **File Reference Tracking**: Multi-entity file references with orphan detection, soft delete, and lifecycle management
 - **File Change Detection**: xxHash-based content hashing for efficient change detection
+- **Content Tagging**: Optional LLM-based content classification at upload time or on-demand via `content_tag` field
+- **Schema Migrations**: Built-in V2/V3 migration utilities for adding reference tracking and content tagging to existing databases
 - **TypeScript**: Full type safety and IntelliSense support
 - **OAuth Integration**: Built-in Google Drive OAuth authentication
 - **Progress Tracking**: Upload/download progress callbacks
@@ -1106,11 +1109,17 @@ const extractionService = new LLMExtractionService((provider, options) => {
   return createLLM({ provider, ...options });
 }, 'gemini');
 
-// Create upload + extract service
+// Create upload + extract service (with optional content tag config)
 const uploadExtract = new UploadExtractService(
   trackedFileManager,
   namingService,
-  extractionService
+  extractionService,
+  {
+    content_tag_set_by_llm: true,
+    content_tag_prompt_area: 'classification',
+    content_tag_prompt_key: 'classify_document',
+    content_tag_prompt_return_fieldname: 'document_type',
+  }
 );
 
 // Upload with extraction and naming convention
@@ -1138,6 +1147,8 @@ if (result.success) {
   console.log('Uploaded to:', result.generatedPath);
   // e.g., '/documents/2024/ACME/ACME_Q4_2024-12-09_001.pdf'
   console.log('Extracted data:', result.extraction?.data);
+  console.log('Content tag:', result.contentTag);
+  // e.g., 'invoice', 'report', 'contract'
 }
 
 // Generate path preview without uploading
@@ -1192,6 +1203,169 @@ const autoResult = await extractionService.extract(
 );
 ```
 
+## Content Tagging
+
+Automatically classify uploaded files using LLM-based content analysis. The `content_tag` field stores a classification string (e.g., "invoice", "report", "contract") determined by an LLM prompt.
+
+### Configuration
+
+```typescript
+import type { ContentTagConfig } from 'hazo_files';
+
+const contentTagConfig: ContentTagConfig = {
+  content_tag_set_by_llm: true,
+  content_tag_prompt_area: 'classification',
+  content_tag_prompt_key: 'classify_document',
+  content_tag_prompt_return_fieldname: 'document_type',
+  content_tag_prompt_variables: { language: 'en' }, // optional
+};
+```
+
+### Automatic Tagging at Upload
+
+Pass `contentTagConfig` to `UploadExtractService` constructor (default for all uploads) or per-upload via options:
+
+```typescript
+// Per-upload override
+const result = await uploadExtract.uploadWithExtract(buffer, 'file.pdf', {
+  basePath: '/docs',
+  contentTagConfig: {
+    content_tag_set_by_llm: true,
+    content_tag_prompt_area: 'classification',
+    content_tag_prompt_key: 'classify_document',
+    content_tag_prompt_return_fieldname: 'document_type',
+  },
+});
+console.log(result.contentTag); // e.g., 'invoice'
+```
+
+### Manual Tagging
+
+Tag existing files by their database record ID:
+
+```typescript
+const tagResult = await uploadExtract.tagFileContent('file-record-id');
+if (tagResult.success) {
+  console.log('Tagged as:', tagResult.data);
+}
+```
+
+### V3 Database Migration
+
+If you have an existing `hazo_files` table, run the V3 migration to add the `content_tag` column:
+
+```typescript
+import { migrateToV3, HAZO_FILES_MIGRATION_V3 } from 'hazo_files';
+
+// Using the migration helper
+await migrateToV3(
+  { run: (sql) => db.run(sql) },
+  'sqlite'
+);
+
+// Or run statements manually
+for (const stmt of HAZO_FILES_MIGRATION_V3.sqlite.alterStatements) {
+  try { await db.run(stmt); } catch { /* column exists */ }
+}
+```
+
+New tables created with `HAZO_FILES_TABLE_SCHEMA` already include the `content_tag` column.
+
+## File Reference Tracking
+
+Track which entities (form fields, chat messages, etc.) reference each file. Multiple entities can reference the same file, enabling shared files without duplication.
+
+### Adding and Removing References
+
+```typescript
+import { TrackedFileManager } from 'hazo_files';
+
+// Upload a file with an initial reference
+const result = await trackedManager.uploadFileWithRef(buffer, '/docs/report.pdf', {
+  scope_id: 'workspace-123',
+  uploaded_by: 'user-456',
+  ref: {
+    entity_type: 'form_field',
+    entity_id: 'field-789',
+    created_by: 'user-456',
+  },
+});
+// result.data.file_id, result.data.ref_id
+
+// Add another reference to the same file
+await trackedManager.addRef(fileId, {
+  entity_type: 'chat_message',
+  entity_id: 'msg-abc',
+});
+
+// Remove a specific reference
+const { remaining_refs } = await trackedManager.removeRef(fileId, refId);
+
+// Get file with status info
+const fileStatus = await trackedManager.getFileById(fileId);
+// { record, refs: FileRef[], is_orphaned: boolean }
+```
+
+### Orphan Detection and Cleanup
+
+```typescript
+// Find files with zero references
+const orphans = await trackedManager.findOrphanedFiles({
+  olderThanMs: 7 * 24 * 60 * 60 * 1000, // 7 days old
+  scope_id: 'workspace-123',
+});
+
+// Clean up orphaned files (delete physical files + DB records)
+const { cleaned, errors } = await trackedManager.cleanupOrphanedFiles({
+  olderThanMs: 30 * 24 * 60 * 60 * 1000,
+  softDeleteOnly: false, // true to only mark as soft_deleted
+});
+
+// Soft-delete a specific file
+await trackedManager.softDeleteFile(fileId);
+
+// Verify physical file existence
+const exists = await trackedManager.verifyFileExistence(fileId);
+```
+
+### Database Migration (Existing Databases)
+
+If you have an existing `hazo_files` table, run the V2 migration to add reference tracking columns:
+
+```typescript
+import { migrateToV2, backfillV2Defaults, HAZO_FILES_MIGRATION_V2 } from 'hazo_files';
+
+// Using the migration helper
+await migrateToV2(
+  { run: (sql) => db.exec(sql) }, // SQLite
+  'sqlite'
+);
+await backfillV2Defaults({ run: (sql) => db.exec(sql) }, 'sqlite');
+
+// Or run statements manually
+for (const stmt of HAZO_FILES_MIGRATION_V2.sqlite.alterStatements) {
+  try { await db.run(stmt); } catch { /* column exists */ }
+}
+for (const idx of HAZO_FILES_MIGRATION_V2.sqlite.indexes) {
+  await db.run(idx);
+}
+```
+
+New tables created with `HAZO_FILES_TABLE_SCHEMA` already include V2 columns. For V3 content tagging migration, see [Content Tagging](#content-tagging) above.
+
+### Reference Tracking Types
+
+```typescript
+import type {
+  FileRef,           // Individual reference from entity to file
+  FileMetadataRecordV2,  // Extended record with refs, status, scope
+  FileWithStatus,    // Rich view: record + parsed refs + is_orphaned
+  FileStatus,        // 'active' | 'orphaned' | 'soft_deleted' | 'missing'
+  AddRefOptions,     // Options for adding a reference
+  RemoveRefsCriteria, // Criteria for bulk ref removal
+} from 'hazo_files';
+```
+
 ## File Change Detection
 
 Detect file content changes using fast xxHash hashing.
@@ -1236,6 +1410,13 @@ const hazoFiles = await createHazoFilesServer({
   },
   enableTracking: true,
   llmFactory: (provider) => createLLM({ provider }),
+  // Optional: enable automatic content tagging for all uploads
+  defaultContentTagConfig: {
+    content_tag_set_by_llm: true,
+    content_tag_prompt_area: 'classification',
+    content_tag_prompt_key: 'classify_document',
+    content_tag_prompt_return_fieldname: 'document_type',
+  },
 });
 
 // Access all services