@arela/uploader 1.0.23 → 1.1.0

package/docs/WATCHER_API_INTEGRATION.md

@@ -0,0 +1,562 @@
+# Integración del Watcher con arela-api
+
+## 📋 Resumen
+Este documento describe cómo exponer la información del WatchService a través de endpoints REST en la arela-api para que el frontend pueda monitorear en tiempo real el estado del uploader en modo watch.
+
+## 🎯 Objetivos
+- Permitir que el frontend vea el estado en tiempo real del watcher
+- Exponer estadísticas de archivos pendientes, listos para subir, procesando, etc.
+- Permitir control remoto del watcher (start/stop/enable-disable auto-processing)
+- Proporcionar información de progreso del pipeline automático
+
+---
+
+## 📡 Endpoints Necesarios
+
+### 1. GET `/api/watcher/status`
+**Descripción:** Obtener estado actual del watcher
+
+**Respuesta:**
+```json
+{
+  "isRunning": true,
+  "autoProcessingEnabled": true,
+  "watchedDirectories": 3,
+  "activeWatchers": 3,
+  "lastUpdated": "2025-11-17T22:30:00.000Z",
+  "stats": {
+    "filesAdded": 145,
+    "filesModified": 23,
+    "filesRemoved": 5,
+    "uploadsTriggered": 12,
+    "pipelinesTriggered": 12,
+    "errorsEncountered": 0
+  }
+}
+```
+
+---
+
+### 2. GET `/api/watcher/queue-stats`
+**Descripción:** Obtener estadísticas de la cola de procesamiento
+
+**Respuesta:**
+```json
+{
+  "queue": {
+    "pending": 25,
+    "readyToUpload": 8,
+    "processing": 2,
+    "uploaded": 156,
+    "failed": 0,
+    "total": 191
+  },
+  "progress": {
+    "percentComplete": 82.2,
+    "estimatedTimeRemaining": "5 minutes",
+    "avgProcessingTimeMs": 3400
+  },
+  "timestamp": "2025-11-17T22:30:00.000Z"
+}
+```
+
+---
+
+### 3. GET `/api/watcher/directories`
+**Descripción:** Obtener lista de directorios siendo monitoreados
+
+**Respuesta:**
+```json
+{
+  "directories": [
+    {
+      "path": "/Users/jjfigueroa/Documents/2023",
+      "isReady": true,
+      "fileCount": 65,
+      "folderStructure": "arela"
+    },
+    {
+      "path": "/Users/jjfigueroa/Documents/2024",
+      "isReady": true,
+      "fileCount": 89,
+      "folderStructure": "arela"
+    },
+    {
+      "path": "/Users/jjfigueroa/Downloads",
+      "isReady": false,
+      "fileCount": 37,
+      "folderStructure": "arela"
+    }
+  ],
+  "totalDirectories": 3
+}
+```
+
+---
+
+### 4. POST `/api/watcher/start`
+**Descripción:** Iniciar el watcher
+
+**Body (opcional):**
+```json
+{
+  "directories": ["/path/to/dir1", "/path/to/dir2"],
+  "autoProcessing": true,
+  "folderStructure": "arela"
+}
+```
+
+**Respuesta:**
+```json
+{
+  "success": true,
+  "message": "Watcher started successfully",
+  "stats": { /* ver GET /api/watcher/status */ }
+}
+```
+
+---
+
+### 5. POST `/api/watcher/stop`
+**Descripción:** Detener el watcher
+
+**Respuesta:**
+```json
+{
+  "success": true,
+  "message": "Watcher stopped successfully",
+  "finalStats": { /* estadísticas finales */ }
+}
+```
+
+---
+
+### 6. POST `/api/watcher/enable-auto-processing`
+**Descripción:** Habilitar procesamiento automático
+
+**Body:**
+```json
+{
+  "batchSize": 10,
+  "timeout": 30000
+}
+```
+
+**Respuesta:**
+```json
+{
+  "success": true,
+  "autoProcessingEnabled": true
+}
+```
+
+---
+
+### 7. POST `/api/watcher/disable-auto-processing`
+**Descripción:** Deshabilitar procesamiento automático
+
+**Respuesta:**
+```json
+{
+  "success": true,
+  "autoProcessingEnabled": false
+}
+```
+
+---
+
+### 8. GET `/api/watcher/pipeline-status`
+**Descripción:** Obtener estado de los pipelines en ejecución
+
+**Respuesta:**
+```json
+{
+  "isProcessing": false,
+  "activePipelines": 0,
+  "recentPipelines": [
+    {
+      "id": "pipeline-1763418030418-ml4dcsgw3",
+      "file": "3429-07-23009203-Simplif.pdf",
+      "directory": "/Users/jjfigueroa/Documents/2023/230734293009203",
+      "startTime": "2025-11-17T22:20:30.000Z",
+      "duration": 653,
+      "status": "success",
+      "steps": {
+        "statsOnly": "success",
+        "detectPdfs": "success",
+        "propagateArelaPath": "success",
+        "uploadByRfc": "success"
+      }
+    }
+  ]
+}
+```
+
+---
+
+## 🔧 Implementación en arela-api
+
+### Archivo: `routes/watcher.routes.js`
+
+```javascript
+import express from 'express';
+import watchService from '../services/WatchService.js'; // Import desde arela-uploader
+import logger from '../services/LoggingService.js';
+
+const router = express.Router();
+
+/**
+ * GET /api/watcher/status
+ * Get watcher status and stats
+ */
+router.get('/status', (req, res) => {
+  try {
+    const stats = watchService.getStats();
+    res.json({
+      isRunning: stats.isRunning,
+      autoProcessingEnabled: stats.autoProcessingEnabled,
+      watchedDirectories: stats.watchedDirectories,
+      activeWatchers: stats.activeWatchers,
+      lastUpdated: new Date().toISOString(),
+      stats: {
+        filesAdded: stats.filesAdded,
+        filesModified: stats.filesModified,
+        filesRemoved: stats.filesRemoved,
+        uploadsTriggered: stats.uploadsTriggered,
+        pipelinesTriggered: stats.pipelinesTriggered,
+        errorsEncountered: stats.errorsEncountered,
+      },
+    });
+  } catch (error) {
+    logger.error(`Error getting watcher status: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * GET /api/watcher/queue-stats
+ * Get queue statistics
+ */
+router.get('/queue-stats', async (req, res) => {
+  try {
+    const queueStats = await watchService.getQueueStats();
+    if (!queueStats) {
+      return res.status(500).json({ error: 'Could not fetch queue stats' });
+    }
+    res.json(queueStats);
+  } catch (error) {
+    logger.error(`Error getting queue stats: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * GET /api/watcher/directories
+ * Get watched directories info
+ */
+router.get('/directories', (req, res) => {
+  try {
+    const directories = Array.from(watchService.watchedDirs).map((dir) => {
+      const config = watchService.directoryConfigs.get(dir);
+      const isReady = watchService.watcherReady.get(dir) === true;
+      
+      return {
+        path: dir,
+        isReady,
+        folderStructure: config?.folderStructure || 'default',
+      };
+    });
+
+    res.json({
+      directories,
+      totalDirectories: directories.length,
+    });
+  } catch (error) {
+    logger.error(`Error getting watched directories: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * POST /api/watcher/start
+ * Start the watcher
+ */
+router.post('/start', async (req, res) => {
+  try {
+    const { directories, autoProcessing = true, folderStructure = 'arela' } =
+      req.body;
+
+    if (!directories || directories.length === 0) {
+      return res.status(400).json({ error: 'No directories specified' });
+    }
+
+    // Add watchers for each directory
+    for (const dir of directories) {
+      await watchService.addWatcher(dir, {}, { folderStructure });
+    }
+
+    // Enable auto-processing if requested
+    if (autoProcessing) {
+      watchService.enableAutoProcessing({ batchSize: 10 });
+    }
+
+    // Start watching
+    await watchService.start();
+
+    res.json({
+      success: true,
+      message: 'Watcher started successfully',
+      stats: watchService.getStats(),
+    });
+  } catch (error) {
+    logger.error(`Error starting watcher: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * POST /api/watcher/stop
+ * Stop the watcher
+ */
+router.post('/stop', async (req, res) => {
+  try {
+    const finalStats = watchService.getStats();
+    await watchService.stop('API request');
+
+    res.json({
+      success: true,
+      message: 'Watcher stopped successfully',
+      finalStats,
+    });
+  } catch (error) {
+    logger.error(`Error stopping watcher: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * POST /api/watcher/enable-auto-processing
+ * Enable automatic processing pipeline
+ */
+router.post('/enable-auto-processing', (req, res) => {
+  try {
+    const { batchSize = 10 } = req.body;
+
+    watchService.enableAutoProcessing({ batchSize });
+
+    res.json({
+      success: true,
+      autoProcessingEnabled: watchService.isAutoProcessingEnabled(),
+    });
+  } catch (error) {
+    logger.error(`Error enabling auto-processing: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+/**
+ * POST /api/watcher/disable-auto-processing
+ * Disable automatic processing pipeline
+ */
+router.post('/disable-auto-processing', (req, res) => {
+  try {
+    watchService.disableAutoProcessing();
+
+    res.json({
+      success: true,
+      autoProcessingEnabled: watchService.isAutoProcessingEnabled(),
+    });
+  } catch (error) {
+    logger.error(`Error disabling auto-processing: ${error.message}`);
+    res.status(500).json({ error: error.message });
+  }
+});
+
+export default router;
+```
+
+---
+
+## 🖥️ Integración en arela-api Express App
+
+### Archivo: `app.js` o `index.js`
+
+```javascript
+import express from 'express';
+import watcherRoutes from './routes/watcher.routes.js';
+
+const app = express();
+
+// ... other middleware ...
+
+// Mount watcher routes
+app.use('/api/watcher', watcherRoutes);
+
+// ... rest of app ...
+```
+
+---
+
+## 📦 Importación de WatchService
+
+Para que arela-api pueda acceder a WatchService desde arela-uploader, hay dos opciones:
+
+### Opción 1: Exportar como módulo NPM (Recomendado)
+1. En `arela-uploader/package.json`:
+```json
+{
+  "name": "@arela/uploader",
+  "exports": {
+    "./services": "./src/services/index.js"
+  }
+}
+```
+
+2. En `arela-uploader/src/services/index.js`:
+```javascript
+export { default as watchService } from './WatchService.js';
+export { default as databaseService } from './DatabaseService.js';
+export { default as autoProcessingService } from './AutoProcessingService.js';
+```
+
+3. En arela-api:
+```bash
+npm install /path/to/arela-uploader
+# o si está publicado en npm
+npm install @arela/uploader
+```
+
+4. Usar en arela-api:
+```javascript
+import { watchService } from '@arela/uploader/services';
+```
+
+### Opción 2: Compartir código vía monorepo
+Usar Lerna o Yarn Workspaces para mantener ambos proyectos en el mismo workspace.
+
+---
+
+## 🎨 Ejemplo de Interfaz Frontend
+
+```javascript
+// hooks/useWatcher.js
+import { useState, useEffect } from 'react';
+
+export const useWatcher = () => {
+  const [status, setStatus] = useState(null);
+  const [queue, setQueue] = useState(null);
+  const [loading, setLoading] = useState(false);
+  const [error, setError] = useState(null);
+
+  useEffect(() => {
+    const interval = setInterval(async () => {
+      try {
+        setLoading(true);
+        const [statusRes, queueRes] = await Promise.all([
+          fetch('/api/watcher/status'),
+          fetch('/api/watcher/queue-stats'),
+        ]);
+
+        const statusData = await statusRes.json();
+        const queueData = await queueRes.json();
+
+        setStatus(statusData);
+        setQueue(queueData);
+        setError(null);
+      } catch (err) {
+        setError(err.message);
+      } finally {
+        setLoading(false);
+      }
+    }, 5000); // Refrescar cada 5 segundos
+
+    return () => clearInterval(interval);
+  }, []);
+
+  const startWatcher = async (directories) => {
+    const res = await fetch('/api/watcher/start', {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({ directories, autoProcessing: true }),
+    });
+    return res.json();
+  };
+
+  const stopWatcher = async () => {
+    const res = await fetch('/api/watcher/stop', { method: 'POST' });
+    return res.json();
+  };
+
+  return {
+    status,
+    queue,
+    loading,
+    error,
+    startWatcher,
+    stopWatcher,
+  };
+};
+```
+
+---
+
+## 🔐 Consideraciones de Seguridad
+
+1. **Autenticación**: Todos los endpoints deben requerir autenticación JWT o similar
+2. **Autorización**: Solo usuarios con rol `admin` o `uploader` pueden controlar el watcher
+3. **Rate Limiting**: Implementar rate limiting en los endpoints de control (start/stop)
+4. **Auditoría**: Registrar todas las acciones de control del watcher
+
+Ejemplo con middleware de autenticación:
+```javascript
+import { verifyAuth } from '../middleware/auth.js';
+
+router.get('/status', verifyAuth, (req, res) => {
+  // ... endpoint
+});
+
+router.post('/start', verifyAuth, requireRole('admin'), async (req, res) => {
+  // ... endpoint
+});
+```
+
+---
+
+## 📊 WebSocket para actualizaciones en tiempo real (Opcional)
+
+Para mayor eficiencia, implementar WebSocket en lugar de polling:
+
+```javascript
+// socket.io integration
+io.on('connection', (socket) => {
+  socket.on('subscribe-watcher', async () => {
+    // Enviar status inicial
+    socket.emit('watcher:status', watchService.getStats());
+
+    // Actualizar cada 5 segundos
+    const interval = setInterval(async () => {
+      socket.emit('watcher:status', watchService.getStats());
+      socket.emit('watcher:queue', await watchService.getQueueStats());
+    }, 5000);
+
+    socket.on('disconnect', () => clearInterval(interval));
+  });
+});
+```
+
+---
+
+## ✅ Checklist de Implementación
+
+- [ ] Crear archivo `routes/watcher.routes.js` con endpoints
+- [ ] Integrar rutas en `app.js`
+- [ ] Exportar WatchService desde arela-uploader
+- [ ] Importar en arela-api
+- [ ] Agregar autenticación a endpoints
+- [ ] Agregar validación de inputs
+- [ ] Agregar logging y error handling
+- [ ] Implementar caché en frontend (SWR, React Query)
+- [ ] Agregar WebSocket para actualizaciones real-time (opcional)
+- [ ] Documentar endpoints en OpenAPI/Swagger
+- [ ] Crear tests de integración
+