@arela/uploader 0.2.13 → 1.0.0

package/docs/API_ENDPOINTS_FOR_DETECTION.md

@@ -0,0 +1,647 @@
+# Backend API Endpoints for Pedimento Detection and Propagation
+
+This document describes the required backend API endpoints needed to support the refactored detection and propagation functionality in the uploader client.
+
+## Overview
+
+The uploader client has been refactored to use API endpoints instead of direct Supabase client queries for:
+1. PDF pedimento detection process
+2. Arela path propagation to related files
+
+This provides better abstraction, security, and control over database operations.
+
+## Required Endpoints
+
+### 1. GET /api/uploader/pdf-records
+
+Fetch PDF records that are ready for pedimento detection processing.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Query Parameters
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `offset` | integer | No | 0 | Pagination offset for records |
+| `limit` | integer | No | 100 | Maximum number of records to return |
+| `status` | string | No | - | Filter by status field |
+| `file_extension` | string | No | - | Filter by file extension |
+| `is_like_simplificado` | boolean | No | - | Filter records likely to be simplificado |
+
+#### Example Request
+
+```bash
+GET /api/uploader/pdf-records?offset=0&limit=100&status=fs-stats&file_extension=pdf&is_like_simplificado=true
+Headers:
+  x-api-key: your-api-key-here
+```
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+[
+  {
+    "id": "uuid-here",
+    "original_path": "/path/to/file.pdf",
+    "filename": "file.pdf",
+    "file_extension": "pdf",
+    "status": "fs-stats"
+  },
+  ...
+]
+```
+
+**Error Response**
+
+```json
+{
+  "error": "Error message here",
+  "message": "Detailed error description"
+}
+```
+
+#### Implementation Notes
+
+- Should query the `uploader` table in Supabase
+- Apply filters based on query parameters
+- Order results by `created_at` for consistent pagination
+- Use efficient indexing on frequently filtered columns (`status`, `file_extension`, `is_like_simplificado`)
+- Should support range queries for pagination: `range(offset, offset + limit - 1)`
+
+#### SQL Query Example
+
+```sql
+SELECT id, original_path, filename, file_extension, status
+FROM uploader
+WHERE status = 'fs-stats'
+  AND file_extension = 'pdf'
+  AND is_like_simplificado = true
+ORDER BY created_at
+LIMIT 100 OFFSET 0;
+```
+
+---
+
+### 2. PATCH /api/uploader/batch-update-detection
+
+Batch update detection results for multiple records.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Request Body
+
+```json
+{
+  "updates": [
+    {
+      "id": "uuid-here",
+      "status": "detected",
+      "document_type": "pedimento-simplificado",
+      "num_pedimento": "23  12  3456  7890123",
+      "arela_path": "RFC123456789/2023/3456/",
+      "year": "2023",
+      "rfc": "RFC123456789",
+      "message": null
+    },
+    {
+      "id": "uuid-here-2",
+      "status": "file-not-found",
+      "message": "File no longer exists at original path"
+    },
+    ...
+  ]
+}
+```
+
+#### Update Object Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `id` | uuid | Yes | Record ID to update |
+| `status` | string | Yes | New status: 'detected', 'not-detected', 'file-not-found', 'detection-error' |
+| `document_type` | string | No | Detected document type (e.g., 'pedimento-simplificado') |
+| `num_pedimento` | string | No | Detected pedimento number |
+| `arela_path` | string | No | Generated Arela storage path |
+| `year` | string | No | Extracted year from pedimento |
+| `rfc` | string | No | Extracted RFC |
+| `message` | string | No | Error message or additional notes |
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+{
+  "success": true,
+  "updated": 25,
+  "errors": []
+}
+```
+
+**Partial Success (200 OK)**
+
+```json
+{
+  "success": true,
+  "updated": 23,
+  "errors": [
+    {
+      "id": "uuid-that-failed",
+      "message": "Update failed: reason"
+    }
+  ]
+}
+```
+
+**Error Response (400/500)**
+
+```json
+{
+  "success": false,
+  "updated": 0,
+  "errors": [
+    {
+      "message": "Validation error or other error description"
+    }
+  ]
+}
+```
+
+#### Implementation Notes
+
+- Should perform batch updates efficiently (use bulk UPDATE or upsert operations)
+- Validate that all required fields are present
+- Handle partial failures gracefully - some updates may succeed while others fail
+- Consider transaction handling for data consistency
+- Log all updates for audit purposes
+- Should handle updates in batches of reasonable size (e.g., 100-500 records)
+
+#### SQL Query Example (PostgreSQL)
+
+```sql
+-- Using a CTE (Common Table Expression) for batch updates
+WITH update_data(id, status, document_type, num_pedimento, arela_path, year, rfc, message) AS (
+  VALUES 
+    ('uuid-1'::uuid, 'detected', 'pedimento-simplificado', '23  12  3456  7890123', 'RFC/2023/3456/', '2023', 'RFC123456789', NULL),
+    ('uuid-2'::uuid, 'file-not-found', NULL, NULL, NULL, NULL, NULL, 'File no longer exists')
+)
+UPDATE uploader AS u
+SET 
+  status = ud.status,
+  document_type = COALESCE(ud.document_type, u.document_type),
+  num_pedimento = COALESCE(ud.num_pedimento, u.num_pedimento),
+  arela_path = COALESCE(ud.arela_path, u.arela_path),
+  year = COALESCE(ud.year, u.year),
+  rfc = COALESCE(ud.rfc, u.rfc),
+  message = COALESCE(ud.message, u.message),
+  updated_at = NOW()
+FROM update_data AS ud
+WHERE u.id = ud.id;
+```
+
+---
+
+### 3. POST /api/uploader/propagate-arela-path
+
+Execute arela_path propagation process on the backend. This endpoint handles all the logic for finding pedimentos and propagating their arela_path to related files in the same directory.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Request Body
+
+```json
+{
+  "years": ["2023", "2024"]
+}
+```
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `years` | array of strings | No | Optional year filter (e.g., ["2023", "2024"]) |
+
+#### Example Request
+
+```bash
+POST /api/uploader/propagate-arela-path
+Headers:
+  x-api-key: your-api-key-here
+  Content-Type: application/json
+Body:
+{
+  "years": ["2023", "2024"]
+}
+```
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+{
+  "success": true,
+  "processedCount": 150,
+  "updatedCount": 1250,
+  "errorCount": 2
+}
+```
+
+**Error Response**
+
+```json
+{
+  "success": false,
+  "processedCount": 0,
+  "updatedCount": 0,
+  "errorCount": 1,
+  "error": "Error message here"
+}
+```
+
+#### Implementation Notes
+
+This endpoint should perform all propagation logic server-side:
+
+1. **Find Pedimentos**: Query `pedimento_simplificado` records that have `arela_path` set
+   - Filter by `years` if provided
+   - Order by `created_at` for consistent processing
+
+2. **For Each Pedimento**:
+   - Extract the base directory path from `original_path`
+   - Extract the folder part of `arela_path` (remove filename if present)
+   - Find all related files in the same directory where `arela_path IS NULL`
+   - Update those files with the pedimento's `arela_path` and `year`
+
+3. **Pagination**: Process records in batches to handle large datasets
+   - Use cursor-based or offset pagination internally
+   - Handle pagination reset when updates change query results
+
+4. **Error Handling**: Continue processing even if some records fail
+   - Track errors but don't stop the entire process
+   - Return counts of processed, updated, and errors
+
+#### SQL Logic Example
+
+```sql
+-- Step 1: Find pedimentos with arela_path
+WITH pedimentos AS (
+  SELECT id, original_path, arela_path, year
+  FROM uploader
+  WHERE document_type = 'pedimento_simplificado'
+    AND arela_path IS NOT NULL
+    AND ($1::text[] IS NULL OR year = ANY($1::text[]))
+  ORDER BY created_at
+)
+-- Step 2: For each pedimento, update related files
+UPDATE uploader u
+SET 
+  arela_path = p.arela_path,
+  year = p.year,
+  updated_at = NOW()
+FROM pedimentos p
+WHERE u.arela_path IS NULL
+  AND u.original_path LIKE (SELECT regexp_replace(p.original_path, '/[^/]+$', '') || '%')
+  AND u.id != p.id;
+```
+
+**Note**: The actual implementation should process pedimentos iteratively with proper pagination and error handling, not in a single query.
+
+#### Performance Considerations
+
+- Process in batches of 50-100 pedimentos at a time
+- Use batch updates (50-100 files per update query)
+- Add appropriate database indexes on `original_path`, `arela_path`, `document_type`, `year`
+- Consider using background jobs for large datasets
+- Log progress for monitoring
+- Handle transaction boundaries appropriately
+
+---
+
+## RFC-Based Upload Endpoints
+
+The following endpoints support RFC-based file upload operations.
+
+### 4. GET /api/uploader/rfc-file-count
+
+Get count of files for specified RFCs.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Query Parameters
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `rfcs` | string | Yes | - | Comma-separated list of RFC values (e.g., "RFC123,RFC456") |
+
+#### Example Request
+
+```bash
+GET /api/uploader/rfc-file-count?rfcs=RFC123456789,RFC987654321
+Headers:
+  x-api-key: your-api-key-here
+```
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+{
+  "count": 1250
+}
+```
+
+#### SQL Query Example
+
+```sql
+SELECT COUNT(*) as count
+FROM uploader
+WHERE rfc = ANY($1::text[])
+  AND arela_path IS NOT NULL;
+```
+
+---
+
+### 5. GET /api/uploader/pedimentos-by-rfc
+
+Fetch pedimento_simplificado records for specified RFCs.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Query Parameters
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `rfcs` | string | Yes | - | Comma-separated list of RFC values |
+| `years` | string | No | - | Optional comma-separated list of years to filter |
+| `offset` | integer | No | 0 | Pagination offset |
+| `limit` | integer | No | 500 | Maximum number of records to return |
+
+#### Example Request
+
+```bash
+GET /api/uploader/pedimentos-by-rfc?rfcs=RFC123456789,RFC987654321&years=2023,2024&offset=0&limit=500
+Headers:
+  x-api-key: your-api-key-here
+```
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+[
+  {
+    "arela_path": "RFC123456789/2023/3456/"
+  },
+  {
+    "arela_path": "RFC123456789/2024/7890/"
+  }
+]
+```
+
+#### SQL Query Example
+
+```sql
+SELECT arela_path
+FROM uploader
+WHERE document_type = 'pedimento_simplificado'
+  AND rfc = ANY($1::text[])
+  AND arela_path IS NOT NULL
+  AND ($2::text[] IS NULL OR year = ANY($2::text[]))
+ORDER BY created_at
+LIMIT $3 OFFSET $4;
+```
+
+---
+
+### 6. GET /api/uploader/files-for-upload
+
+Fetch files ready for upload by arela_path.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Query Parameters
+
+| Parameter | Type | Required | Default | Description |
+|-----------|------|----------|---------|-------------|
+| `arela_paths` | string | Yes | - | Pipe-separated list of arela_path values |
+| `offset` | integer | No | 0 | Pagination offset |
+| `limit` | integer | No | 1000 | Maximum number of records to return |
+
+#### Example Request
+
+```bash
+GET /api/uploader/files-for-upload?arela_paths=RFC123/2023/3456/|RFC456/2024/7890/&offset=0&limit=1000
+Headers:
+  x-api-key: your-api-key-here
+```
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+[
+  {
+    "id": "uuid-here",
+    "original_path": "/path/to/file.xml",
+    "arela_path": "RFC123456789/2023/3456/",
+    "filename": "file.xml",
+    "rfc": "RFC123456789",
+    "document_type": "support_document"
+  },
+  ...
+]
+```
+
+#### SQL Query Example
+
+```sql
+SELECT id, original_path, arela_path, filename, rfc, document_type
+FROM uploader
+WHERE arela_path = ANY($1::text[])
+  AND status != 'file-uploaded'
+  AND original_path IS NOT NULL
+ORDER BY created_at
+LIMIT $2 OFFSET $3;
+```
+
+---
+
+### 7. PATCH /api/uploader/batch-update-status
+
+Batch update file status after upload operations.
+
+#### Authentication
+- Requires `x-api-key` header
+
+#### Request Body
+
+```json
+{
+  "updates": [
+    {
+      "id": "uuid-here",
+      "status": "file-uploaded",
+      "message": "Successfully uploaded",
+      "processing_status": "UPLOADED"
+    },
+    {
+      "id": "uuid-here-2",
+      "status": "upload-error",
+      "message": "Upload failed: connection timeout"
+    }
+  ]
+}
+```
+
+#### Update Object Fields
+
+| Field | Type | Required | Description |
+|-------|------|----------|-------------|
+| `id` | uuid | Yes | Record ID to update |
+| `status` | string | Yes | New status value |
+| `message` | string | No | Status message |
+| `processing_status` | string | No | Processing queue status |
+
+#### Response Format
+
+**Success (200 OK)**
+
+```json
+{
+  "success": true,
+  "updated": 25,
+  "errors": []
+}
+```
+
+**Partial Success (200 OK)**
+
+```json
+{
+  "success": true,
+  "updated": 23,
+  "errors": [
+    {
+      "id": "uuid-that-failed",
+      "message": "Update failed: reason"
+    }
+  ]
+}
+```
+
+#### SQL Query Example
+
+```sql
+WITH update_data(id, status, message, processing_status) AS (
+  VALUES 
+    ('uuid-1'::uuid, 'file-uploaded', 'Success', 'UPLOADED'),
+    ('uuid-2'::uuid, 'upload-error', 'Failed', NULL)
+)
+UPDATE uploader AS u
+SET 
+  status = COALESCE(ud.status, u.status),
+  message = COALESCE(ud.message, u.message),
+  processing_status = COALESCE(ud.processing_status, u.processing_status),
+  updated_at = NOW()
+FROM update_data AS ud
+WHERE u.id = ud.id;
+```
+
+---
+
+## Migration Path
+
+### Phase 1: Implementation (Backend)
+1. Implement all five endpoints in the backend API
+2. Add proper authentication and authorization
+3. Add rate limiting and request validation
+4. Test endpoints with sample data
+
+### Phase 2: Testing (Client)
+1. Test the refactored client code with the new endpoints
+2. Performance testing with large datasets
+3. Verify pagination works correctly
+
+### Phase 3: Deployment
+1. Deploy backend API changes
+2. Update client configuration to use API endpoints
+3. Monitor logs for any issues
+4. Phase out direct Supabase access completely
+
+## Security Considerations
+
+1. **Authentication**: All endpoints require valid `x-api-key` header
+2. **Rate Limiting**: Implement rate limiting to prevent abuse
+3. **Input Validation**: Validate all query parameters and request body data
+4. **SQL Injection Prevention**: Use parameterized queries or ORM
+5. **Authorization**: Ensure API keys have appropriate permissions
+6. **Audit Logging**: Log all data access and modifications
+7. **Path Traversal**: Validate `basePath` parameter to prevent directory traversal attacks
+
+## Performance Considerations
+
+1. **Indexing**: Add database indexes on commonly filtered columns:
+   - `status`
+   - `file_extension`
+   - `is_like_simplificado`
+   - `created_at`
+   - `document_type`
+   - `arela_path`
+   - `original_path` (consider using pg_trgm for LIKE queries)
+   - `year`
+
+2. **Batch Size**: Optimize batch sizes for updates (50-100 records per batch for propagation)
+
+3. **Connection Pooling**: Use database connection pooling to handle concurrent requests
+
+4. **Caching**: Consider caching frequently accessed data
+
+5. **Pagination**: Implement efficient pagination with cursors or offset/limit
+
+6. **LIKE Query Optimization**: Use proper indexes or consider full-text search for path matching
+
+## Error Handling
+
+Both endpoints should return appropriate HTTP status codes:
+
+- **200 OK**: Successful operation
+- **400 Bad Request**: Invalid parameters or request body
+- **401 Unauthorized**: Missing or invalid API key
+- **429 Too Many Requests**: Rate limit exceeded
+- **500 Internal Server Error**: Server-side error
+
+## Testing Checklist
+
+- [ ] GET /api/uploader/pdf-records returns correct records with filters
+- [ ] GET /api/uploader/pdf-records handles pagination correctly
+- [ ] GET /api/uploader/pdf-records returns empty array when no records found
+- [ ] PATCH /api/uploader/batch-update-detection successfully updates records
+- [ ] PATCH /api/uploader/batch-update-detection handles partial failures gracefully
+- [ ] PATCH /api/uploader/batch-update-detection validates required fields
+- [ ] GET /api/uploader/pedimentos-for-propagation returns correct pedimento records
+- [ ] GET /api/uploader/pedimentos-for-propagation handles year filtering correctly
+- [ ] GET /api/uploader/related-files finds files in same directory
+- [ ] GET /api/uploader/related-files excludes pedimento record correctly
+- [ ] PATCH /api/uploader/batch-update-arela-path successfully updates arela_path and year
+- [ ] All endpoints require valid authentication
+- [ ] All endpoints handle errors appropriately
+- [ ] Performance testing with 10,000+ records
+- [ ] Rate limiting works as expected
+
+## Example Integration Code
+
+The client code has been updated in:
+- `/src/services/upload/ApiUploadService.js` - API methods for detection and propagation
+- `/src/services/DatabaseService.js` - Detection and propagation logic using API
+
+See the implementation for reference on how to integrate these endpoints.