@tfw.in/structura-lib 0.2.0

package/PRODUCTION_ARCHITECTURE.md

@@ -0,0 +1,511 @@
+# Production Architecture - Structura Edit System
+
+## Overview
+
+A scalable, production-ready system for editing structured documents with persistent storage and version history.
+
+## Design Principles
+
+1. **Auto-initialization**: Documents are created automatically on first save
+2. **Version tracking**: Every edit is stored with full history
+3. **Baseline preservation**: Original JSON is preserved as the baseline for diffs
+4. **Scalability**: Works with any PDF/JSON combination without pre-seeding
+5. **Zero configuration**: No manual database setup required
+
+## Architecture
+
+```
+┌──────────────────────┐
+│   Client Browser     │
+│                      │
+│  1. Load PDF + JSON  │ ──────┐
+│  2. User edits       │       │
+│  3. Click Save       │       │
+└──────────────────────┘       │
+           │                   │
+           │ HTTP POST         │ Initial Load
+           │ /api/save         │ (from file)
+           ▼                   │
+┌──────────────────────┐       │
+│   Express Server     │       │
+│   (port 3002)        │◄──────┘
+│                      │
+│  Smart Save Logic:   │
+│  - First save?       │
+│    Store original    │
+│  - Subsequent?       │
+│    Store edit only   │
+└──────────────────────┘
+           │
+           │ SQL Insert
+           ▼
+┌──────────────────────┐
+│   SQLite Database    │
+│   (edits.db)         │
+│                      │
+│  documents table     │
+│  - pdf_name (key)    │
+│  - original_json     │
+│                      │
+│  edits table         │
+│  - document_id       │
+│  - edited_json       │
+│  - edit_summary      │
+│  - created_at        │
+└──────────────────────┘
+```
+
+## Data Flow
+
+### First Save (Document Creation)
+
+```
+User loads: ?pdf=/doc.pdf&json=/data.json
+     │
+     ├─ Frontend loads JSON from file
+     │  - Stores as mockData (for display)
+     │  - Stores as originalData (for save)
+     │
+User makes edits
+     │
+     └─ User clicks Save
+            │
+            ▼
+        Frontend sends:
+        {
+          pdfName: "doc.pdf",
+          editedJson: {...},      // Current state
+          originalJson: {...}     // Baseline
+        }
+            │
+            ▼
+        Backend:
+        - Creates document with originalJson
+        - Creates first edit with editedJson
+        - Returns documentId + editId
+            │
+            ▼
+        Frontend:
+        - Clears originalData (already saved)
+        - Shows success message
+```
+
+### Subsequent Saves (Edit Updates)
+
+```
+User makes more edits
+     │
+     └─ User clicks Save
+            │
+            ▼
+        Frontend sends:
+        {
+          pdfName: "doc.pdf",
+          editedJson: {...}       // New state
+          // No originalJson
+        }
+            │
+            ▼
+        Backend:
+        - Finds existing document
+        - Creates new edit entry
+        - Returns documentId + editId
+```
+
+## API Design
+
+### POST /api/save
+
+**Request:**
+```json
+{
+  "pdfName": "document.pdf",
+  "editedJson": { /* current state */ },
+  "originalJson": { /* baseline (first save only) */ },
+  "summary": "Optional description"
+}
+```
+
+**Logic:**
+1. Check if document exists (by pdfName)
+2. If new:
+   - Create document with originalJson (or editedJson as fallback)
+   - Create first edit with editedJson
+3. If existing:
+   - Create new edit with editedJson
+4. Return documentId and editId
+
+**Response:**
+```json
+{
+  "success": true,
+  "documentId": 123,
+  "editId": 456,
+  "message": "Document saved successfully"
+}
+```
+
+### GET /api/load/:pdfName
+
+**Response:**
+```json
+{
+  "success": true,
+  "document": {
+    "id": 123,
+    "pdfName": "document.pdf",
+    "originalJson": { /* baseline */ },
+    "currentJson": { /* latest state */ },
+    "latestEdit": {
+      "id": 456,
+      "edit_summary": "Edit via UI",
+      "created_at": "2025-11-18 10:00:00"
+    }
+  }
+}
+```
+
+### GET /api/history/:pdfName
+
+**Response:**
+```json
+{
+  "success": true,
+  "document": { "id": 123, "pdfName": "document.pdf" },
+  "history": [
+    {
+      "id": 456,
+      "edited_json": { /* state */ },
+      "edit_summary": "Edit via UI",
+      "created_at": "2025-11-18 10:00:00"
+    },
+    // ... older edits
+  ]
+}
+```
+
+## Database Schema
+
+### documents table
+```sql
+CREATE TABLE documents (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  pdf_name TEXT NOT NULL,           -- Unique identifier
+  original_json TEXT NOT NULL,      -- Baseline for diffs
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
+);
+
+CREATE INDEX idx_document_pdf ON documents(pdf_name);
+```
+
+### edits table
+```sql
+CREATE TABLE edits (
+  id INTEGER PRIMARY KEY AUTOINCREMENT,
+  document_id INTEGER NOT NULL,
+  edited_json TEXT NOT NULL,        -- Full state at this edit
+  edit_summary TEXT,                -- Optional description
+  created_at DATETIME DEFAULT CURRENT_TIMESTAMP,
+  FOREIGN KEY (document_id) REFERENCES documents(id)
+);
+
+CREATE INDEX idx_edits_document ON edits(document_id);
+```
+
+## Scalability Features
+
+### 1. Automatic Document Creation
+- No pre-seeding required
+- Works with any PDF/JSON combination
+- Document is created on first save
+
+### 2. Version History
+- Every edit is stored completely
+- No data loss
+- Can restore any previous version
+
+### 3. Efficient Storage
+- Original JSON stored once in documents table
+- Edits stored in separate table
+- Indexed for fast lookups
+
+### 4. Query Patterns
+
+**Get latest state:**
+```javascript
+const doc = getDocumentWithLatestEdit(pdfName);
+// Returns: original + latest edit
+```
+
+**Get full history:**
+```javascript
+const history = getEditHistory(documentId);
+// Returns: all edits ordered by timestamp
+```
+
+**Restore previous version:**
+```javascript
+const edit = getEdit(editId);
+// Use edit.edited_json to restore
+```
+
+## Production Deployment
+
+### Environment Variables
+
+```bash
+# Server configuration
+PORT=3002                    # Server port
+NODE_ENV=production          # Environment
+
+# Database configuration
+DATABASE_PATH=./server/edits.db   # SQLite path
+MAX_JSON_SIZE=50mb          # Request body limit
+```
+
+### Process Management
+
+```bash
+# Development
+npm run server              # Basic server
+npm run server:dev          # With nodemon
+
+# Production (recommended)
+pm2 start server/server.js --name structura-edit
+pm2 save
+pm2 startup
+```
+
+### Nginx Configuration
+
+```nginx
+server {
+  listen 80;
+  server_name your-domain.com;
+
+  # API endpoints
+  location /api/ {
+    proxy_pass http://localhost:3002;
+    proxy_http_version 1.1;
+    proxy_set_header Upgrade $http_upgrade;
+    proxy_set_header Connection 'upgrade';
+    proxy_set_header Host $host;
+    proxy_cache_bypass $http_upgrade;
+    client_max_body_size 50M;  # Match MAX_JSON_SIZE
+  }
+
+  # Frontend
+  location / {
+    proxy_pass http://localhost:5175;
+    # ... similar proxy settings
+  }
+}
+```
+
+## Usage Patterns
+
+### Pattern 1: Direct File Load (Current)
+
+```typescript
+// Load from file system
+const url = 'http://localhost:5175/?pdf=/doc.pdf&json=/data.json';
+
+// First save:
+// - Frontend sends originalJson + editedJson
+// - Backend creates document + first edit
+
+// Subsequent saves:
+// - Frontend sends only editedJson
+// - Backend creates new edit
+```
+
+### Pattern 2: API-Generated JSON
+
+```typescript
+// Load from API
+const url = 'http://localhost:5175/?pdf=/doc.pdf';
+// API processes PDF, returns JSON
+// Store in mockData + originalData
+
+// Save flow same as Pattern 1
+```
+
+### Pattern 3: Resume Previous Session
+
+```typescript
+// Load from database
+fetch('/api/load/doc.pdf')
+  .then(res => res.json())
+  .then(data => {
+    // Use data.document.currentJson
+    // Continue editing from last state
+  });
+```
+
+## Monitoring & Observability
+
+### Logging
+
+Server logs include:
+- Document creation: `[Server] Created new document for {pdfName}`
+- Edit saves: `[Server] Saved edit {editId} for document {documentId}`
+- Errors: `[Server] Error saving document: {details}`
+
+### Metrics to Track
+
+1. **Save Operations**
+   - New documents created per day
+   - Edits saved per document
+   - Average time between edits
+
+2. **Database Health**
+   - Database size growth
+   - Query performance
+   - Index usage
+
+3. **API Performance**
+   - Request latency
+   - Error rates
+   - Payload sizes
+
+## Security Considerations
+
+### Current (Development)
+
+- CORS enabled for all origins
+- No authentication
+- No rate limiting
+- SQLite file-based storage
+
+### Production Recommendations
+
+1. **Authentication**
+   ```javascript
+   app.use('/api', authenticateToken);
+   ```
+
+2. **Authorization**
+   ```javascript
+   // Store user_id in documents table
+   // Verify ownership before save/load
+   ```
+
+3. **Rate Limiting**
+   ```javascript
+   const rateLimit = require('express-rate-limit');
+   app.use('/api/save', rateLimit({
+     windowMs: 15 * 60 * 1000,
+     max: 100
+   }));
+   ```
+
+4. **Input Validation**
+   ```javascript
+   const { body, validationResult } = require('express-validator');
+   app.post('/api/save',
+     body('pdfName').isString().trim().escape(),
+     body('editedJson').isObject(),
+     // ... validate
+   );
+   ```
+
+5. **Database Migration**
+   - Consider PostgreSQL for production
+   - Supports concurrent writes
+   - Better for multi-user scenarios
+
+## Testing
+
+### Unit Tests
+
+```javascript
+// Test document creation
+const docId = saveDocument('test.pdf', originalData);
+expect(docId).toBeGreaterThan(0);
+
+// Test edit saves
+const editId = saveEdit(docId, editedData, 'Test edit');
+expect(editId).toBeGreaterThan(0);
+
+// Test retrieval
+const doc = getDocumentWithLatestEdit('test.pdf');
+expect(doc.current_json).toEqual(editedData);
+```
+
+### Integration Tests
+
+```bash
+# Test save endpoint
+curl -X POST http://localhost:3002/api/save \
+  -H "Content-Type: application/json" \
+  -d @test-data.json
+
+# Test load endpoint
+curl http://localhost:3002/api/load/test.pdf
+
+# Test history
+curl http://localhost:3002/api/history/test.pdf
+```
+
+## Backup & Recovery
+
+### Backup Strategy
+
+```bash
+# Automated backups
+0 2 * * * cp /path/to/edits.db /backups/edits-$(date +\%Y\%m\%d).db
+
+# Retention: 30 days
+find /backups -name "edits-*.db" -mtime +30 -delete
+```
+
+### Recovery
+
+```bash
+# Restore from backup
+cp /backups/edits-20251118.db /path/to/edits.db
+pm2 restart structura-edit
+```
+
+## Future Enhancements
+
+1. **Real-time Collaboration**
+   - WebSocket for live updates
+   - Operational transform for conflict resolution
+
+2. **Diff View**
+   - Compare original vs current
+   - Highlight changes per edit
+
+3. **Undo/Redo**
+   - Restore previous edit states
+   - Apply/revert specific edits
+
+4. **Export Options**
+   - Export to PDF with changes
+   - Export diff report
+   - Export full history
+
+5. **Search & Filter**
+   - Search within documents
+   - Filter by edit date/user
+   - Full-text search in SQLite
+
+## Status
+
+✅ **Production-Ready Features:**
+- Automatic document creation
+- Version history tracking
+- Scalable for any PDF/JSON
+- RESTful API design
+- SQLite storage
+- Error handling
+- Logging
+
+🔄 **TODO for Production:**
+- Add authentication
+- Add rate limiting
+- Migrate to PostgreSQL
+- Add monitoring/metrics
+- Add automated backups
+- Write comprehensive tests