@easyling/sanity-auto-translate 0.0.0

package/README.md

@@ -0,0 +1,748 @@
+# Sanity Function: Auto-Translate on Publish
+
+Serverless function that automatically translates Sanity documents when published. Hooks into the `document.publish` lifecycle event to create translated versions without blocking the publish operation.
+
+## Features
+
+- **Automatic Translation**: Translates documents on publish without manual intervention
+- **Configurable Document Types**: Control which document types are auto-translated
+- **Multi-Locale Support**: Translate to multiple target languages simultaneously
+- **Shared Configuration**: Uses the same configuration as the Studio plugin (no environment variables needed)
+- **Non-Blocking**: Never blocks document publishing - errors are logged and gracefully handled
+- **Deadline Monitoring**: Tracks execution time and terminates gracefully before timeout
+- **Size Limit Protection**: Skips documents exceeding size limits to prevent timeouts
+- **Comprehensive Logging**: Structured JSON logs for monitoring and debugging
+- **Metrics Tracking**: Tracks success/failure rates per locale and error types
+
+## How It Works
+
+1. **Trigger**: Function is invoked when any document is published in Sanity
+2. **Filter**: Checks if document type is in `autoTranslateDocumentTypes` configuration
+3. **Extract**: Uses `ContentExtractor` from shared library to extract translatable content
+4. **Translate**: Sends translation request to configured API using `TranslationService`
+5. **Create**: Creates translated documents as drafts using `DocumentCreationService`
+6. **Log**: Records all operations with structured logging for monitoring
+
+The function uses the same core services as the Studio plugin, ensuring identical translation behavior across both contexts.
+
+## Deployment
+
+### Prerequisites
+
+- Sanity CLI installed: `npm install -g @sanity/cli`
+- Sanity project with dataset
+- Write access to the Sanity project
+
+### Deploy the Function
+
+1. Navigate to the function directory:
+
+```bash
+cd function
+```
+
+2. Build the function:
+
+```bash
+npm run build
+```
+
+3. Deploy using Sanity CLI:
+
+```bash
+sanity function deploy
+```
+
+The CLI will:
+- Upload the function code to Sanity's infrastructure
+- Register the `document.publish` trigger
+- Set the 120-second execution timeout
+- Activate the function for your project
+
+### Verify Deployment
+
+Check function status:
+
+```bash
+sanity function list
+```
+
+View function logs:
+
+```bash
+sanity function logs translate-on-publish
+```
+
+## Configuration
+
+The function reads all configuration from your Sanity dataset using the `EL_PluginConfiguration` document. This is the same configuration used by the Studio plugin, providing a single source of truth.
+
+### Configuration Document
+
+- **Document ID**: `sanity-translation-plugin.config`
+- **Document Type**: `EL_PluginConfiguration`
+
+### Required Configuration Fields
+
+#### OAuth Credentials
+
+The function requires OAuth credentials to authenticate with the translation service:
+
+- `projectId`: Your translation project identifier
+- `accessToken`: Long-lived access token for API authentication
+
+Configure through the Studio plugin UI or directly in the configuration document.
+
+#### Target Locales
+
+Define which languages to translate to:
+
+```typescript
+{
+  locales: [
+    {
+      code: 'en',
+      title: 'English',
+      enabled: true,
+      isDefault: true  // Source language
+    },
+    {
+      code: 'ja',
+      title: 'Japanese',
+      enabled: true,
+      isDefault: false
+    },
+    {
+      code: 'de',
+      title: 'German',
+      enabled: true,
+      isDefault: false
+    }
+  ]
+}
+```
+
+Only locales with `enabled: true` will be used as translation targets. The locale with `isDefault: true` is used as the source language.
+
+#### Auto-Translate Document Types
+
+Specify which document types should be automatically translated:
+
+```typescript
+{
+  autoTranslateDocumentTypes: ['article', 'blogPost', 'page']
+}
+```
+
+**Important**: Only document types in this array will trigger automatic translation. This prevents unwanted translations of configuration documents, user profiles, etc.
+
+### Optional Configuration Fields
+
+#### Translation API Endpoint
+
+Customize the translation service endpoint:
+
+```typescript
+{
+  translationApiEndpoint: 'https://api.easyling.com/translate'
+}
+```
+
+**Default**: Easyling translation API endpoint
+
+#### Content Type Settings
+
+Configure request/response format:
+
+```typescript
+{
+  requestContentType: 'application/x-protobuf',  // or 'application/json'
+  responseAcceptHeader: 'application/x-protobuf'  // or 'application/json'
+}
+```
+
+**Default**: `application/json` for both
+
+#### Document Creation Mode
+
+Control how translated documents are created:
+
+```typescript
+{
+  defaultDocumentCreationMode: 'draft'  // or 'published'
+}
+```
+
+**Default**: `draft` (recommended to allow review before publishing)
+
+### Configuration Example
+
+Complete configuration document:
+
+```json
+{
+  "_id": "sanity-translation-plugin.config",
+  "_type": "EL_PluginConfiguration",
+  "projectId": "your-easyling-project-id",
+  "accessToken": "your-access-token",
+  "translationApiEndpoint": "https://app.easyling.com/_el/ext/direct/sanity",
+  "requestContentType": "application/json",
+  "responseAcceptHeader": "application/json",
+  "defaultDocumentCreationMode": "draft",
+  "autoTranslateDocumentTypes": ["article", "blogPost", "page"],
+  "locales": [
+    {
+      "code": "en",
+      "title": "English",
+      "enabled": true,
+      "isDefault": true
+    },
+    {
+      "code": "ja",
+      "title": "Japanese",
+      "enabled": true,
+      "isDefault": false
+    },
+    {
+      "code": "de",
+      "title": "German",
+      "enabled": true,
+      "isDefault": false
+    }
+  ]
+}
+```
+
+## Blueprint Configuration
+
+The function is configured via `sanity.function.json`:
+
+```json
+{
+  "name": "translate-on-publish",
+  "description": "Automatically translate documents when published",
+  "runtime": "nodejs20",
+  "trigger": {
+    "type": "document.publish"
+  },
+  "timeout": 120
+}
+```
+
+### Blueprint Fields
+
+- **name**: Function identifier (used in CLI commands and logs)
+- **description**: Human-readable description
+- **runtime**: Node.js version (nodejs20)
+- **trigger.type**: Lifecycle event (`document.publish`)
+- **timeout**: Maximum execution time in seconds (120)
+
+### No Environment Variables Required
+
+Unlike typical serverless functions, this function does not require environment variables. All configuration is read from the Sanity dataset, which provides several benefits:
+
+- **Single Source of Truth**: Configuration is shared with Studio plugin
+- **Easy Updates**: Change configuration through Studio UI without redeploying
+- **No Secrets Management**: Credentials stored securely in Sanity dataset
+- **Immediate Effect**: Configuration changes take effect on next function invocation
+
+## Monitoring and Logging
+
+### Structured Logs
+
+The function emits structured JSON logs for easy parsing:
+
+```json
+{
+  "timestamp": "2024-01-15T10:30:45.123Z",
+  "level": "info",
+  "message": "Translation workflow completed",
+  "context": {
+    "documentId": "article-123",
+    "documentType": "article",
+    "operation": "translateOnPublish",
+    "successCount": 2,
+    "failureCount": 0,
+    "elapsedMs": 3456
+  }
+}
+```
+
+### Log Levels
+
+- **info**: Normal operation events (function start, translation complete, etc.)
+- **warn**: Non-critical issues (approaching deadline, document skipped, etc.)
+- **error**: Failures that don't block publish (translation failed, document creation failed, etc.)
+
+### Key Log Events
+
+#### Function Invocation
+
+```json
+{
+  "level": "info",
+  "message": "Function invoked",
+  "context": {
+    "documentId": "article-123",
+    "documentType": "article",
+    "startTime": 1705315845123
+  }
+}
+```
+
+#### Configuration Loaded
+
+```json
+{
+  "level": "info",
+  "message": "Configuration validated",
+  "context": {
+    "targetLocales": ["ja", "de"],
+    "autoTranslateDocumentTypes": ["article", "blogPost"]
+  }
+}
+```
+
+#### Document Skipped
+
+```json
+{
+  "level": "info",
+  "message": "Document type not configured for auto-translation",
+  "context": {
+    "documentType": "user",
+    "configuredTypes": ["article", "blogPost"]
+  }
+}
+```
+
+#### Translation Completed
+
+```json
+{
+  "level": "info",
+  "message": "Translation workflow completed",
+  "context": {
+    "totalResults": 2,
+    "successCount": 2,
+    "failureCount": 0,
+    "elapsedMs": 3456,
+    "metrics": {
+      "successfulTranslations": 2,
+      "failedTranslations": 0,
+      "localeMetrics": {
+        "ja": { "successCount": 1, "failureCount": 0 },
+        "de": { "successCount": 1, "failureCount": 0 }
+      }
+    }
+  }
+}
+```
+
+#### Error Occurred
+
+```json
+{
+  "level": "error",
+  "message": "Translation service call failed",
+  "context": {
+    "documentId": "article-123",
+    "documentType": "article",
+    "targetLocales": ["ja", "de"],
+    "error": "Network timeout",
+    "stack": "Error: Network timeout\n  at ..."
+  }
+}
+```
+
+### Viewing Logs
+
+View real-time logs:
+
+```bash
+sanity function logs translate-on-publish --follow
+```
+
+View recent logs:
+
+```bash
+sanity function logs translate-on-publish --limit 100
+```
+
+Filter by log level:
+
+```bash
+sanity function logs translate-on-publish --level error
+```
+
+### Metrics Tracking
+
+The function tracks comprehensive metrics for monitoring:
+
+```typescript
+{
+  totalInvocations: 1,
+  successfulTranslations: 2,
+  failedTranslations: 0,
+  skippedTranslations: 0,
+  localeMetrics: {
+    "ja": { successCount: 1, failureCount: 0 },
+    "de": { successCount: 1, failureCount: 0 }
+  },
+  errorsByStage: {
+    configurationLoad: 0,
+    configurationValidation: 0,
+    contentExtraction: 0,
+    translationService: 0,
+    documentCreation: 0,
+    timeout: 0,
+    sizeLimit: 0,
+    other: 0
+  }
+}
+```
+
+Use these metrics to:
+- Track translation success rates per locale
+- Identify problematic document types
+- Monitor timeout and size limit issues
+- Detect configuration problems
+
+## Troubleshooting
+
+### Function Not Triggering
+
+**Problem**: Documents are published but function doesn't run
+
+**Solutions**:
+
+1. Verify function is deployed:
+   ```bash
+   sanity function list
+   ```
+
+2. Check function status is "active"
+
+3. Verify document type is in `autoTranslateDocumentTypes`:
+   ```bash
+   sanity function logs translate-on-publish --limit 10
+   ```
+   Look for "Document type not configured for auto-translation" messages
+
+4. Check configuration document exists:
+   - Open Studio
+   - Navigate to `EL_PluginConfiguration` document
+   - Verify `autoTranslateDocumentTypes` array is populated
+
+### Translation Fails with Authentication Error
+
+**Problem**: Function logs show 401 or 403 errors
+
+**Solutions**:
+
+1. Verify OAuth credentials in configuration:
+   - Check `projectId` is correct
+   - Check `accessToken` is valid and not expired
+
+2. Test credentials with Studio plugin:
+   - Try manual translation in Studio
+   - If Studio works, function should work too
+
+3. Check translation API endpoint:
+   - Verify `translationApiEndpoint` is correct
+   - Ensure endpoint is accessible from Sanity's infrastructure
+
+4. Contact translation service provider:
+   - Request new access token
+   - Verify project ID is active
+
+### Documents Not Created
+
+**Problem**: Translation succeeds but no documents appear
+
+**Solutions**:
+
+1. Check function logs for document creation errors:
+   ```bash
+   sanity function logs translate-on-publish --level error
+   ```
+
+2. Verify creation mode:
+   - If `defaultDocumentCreationMode` is "draft", check Drafts view
+   - If "published", check published documents
+
+3. Check for collision resolution:
+   - Function may skip creation if document already exists
+   - Look for "Collision detected" in logs
+
+4. Verify Sanity client permissions:
+   - Function needs write access to dataset
+   - Check project permissions in Sanity dashboard
+
+### Function Timeout
+
+**Problem**: Function times out before completing
+
+**Solutions**:
+
+1. Check document size:
+   - Function skips documents over 10MB
+   - Look for "Document exceeds size limit" in logs
+
+2. Reduce target locales:
+   - Translating to many locales increases execution time
+   - Consider translating to fewer locales per publish
+
+3. Check translation API response time:
+   - Slow API responses can cause timeouts
+   - Monitor "elapsedMs" in logs
+
+4. Review deadline warnings:
+   - Function logs warnings at 100 seconds
+   - If consistently hitting warnings, optimize document size or locale count
+
+### Configuration Not Loading
+
+**Problem**: Function logs show "Configuration not found"
+
+**Solutions**:
+
+1. Verify configuration document exists:
+   ```bash
+   sanity documents get sanity-translation-plugin.config
+   ```
+
+2. Check document type:
+   - Must be `EL_PluginConfiguration`
+   - Must have document ID `sanity-translation-plugin.config`
+
+3. Create configuration through Studio plugin:
+   - Install and configure Studio plugin first
+   - Plugin will create configuration document
+
+4. Manually create configuration document:
+   ```bash
+   sanity documents create --replace <<EOF
+   {
+     "_id": "sanity-translation-plugin.config",
+     "_type": "EL_PluginConfiguration",
+     "projectId": "your-project-id",
+     "accessToken": "your-token",
+     "autoTranslateDocumentTypes": ["article"],
+     "locales": [
+       {"code": "en", "title": "English", "enabled": true, "isDefault": true},
+       {"code": "ja", "title": "Japanese", "enabled": true, "isDefault": false}
+     ]
+   }
+   EOF
+   ```
+
+### Partial Translations
+
+**Problem**: Some locales succeed, others fail
+
+**Solutions**:
+
+1. Check locale-specific metrics in logs:
+   ```json
+   {
+     "localeMetrics": {
+       "ja": { "successCount": 1, "failureCount": 0 },
+       "de": { "successCount": 0, "failureCount": 1 }
+     }
+   }
+   ```
+
+2. Review error messages for failed locales:
+   - Look for locale-specific errors in logs
+   - May indicate API issues for specific languages
+
+3. Verify locale codes:
+   - Ensure locale codes match translation service expectations
+   - Check for typos in locale configuration
+
+4. Test individual locales:
+   - Temporarily disable problematic locales
+   - Test with Studio plugin to isolate issue
+
+### High Error Rate
+
+**Problem**: Many translations failing
+
+**Solutions**:
+
+1. Review error breakdown by stage:
+   ```json
+   {
+     "errorsByStage": {
+       "contentExtraction": 5,
+       "translationService": 12,
+       "documentCreation": 3
+     }
+   }
+   ```
+
+2. Address most common error stage:
+   - **contentExtraction**: Document structure issues, invalid portable text
+   - **translationService**: API issues, authentication, network problems
+   - **documentCreation**: Permission issues, validation errors, collisions
+
+3. Check for systematic issues:
+   - Specific document type causing problems
+   - Specific locale consistently failing
+   - Time-based patterns (API rate limits, downtime)
+
+4. Enable debug mode in Studio plugin:
+   - Set `debugMode: true` in configuration
+   - Review detailed field information
+   - Identify problematic fields or content
+
+## Performance Considerations
+
+### Execution Time
+
+- **Typical**: 3-5 seconds per document with 2-3 target locales
+- **Maximum**: 120 seconds (hard timeout)
+- **Warning Threshold**: 100 seconds (function logs warning)
+- **Termination Threshold**: 110 seconds (function terminates gracefully)
+
+### Document Size Limits
+
+- **Maximum**: 10MB per document
+- Documents exceeding limit are skipped with warning log
+- Consider splitting large documents into smaller pieces
+
+### Concurrent Translations
+
+The function processes target locales sequentially to:
+- Avoid overwhelming translation API
+- Provide better error isolation
+- Enable partial success (some locales succeed even if others fail)
+
+### Optimization Tips
+
+1. **Limit Target Locales**: Fewer locales = faster execution
+2. **Use Draft Mode**: Creating drafts is faster than published documents
+3. **Optimize Document Structure**: Simpler structures extract faster
+4. **Monitor Execution Time**: Review "elapsedMs" in logs to identify slow documents
+5. **Batch Similar Documents**: Publish related documents together for better caching
+
+## Shared Configuration with Studio Plugin
+
+The function and Studio plugin share the same configuration document, providing several benefits:
+
+### Single Source of Truth
+
+- No duplicate configuration to maintain
+- Changes apply to both function and plugin
+- Consistent behavior across manual and automatic translation
+
+### Easy Management
+
+- Configure through Studio UI (user-friendly)
+- Or edit configuration document directly (programmatic)
+- No need to redeploy function after configuration changes
+
+### Configuration Fields Used by Function
+
+The function reads these fields from `EL_PluginConfiguration`:
+
+- `projectId`: OAuth project identifier
+- `accessToken`: OAuth access token
+- `translationApiEndpoint`: Translation API URL
+- `requestContentType`: Request format (JSON or protobuf)
+- `responseAcceptHeader`: Response format (JSON or protobuf)
+- `defaultDocumentCreationMode`: Draft or published
+- `autoTranslateDocumentTypes`: Document types to auto-translate
+- `locales`: Locale definitions with enabled flags
+- `defaultLocale`: Source language code
+
+### Configuration Fields Ignored by Function
+
+These fields are used only by the Studio plugin:
+
+- `collisionResolutionMode`: UI-specific collision handling
+- `existingDocumentHandling`: UI-specific document handling
+- `dntFieldConfigurations`: DNT preferences (function uses defaults)
+- `debugMode`: UI-specific debug display
+
+## Development
+
+### Local Testing
+
+Test the function locally before deploying:
+
+```bash
+# Build the function
+npm run build
+
+# Run tests
+npm test
+
+# Run specific test file
+npm test -- config.test.ts
+```
+
+### Function Structure
+
+```
+function/
+├── src/
+│   ├── index.ts           # Main function entry point
+│   ├── config.ts          # Configuration loading and validation
+│   ├── logger.ts          # Structured logging
+│   └── __tests__/         # Unit and integration tests
+├── sanity.function.json   # Blueprint configuration
+├── package.json           # Dependencies and scripts
+└── tsconfig.json          # TypeScript configuration
+```
+
+### Key Files
+
+- **index.ts**: Main function logic, handles publish events
+- **config.ts**: Loads configuration from Sanity dataset
+- **logger.ts**: Structured logging with context
+- **sanity.function.json**: Deployment configuration
+
+### Testing
+
+```bash
+# Run all tests
+npm test
+
+# Run with coverage
+npm run test:coverage
+
+# Run in watch mode
+npm run test:watch
+```
+
+### Building
+
+```bash
+# Production build
+npm run build
+
+# Clean build artifacts
+npm run clean
+
+# Type check
+npm run typecheck
+```
+
+## Related Packages
+
+- [@easyling/sanity-connector](../plugin/README.md) - Studio plugin for manual translation
+- [@easyling/sanity-connector-shared](../shared/README.md) - Shared translation logic
+
+## Support
+
+For questions, issues, or feature requests:
+
+- **Email**: support@easyling.com
+- **Issues**: [GitHub Issues](https://github.com/easyling/el-sanity-connector/issues)
+- **Documentation**: [Main README](../README.md) | [Plugin README](../plugin/README.md) | [Shared Library](../shared/README.md)
+
+## License
+
+MIT