ldap-rest 0.0.1

package/docs/ldapBulkImport.md

@@ -0,0 +1,722 @@
+# LDAP Bulk Import Plugin
+
+Generic CSV-based bulk import plugin for LDAP resources, automatically configured from JSON schemas.
+
+## Overview
+
+The `ldapBulkImport` plugin provides CSV import functionality for any LDAP resource type (users, groups, etc.) by reading JSON schema definitions. It automatically:
+
+- Generates CSV templates based on schema attributes
+- Excludes fixed attributes from CSV (auto-populated from schema)
+- Auto-calculates organization link and path from a single `organizationDn` column
+- Validates data against schema requirements
+- Supports dry-run mode for testing
+- Provides detailed success/failure reports
+
+## Configuration
+
+### CLI Arguments
+
+```bash
+--plugin core/ldap/bulkImport \
+--bulk-import-schemas "users:./static/schemas/twake/users.json,groups:./static/schemas/twake/groups.json" \
+--bulk-import-max-file-size 10485760 \
+--bulk-import-batch-size 100
+```
+
+### Configuration Options
+
+| Argument                      | Environment Variable           | Default    | Description                                   |
+| ----------------------------- | ------------------------------ | ---------- | --------------------------------------------- |
+| `--bulk-import-schemas`       | `DM_BULK_IMPORT_SCHEMAS`       | (required) | Comma-separated list of resource:schema pairs |
+| `--bulk-import-max-file-size` | `DM_BULK_IMPORT_MAX_FILE_SIZE` | `10485760` | Maximum upload file size in bytes (10MB)      |
+| `--bulk-import-batch-size`    | `DM_BULK_IMPORT_BATCH_SIZE`    | `100`      | Number of records to process at once          |
+
+### Schema Configuration Format
+
+Format: `"resourceName:path/to/schema.json,resourceName2:path/to/schema2.json"`
+
+**Example**:
+
+```bash
+--bulk-import-schemas "users:./static/schemas/twake/users.json,groups:./static/schemas/twake/groups.json"
+```
+
+This creates two import APIs:
+
+- `/api/v1/ldap/bulk-import/users/*`
+- `/api/v1/ldap/bulk-import/groups/*`
+
+## REST API
+
+For each configured schema, the plugin automatically generates two endpoints:
+
+### GET /api/v1/ldap/bulk-import/{resource}/template.csv
+
+Downloads a CSV template file with headers based on the schema.
+
+**Response**: CSV file with headers
+
+**Example**:
+
+```bash
+curl -O "http://localhost:8081/api/v1/ldap/bulk-import/users/template.csv"
+```
+
+**Generated template** (for users):
+
+```csv
+uid,cn,sn,givenName,mail,userPassword,telephoneNumber,organizationDn
+```
+
+**Notes**:
+
+- Excludes attributes with `"fixed": true` (auto-populated)
+- Excludes attributes with `"role": ["organizationLink"]` or `"role": ["organizationPath"]` (auto-calculated)
+- Includes special column `organizationDn` for organization assignment
+
+### POST /api/v1/ldap/bulk-import/{resource}
+
+Bulk import entries from CSV file.
+
+**Request** (multipart/form-data):
+
+| Field             | Type    | Required | Default | Description                                   |
+| ----------------- | ------- | -------- | ------- | --------------------------------------------- |
+| `file`            | File    | Yes      | -       | CSV file to import                            |
+| `dryRun`          | Boolean | No       | `false` | Validate without creating entries             |
+| `updateExisting`  | Boolean | No       | `false` | Update entries if they already exist          |
+| `continueOnError` | Boolean | No       | `true`  | Continue processing even if some entries fail |
+
+**Response (200)**:
+
+```json
+{
+  "success": true,
+  "total": 100,
+  "created": 95,
+  "updated": 0,
+  "skipped": 3,
+  "failed": 2,
+  "errors": [
+    {
+      "line": 15,
+      "identifier": "baduser",
+      "error": "Missing required attribute: sn"
+    },
+    {
+      "line": 42,
+      "identifier": "duplicate",
+      "error": "Organization not found: ou=InvalidDept,dc=example,dc=com"
+    }
+  ],
+  "details": {
+    "duration": "2.5s",
+    "linesProcessed": 100
+  }
+}
+```
+
+**Example**:
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "dryRun=false" \
+  -F "updateExisting=false" \
+  -F "continueOnError=true"
+```
+
+## CSV Format
+
+### Basic Structure
+
+```csv
+uid,cn,sn,givenName,mail,userPassword,organizationDn
+jdoe,John Doe,Doe,John,john.doe@example.com,password123,ou=IT,ou=organization,dc=example,dc=com
+asmith,Alice Smith,Smith,Alice,alice.smith@example.com,secret456,ou=HR,ou=organization,dc=example,dc=com
+```
+
+### Special Column: `organizationDn`
+
+The `organizationDn` column is special and triggers automatic calculation of:
+
+1. **Organization Link** - Attribute with `role: ["organizationLink"]` in schema
+   - Automatically set to the value of `organizationDn`
+
+2. **Organization Path** - Attribute with `role: ["organizationPath"]` in schema
+   - Automatically fetched from the organization's LDAP entry
+   - Example: `"IT / organization"` for `ou=IT,ou=organization,dc=example,dc=com`
+
+### Multi-Value Attributes
+
+Use semicolon (`;`) as separator for multi-value attributes:
+
+```csv
+uid,cn,sn,mail,organizationDn
+jdoe,John Doe,Doe,john@example.com;j.doe@example.com;jdoe@company.com,ou=IT,dc=example,dc=com
+```
+
+This creates:
+
+```
+mail: john@example.com
+mail: j.doe@example.com
+mail: jdoe@company.com
+```
+
+### Fixed Attributes
+
+Attributes marked as `"fixed": true` in the schema are automatically added and should NOT be included in the CSV:
+
+**Schema**:
+
+```json
+{
+  "properties": {
+    "objectClass": {
+      "type": "array",
+      "fixed": true,
+      "default": ["inetOrgPerson", "organizationalPerson", "person", "top"]
+    }
+  }
+}
+```
+
+**CSV** (objectClass NOT included):
+
+```csv
+uid,cn,sn,mail
+jdoe,John Doe,Doe,jdoe@example.com
+```
+
+**Result in LDAP**:
+
+```
+objectClass: inetOrgPerson
+objectClass: organizationalPerson
+objectClass: person
+objectClass: top
+uid: jdoe
+cn: John Doe
+sn: Doe
+mail: jdoe@example.com
+```
+
+## Complete Example
+
+### Step 1: Configure Plugin
+
+```bash
+--plugin core/ldap/bulkImport \
+--bulk-import-schemas "users:./static/schemas/twake/users.json"
+```
+
+### Step 2: Download Template
+
+```bash
+curl -O "http://localhost:8081/api/v1/ldap/bulk-import/users/template.csv"
+```
+
+**Generated `users-template.csv`**:
+
+```csv
+uid,cn,sn,givenName,mail,userPassword,telephoneNumber,organizationDn
+```
+
+### Step 3: Fill CSV File
+
+**users.csv**:
+
+```csv
+uid,cn,sn,givenName,mail,userPassword,telephoneNumber,organizationDn
+jdoe,John Doe,Doe,John,john.doe@example.com,SecurePass123,+1-555-0100,ou=Engineering,ou=organization,dc=example,dc=com
+asmith,Alice Smith,Smith,Alice,alice.smith@example.com,SecretPass456,+1-555-0101,ou=Marketing,ou=organization,dc=example,dc=com
+bwilson,Bob Wilson,Wilson,Bob,bob.wilson@example.com,MyPass789,+1-555-0102,ou=Engineering,ou=organization,dc=example,dc=com
+```
+
+### Step 4: Test with Dry Run
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "dryRun=true"
+```
+
+**Response**:
+
+```json
+{
+  "success": true,
+  "total": 3,
+  "created": 3,
+  "failed": 0,
+  "errors": [],
+  "details": {
+    "duration": "0.5s",
+    "linesProcessed": 3
+  }
+}
+```
+
+### Step 5: Import for Real
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "dryRun=false"
+```
+
+### Step 6: Verify in LDAP
+
+```bash
+ldapsearch -x -b "ou=users,dc=example,dc=com" "(uid=jdoe)"
+```
+
+**Result**:
+
+```
+dn: uid=jdoe,ou=users,dc=example,dc=com
+objectClass: inetOrgPerson
+objectClass: organizationalPerson
+objectClass: person
+objectClass: top
+uid: jdoe
+cn: John Doe
+sn: Doe
+givenName: John
+mail: john.doe@example.com
+userPassword: {SSHA}...
+telephoneNumber: +1-555-0100
+twakeDepartmentLink: ou=Engineering,ou=organization,dc=example,dc=com
+twakeDepartmentPath: Engineering / organization
+```
+
+## Operation Modes
+
+### Dry Run Mode
+
+Validate CSV without creating entries:
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "dryRun=true"
+```
+
+**Use cases**:
+
+- Test CSV format before actual import
+- Validate required attributes
+- Check organization DNs exist
+- Estimate import duration
+
+### Update Existing Mode
+
+Update entries that already exist:
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "updateExisting=true"
+```
+
+**Behavior**:
+
+- If entry exists: Updates all attributes from CSV
+- If entry doesn't exist: Creates new entry
+- Uses LDAP `modify` operation for updates
+
+### Continue on Error Mode
+
+Process all records even if some fail:
+
+```bash
+curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+  -F "file=@users.csv" \
+  -F "continueOnError=true"
+```
+
+**Behavior** (default: `true`):
+
+- Continues processing remaining records after errors
+- Collects all errors in response
+- Reports partial success
+- Successfully created entries remain in LDAP even if later entries fail
+
+**Example with 5 entries**:
+
+- Entry 1: ✅ Created
+- Entry 2: ✅ Created
+- Entry 3: ❌ Failed (invalid organization)
+- Entry 4: ✅ Created (processed despite previous error)
+- Entry 5: ✅ Created
+
+**Result**: 4 created, 1 failed, all errors reported
+
+**Opposite** (`continueOnError=false`):
+
+- Stops at first error
+- Returns immediately with error
+- Previously created entries remain in LDAP
+- No rollback (LDAP doesn't support transactions)
+
+**Example with same 5 entries**:
+
+- Entry 1: ✅ Created
+- Entry 2: ✅ Created
+- Entry 3: ❌ Failed → **STOPS HERE**
+- Entry 4: ⏭️ Not processed
+- Entry 5: ⏭️ Not processed
+
+**Result**: 2 created, 1 failed, entries 4-5 not processed
+
+**Important**: Entries already created are NOT rolled back. If you need to undo, you must manually delete them or re-run with correct data using `updateExisting=true`.
+
+## Error Handling
+
+### Common Errors
+
+#### Missing Required Attribute
+
+**CSV**:
+
+```csv
+uid,cn,mail
+jdoe,John Doe,jdoe@example.com
+```
+
+**Error**:
+
+```json
+{
+  "line": 2,
+  "identifier": "jdoe",
+  "error": "Missing required attribute: sn"
+}
+```
+
+#### Invalid Organization DN
+
+**CSV**:
+
+```csv
+uid,cn,sn,organizationDn
+jdoe,John Doe,Doe,ou=NonExistent,dc=example,dc=com
+```
+
+**Error**:
+
+```json
+{
+  "line": 2,
+  "identifier": "jdoe",
+  "error": "Organization not found: ou=NonExistent,dc=example,dc=com"
+}
+```
+
+#### Duplicate Entry
+
+**Scenario**: User already exists in LDAP
+
+**CSV**:
+
+```csv
+uid,cn,sn,mail
+existinguser,Updated Name,User,new@example.com
+newuser,New User,User,new2@example.com
+```
+
+**With `updateExisting=false` (default)**:
+
+```json
+{
+  "success": true,
+  "total": 2,
+  "created": 1,
+  "skipped": 1,
+  "failed": 0,
+  "errors": []
+}
+```
+
+- `existinguser`: ⏭️ Skipped (already exists)
+- `newuser`: ✅ Created
+
+**With `updateExisting=true`**:
+
+```json
+{
+  "success": true,
+  "total": 2,
+  "created": 1,
+  "updated": 1,
+  "skipped": 0,
+  "failed": 0,
+  "errors": []
+}
+```
+
+- `existinguser`: ✏️ Updated with new values
+- `newuser`: ✅ Created
+
+**Behavior**:
+
+- With `updateExisting=false`: Skipped (not an error)
+- With `updateExisting=true`: Updated
+
+### Error Response Format
+
+```json
+{
+  "success": true,
+  "total": 10,
+  "created": 7,
+  "failed": 3,
+  "errors": [
+    {
+      "line": 5,
+      "identifier": "user004",
+      "error": "Missing required attribute: mail"
+    },
+    {
+      "line": 8,
+      "identifier": "user007",
+      "error": "Organization not found: ou=Invalid,dc=example,dc=com"
+    },
+    {
+      "line": 10,
+      "identifier": "user009",
+      "error": "Invalid DN format"
+    }
+  ]
+}
+```
+
+## Best Practices
+
+### Handling Partial Failures
+
+When importing large batches, some entries may fail while others succeed:
+
+**Recommended workflow**:
+
+1. **First import with continueOnError=true**:
+
+   ```bash
+   curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+     -F "file=@users.csv" \
+     -F "continueOnError=true"
+   ```
+
+2. **Check response for errors**:
+
+   ```json
+   {
+     "created": 95,
+     "failed": 5,
+     "errors": [
+       { "line": 10, "identifier": "user10", "error": "..." },
+       { "line": 25, "identifier": "user25", "error": "..." }
+     ]
+   }
+   ```
+
+3. **Fix failed entries in CSV**:
+   - Extract lines 10, 25, etc.
+   - Correct the errors
+   - Create a new CSV with only failed entries
+
+4. **Re-import fixed entries**:
+   ```bash
+   curl -X POST "http://localhost:8081/api/v1/ldap/bulk-import/users" \
+     -F "file=@failed-users-fixed.csv" \
+     -F "continueOnError=true"
+   ```
+
+**Result**: All entries successfully imported without duplicates.
+
+### Re-running an Import
+
+If you need to re-run an import (e.g., after partial failure):
+
+**Option 1: Skip existing entries** (default)
+
+```bash
+# Only creates new entries, skips existing ones
+-F "updateExisting=false"
+```
+
+- Safe for re-runs
+- Won't modify existing data
+- Good for: Adding new entries only
+
+**Option 2: Update existing entries**
+
+```bash
+# Updates existing entries with CSV data
+-F "updateExisting=true"
+```
+
+- Updates all attributes from CSV
+- Good for: Correcting bulk data, synchronization
+
+**Option 3: Delete and re-create**
+
+```bash
+# First, delete all entries manually or via API
+# Then re-import with fresh data
+```
+
+- Clean slate approach
+- Good for: Testing, development
+
+## Authorization
+
+The bulk import plugin respects all configured authorization plugins:
+
+- Uses `ldapaddrequest` hook for each entry
+- Checks write permissions on target branch
+- Checks organization access permissions
+- Supports `authzPerBranch` plugin
+
+**Example with authzPerBranch**:
+
+If user only has write permission on `ou=Engineering`:
+
+- Can import users to `ou=Engineering`
+- Cannot import users to `ou=Marketing` (fails with permission error)
+
+## Performance
+
+### Batch Processing
+
+The plugin processes records in batches (default: 100):
+
+```bash
+--bulk-import-batch-size 100
+```
+
+**Recommendations**:
+
+- **Small files (<1000 records)**: Use default batch size
+- **Large files (>10000 records)**: Increase to 500-1000
+- **Slow LDAP server**: Decrease to 50
+
+### File Size Limits
+
+Default maximum file size: 10MB
+
+```bash
+--bulk-import-max-file-size 10485760  # 10MB
+--bulk-import-max-file-size 52428800  # 50MB
+```
+
+**File size estimates**:
+
+- ~100 bytes per user record
+- 10MB ≈ 100,000 users
+- 50MB ≈ 500,000 users
+
+### Processing Speed
+
+Typical performance (depends on LDAP server and network):
+
+- **Dry run**: ~1000 records/second
+- **Create**: ~100-500 records/second
+- **Update**: ~50-200 records/second
+
+## Integration with Other Plugins
+
+### With ldapFlat
+
+```bash
+--plugin core/ldap/flatGeneric \
+--flat-resources "users:./static/schemas/twake/users.json" \
+--plugin core/ldap/bulkImport \
+--bulk-import-schemas "users:./static/schemas/twake/users.json"
+```
+
+Both plugins share the same schema for consistency.
+
+### With authzPerBranch
+
+```bash
+--plugin core/auth/authzPerBranch \
+--authz-per-branch-config '{"users":{"admin":{"ou=users,dc=example,dc=com":{"write":true}}}}' \
+--plugin core/ldap/bulkImport \
+--bulk-import-schemas "users:./static/schemas/twake/users.json"
+```
+
+Bulk import respects authorization rules.
+
+### With onChange/James
+
+```bash
+--plugin core/ldap/onChange \
+--plugin twake/james \
+--plugin core/ldap/bulkImport \
+--bulk-import-schemas "users:./static/schemas/twake/users.json"
+```
+
+Each imported user triggers `onChange` hooks (e.g., creates mailbox in James).
+
+## Troubleshooting
+
+### Problem: "No file uploaded"
+
+**Cause**: Missing file in request
+
+**Solution**:
+
+```bash
+curl -X POST "..." -F "file=@users.csv"  # Correct
+curl -X POST "..." -d "file=users.csv"   # Wrong
+```
+
+### Problem: "Only CSV files are allowed"
+
+**Cause**: File doesn't have `.csv` extension or correct mimetype
+
+**Solution**:
+
+- Rename file to `*.csv`
+- Check file mimetype: `file --mime-type users.csv`
+
+### Problem: All records failed with "Missing required attribute"
+
+**Cause**: CSV headers don't match schema attribute names
+
+**Solution**:
+
+- Download template with `GET /template.csv`
+- Ensure CSV headers match exactly (case-sensitive)
+- Check for extra spaces in headers
+
+### Problem: "Organization not found"
+
+**Cause**: `organizationDn` in CSV points to non-existent organization
+
+**Solution**:
+
+- Verify organization exists: `ldapsearch -x -b "ou=IT,..." -s base`
+- Check DN syntax (commas, spaces, escaping)
+- Create missing organizations first
+
+### Problem: Slow import (timeout)
+
+**Cause**: Too many records or slow LDAP server
+
+**Solution**:
+
+- Split large CSV into smaller files
+- Increase timeout in HTTP client
+- Reduce `--bulk-import-batch-size`
+- Use `dryRun` to estimate duration first
+
+## See Also
+
+- [LDAP Flat Generic Plugin](ldapFlatGeneric.md) - For individual CRUD operations
+- [Authorization Plugin](authzPerBranch.md) - For access control
+- [JSON Schemas](schemas/SCHEMAS.md) - For schema format reference
+- [LDAP Organizations](ldapOrganizations.md) - For organization management