convert-csv-to-json 3.28.0 → 4.1.0

package/migration/RFC4180_MIGRATION_GUIDE.md

@@ -0,0 +1,737 @@
+# RFC 4180 Migration Guide
+
+## Overview
+
+This guide helps developers migrate to csvToJson v3.21.0, which introduces **full RFC 4180 compliance**. RFC 4180 is the standard specification for CSV (Comma-Separated Values) files.
+
+**⏱️ Estimated reading time: 15-20 minutes**
+
+---
+
+## Table of Contents
+
+1. [What is RFC 4180?](#what-is-rfc-4180)
+2. [Breaking Changes](#breaking-changes)
+3. [Migration Checklist](#migration-checklist)
+4. [Quick Migration Examples](#quick-migration-examples)
+5. [Detailed Migration Scenarios](#detailed-migration-scenarios)
+6. [Testing Your Migration](#testing-your-migration)
+7. [Troubleshooting](#troubleshooting)
+8. [FAQ](#faq)
+
+---
+
+## What is RFC 4180?
+
+RFC 4180 is the Internet Engineering Task Force (IETF) standard specification for CSV files. It defines:
+
+### Key Specifications
+
+| Aspect | RFC 4180 Standard |
+|--------|-------------------|
+| **Field Delimiter** | Comma (`,`) |
+| **Record Delimiter** | CRLF (`\r\n`) or LF (`\n`) |
+| **Quote Character** | Double-quote (`"`) |
+| **Quoted Fields** | Allowed for any field |
+| **Quote Escaping** | Use double quotes (`""`) |
+| **MIME Type** | `text/csv` |
+| **Charset** | UTF-8 (recommended) |
+
+### RFC 4180 CSV Example
+
+```csv
+firstName,lastName,email,status
+"Smith, John",Smith,john@example.com,"Active, Verified"
+Jane,Doe,jane@example.com,"Inactive"
+"Cooper, Andy",Cooper,"andy@company.com, andy.cooper@home.com","Active"
+```
+
+**Key Features Demonstrated:**
+- Comma delimiters
+- Quoted fields containing commas
+- Escaped quotes using `""`
+- Field names (header row)
+
+---
+
+## Breaking Changes
+
+### 1. Default Delimiter: Semicolon → Comma
+
+**The most significant breaking change**
+
+#### Context
+- **Before**: Default delimiter was semicolon (`;`)
+- **After**: Default delimiter is comma (`,`) - per RFC 4180 standard
+- **Reason**: RFC 4180 specifies comma as the standard delimiter
+
+#### Impact Analysis
+
+| Scenario | Impact | Action Required |
+|----------|--------|-----------------|
+| Using explicit `.fieldDelimiter()` | ✅ None | No changes needed |
+| Parsing comma-delimited CSV | ✅ Improved | May see better results |
+| Parsing semicolon-delimited CSV | ❌ Breaking | Add `.fieldDelimiter(';')` |
+| No delimiter configuration | ❌ Breaking | Verify CSV format |
+
+#### Before Code (Old Behavior)
+```javascript
+const csvToJson = require('convert-csv-to-json');
+
+// Assuming file uses semicolons as delimiter
+// data.csv:
+// name;email;age
+// John;john@example.com;30
+
+let result = csvToJson.getJsonFromCsv('data.csv');
+// Works - automatically used semicolon
+```
+
+#### After Code (New RFC 4180 Behavior)
+```javascript
+const csvToJson = require('convert-csv-to-json');
+
+// data.csv (with semicolons):
+// name;email;age
+// John;john@example.com;30
+
+let result = csvToJson.getJsonFromCsv('data.csv');
+// ❌ BREAKS - expects comma delimiter now
+// Result: { 'name;email;age': 'John;john@example.com;30' }
+
+// ✅ FIX: Explicitly set semicolon delimiter
+let result = csvToJson.fieldDelimiter(';').getJsonFromCsv('data.csv');
+// Works correctly
+```
+
+### 2. Removed Deprecated Function `jsonToCsv()`
+
+**This function has been completely removed**
+
+#### Before
+```javascript
+csvToJson.jsonToCsv('input.csv', 'output.json');
+```
+
+#### After
+```javascript
+csvToJson.generateJsonFileFromCsv('input.csv', 'output.json');
+```
+
+### 3. Stricter Quoted Field Parsing
+
+RFC 4180 compliance means stricter adherence to the standard.
+
+#### Quote Escaping Changes
+
+**Before (Non-compliant):**
+```javascript
+// Input CSV: name,description
+//            "John","He said ""Hello"""
+
+let result = csvToJson.supportQuotedField(true).csvStringToJson(csv);
+// Result might have inconsistent quote handling
+```
+
+**After (RFC 4180 Compliant):**
+```javascript
+// Input CSV: name,description
+//            "John","He said ""Hello"""
+
+let result = csvToJson.supportQuotedField(true).csvStringToJson(csv);
+// Result correctly unescapes to: He said "Hello"
+```
+
+#### Multi-line Field Handling
+
+**Before:**
+- Multi-line fields in quoted regions might not parse correctly
+
+**After:**
+- Full support for multi-line fields within quoted regions
+- Newlines preserved inside quoted fields
+- Line ending detection: CRLF, LF, CR all supported
+
+```javascript
+let csv = 'id,description\n' +
+          '1,"Multi-line\ndescription\nhere"\n' +
+          '2,"Single line"\n';
+
+let result = csvToJson.supportQuotedField(true).csvStringToJson(csv);
+// Correctly handles newlines inside quoted fields
+```
+
+---
+
+## Migration Checklist
+
+Use this checklist to identify and fix breaking changes:
+
+- [ ] **Audit CSV files** - Identify delimiter used (`,`, `;`, `|`, `\t`, etc.)
+- [ ] **Search codebase** - Find all `csvToJson` calls
+- [ ] **Check default usage** - Look for calls without `.fieldDelimiter()`
+- [ ] **Update non-comma delimiters** - Add `.fieldDelimiter()` calls
+- [ ] **Replace deprecated functions** - Change `jsonToCsv()` to `generateJsonFileFromCsv()`
+- [ ] **Test quoted fields** - Verify fields with quotes parse correctly
+- [ ] **Test multi-line fields** - Verify newlines in fields work
+- [ ] **Run test suite** - Execute existing tests to find issues
+- [ ] **Update documentation** - Document any changes to your API
+
+---
+
+## Quick Migration Examples
+
+### Scenario 1: Semicolon-Delimited Files
+
+**Affected Code:**
+```javascript
+// Before (v3.20 and earlier)
+const csvToJson = require('convert-csv-to-json');
+let data = csvToJson.getJsonFromCsv('data.csv'); // Worked with semicolons
+```
+
+**Migration:**
+```javascript
+// After (v3.21)
+const csvToJson = require('convert-csv-to-json');
+let data = csvToJson.fieldDelimiter(';').getJsonFromCsv('data.csv');
+```
+
+### Scenario 2: Tab-Delimited Files
+
+**Affected Code:**
+```javascript
+// Before
+let data = csvToJson.getJsonFromCsv('data.tsv');
+```
+
+**Migration:**
+```javascript
+// After
+let data = csvToJson.fieldDelimiter('\t').getJsonFromCsv('data.tsv');
+```
+
+### Scenario 3: Pipe-Delimited Files
+
+**Affected Code:**
+```javascript
+// Before
+let data = csvToJson.getJsonFromCsv('data.psv');
+```
+
+**Migration:**
+```javascript
+// After
+let data = csvToJson.fieldDelimiter('|').getJsonFromCsv('data.psv');
+```
+
+### Scenario 4: Standard Comma-Delimited CSV
+
+**No Changes Needed:**
+```javascript
+// Before (already works the same)
+let data = csvToJson.getJsonFromCsv('data.csv');
+
+// After (still works the same)
+let data = csvToJson.getJsonFromCsv('data.csv');
+// Now works out-of-the-box without explicit delimiter!
+```
+
+### Scenario 5: Remove Deprecated Function
+
+**Before:**
+```javascript
+csvToJson.jsonToCsv('input.csv', 'output.json');
+```
+
+**After:**
+```javascript
+csvToJson.generateJsonFileFromCsv('input.csv', 'output.json');
+```
+
+---
+
+## Detailed Migration Scenarios
+
+### Scenario A: Express.js CSV Upload Handler
+
+**Original Code (Pre-RFC 4180):**
+```javascript
+const express = require('express');
+const csvToJson = require('convert-csv-to-json');
+const app = express();
+
+app.post('/upload', (req, res) => {
+  try {
+    // Assumes semicolon delimiter (old default)
+    let result = csvToJson.getJsonFromCsv(req.file.path);
+    res.json(result);
+  } catch (error) {
+    res.status(400).json({ error: error.message });
+  }
+});
+```
+
+**Migrated Code (RFC 4180 Compliant):**
+```javascript
+const express = require('express');
+const csvToJson = require('convert-csv-to-json');
+const app = express();
+
+app.post('/upload', (req, res) => {
+  try {
+    // Detect delimiter or use user-provided delimiter
+    const delimiter = req.body.delimiter || ','; // Default to comma (RFC 4180)
+    
+    let result = csvToJson
+      .fieldDelimiter(delimiter)
+      .supportQuotedField(true)
+      .getJsonFromCsv(req.file.path);
+    
+    res.json(result);
+  } catch (error) {
+    res.status(400).json({ error: error.message });
+  }
+});
+```
+
+### Scenario B: Batch CSV Processing Script
+
+**Original Code:**
+```javascript
+const csvToJson = require('convert-csv-to-json');
+const fs = require('fs');
+const path = require('path');
+
+// Process all CSV files in a directory
+const csvDir = './data/csv';
+fs.readdirSync(csvDir).forEach(file => {
+  if (file.endsWith('.csv')) {
+    const csvPath = path.join(csvDir, file);
+    const result = csvToJson.getJsonFromCsv(csvPath);
+    console.log(`Processed ${file}: ${result.length} records`);
+  }
+});
+```
+
+**Migrated Code:**
+```javascript
+const csvToJson = require('convert-csv-to-json');
+const fs = require('fs');
+const path = require('path');
+
+// Configuration by file extension or name pattern
+const delimiterConfig = {
+  'semicolon': ';',
+  'tab': '\t',
+  'pipe': '|',
+  'default': ','
+};
+
+function detectDelimiter(filename) {
+  // Example: detect based on filename pattern
+  if (filename.includes('semi')) return delimiterConfig.semicolon;
+  if (filename.includes('tsv')) return delimiterConfig.tab;
+  if (filename.includes('pipe')) return delimiterConfig.pipe;
+  return delimiterConfig.default; // RFC 4180 default
+}
+
+// Process all CSV files in a directory
+const csvDir = './data/csv';
+fs.readdirSync(csvDir).forEach(file => {
+  if (file.endsWith('.csv') || file.endsWith('.tsv')) {
+    const csvPath = path.join(csvDir, file);
+    const delimiter = detectDelimiter(file);
+    
+    try {
+      const result = csvToJson
+        .fieldDelimiter(delimiter)
+        .supportQuotedField(true)
+        .getJsonFromCsv(csvPath);
+      console.log(`✓ Processed ${file}: ${result.length} records`);
+    } catch (error) {
+      console.error(`✗ Error processing ${file}: ${error.message}`);
+    }
+  }
+});
+```
+
+### Scenario C: Data Import with Validation
+
+**Original Code:**
+```javascript
+const csvToJson = require('convert-csv-to-json');
+
+function importUserData(csvFile) {
+  let data = csvToJson.getJsonFromCsv(csvFile);
+  
+  // Validate and import
+  data.forEach(user => {
+    if (user.email && user.name) {
+      saveUser(user);
+    }
+  });
+}
+```
+
+**Migrated Code:**
+```javascript
+const csvToJson = require('convert-csv-to-json');
+
+function importUserData(csvFile, options = {}) {
+  const delimiter = options.delimiter || ','; // RFC 4180 default
+  const hasQuotedFields = options.hasQuotedFields !== false; // Default true
+  
+  try {
+    let data = csvToJson
+      .fieldDelimiter(delimiter)
+      .supportQuotedField(hasQuotedFields)
+      .formatValueByType(true) // Format values by type
+      .getJsonFromCsv(csvFile);
+    
+    // Validate and import
+    const imported = [];
+    const errors = [];
+    
+    data.forEach((user, index) => {
+      if (user.email && user.name) {
+        try {
+          saveUser(user);
+          imported.push(user.email);
+        } catch (error) {
+          errors.push(`Row ${index + 2}: ${error.message}`);
+        }
+      } else {
+        errors.push(`Row ${index + 2}: Missing email or name`);
+      }
+    });
+    
+    return { imported, errors };
+  } catch (error) {
+    throw new Error(`CSV processing failed: ${error.message}`);
+  }
+}
+```
+
+### Scenario D: Backward Compatibility Wrapper
+
+**If you need to maintain backward compatibility:**
+
+```javascript
+const csvToJsonLib = require('convert-csv-to-json');
+
+// Wrapper with backward-compatible defaults
+class CSVConverter {
+  constructor(options = {}) {
+    // Use RFC 4180 standard by default, but allow override
+    this.defaultDelimiter = options.defaultDelimiter || ',';
+    this.legacyMode = options.legacyMode || false;
+  }
+  
+  getJsonFromCsv(filepath, options = {}) {
+    const delimiter = options.delimiter || this.defaultDelimiter;
+    
+    // In legacy mode, default to semicolon
+    if (this.legacyMode && !options.delimiter) {
+      return csvToJsonLib
+        .fieldDelimiter(';')
+        .getJsonFromCsv(filepath);
+    }
+    
+    return csvToJsonLib
+      .fieldDelimiter(delimiter)
+      .supportQuotedField(options.supportQuotedField !== false)
+      .getJsonFromCsv(filepath);
+  }
+}
+
+// Usage - RFC 4180 mode (default)
+const converter = new CSVConverter();
+let data = converter.getJsonFromCsv('data.csv');
+
+// Usage - Legacy mode (backward compatible)
+const legacyConverter = new CSVConverter({ legacyMode: true });
+let legacyData = legacyConverter.getJsonFromCsv('legacy_data.csv');
+
+// Usage - Custom delimiter
+let tabData = converter.getJsonFromCsv('data.tsv', { delimiter: '\t' });
+```
+
+---
+
+## Testing Your Migration
+
+### Step 1: Unit Tests
+
+Create tests to verify both old and new behavior:
+
+```javascript
+const csvToJson = require('convert-csv-to-json');
+
+describe('RFC 4180 Migration Tests', () => {
+  test('should parse comma-delimited CSV (RFC 4180 default)', () => {
+    const csv = 'name,age\nJohn,30\nJane,25\n';
+    const result = csvToJson.csvStringToJson(csv);
+    
+    expect(result.length).toBe(2);
+    expect(result[0].name).toBe('John');
+  });
+  
+  test('should parse semicolon-delimited CSV with explicit delimiter', () => {
+    const csv = 'name;age\nJohn;30\nJane;25\n';
+    const result = csvToJson
+      .fieldDelimiter(';')
+      .csvStringToJson(csv);
+    
+    expect(result.length).toBe(2);
+    expect(result[0].name).toBe('John');
+  });
+  
+  test('should handle quoted fields per RFC 4180', () => {
+    const csv = 'name,description\n"Smith, John","He said ""Hello"""\n';
+    const result = csvToJson
+      .supportQuotedField(true)
+      .csvStringToJson(csv);
+    
+    expect(result[0].name).toBe('Smith, John');
+    expect(result[0].description).toBe('He said "Hello"');
+  });
+});
+```
+
+### Step 2: Integration Tests
+
+Test with real CSV files:
+
+```javascript
+describe('Real CSV File Integration', () => {
+  test('should process standard RFC 4180 CSV', () => {
+    const result = csvToJson.getJsonFromCsv('./test/data/standard.csv');
+    expect(result.length).toBeGreaterThan(0);
+  });
+  
+  test('should process legacy semicolon-delimited CSV', () => {
+    const result = csvToJson
+      .fieldDelimiter(';')
+      .getJsonFromCsv('./test/data/legacy.csv');
+    expect(result.length).toBeGreaterThan(0);
+  });
+});
+```
+
+### Step 3: Validation Checklist
+
+```javascript
+// Test for correct parsing
+✓ Basic parsing works
+✓ Field count matches headers
+✓ Empty fields handled correctly
+✓ Quoted fields with commas work
+✓ Escaped quotes unescaped correctly
+✓ Multi-line fields preserved
+✓ Line endings detected correctly (CRLF, LF, CR)
+✓ Special characters preserved
+✓ Type formatting works
+✓ Boolean/number conversion works
+✓ Error handling for malformed CSV
+```
+
+---
+
+## Troubleshooting
+
+### Problem 1: "Headers Not Found" Error
+
+**Symptom:**
+```javascript
+// Works in v3.20, breaks in v3.21
+csvToJson.getJsonFromCsv('mydata.csv');
+// Error: No header row found in CSV
+```
+
+**Cause:** File likely uses semicolon delimiter (old default changed to comma)
+
+**Solution:**
+```javascript
+// Add explicit delimiter
+csvToJson.fieldDelimiter(';').getJsonFromCsv('mydata.csv');
+
+// Or inspect the file first
+const fs = require('fs');
+const firstLine = fs.readFileSync('mydata.csv', 'utf8').split('\n')[0];
+console.log('First line:', firstLine);
+// Count delimiters to detect type
+```
+
+### Problem 2: Fields Not Splitting Correctly
+
+**Symptom:**
+```javascript
+// One field shows entire row content
+[{ 'field1;field2;field3': '...value...value...value' }]
+```
+
+**Cause:** Wrong delimiter configured or auto-detection failed
+
+**Solution:**
+```javascript
+// Explicitly set the delimiter
+csvToJson.fieldDelimiter(';').getJsonFromCsv('file.csv');
+
+// Or auto-detect:
+function detectDelimiter(filePath) {
+  const fs = require('fs');
+  const line = fs.readFileSync(filePath, 'utf8').split('\n')[0];
+  
+  const delimiters = [',', ';', '|', '\t'];
+  for (let delim of delimiters) {
+    if (line.includes(delim)) {
+      return delim;
+    }
+  }
+  return ','; // Default to RFC 4180
+}
+
+const delimiter = detectDelimiter('file.csv');
+csvToJson.fieldDelimiter(delimiter).getJsonFromCsv('file.csv');
+```
+
+### Problem 3: Quoted Fields Not Working
+
+**Symptom:**
+```javascript
+// With input: name,value
+//             "John","He said ""Hi"""
+let result = csvToJson.getJsonFromCsv('file.csv');
+// Result shows: { name: '"John"', value: '"He said ""Hi"""' }
+```
+
+**Cause:** `.supportQuotedField()` not enabled
+
+**Solution:**
+```javascript
+let result = csvToJson
+  .supportQuotedField(true)
+  .getJsonFromCsv('file.csv');
+// Now works correctly
+```
+
+### Problem 4: `jsonToCsv` Function Not Found
+
+**Symptom:**
+```javascript
+csvToJson.jsonToCsv('input.csv', 'output.json');
+// TypeError: csvToJson.jsonToCsv is not a function
+```
+
+**Cause:** Function was removed (it was deprecated)
+
+**Solution:**
+```javascript
+// Replace with the non-deprecated function
+csvToJson.generateJsonFileFromCsv('input.csv', 'output.json');
+```
+
+### Problem 5: Multi-line Fields Not Preserved
+
+**Symptom:**
+```javascript
+// Input has field with newlines inside quotes
+// But newlines are stripped or cause parsing errors
+```
+
+**Cause:** Parsing multi-line data requires `supportQuotedField(true)`
+
+**Solution:**
+```javascript
+let result = csvToJson
+  .supportQuotedField(true)
+  .getJsonFromCsv('file.csv');
+// Newlines inside quoted fields now preserved
+```
+
+---
+
+## FAQ
+
+### Q: Do I need to update my code?
+
+**A:** Only if you:
+- Don't explicitly set `.fieldDelimiter()` and use non-comma delimiters
+- Use the removed `jsonToCsv()` function
+- Parse quoted CSV files without `.supportQuotedField(true)`
+
+### Q: Is this a breaking change?
+
+**A:** Yes, the default delimiter changed from semicolon to comma. However, most code using explicit `.fieldDelimiter()` is unaffected.
+
+### Q: Why change the default delimiter?
+
+**A:** RFC 4180 (the standard specification) defines comma as the standard delimiter. This makes the library RFC 4180 compliant out-of-the-box.
+
+### Q: Can I still use semicolons?
+
+**A:** Yes, just use: `.fieldDelimiter(';').getJsonFromCsv('file.csv')`
+
+### Q: What about backward compatibility?
+
+**A:** Existing code with explicit delimiters works without changes. Only code relying on the old default delimiter needs updates.
+
+### Q: Can I detect the delimiter automatically?
+
+**A:** Yes, see the troubleshooting section for delimiter detection code.
+
+### Q: Are all my CSV files now broken?
+
+**A:** Only if they use non-comma delimiters AND you were relying on the old default. Explicitly set the delimiter to fix.
+
+### Q: What's RFC 4180?
+
+**A:** It's the official IETF standard for CSV file format. Read more: https://tools.ietf.org/html/rfc4180
+
+### Q: Do I need to update my tests?
+
+**A:** Yes, update tests that assume the old semicolon default or test the removed `jsonToCsv()` function.
+
+### Q: How do I test my migration?
+
+**A:** Run existing tests first to identify failures, then update code and delimiters as needed. See "Testing Your Migration" section above.
+
+### Q: What if I have legacy code I can't change?
+
+**A:** You can create a wrapper class (see Scenario D) that uses the old default delimiter and maintains backward compatibility.
+
+### Q: Where can I get help?
+
+**A:** Check GitHub Issues: https://github.com/iuccio/csvToJson/issues
+
+---
+
+## Summary
+
+**Key Points to Remember:**
+
+1. **Default delimiter is now comma (`,`)** - per RFC 4180 standard
+2. **Explicitly set delimiters** for non-comma files: `.fieldDelimiter(';')`
+3. **Remove `jsonToCsv()` calls** - use `generateJsonFileFromCsv()` instead
+4. **Enable quoted field support** for complex CSV: `.supportQuotedField(true)`
+5. **Test thoroughly** - especially if not using explicit delimiters
+6. **Refer to RFC 4180** - for standard CSV format specification
+
+---
+
+## Next Steps
+
+1. ✅ Review breaking changes above
+2. ✅ Audit your codebase for affected code
+3. ✅ Update code with explicit delimiters as needed
+4. ✅ Run tests to identify issues
+5. ✅ Update affected tests
+6. ✅ Deploy with confidence!
+
+---
+
+**For more information, see:**
+- [Release Notes](./RELEASE_NOTES.md)
+- [README](./README.md)
+- [RFC 4180 Specification](https://tools.ietf.org/html/rfc4180)