convert-csv-to-json 3.28.0 → 4.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -1
- package/README.md +173 -833
- package/docs/ASYNC.md +485 -0
- package/docs/BROWSER.md +425 -0
- package/docs/ERROR_HANDLING.md +367 -0
- package/docs/MAPROWS.md +196 -0
- package/docs/SYNC.md +239 -0
- package/index.d.ts +10 -8
- package/index.js +13 -10
- package/migration/RFC4180_MIGRATION_GUIDE.md +737 -0
- package/package.json +1 -1
- package/release-notes/RELEASE_NOTES_v4.0.0.md +320 -0
- package/src/browserApi.js +29 -6
- package/src/csvToJson.js +233 -82
- package/src/csvToJsonAsync.js +15 -1
- package/src/util/errors.js +227 -0
- package/src/util/fileUtils.js +18 -7
- package/src/util/jsonUtils.js +3 -1
- /package/{MIGRATION.md → migration/MIGRATION_TO_ASYNC.md} +0 -0
package/README.md
CHANGED
|
@@ -1,4 +1,5 @@
|
|
|
1
|
-
# CSVtoJSON
|
|
1
|
+
# CSVtoJSON
|
|
2
|
+
|
|
2
3
|
[](https://github.com/iuccio/csvToJson/actions/workflows/ci-cd.yml)
|
|
3
4
|

|
|
4
5
|
[](https://snyk.io/test/github/iuccio/csvToJson)
|
|
@@ -7,952 +8,291 @@
|
|
|
7
8
|

|
|
8
9
|
[](https://npmjs.org/package/convert-csv-to-json)
|
|
9
10
|
[](https://npmjs.org/package/convert-csv-to-json)
|
|
11
|
+
[](https://socket.dev/npm/package/convert-csv-to-json)
|
|
10
12
|
|
|
11
13
|
|
|
12
14
|

|
|
13
|
-

|
|
14
|
-

|
|
15
|
-

|
|
16
|
-
|
|
17
|
-
**This project is not dependent on others packages or libraries, and supports both synchronous and Promise-based asynchronous APIs.**
|
|
18
|
-
|
|
19
|
-
This repository uses [](https://github.com/marketplace/actions/npm-semver-publish)
|
|
20
|
-
|
|
21
|
-
Follow [me](https://github.com/iuccio), and consider starring the project to
|
|
22
|
-
show your :heart: and support.
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
## Table of Contents
|
|
26
|
-
|
|
27
|
-
<!-- toc -->
|
|
28
|
-
|
|
29
|
-
- [Description](#description)
|
|
30
|
-
- [Support for NodeJS, Browser, JS, TS](#support-for-nodejs-browser-js-ts)
|
|
31
|
-
- [Prerequisites](#prerequisites)
|
|
32
|
-
- [Install npm *convert-csv-to-json package*](#install-npm-convert-csv-to-json-package)
|
|
33
|
-
- [Sync API Usage](#sync-api-usage)
|
|
34
|
-
* [Generate JSON file](#generate-json-file)
|
|
35
|
-
* [Generate Array of Object in JSON format](#generate-array-of-object-in-json-format)
|
|
36
|
-
* [Generate Object with sub array](#generate-object-with-sub-array)
|
|
37
|
-
* [Define field delimiter](#define-field-delimiter)
|
|
38
|
-
* [Trim header field](#trim-header-field)
|
|
39
|
-
* [Trim header field with whitespaces](#trim-header-field-with-whitespaces)
|
|
40
|
-
* [Support Quoted Fields](#support-quoted-fields)
|
|
41
|
-
* [Index header](#index-header)
|
|
42
|
-
* [Empty rows](#empty-rows)
|
|
43
|
-
* [Format property value by type](#format-property-value-by-type)
|
|
44
|
-
+ [Numbers](#numbers)
|
|
45
|
-
+ [Boolean](#boolean)
|
|
46
|
-
+ [Complete Example](#complete-example)
|
|
47
|
-
* [Encoding](#encoding)
|
|
48
|
-
* [Working with CSV strings directly](#working-with-csv-strings-directly)
|
|
49
|
-
* [Sync API (TypeScript)](#sync-api-typescript)
|
|
50
|
-
- [Browser API Usage](#browser-api-usage)
|
|
51
|
-
* [Basic Browser Operations](#basic-browser-operations)
|
|
52
|
-
* [Parsing File/Blob](#parsing-fileblob)
|
|
53
|
-
* [Browser API Notes](#browser-api-notes)
|
|
54
|
-
* [Browser API (TypeScript)](#browser-api-typescript)
|
|
55
|
-
- [Async API Usage](#async-api-usage)
|
|
56
|
-
* [Async API (TypeScript)](#async-api-typescript)
|
|
57
|
-
* [Basic Async Operations](#basic-async-operations)
|
|
58
|
-
* [Working with Raw CSV Data](#working-with-raw-csv-data)
|
|
59
|
-
* [Processing Large Files](#processing-large-files)
|
|
60
|
-
* [Error Handling and Retries](#error-handling-and-retries)
|
|
61
|
-
* [Batch Processing](#batch-processing)
|
|
62
|
-
- [Chaining Pattern](#chaining-pattern)
|
|
63
|
-
* [Synchronous Chaining](#synchronous-chaining)
|
|
64
|
-
* [Asynchronous Chaining](#asynchronous-chaining)
|
|
65
|
-
- [Common Use Cases](#common-use-cases)
|
|
66
|
-
* [1. Processing CSV from HTTP Response](#1-processing-csv-from-http-response)
|
|
67
|
-
* [2. Batch Processing Multiple Files](#2-batch-processing-multiple-files)
|
|
68
|
-
* [3. Data Transformation Pipeline](#3-data-transformation-pipeline)
|
|
69
|
-
* [4. Error Recovery and Logging](#4-error-recovery-and-logging)
|
|
70
|
-
- [Troubleshooting](#troubleshooting)
|
|
71
|
-
* [Memory Issues with Large Files](#memory-issues-with-large-files)
|
|
72
|
-
* [Handling Different CSV Formats](#handling-different-csv-formats)
|
|
73
|
-
* [Common Error Solutions](#common-error-solutions)
|
|
74
|
-
* [Performance Optimization](#performance-optimization)
|
|
75
|
-
* [TypeScript Support](#typescript-support)
|
|
76
|
-
- [Development](#development)
|
|
77
|
-
- [CI CD github action](#ci-cd-github-action)
|
|
78
|
-
- [License](#license)
|
|
79
|
-
- [Buy me a Coffee](#buy-me-a-coffee)
|
|
80
|
-
|
|
81
|
-
<!-- tocstop -->
|
|
82
|
-
|
|
83
|
-
## Description
|
|
84
|
-
Converts *csv* files to *JSON* files with Node.js. Supports both synchronous operations and Promise-based asynchronous operations, allowing integration with modern async/await patterns.
|
|
85
|
-
|
|
86
|
-
Give an input file like:
|
|
87
|
-
|
|
88
|
-
|first_name|last_name|email|gender|age|zip|registered|
|
|
89
|
-
|:----------:|:-------:|:---:|:----:|:---:|:---:|:---:|
|
|
90
|
-
|Constantin|Langsdon|clangsdon0@hc360.com|Male|96|123|true|
|
|
91
|
-
|Norah|Raison|nraison1@wired.com|Female|32| |false|
|
|
92
|
-
|
|
93
|
-
e.g. :
|
|
94
|
-
~~~
|
|
95
|
-
first_name;last_name;email;gender;age;zip;registered
|
|
96
|
-
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;123;true
|
|
97
|
-
Norah;Raison;nraison1@wired.com;Female;32;;false
|
|
98
|
-
~~~
|
|
99
|
-
|
|
100
|
-
will generate:
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
```json
|
|
104
|
-
[
|
|
105
|
-
{
|
|
106
|
-
"first_name": "Constantin",
|
|
107
|
-
"last_name": "Langsdon",
|
|
108
|
-
"email": "clangsdon0@hc360.com",
|
|
109
|
-
"gender": "Male",
|
|
110
|
-
"age": "96",
|
|
111
|
-
"zip": "123",
|
|
112
|
-
"registered": "true"
|
|
113
|
-
},
|
|
114
|
-
{
|
|
115
|
-
"first_name": "Norah",
|
|
116
|
-
"last_name": "Raison",
|
|
117
|
-
"email": "nraison1@wired.com",
|
|
118
|
-
"gender": "Female",
|
|
119
|
-
"age": "32",
|
|
120
|
-
"zip": "",
|
|
121
|
-
"registered": "false"
|
|
122
|
-
}
|
|
123
|
-
]
|
|
124
|
-
```
|
|
125
|
-
## Support for NodeJS, Browser, JS, TS
|
|
126
|
-
|
|
127
|
-
This package is compatible with:
|
|
128
|
-
|
|
129
|
-

|
|
130
|
-

|
|
15
|
+

|
|
131
16
|

|
|
132
|
-

|
|
17
|
+

|
|
133
18
|
|
|
134
|
-
|
|
135
|
-
**
|
|
19
|
+
>
|
|
20
|
+
Convert CSV files to JSON with **no dependencies**. Supports Node.js (Sync & Async), and Browser environments with full RFC 4180 compliance.
|
|
136
21
|
|
|
137
|
-
##
|
|
138
|
-
Go to NPM package [convert-csv-to-json](https://www.npmjs.com/package/convert-csv-to-json).
|
|
22
|
+
## Overview
|
|
139
23
|
|
|
140
|
-
|
|
141
|
-
```bash
|
|
142
|
-
$ npm install convert-csv-to-json --save
|
|
143
|
-
```
|
|
144
|
-
Install package on your machine
|
|
145
|
-
```bash
|
|
146
|
-
$ npm install -g convert-csv-to-json
|
|
147
|
-
```
|
|
148
|
-
|
|
149
|
-
## Sync API Usage
|
|
24
|
+
Transform CSV data into JSON with a simple, chainable API. Choose your implementation style:
|
|
150
25
|
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
26
|
+
- **[Synchronous API](docs/SYNC.md)** - Blocking operations for simple workflows
|
|
27
|
+
- **[Asynchronous API](docs/ASYNC.md)** - Promise-based for modern async/await patterns
|
|
28
|
+
- **[Browser API](docs/BROWSER.md)** - Client-side CSV parsing for web applications
|
|
154
29
|
|
|
155
|
-
|
|
156
|
-
let fileOutputName = 'myOutputFile.json';
|
|
30
|
+
## Features
|
|
157
31
|
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
32
|
+
✅ **RFC 4180 Compliant** - Proper handling of quoted fields, delimiters, newlines, and escape sequences
|
|
33
|
+
✅ **Zero Dependencies** - No external packages required
|
|
34
|
+
✅ **Full TypeScript Support** - Included type definitions for all APIs
|
|
35
|
+
✅ **Flexible Configuration** - Custom delimiters, encoding, trimming, and more
|
|
36
|
+
✅ **Method Chaining** - Fluent API for readable code
|
|
37
|
+
✅ **Large File Support** - Stream processing for memory-efficient handling
|
|
38
|
+
✅ **Comprehensive Error Handling** - Detailed, actionable error messages with solutions (see [ERROR_HANDLING.md](docs/ERROR_HANDLING.md))
|
|
163
39
|
|
|
164
|
-
|
|
165
|
-
for(let i=0; i<json.length;i++){
|
|
166
|
-
console.log(json[i]);
|
|
167
|
-
}
|
|
168
|
-
```
|
|
40
|
+
## RFC 4180 Standard
|
|
169
41
|
|
|
170
|
-
|
|
171
|
-
```
|
|
172
|
-
firstName;lastName;email;gender;age;birth;sons
|
|
173
|
-
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;10.02.1965;*diego,marek,dries*
|
|
174
|
-
```
|
|
175
|
-
Given the above CSV example, to generate a JSON Object with properties that contains sub Array, like the property **sons**
|
|
176
|
-
with the values <b>*diego,marek,dries*</b> you have to call the function ```parseSubArray(delimiter, separator)``` .
|
|
177
|
-
To generate the JSON Object with sub array from the above CSV example:
|
|
178
|
-
```js
|
|
179
|
-
csvToJson.parseSubArray('*',',')
|
|
180
|
-
.getJsonFromCsv('myInputFile.csv');
|
|
181
|
-
```
|
|
182
|
-
The result will be:
|
|
183
|
-
```json
|
|
184
|
-
[
|
|
185
|
-
{
|
|
186
|
-
"firstName": "Constantin",
|
|
187
|
-
"lastName": "Langsdon",
|
|
188
|
-
"email": "clangsdon0@hc360.com",
|
|
189
|
-
"gender": "Male",
|
|
190
|
-
"age": "96",
|
|
191
|
-
"birth": "10.02.1965",
|
|
192
|
-
"sons": ["diego","marek","dries"]
|
|
193
|
-
}
|
|
194
|
-
]
|
|
195
|
-
```
|
|
42
|
+
**[RFC 4180](https://datatracker.ietf.org/doc/html/rfc4180)** is the IETF standard specification for CSV (Comma-Separated Values) files. This library is fully compliant with RFC 4180, ensuring proper handling of:
|
|
196
43
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
44
|
+
| Aspect | RFC 4180 Specification |
|
|
45
|
+
|--------|------------------------|
|
|
46
|
+
| **Default Delimiter** | Comma (`,`) |
|
|
47
|
+
| **Record Delimiter** | CRLF (`\r\n`) or LF (`\n`) |
|
|
48
|
+
| **Quote Character** | Double-quote (`"`) |
|
|
49
|
+
| **Quote Escaping** | Double quotes (`""`) |
|
|
200
50
|
|
|
201
|
-
|
|
51
|
+
### RFC 4180 Example
|
|
202
52
|
|
|
203
|
-
```
|
|
204
|
-
|
|
205
|
-
|
|
53
|
+
```csv
|
|
54
|
+
firstName,lastName,email
|
|
55
|
+
"Smith, John",Smith,john@example.com
|
|
56
|
+
Jane,Doe,jane@example.com
|
|
57
|
+
"Cooper, Andy",Cooper,andy@company.com
|
|
206
58
|
```
|
|
207
59
|
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
The content of the field header is cut off at the beginning and end of the string. E.g. " Last Name " -> "Last Name".
|
|
60
|
+
Note the quoted fields containing commas are properly handled. See [RFC4180_MIGRATION_GUIDE.md](RFC4180_MIGRATION_GUIDE.md) for breaking changes and migration details.
|
|
211
61
|
|
|
212
|
-
|
|
62
|
+
## Quick Start
|
|
213
63
|
|
|
214
|
-
|
|
64
|
+
### Installation
|
|
215
65
|
|
|
216
|
-
```
|
|
217
|
-
|
|
218
|
-
.getJsonFromCsv(fileInputName);
|
|
66
|
+
```bash
|
|
67
|
+
npm install convert-csv-to-json
|
|
219
68
|
```
|
|
220
69
|
|
|
221
|
-
###
|
|
222
|
-
To be able to parse correctly fields wrapped in quote, like the **last_name** in the first row in the following example:
|
|
223
|
-
|
|
224
|
-
|first_name| last_name |email|
|
|
225
|
-
|:----------:|:--------------------------:|:---:|
|
|
226
|
-
|Constantin| "Langsdon,Nandson,Gangson" |clangsdon0@hc360.com|
|
|
227
|
-
|
|
228
|
-
you need to activate the support quoted fields feature:
|
|
70
|
+
### Synchronous (Simple)
|
|
229
71
|
|
|
230
72
|
```js
|
|
231
|
-
csvToJson
|
|
232
|
-
|
|
73
|
+
const csvToJson = require('convert-csv-to-json');
|
|
74
|
+
const json = csvToJson.getJsonFromCsv('input.csv');
|
|
233
75
|
```
|
|
234
76
|
|
|
235
|
-
|
|
236
|
-
```json
|
|
237
|
-
[
|
|
238
|
-
{
|
|
239
|
-
"firstName": "Constantin",
|
|
240
|
-
"lastName": "Langsdon,Nandson,Gangson",
|
|
241
|
-
"email": "clangsdon0@hc360.com"
|
|
242
|
-
}
|
|
243
|
-
]
|
|
244
|
-
```
|
|
245
|
-
|
|
246
|
-
### Index header
|
|
247
|
-
If the header is not on the first line you can define the header index like:
|
|
77
|
+
### Asynchronous (Modern)
|
|
248
78
|
|
|
249
79
|
```js
|
|
250
|
-
csvToJson
|
|
251
|
-
|
|
80
|
+
const csvToJson = require('convert-csv-to-json');
|
|
81
|
+
const json = await csvToJson.getJsonFromCsvAsync('input.csv');
|
|
252
82
|
```
|
|
253
83
|
|
|
254
|
-
###
|
|
255
|
-
Empty rows are ignored and not parsed.
|
|
256
|
-
|
|
257
|
-
### Format property value by type
|
|
258
|
-
The `formatValueByType()` function intelligently converts string values to their appropriate types while preserving data integrity. To enable automatic type conversion:
|
|
84
|
+
### Browser
|
|
259
85
|
|
|
260
86
|
```js
|
|
261
|
-
|
|
262
|
-
|
|
263
|
-
```
|
|
264
|
-
|
|
265
|
-
This conversion follows these rules:
|
|
266
|
-
|
|
267
|
-
#### Numbers
|
|
268
|
-
- Regular integers and decimals are converted to Number type
|
|
269
|
-
- Numbers with leading zeros are preserved as strings (e.g., "0012" stays "0012")
|
|
270
|
-
- Large integers outside JavaScript's safe range are preserved as strings
|
|
271
|
-
- Valid decimal numbers are converted to Number type
|
|
272
|
-
|
|
273
|
-
For example:
|
|
274
|
-
```json
|
|
275
|
-
{
|
|
276
|
-
"normalInteger": 42, // Converted to number
|
|
277
|
-
"decimal": 3.14, // Converted to number
|
|
278
|
-
"leadingZeros": "0012345", // Kept as string to preserve leading zeros
|
|
279
|
-
"largeNumber": "9007199254740992" // Kept as string to preserve precision
|
|
280
|
-
}
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
#### Boolean
|
|
284
|
-
Case-insensitive "true" or "false" strings are converted to boolean values:
|
|
285
|
-
```json
|
|
286
|
-
{
|
|
287
|
-
"registered": true, // From "true" or "TRUE" or "True"
|
|
288
|
-
"active": false // From "false" or "FALSE" or "False"
|
|
289
|
-
}
|
|
87
|
+
const convert = require('convert-csv-to-json');
|
|
88
|
+
const json = await convert.browser.parseFile(file);
|
|
290
89
|
```
|
|
291
90
|
|
|
292
|
-
|
|
293
|
-
Input CSV:
|
|
294
|
-
```csv
|
|
295
|
-
first_name;last_name;email;gender;age;id;zip;registered
|
|
296
|
-
Constantin;Langsdon;clangsdon0@hc360.com;Male;96;00123;123;true
|
|
297
|
-
Norah;Raison;nraison1@wired.com;Female;32;987;00456;FALSE
|
|
298
|
-
```
|
|
91
|
+
## Documentation
|
|
299
92
|
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
[
|
|
303
|
-
|
|
304
|
-
|
|
305
|
-
"last_name": "Langsdon",
|
|
306
|
-
"email": "clangsdon0@hc360.com",
|
|
307
|
-
"gender": "Male",
|
|
308
|
-
"age": 96,
|
|
309
|
-
"id": "00123", // Preserved leading zeros
|
|
310
|
-
"zip": 123, // Converted to number
|
|
311
|
-
"registered": true // Converted to boolean
|
|
312
|
-
},
|
|
313
|
-
{
|
|
314
|
-
"first_name": "Norah",
|
|
315
|
-
"last_name": "Raison",
|
|
316
|
-
"email": "nraison1@wired.com",
|
|
317
|
-
"gender": "Female",
|
|
318
|
-
"age": 32,
|
|
319
|
-
"id": "987",
|
|
320
|
-
"zip": "00456", // Preserved leading zeros
|
|
321
|
-
"registered": false // Case-insensitive boolean conversion
|
|
322
|
-
}
|
|
323
|
-
]
|
|
324
|
-
```
|
|
93
|
+
| Implementation | Use Case | Learn More |
|
|
94
|
+
|---|---|---|
|
|
95
|
+
| **Sync API** | Simple, blocking operations | [Read SYNC.md](docs/SYNC.md) |
|
|
96
|
+
| **Async API** | Concurrent operations, large files | [Read ASYNC.md](docs/ASYNC.md) |
|
|
97
|
+
| **Browser API** | Client-side file parsing | [Read BROWSER.md](docs/BROWSER.md) |
|
|
325
98
|
|
|
326
|
-
|
|
327
|
-
You can read and decode files with the following encoding:
|
|
328
|
-
* utf8:
|
|
329
|
-
```js
|
|
330
|
-
csvToJson.utf8Encoding()
|
|
331
|
-
.getJsonFromCsv(fileInputName);
|
|
332
|
-
```
|
|
333
|
-
* ucs2:
|
|
334
|
-
```js
|
|
335
|
-
csvToJson.ucs2Encoding()
|
|
336
|
-
.getJsonFromCsv(fileInputName);
|
|
337
|
-
```
|
|
338
|
-
* utf16le:
|
|
339
|
-
```js
|
|
340
|
-
csvToJson.utf16leEncoding()
|
|
341
|
-
.getJsonFromCsv(fileInputName);
|
|
342
|
-
```
|
|
343
|
-
* latin1:
|
|
344
|
-
```js
|
|
345
|
-
csvToJson.latin1Encoding()
|
|
346
|
-
.getJsonFromCsv(fileInputName);
|
|
347
|
-
```
|
|
348
|
-
* ascii:
|
|
349
|
-
```js
|
|
350
|
-
csvToJson.asciiEncoding()
|
|
351
|
-
.getJsonFromCsv(fileInputName);
|
|
352
|
-
```
|
|
353
|
-
* base64:
|
|
354
|
-
```js
|
|
355
|
-
csvToJson.base64Encoding()
|
|
356
|
-
.getJsonFromCsv(fileInputName);
|
|
357
|
-
```
|
|
358
|
-
* hex:
|
|
359
|
-
```js
|
|
360
|
-
csvToJson.hexEncoding()
|
|
361
|
-
.getJsonFromCsv(fileInputName);
|
|
362
|
-
```
|
|
363
|
-
|
|
364
|
-
### Working with CSV strings directly
|
|
365
|
-
If you have CSV content as a string (for example, from an API response or test data), you can parse it directly without writing to a file:
|
|
99
|
+
## Common Tasks
|
|
366
100
|
|
|
101
|
+
### Parse CSV String
|
|
367
102
|
```js
|
|
368
|
-
|
|
369
|
-
// Parse CSV string to array of objects
|
|
370
|
-
let csvString = 'firstName;lastName\nJohn;Doe\nJane;Smith';
|
|
371
|
-
let jsonArray = csvToJson.csvStringToJson(csvString);
|
|
372
|
-
// Output: [{"firstName":"John","lastName":"Doe"},{"firstName":"Jane","lastName":"Smith"}]
|
|
373
|
-
|
|
374
|
-
// Parse CSV string to JSON string (validated)
|
|
375
|
-
let jsonString = csvToJson.csvStringToJsonStringified(csvString);
|
|
376
|
-
// Output: "[\n {\n \"firstName\": \"John\",\n \"lastName\": \"Doe\"\n },\n {\n \"firstName\": \"Jane\",\n \"lastName\": \"Smith\"\n }\n]"
|
|
103
|
+
const json = csvToJson.csvStringToJson('name,age\nAlice,30');
|
|
377
104
|
```
|
|
378
105
|
|
|
379
|
-
|
|
380
|
-
|
|
106
|
+
### Custom Delimiter
|
|
381
107
|
```js
|
|
382
|
-
|
|
383
|
-
.fieldDelimiter('
|
|
384
|
-
.
|
|
385
|
-
.csvStringToJson(csvString);
|
|
108
|
+
const json = csvToJson
|
|
109
|
+
.fieldDelimiter(';')
|
|
110
|
+
.getJsonFromCsv('input.csv');
|
|
386
111
|
```
|
|
387
112
|
|
|
388
|
-
###
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
```ts
|
|
393
|
-
// Named import (recommended when using ES modules)
|
|
394
|
-
import converter, { /* or */ } from 'convert-csv-to-json';
|
|
395
|
-
// Access the default converter
|
|
396
|
-
const csvToJson = require('convert-csv-to-json');
|
|
397
|
-
|
|
398
|
-
// Define a type for your CSV records
|
|
399
|
-
interface Person {
|
|
400
|
-
name: string;
|
|
401
|
-
age: number;
|
|
402
|
-
}
|
|
403
|
-
|
|
404
|
-
// Parse CSV string synchronously and assert the returned type
|
|
405
|
-
const csv = 'name,age\nAlice,30';
|
|
406
|
-
const parsed = csvToJson.csvStringToJson(csv) as Person[];
|
|
407
|
-
|
|
408
|
-
// Chain configuration and call sync methods
|
|
409
|
-
const result = csvToJson
|
|
410
|
-
.fieldDelimiter(',')
|
|
113
|
+
### Format Values
|
|
114
|
+
```js
|
|
115
|
+
const json = csvToJson
|
|
411
116
|
.formatValueByType()
|
|
412
|
-
.
|
|
117
|
+
.getJsonFromCsv('input.csv');
|
|
118
|
+
// Converts "30" → 30, "true" → true, etc.
|
|
413
119
|
```
|
|
414
120
|
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
The package exposes a `browser` helper that reuses the library's parsing logic but provides browser-friendly helpers for parsing CSV strings and `File`/`Blob` objects. The API mirrors the synchronous and asynchronous Node APIs and supports method chaining for configuration.
|
|
418
|
-
|
|
419
|
-
### Basic Browser Operations
|
|
420
|
-
|
|
121
|
+
### Handle Quoted Fields
|
|
421
122
|
```js
|
|
422
|
-
const
|
|
423
|
-
|
|
424
|
-
// Parse CSV string synchronously
|
|
425
|
-
const arr = convert.browser
|
|
123
|
+
const json = csvToJson
|
|
426
124
|
.supportQuotedField(true)
|
|
427
|
-
.
|
|
428
|
-
.csvStringToJson('name,age\nAlice,30');
|
|
429
|
-
|
|
430
|
-
// Parse CSV string asynchronously (returns Promise)
|
|
431
|
-
const arrAsync = await convert.browser.csvStringToJsonAsync('name;age\nBob;25');
|
|
432
|
-
|
|
433
|
-
// Get stringified JSON synchronously
|
|
434
|
-
const jsonString = convert.browser.csvStringToJsonStringified('name;age\nEve;40');
|
|
125
|
+
.getJsonFromCsv('input.csv');
|
|
435
126
|
```
|
|
436
127
|
|
|
437
|
-
###
|
|
438
|
-
|
|
439
|
-
`parseFile(file, options)` reads a `File` or `Blob` and returns a Promise that resolves with the parsed array of objects.
|
|
440
|
-
|
|
128
|
+
### Batch Process Files (Async)
|
|
441
129
|
```js
|
|
442
|
-
|
|
443
|
-
const
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
.formatValueByType()
|
|
447
|
-
.parseFile(file)
|
|
448
|
-
.then(json => console.log(json))
|
|
449
|
-
.catch(err => console.error(err));
|
|
130
|
+
const files = ['file1.csv', 'file2.csv', 'file3.csv'];
|
|
131
|
+
const results = await Promise.all(
|
|
132
|
+
files.map(f => csvToJson.getJsonFromCsvAsync(f))
|
|
133
|
+
);
|
|
450
134
|
```
|
|
451
135
|
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
### Browser API Notes
|
|
455
|
-
|
|
456
|
-
- The `browser` API proxies the same configuration methods as the Node API and follows the same behavior for quoted fields, sub-array parsing, trimming, and value formatting.
|
|
457
|
-
- `parseFile` depends on the browser `FileReader` API; calling it in Node.js will reject with an informative error.
|
|
458
|
-
|
|
459
|
-
### Browser API (TypeScript)
|
|
460
|
-
|
|
461
|
-
TypeScript typings are provided via the included `index.d.ts`. You can import the default converter and access the `browser` helper, or import `browser` directly. Below are common usage patterns.
|
|
462
|
-
|
|
463
|
-
```ts
|
|
464
|
-
// Named import (recommended for direct use)
|
|
465
|
-
import { browser } from 'convert-csv-to-json';
|
|
136
|
+
## Configuration Options
|
|
466
137
|
|
|
467
|
-
|
|
468
|
-
import converter from 'convert-csv-to-json';
|
|
469
|
-
const browserApi = converter.browser;
|
|
470
|
-
|
|
471
|
-
// Define a type for your CSV records
|
|
472
|
-
interface Person {
|
|
473
|
-
name: string;
|
|
474
|
-
age: number;
|
|
475
|
-
}
|
|
476
|
-
|
|
477
|
-
// Synchronous parse (assert the returned type)
|
|
478
|
-
const csv = 'name,age\nAlice,30';
|
|
479
|
-
const parsed = browser.csvStringToJson(csv) as Person[];
|
|
480
|
-
|
|
481
|
-
// Async parse
|
|
482
|
-
const parsedAsync = await browser.csvStringToJsonAsync(csv) as Person[];
|
|
483
|
-
|
|
484
|
-
// Parse a File in the browser
|
|
485
|
-
const inputEl = document.querySelector('input[type=file]') as HTMLInputElement;
|
|
486
|
-
const file = inputEl.files![0];
|
|
487
|
-
const data = await browser.parseFile(file) as Person[];
|
|
488
|
-
```
|
|
138
|
+
All APIs (Sync, Async and Browser) support the same configuration methods:
|
|
489
139
|
|
|
490
|
-
|
|
140
|
+
- `fieldDelimiter(char)` - Set field delimiter (default: `,`)
|
|
141
|
+
- `formatValueByType()` - Auto-convert numbers, booleans
|
|
142
|
+
- `supportQuotedField(bool)` - Handle quoted fields with embedded delimiters
|
|
143
|
+
- `indexHeader(num)` - Specify header row (default: 0)
|
|
144
|
+
- `trimHeaderFieldWhiteSpace(bool)` - Remove spaces from headers
|
|
145
|
+
- `parseSubArray(delim, sep)` - Parse delimited arrays
|
|
146
|
+
- `mapRows(fn)` - Transform, filter, or enrich each row
|
|
147
|
+
- `utf8Encoding()`, `latin1Encoding()`, etc. - Set file encoding
|
|
491
148
|
|
|
492
|
-
|
|
149
|
+
### Examples
|
|
493
150
|
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
The async API also has TypeScript typings. Typical usage in TypeScript looks like this:
|
|
499
|
-
|
|
500
|
-
```ts
|
|
501
|
-
import csvToJson from 'convert-csv-to-json';
|
|
151
|
+
#### `fieldDelimiter(char)` - Set field delimiter (default: `,`)
|
|
152
|
+
```js
|
|
153
|
+
// Semicolon-delimited
|
|
154
|
+
csvToJson.fieldDelimiter(';').getJsonFromCsv('data.csv');
|
|
502
155
|
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
age: number;
|
|
506
|
-
}
|
|
156
|
+
// Tab-delimited
|
|
157
|
+
csvToJson.fieldDelimiter('\t').getJsonFromCsv('data.tsv');
|
|
507
158
|
|
|
508
|
-
//
|
|
509
|
-
|
|
510
|
-
const csv = 'name,age\nAlice,30';
|
|
511
|
-
const parsed = await csvToJson.getJsonFromCsvAsync(csv, { raw: true }) as Person[];
|
|
512
|
-
return parsed;
|
|
513
|
-
}
|
|
514
|
-
|
|
515
|
-
// Using the async helper that parses CSV strings
|
|
516
|
-
const parsedDirect = await csvToJson.csvStringToJsonAsync('name;age\nBob;25') as Person[];
|
|
159
|
+
// Pipe-delimited
|
|
160
|
+
csvToJson.fieldDelimiter('|').getJsonFromCsv('data.psv');
|
|
517
161
|
```
|
|
518
162
|
|
|
519
|
-
|
|
520
|
-
### Basic Async Operations
|
|
521
|
-
|
|
522
|
-
1. Convert CSV file to JSON:
|
|
163
|
+
#### `formatValueByType()` - Auto-convert numbers, booleans
|
|
523
164
|
```js
|
|
524
|
-
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
.then(json => console.log(json))
|
|
529
|
-
.catch(err => console.error('Error:', err));
|
|
530
|
-
|
|
531
|
-
// Using async/await
|
|
532
|
-
async function convertCsv() {
|
|
533
|
-
try {
|
|
534
|
-
const json = await csvToJson.getJsonFromCsvAsync('input.csv');
|
|
535
|
-
console.log(json);
|
|
536
|
-
} catch (err) {
|
|
537
|
-
console.error('Error:', err);
|
|
538
|
-
}
|
|
539
|
-
}
|
|
165
|
+
// Input: name,age,active
|
|
166
|
+
// John,30,true
|
|
167
|
+
csvToJson.formatValueByType().getJsonFromCsv('data.csv');
|
|
168
|
+
// Output: { name: 'John', age: 30, active: true }
|
|
540
169
|
```
|
|
541
170
|
|
|
542
|
-
|
|
171
|
+
#### `supportQuotedField(bool)` - Handle quoted fields with embedded delimiters
|
|
543
172
|
```js
|
|
544
|
-
//
|
|
545
|
-
|
|
546
|
-
|
|
547
|
-
|
|
548
|
-
.formatValueByType()
|
|
549
|
-
.generateJsonFileFromCsvAsync('input.csv', 'output.json');
|
|
550
|
-
}
|
|
173
|
+
// Input: name,description
|
|
174
|
+
// "Smith, John","He said ""Hello"""
|
|
175
|
+
csvToJson.supportQuotedField(true).getJsonFromCsv('data.csv');
|
|
176
|
+
// Output: { name: 'Smith, John', description: 'He said "Hello"' }
|
|
551
177
|
```
|
|
552
178
|
|
|
553
|
-
|
|
554
|
-
|
|
555
|
-
Process CSV data from memory or network sources:
|
|
556
|
-
|
|
179
|
+
#### `indexHeader(num)` - Specify header row (default: 0)
|
|
557
180
|
```js
|
|
558
|
-
//
|
|
559
|
-
|
|
560
|
-
const response = await fetch('https://api.example.com/data.csv');
|
|
561
|
-
const csvText = await response.text();
|
|
562
|
-
|
|
563
|
-
const json = await csvToJson
|
|
564
|
-
.formatValueByType()
|
|
565
|
-
.getJsonFromCsvAsync(csvText, { raw: true });
|
|
566
|
-
|
|
567
|
-
return json;
|
|
568
|
-
}
|
|
181
|
+
// If headers are in row 2 (3rd line):
|
|
182
|
+
csvToJson.indexHeader(2).getJsonFromCsv('data.csv');
|
|
569
183
|
```
|
|
570
184
|
|
|
571
|
-
|
|
572
|
-
|
|
573
|
-
For large files, use streaming to manage memory efficiently:
|
|
574
|
-
|
|
185
|
+
#### `trimHeaderFieldWhiteSpace(bool)` - Remove spaces from headers
|
|
575
186
|
```js
|
|
576
|
-
|
|
577
|
-
|
|
578
|
-
|
|
579
|
-
async function* processLargeFile(filePath) {
|
|
580
|
-
const fileStream = createReadStream(filePath);
|
|
581
|
-
const rl = createInterface({
|
|
582
|
-
input: fileStream,
|
|
583
|
-
crlfDelay: Infinity
|
|
584
|
-
});
|
|
585
|
-
|
|
586
|
-
for await (const line of rl) {
|
|
587
|
-
yield await csvToJson.getJsonFromCsvAsync(line, { raw: true });
|
|
588
|
-
}
|
|
589
|
-
}
|
|
590
|
-
|
|
591
|
-
// Usage
|
|
592
|
-
async function processData() {
|
|
593
|
-
for await (const record of processLargeFile('large.csv')) {
|
|
594
|
-
await saveToDatabase(record);
|
|
595
|
-
}
|
|
596
|
-
}
|
|
187
|
+
// Input: " First Name ", " Last Name "
|
|
188
|
+
csvToJson.trimHeaderFieldWhiteSpace(true).getJsonFromCsv('data.csv');
|
|
189
|
+
// Output: { FirstName: 'John', LastName: 'Doe' }
|
|
597
190
|
```
|
|
598
191
|
|
|
599
|
-
|
|
600
|
-
|
|
601
|
-
Implement robust error handling with retries:
|
|
602
|
-
|
|
192
|
+
#### `parseSubArray(delim, sep)` - Parse delimited arrays
|
|
603
193
|
```js
|
|
604
|
-
|
|
605
|
-
|
|
606
|
-
|
|
607
|
-
|
|
608
|
-
.formatValueByType()
|
|
609
|
-
.getJsonFromCsvAsync(filePath);
|
|
610
|
-
|
|
611
|
-
return json;
|
|
612
|
-
} catch (err) {
|
|
613
|
-
if (i === maxRetries - 1) throw err;
|
|
614
|
-
// Exponential backoff
|
|
615
|
-
await new Promise(resolve =>
|
|
616
|
-
setTimeout(resolve, Math.pow(2, i) * 1000)
|
|
617
|
-
);
|
|
618
|
-
}
|
|
619
|
-
}
|
|
620
|
-
}
|
|
194
|
+
// Input: name,tags
|
|
195
|
+
// John,*javascript,nodejs,typescript*
|
|
196
|
+
csvToJson.parseSubArray('*', ',').getJsonFromCsv('data.csv');
|
|
197
|
+
// Output: { name: 'John', tags: ['javascript', 'nodejs', 'typescript'] }
|
|
621
198
|
```
|
|
622
199
|
|
|
623
|
-
|
|
624
|
-
|
|
625
|
-
Process multiple files efficiently:
|
|
200
|
+
#### `mapRows(fn)` - Transform, filter, or enrich each row
|
|
626
201
|
|
|
627
202
|
```js
|
|
628
|
-
|
|
629
|
-
|
|
630
|
-
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
processed.forEach(([file, json]) => results.set(file, json));
|
|
641
|
-
}
|
|
642
|
-
|
|
643
|
-
return results;
|
|
644
|
-
}
|
|
645
|
-
|
|
646
|
-
// Usage
|
|
647
|
-
const files = ['data1.csv', 'data2.csv', 'data3.csv', 'data4.csv'];
|
|
648
|
-
const results = await batchProcess(files, 2);
|
|
203
|
+
// Filter out rows that don't match a condition
|
|
204
|
+
const result = csvToJson
|
|
205
|
+
.fieldDelimiter(',')
|
|
206
|
+
.mapRows((row) => {
|
|
207
|
+
// Only keep rows where age >= 30
|
|
208
|
+
if (parseInt(row.age) >= 30) {
|
|
209
|
+
return row;
|
|
210
|
+
}
|
|
211
|
+
return null; // Filters out this row
|
|
212
|
+
})
|
|
213
|
+
.getJsonFromCsv('input.csv');
|
|
649
214
|
```
|
|
650
215
|
|
|
651
|
-
|
|
652
|
-
|
|
653
|
-
The exposed API is implemented with the [Method Chaining Pattern](https://en.wikipedia.org/wiki/Method_chaining), which means that multiple methods can be chained. This pattern works with both synchronous and asynchronous methods:
|
|
654
|
-
|
|
655
|
-
### Synchronous Chaining
|
|
216
|
+
See [mapRows Feature - Usage Guide](docs/MAPROWS.md).
|
|
656
217
|
|
|
218
|
+
#### `utf8Encoding()`, `latin1Encoding()`, etc. - Set file encoding
|
|
657
219
|
```js
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
// Chain configuration methods with sync operation
|
|
661
|
-
const json = csvToJson
|
|
662
|
-
.fieldDelimiter(',')
|
|
663
|
-
.formatValueByType()
|
|
664
|
-
.parseSubArray("*", ',')
|
|
665
|
-
.supportQuotedField(true)
|
|
666
|
-
.getJsonFromCsv('myInputFile.csv');
|
|
220
|
+
// UTF-8 encoding
|
|
221
|
+
csvToJson.utf8Encoding().getJsonFromCsv('data.csv');
|
|
667
222
|
|
|
668
|
-
//
|
|
669
|
-
csvToJson
|
|
670
|
-
.fieldDelimiter(';')
|
|
671
|
-
.utf8Encoding()
|
|
672
|
-
.formatValueByType()
|
|
673
|
-
.generateJsonFileFromCsv('input.csv', 'output.json');
|
|
223
|
+
// Latin-1 encoding
|
|
224
|
+
csvToJson.latin1Encoding().getJsonFromCsv('data.csv');
|
|
674
225
|
|
|
675
|
-
//
|
|
676
|
-
|
|
677
|
-
.fieldDelimiter(',')
|
|
678
|
-
.trimHeaderFieldWhiteSpace(true)
|
|
679
|
-
.csvStringToJson('name,age\nJohn,30\nJane,25');
|
|
226
|
+
// Custom encoding
|
|
227
|
+
csvToJson.customEncoding('ucs2').getJsonFromCsv('data.csv');
|
|
680
228
|
```
|
|
681
229
|
|
|
682
|
-
|
|
230
|
+
See [SYNC.md](docs/SYNC.md), [ASYNC.md](docs/ASYNC.md) or [BROWSER.md](docs/BROWSER.md) for complete configuration details.
|
|
231
|
+
|
|
232
|
+
## Example: Complete Workflow
|
|
683
233
|
|
|
684
234
|
```js
|
|
685
235
|
const csvToJson = require('convert-csv-to-json');
|
|
686
236
|
|
|
687
|
-
// Using async/await
|
|
688
237
|
async function processCSV() {
|
|
689
|
-
|
|
690
|
-
const json = await csvToJson
|
|
691
|
-
.fieldDelimiter(',')
|
|
692
|
-
.formatValueByType()
|
|
693
|
-
.parseSubArray("*", ',')
|
|
694
|
-
.supportQuotedField(true)
|
|
695
|
-
.getJsonFromCsvAsync('myInputFile.csv');
|
|
696
|
-
|
|
697
|
-
// Chain with async file generation
|
|
698
|
-
await csvToJson
|
|
699
|
-
.fieldDelimiter(';')
|
|
700
|
-
.utf8Encoding()
|
|
701
|
-
.formatValueByType()
|
|
702
|
-
.generateJsonFileFromCsvAsync('input.csv', 'output.json');
|
|
703
|
-
}
|
|
704
|
-
|
|
705
|
-
// Using Promises
|
|
706
|
-
csvToJson
|
|
238
|
+
const data = await csvToJson
|
|
707
239
|
.fieldDelimiter(',')
|
|
708
240
|
.formatValueByType()
|
|
709
|
-
.
|
|
710
|
-
.
|
|
711
|
-
.catch(err => console.error('Error:', err));
|
|
712
|
-
```
|
|
713
|
-
|
|
714
|
-
All configuration methods can be chained in any order before calling the final operation method (like `getJsonFromCsv`, `getJsonFromCsvAsync`, etc.). The configuration will be applied in the order it is chained.
|
|
715
|
-
|
|
716
|
-
## Common Use Cases
|
|
717
|
-
|
|
718
|
-
Here are some common use cases and how to implement them:
|
|
719
|
-
|
|
720
|
-
### 1. Processing CSV from HTTP Response
|
|
721
|
-
```js
|
|
722
|
-
const https = require('https');
|
|
723
|
-
|
|
724
|
-
async function processRemoteCsv(url) {
|
|
725
|
-
const csvData = await new Promise((resolve, reject) => {
|
|
726
|
-
https.get(url, (res) => {
|
|
727
|
-
let data = '';
|
|
728
|
-
res.on('data', chunk => data += chunk);
|
|
729
|
-
res.on('end', () => resolve(data));
|
|
730
|
-
res.on('error', reject);
|
|
731
|
-
});
|
|
732
|
-
});
|
|
733
|
-
|
|
734
|
-
return csvToJson.getJsonFromCsvAsync(csvData, { raw: true });
|
|
735
|
-
}
|
|
736
|
-
```
|
|
737
|
-
|
|
738
|
-
### 2. Batch Processing Multiple Files
|
|
739
|
-
```js
|
|
740
|
-
async function batchProcess(files) {
|
|
741
|
-
const results = new Map();
|
|
742
|
-
|
|
743
|
-
// Process in chunks of 3 files at a time
|
|
744
|
-
for (let i = 0; i < files.length; i += 3) {
|
|
745
|
-
const chunk = files.slice(i, i + 3);
|
|
746
|
-
const processed = await Promise.all(
|
|
747
|
-
chunk.map(async file => {
|
|
748
|
-
const json = await csvToJson.getJsonFromCsvAsync(file);
|
|
749
|
-
return [file, json];
|
|
750
|
-
})
|
|
751
|
-
);
|
|
752
|
-
|
|
753
|
-
processed.forEach(([file, json]) => results.set(file, json));
|
|
754
|
-
}
|
|
241
|
+
.supportQuotedField(true)
|
|
242
|
+
.getJsonFromCsvAsync('data.csv');
|
|
755
243
|
|
|
756
|
-
|
|
244
|
+
console.log(`Parsed ${data.length} records`);
|
|
245
|
+
return data;
|
|
757
246
|
}
|
|
758
247
|
```
|
|
759
248
|
|
|
760
|
-
|
|
761
|
-
```js
|
|
762
|
-
async function transformData(csvFile) {
|
|
763
|
-
// Step 1: Parse CSV
|
|
764
|
-
const json = await csvToJson
|
|
765
|
-
.formatValueByType()
|
|
766
|
-
.getJsonFromCsvAsync(csvFile);
|
|
767
|
-
|
|
768
|
-
// Step 2: Transform data
|
|
769
|
-
const transformed = json.map(record => ({
|
|
770
|
-
id: record.id,
|
|
771
|
-
fullName: `${record.firstName} ${record.lastName}`,
|
|
772
|
-
age: Number(record.age),
|
|
773
|
-
isAdult: Number(record.age) >= 18,
|
|
774
|
-
email: record.email.toLowerCase()
|
|
775
|
-
}));
|
|
776
|
-
|
|
777
|
-
// Step 3: Filter invalid records
|
|
778
|
-
return transformed.filter(record =>
|
|
779
|
-
record.id &&
|
|
780
|
-
record.fullName.length > 0 &&
|
|
781
|
-
!isNaN(record.age)
|
|
782
|
-
);
|
|
783
|
-
}
|
|
784
|
-
```
|
|
249
|
+
## Migration Guides
|
|
785
250
|
|
|
786
|
-
|
|
787
|
-
|
|
788
|
-
|
|
789
|
-
const logger = {
|
|
790
|
-
info: (msg) => console.log(`[INFO] ${msg}`),
|
|
791
|
-
error: (msg, err) => console.error(`[ERROR] ${msg}`, err)
|
|
792
|
-
};
|
|
793
|
-
|
|
794
|
-
try {
|
|
795
|
-
logger.info(`Starting processing ${file}`);
|
|
796
|
-
const startTime = Date.now();
|
|
797
|
-
|
|
798
|
-
const json = await csvToJson.getJsonFromCsvAsync(file);
|
|
799
|
-
|
|
800
|
-
const duration = Date.now() - startTime;
|
|
801
|
-
logger.info(`Processed ${file} in ${duration}ms`);
|
|
802
|
-
|
|
803
|
-
return json;
|
|
804
|
-
} catch (err) {
|
|
805
|
-
logger.error(`Failed to process ${file}`, err);
|
|
806
|
-
throw err;
|
|
807
|
-
}
|
|
808
|
-
}
|
|
809
|
-
```
|
|
251
|
+
- **RFC 4180 Breaking Changes** - See [RFC4180_MIGRATION_GUIDE.md](migration/RFC4180_MIGRATION_GUIDE.md)
|
|
252
|
+
- **Sync to Async Migration** - See [MIGRATION.md](migration/MIGRATION_TO_ASYNC.md)
|
|
253
|
+
- **Error Handling** - See [docs/ERROR_HANDLING.md](docs/ERROR_HANDLING.md) for comprehensive error documentation
|
|
810
254
|
|
|
811
|
-
##
|
|
812
|
-
|
|
813
|
-
Here are solutions to common issues you might encounter:
|
|
814
|
-
|
|
815
|
-
### Memory Issues with Large Files
|
|
816
|
-
|
|
817
|
-
If you're processing large CSV files and encountering memory issues:
|
|
818
|
-
|
|
819
|
-
```js
|
|
820
|
-
// Instead of loading the entire file
|
|
821
|
-
const json = await csvToJson.getJsonFromCsvAsync('large.csv'); // ❌
|
|
822
|
-
|
|
823
|
-
// Use streaming with async iteration
|
|
824
|
-
for await (const record of processLargeCsv('large.csv')) { // ✅
|
|
825
|
-
// Process one record at a time
|
|
826
|
-
await processRecord(record);
|
|
827
|
-
}
|
|
828
|
-
```
|
|
829
|
-
|
|
830
|
-
### Handling Different CSV Formats
|
|
255
|
+
## Development
|
|
831
256
|
|
|
832
|
-
|
|
833
|
-
```
|
|
834
|
-
|
|
835
|
-
.supportQuotedField(true) // Enable quoted field support
|
|
836
|
-
.getJsonFromCsvAsync(file);
|
|
257
|
+
Install dependencies:
|
|
258
|
+
```bash
|
|
259
|
+
npm install
|
|
837
260
|
```
|
|
838
261
|
|
|
839
|
-
|
|
840
|
-
```
|
|
841
|
-
|
|
842
|
-
.fieldDelimiter(';') // Change delimiter
|
|
843
|
-
.getJsonFromCsvAsync(file);
|
|
262
|
+
Run tests:
|
|
263
|
+
```bash
|
|
264
|
+
npm test
|
|
844
265
|
```
|
|
845
266
|
|
|
846
|
-
|
|
847
|
-
```
|
|
848
|
-
|
|
849
|
-
.encoding('utf8') // Specify encoding
|
|
850
|
-
.getJsonFromCsvAsync(file);
|
|
267
|
+
Debug tests:
|
|
268
|
+
```bash
|
|
269
|
+
npm run test-debug
|
|
851
270
|
```
|
|
852
271
|
|
|
853
|
-
|
|
854
|
-
|
|
855
|
-
1. **ENOENT: no such file or directory**
|
|
856
|
-
- Check if the file path is correct and absolute
|
|
857
|
-
- Verify file permissions
|
|
858
|
-
- Ensure the file exists in the specified location
|
|
272
|
+
## CI/CD GitHub Action
|
|
859
273
|
|
|
860
|
-
|
|
861
|
-
- Verify CSV format matches expected structure
|
|
862
|
-
- Check for missing or extra delimiters
|
|
863
|
-
- Validate header row exists if expected
|
|
274
|
+
See [CI/CD GitHub Action](.github/workflows/ci-cd.yml).
|
|
864
275
|
|
|
865
|
-
|
|
866
|
-
|
|
867
|
-
|
|
868
|
-
|
|
276
|
+
### Release
|
|
277
|
+
When pushing to the `master` branch:
|
|
278
|
+
- Include `[MAJOR]` in commit message for major release (e.g., v1.0.0 → v2.0.0)
|
|
279
|
+
- Include `[PATCH]` in commit message for patch release (e.g., v1.0.0 → v1.0.1)
|
|
280
|
+
- Minor release is applied by default (e.g., v1.0.0 → v1.1.0)
|
|
869
281
|
|
|
870
|
-
|
|
871
|
-
- Specify correct encoding using .encoding()
|
|
872
|
-
- Check for BOM markers
|
|
873
|
-
- Verify source file encoding
|
|
874
|
-
|
|
875
|
-
### Performance Optimization
|
|
876
|
-
|
|
877
|
-
1. **Parallel Processing**:
|
|
878
|
-
```js
|
|
879
|
-
// Instead of sequential processing
|
|
880
|
-
for (const file of files) {
|
|
881
|
-
await process(file); // ❌
|
|
882
|
-
}
|
|
883
|
-
|
|
884
|
-
// Use parallel processing with limits
|
|
885
|
-
async function processWithLimit(files, limit = 3) {
|
|
886
|
-
const results = [];
|
|
887
|
-
for (let i = 0; i < files.length; i += limit) {
|
|
888
|
-
const chunk = files.slice(i, i + limit);
|
|
889
|
-
const chunkResults = await Promise.all(
|
|
890
|
-
chunk.map(file => csvToJson.getJsonFromCsvAsync(file))
|
|
891
|
-
);
|
|
892
|
-
results.push(...chunkResults);
|
|
893
|
-
}
|
|
894
|
-
return results;
|
|
895
|
-
} // ✅
|
|
896
|
-
```
|
|
282
|
+
## License
|
|
897
283
|
|
|
898
|
-
|
|
899
|
-
```js
|
|
900
|
-
// Clear references when done
|
|
901
|
-
async function processWithCleanup(file) {
|
|
902
|
-
let json;
|
|
903
|
-
try {
|
|
904
|
-
json = await csvToJson.getJsonFromCsvAsync(file);
|
|
905
|
-
return await processData(json);
|
|
906
|
-
} finally {
|
|
907
|
-
json = null; // Clear reference
|
|
908
|
-
}
|
|
909
|
-
}
|
|
910
|
-
```
|
|
284
|
+
CSVtoJSON is licensed under the MIT [License](LICENSE).
|
|
911
285
|
|
|
912
|
-
|
|
286
|
+
---
|
|
913
287
|
|
|
914
|
-
|
|
288
|
+
## Support
|
|
915
289
|
|
|
916
|
-
|
|
917
|
-
// Define custom types for your CSV structure
|
|
918
|
-
interface MyCsvRecord {
|
|
919
|
-
id: number;
|
|
920
|
-
name: string;
|
|
921
|
-
age?: number;
|
|
922
|
-
}
|
|
290
|
+
Found a bug or need a feature? Open an issue on [GitHub](https://github.com/iuccio/csvToJson/issues).
|
|
923
291
|
|
|
924
|
-
|
|
925
|
-
const json = await csvToJson.getJsonFromCsvAsync<MyCsvRecord>('data.csv');
|
|
926
|
-
```
|
|
927
|
-
|
|
928
|
-
## Development
|
|
929
|
-
* Download all csvToJson dependencies:
|
|
930
|
-
~~~
|
|
931
|
-
npm install
|
|
932
|
-
~~~
|
|
933
|
-
* Run Tests
|
|
934
|
-
~~~
|
|
935
|
-
npm test
|
|
936
|
-
~~~
|
|
937
|
-
* Debug Tests
|
|
938
|
-
~~~
|
|
939
|
-
npm run test-debug
|
|
940
|
-
~~~
|
|
941
|
-
|
|
942
|
-
## CI CD github action
|
|
943
|
-
|
|
944
|
-
This repository uses the [GitHub Action iuccio/npm-semantic-publish-action@latest](https://github.com/marketplace/actions/npm-semver-publish) to publish the npm packeges.
|
|
945
|
-
Pushing on the master branch, depending on the git message, an new version will always be released.
|
|
946
|
-
If the commit message contains the keyword:
|
|
947
|
-
* **[MAJOR]**: new major relase, e.g. v1.0.0 -> v2.0.0
|
|
948
|
-
* **[PATCH]**: new patch relase, e.g. v1.0.0 -> v1.0.1
|
|
949
|
-
* without any of the above keywords a new minor relase will be applied, e.g. v1.0.0 -> v1.1.0
|
|
292
|
+
Follow [me](https://github.com/iuccio) and consider starring the project to show your support ⭐
|
|
950
293
|
|
|
294
|
+
### Buy Me a Coffee
|
|
951
295
|
|
|
952
|
-
|
|
953
|
-
CSVtoJSON is licensed under the MIT [License](LICENSE).
|
|
296
|
+
If you find this project helpful and would like to support its development:
|
|
954
297
|
|
|
955
|
-
|
|
956
|
-
Just if you want to support this repository:
|
|
957
|
-
* **BTC** tip address:
|
|
958
|
-
37vdjQhbaR7k7XzhMKWzMcnqUxfw1njBNk
|
|
298
|
+
**BTC**: `37vdjQhbaR7k7XzhMKWzMcnqUxfw1njBNk`
|