fastcsv 0.0.4 → 0.0.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/.travis.yml +1 -0
- data/Gemfile +1 -1
- data/LICENSE +1 -1
- data/README.md +12 -17
- data/TESTS.md +4 -5
- data/ext/fastcsv/fastcsv.c +2494 -153
- data/ext/fastcsv/fastcsv.rl +43 -8
- data/fastcsv.gemspec +5 -6
- data/lib/fastcsv.rb +1 -1
- data/spec/fastcsv_spec.rb +0 -1
- data/test/csv/test_csv_parsing.rb +10 -10
- data/test/csv/test_encodings.rb +23 -23
- data/test/csv/test_features.rb +17 -17
- data/test/csv/test_headers.rb +14 -14
- data/test/csv/test_interface.rb +39 -39
- metadata +9 -11
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA1:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a4bd98782c7284ce6b9041b789f9048aa7d9772b
|
|
4
|
+
data.tar.gz: b970c8d7cd54e52aaa56ab5b88f61b575a7dd811
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 407c12a75c2e24a14f0f65d144e9b525c79ae8c531b8dcaa67f12520083a6474a6af9ceebcf2eecaed86516730a6346a9c3323d7f0b2c2c63e9c50a197f18f5b
|
|
7
|
+
data.tar.gz: 8b8880a373dc298da5e40c7b038ac04b9156c004222ba8ba7f315354b008805388da72b9e954d450bda6ccddec85559aaedc2e076354646c5b62b17bdc8eee94
|
data/.travis.yml
CHANGED
data/Gemfile
CHANGED
data/LICENSE
CHANGED
data/README.md
CHANGED
|
@@ -1,14 +1,12 @@
|
|
|
1
1
|
# FastCSV
|
|
2
2
|
|
|
3
|
-
[](
|
|
4
|
-
[](https://badge.fury.io/rb/fastcsv)
|
|
4
|
+
[](https://travis-ci.org/jpmckinney/fastcsv)
|
|
5
|
+
[](https://gemnasium.com/jpmckinney/fastcsv)
|
|
6
|
+
[](https://coveralls.io/r/jpmckinney/fastcsv)
|
|
7
|
+
[](https://codeclimate.com/github/jpmckinney/fastcsv)
|
|
8
8
|
|
|
9
|
-
A fast [Ragel](http://www.colm.net/open-source/ragel/)-based CSV parser.
|
|
10
|
-
|
|
11
|
-
**Only reads CSVs using `"` as the quote character, `,` as the delimiter and `\r`, `\n` or `\r\n` as the line terminator.**
|
|
9
|
+
A fast [Ragel](http://www.colm.net/open-source/ragel/)-based CSV parser, compatible with Ruby's CSV.
|
|
12
10
|
|
|
13
11
|
## Usage
|
|
14
12
|
|
|
@@ -42,8 +40,9 @@ end
|
|
|
42
40
|
|
|
43
41
|
FastCSV can be used as a drop-in replacement for [CSV](http://ruby-doc.org/stdlib-2.1.1/libdoc/csv/rdoc/CSV.html) (replace `CSV` with `FastCSV`) except:
|
|
44
42
|
|
|
45
|
-
* The `:
|
|
46
|
-
*
|
|
43
|
+
* The `:row_sep` option is ignored. The default `:auto` is implemented. [#9](https://github.com/jpmckinney/fastcsv/issues/9)
|
|
44
|
+
* The `:col_sep` option must be a single-byte string, like the default `,`. [#8](https://github.com/jpmckinney/fastcsv/issues/8)
|
|
45
|
+
* If FastCSV raises an error, you can't continue reading. [#3](https://github.com/jpmckinney/fastcsv/issues/3) Its error messages don't perfectly match those of CSV.
|
|
47
46
|
|
|
48
47
|
A few minor caveats:
|
|
49
48
|
|
|
@@ -74,18 +73,14 @@ CSV delegates IO methods to the IO object it's reading. IO methods that move the
|
|
|
74
73
|
|
|
75
74
|
CSV's `#shift` runs the regular expression in the `:skip_lines` option against a row's raw text. `FastCSV::Parser` implements a `row` method, which returns the most recently parsed row's raw text.
|
|
76
75
|
|
|
77
|
-
FastCSV is tested against the same tests as CSV. See [TESTS.md](https://github.com/
|
|
76
|
+
FastCSV is tested against the same tests as CSV. See [TESTS.md](https://github.com/jpmckinney/fastcsv/blob/master/TESTS.md) for details.
|
|
78
77
|
|
|
79
78
|
## Why?
|
|
80
79
|
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
## Bugs? Questions?
|
|
84
|
-
|
|
85
|
-
This project's main repository is on GitHub: [http://github.com/opennorth/fastcsv](http://github.com/opennorth/fastcsv), where your contributions, forks, bug reports, feature requests, and feedback are greatly welcomed.
|
|
80
|
+
I evaluated [many CSV Ruby gems](https://github.com/jpmckinney/csv-benchmark#benchmark), and they were either too slow or had implementation errors. [rcsv](https://github.com/fiksu/rcsv) is fast and [libcsv](http://sourceforge.net/projects/libcsv/)-based, but it skips blank rows (Ruby's CSV module returns an empty array) and silently fails on input with an unclosed quote. [bamfcsv](https://github.com/jondistad/bamfcsv) is well implemented, but it's considerably slower on large files. I looked for Ragel-based CSV parsers to copy, but they either had implementation errors or could not handle large files. [commas](https://github.com/aklt/commas/blob/master/csv.rl) looks good, but it performs a memory check on each character, which is overkill.
|
|
86
81
|
|
|
87
82
|
## Acknowledgements
|
|
88
83
|
|
|
89
84
|
Started as a Ruby 2.1 fork of MoonWolf <moonwolf@moonwolf.com>'s CSVScan, found in [this commit](https://github.com/nickstenning/csvscan/commit/11ec30f71a27cc673bca09738ee8a63942f416f0.patch). CSVScan uses Ragel code from [HPricot](https://github.com/hpricot/hpricot/blob/master/ext/hpricot_scan/hpricot_scan.rl) from [this commit](https://github.com/hpricot/hpricot/blob/908a4ae64bc8b935c4415c47ca6aea6492c6ce0a/ext/hpricot_scan/hpricot_scan.rl). Most of the Ruby (i.e. non-C, non-Ragel) methods are copied from [CSV](https://github.com/ruby/ruby/blob/ab337e61ecb5f42384ba7d710c36faf96a454e5c/lib/csv.rb).
|
|
90
85
|
|
|
91
|
-
Copyright (c) 2014
|
|
86
|
+
Copyright (c) 2014 James McKinney, released under the MIT license
|
data/TESTS.md
CHANGED
|
@@ -23,15 +23,14 @@ Here are some notes on maintaining the `test/` directory.
|
|
|
23
23
|
sed -i.bak '1s;^;require "fastcsv"\
|
|
24
24
|
;' test/runner.rb
|
|
25
25
|
|
|
26
|
-
1. In `
|
|
26
|
+
1. In `test_encodings.rb`, replace `Encoding.list` with `Encoding.list.reject{|e| e.name[/\AUTF-\d\d/]}`, because UTF-16 and UTF-32 aren't supported.
|
|
27
27
|
|
|
28
|
-
1. Comment these tests because `:
|
|
28
|
+
1. Comment these tests because `:row_sep` is ignored and multibyte `:quote_char` is unsupported:
|
|
29
29
|
|
|
30
30
|
* `test_csv_parsing.rb`: the first part of `test_malformed_csv`
|
|
31
|
-
* `test_features.rb`: `
|
|
32
|
-
* `test_headers.rb`: `test_csv_header_string_inherits_separators`
|
|
31
|
+
* `test_features.rb`: `test_row_sep`, `test_leading_empty_fields_with_multibyte_col_sep_bug_fix`
|
|
33
32
|
|
|
34
|
-
1. Comment these tests in `
|
|
33
|
+
1. Comment these tests in `test_encodings.rb` because UTF-16 and UTF-32 aren't supported:
|
|
35
34
|
|
|
36
35
|
* `test_parses_utf16be_encoding`
|
|
37
36
|
* the second part of `test_open_allows_you_to_set_encodings`
|
data/ext/fastcsv/fastcsv.c
CHANGED
|
@@ -24,6 +24,9 @@ if (enc2 != NULL) { \
|
|
|
24
24
|
#define FREE \
|
|
25
25
|
if (buf != NULL) { \
|
|
26
26
|
free(buf); \
|
|
27
|
+
} \
|
|
28
|
+
if (row_sep != NULL) { \
|
|
29
|
+
free(row_sep); \
|
|
27
30
|
}
|
|
28
31
|
|
|
29
32
|
static VALUE cClass, cParser, eError;
|
|
@@ -35,19 +38,19 @@ typedef struct {
|
|
|
35
38
|
} Data;
|
|
36
39
|
|
|
37
40
|
|
|
38
|
-
#line
|
|
41
|
+
#line 169 "ext/fastcsv/fastcsv.rl"
|
|
39
42
|
|
|
40
43
|
|
|
41
44
|
|
|
42
|
-
#line
|
|
43
|
-
static const int raw_parse_start =
|
|
44
|
-
static const int raw_parse_first_final =
|
|
45
|
+
#line 46 "ext/fastcsv/fastcsv.c"
|
|
46
|
+
static const int raw_parse_start = 5;
|
|
47
|
+
static const int raw_parse_first_final = 5;
|
|
45
48
|
static const int raw_parse_error = 0;
|
|
46
49
|
|
|
47
|
-
static const int raw_parse_en_main =
|
|
50
|
+
static const int raw_parse_en_main = 5;
|
|
48
51
|
|
|
49
52
|
|
|
50
|
-
#line
|
|
53
|
+
#line 172 "ext/fastcsv/fastcsv.rl"
|
|
51
54
|
|
|
52
55
|
// 16 kB
|
|
53
56
|
#define BUFSIZE 16384
|
|
@@ -80,18 +83,18 @@ static void rb_io_ext_int_to_encs(rb_encoding *ext, rb_encoding *intern, rb_enco
|
|
|
80
83
|
|
|
81
84
|
static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
82
85
|
int cs, act, have = 0, curline = 1, io = 0;
|
|
83
|
-
char *ts = 0, *te = 0, *buf = 0, *eof = 0;
|
|
86
|
+
char *ts = 0, *te = 0, *buf = 0, *eof = 0, *mark_row_sep = 0, *row_sep = 0;
|
|
84
87
|
|
|
85
88
|
VALUE port, opts, r_encoding;
|
|
86
89
|
VALUE row = rb_ary_new(), field = Qnil, bufsize = Qnil;
|
|
87
|
-
int done = 0, unclosed_line = 0, buffer_size = 0, taint = 0;
|
|
90
|
+
int done = 0, unclosed_line = 0, buffer_size = 0, taint = 0, len_row_sep = 0;
|
|
88
91
|
rb_encoding *enc = NULL, *enc2 = NULL, *encoding = NULL;
|
|
89
92
|
|
|
90
93
|
Data *d;
|
|
91
94
|
Data_Get_Struct(self, Data, d);
|
|
92
95
|
|
|
93
96
|
VALUE option;
|
|
94
|
-
char quote_char = '"';
|
|
97
|
+
char quote_char = '"', col_sep = ',';
|
|
95
98
|
|
|
96
99
|
rb_scan_args(argc, argv, "11", &port, &opts);
|
|
97
100
|
taint = OBJ_TAINTED(port);
|
|
@@ -113,6 +116,22 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
113
116
|
rb_raise(rb_eArgError, "options has to be a Hash or nil");
|
|
114
117
|
}
|
|
115
118
|
|
|
119
|
+
option = rb_hash_aref(opts, ID2SYM(rb_intern("quote_char")));
|
|
120
|
+
if (TYPE(option) == T_STRING && RSTRING_LEN(option) == 1) {
|
|
121
|
+
quote_char = *StringValueCStr(option);
|
|
122
|
+
}
|
|
123
|
+
else if (!NIL_P(option)) {
|
|
124
|
+
rb_raise(rb_eArgError, ":quote_char has to be a single character String");
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
option = rb_hash_aref(opts, ID2SYM(rb_intern("col_sep")));
|
|
128
|
+
if (TYPE(option) == T_STRING && RSTRING_LEN(option) == 1) {
|
|
129
|
+
col_sep = *StringValueCStr(option);
|
|
130
|
+
}
|
|
131
|
+
else if (!NIL_P(option)) {
|
|
132
|
+
rb_raise(rb_eArgError, ":col_sep has to be a single character String");
|
|
133
|
+
}
|
|
134
|
+
|
|
116
135
|
// @see rb_io_extract_modeenc
|
|
117
136
|
/* Set to defaults */
|
|
118
137
|
rb_io_ext_int_to_encs(NULL, NULL, &enc, &enc2, 0);
|
|
@@ -243,7 +262,7 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
243
262
|
}
|
|
244
263
|
|
|
245
264
|
|
|
246
|
-
#line
|
|
265
|
+
#line 266 "ext/fastcsv/fastcsv.c"
|
|
247
266
|
{
|
|
248
267
|
cs = raw_parse_start;
|
|
249
268
|
ts = 0;
|
|
@@ -251,12 +270,12 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
251
270
|
act = 0;
|
|
252
271
|
}
|
|
253
272
|
|
|
254
|
-
#line
|
|
273
|
+
#line 383 "ext/fastcsv/fastcsv.rl"
|
|
255
274
|
|
|
256
275
|
while (!done) {
|
|
257
276
|
VALUE str;
|
|
258
277
|
char *p, *pe;
|
|
259
|
-
int len, space = buffer_size - have, tokstart_diff, tokend_diff, start_diff;
|
|
278
|
+
int len, space = buffer_size - have, tokstart_diff, tokend_diff, start_diff, mark_row_sep_diff;
|
|
260
279
|
|
|
261
280
|
if (io) {
|
|
262
281
|
if (space == 0) {
|
|
@@ -264,6 +283,7 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
264
283
|
tokstart_diff = ts - buf;
|
|
265
284
|
tokend_diff = te - buf;
|
|
266
285
|
start_diff = d->start - buf;
|
|
286
|
+
mark_row_sep_diff = mark_row_sep - buf;
|
|
267
287
|
|
|
268
288
|
buffer_size += BUFSIZE;
|
|
269
289
|
REALLOC_N(buf, char, buffer_size);
|
|
@@ -273,6 +293,7 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
273
293
|
ts = buf + tokstart_diff;
|
|
274
294
|
te = buf + tokend_diff;
|
|
275
295
|
d->start = buf + start_diff;
|
|
296
|
+
mark_row_sep = buf + mark_row_sep_diff;
|
|
276
297
|
}
|
|
277
298
|
p = buf + have;
|
|
278
299
|
|
|
@@ -311,8 +332,9 @@ static VALUE raw_parse(int argc, VALUE *argv, VALUE self) {
|
|
|
311
332
|
|
|
312
333
|
pe = p + len;
|
|
313
334
|
|
|
314
|
-
#line
|
|
335
|
+
#line 336 "ext/fastcsv/fastcsv.c"
|
|
315
336
|
{
|
|
337
|
+
short _widec;
|
|
316
338
|
if ( p == pe )
|
|
317
339
|
goto _test_eof;
|
|
318
340
|
switch ( cs )
|
|
@@ -328,9 +350,41 @@ tr0:
|
|
|
328
350
|
break;
|
|
329
351
|
}
|
|
330
352
|
}
|
|
331
|
-
goto
|
|
353
|
+
goto st5;
|
|
332
354
|
tr5:
|
|
333
|
-
#line
|
|
355
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
356
|
+
{
|
|
357
|
+
if (p == ts) {
|
|
358
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
359
|
+
field = Qnil;
|
|
360
|
+
}
|
|
361
|
+
else if (p > ts) {
|
|
362
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
363
|
+
ENCODE;
|
|
364
|
+
}
|
|
365
|
+
}
|
|
366
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
367
|
+
{
|
|
368
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
369
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
370
|
+
}
|
|
371
|
+
else if (p > d->start) {
|
|
372
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
373
|
+
}
|
|
374
|
+
|
|
375
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
376
|
+
rb_ary_push(row, field);
|
|
377
|
+
}
|
|
378
|
+
|
|
379
|
+
if (RARRAY_LEN(row)) {
|
|
380
|
+
rb_yield(row);
|
|
381
|
+
}
|
|
382
|
+
}
|
|
383
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
384
|
+
{te = p+1;}
|
|
385
|
+
goto st5;
|
|
386
|
+
tr6:
|
|
387
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
334
388
|
{
|
|
335
389
|
if (p == ts) {
|
|
336
390
|
// Unquoted empty fields are nil, not "", in Ruby.
|
|
@@ -341,18 +395,55 @@ tr5:
|
|
|
341
395
|
ENCODE;
|
|
342
396
|
}
|
|
343
397
|
}
|
|
344
|
-
#line
|
|
398
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
345
399
|
{
|
|
346
400
|
rb_ary_push(row, field);
|
|
347
401
|
field = Qnil;
|
|
348
402
|
}
|
|
349
|
-
#line
|
|
403
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
350
404
|
{te = p+1;}
|
|
351
|
-
goto
|
|
352
|
-
|
|
353
|
-
#line
|
|
405
|
+
goto st5;
|
|
406
|
+
tr7:
|
|
407
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
354
408
|
{
|
|
355
|
-
if (
|
|
409
|
+
if (p == ts) {
|
|
410
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
411
|
+
field = Qnil;
|
|
412
|
+
}
|
|
413
|
+
else if (p > ts) {
|
|
414
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
415
|
+
ENCODE;
|
|
416
|
+
}
|
|
417
|
+
}
|
|
418
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
419
|
+
{
|
|
420
|
+
rb_ary_push(row, field);
|
|
421
|
+
field = Qnil;
|
|
422
|
+
}
|
|
423
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
424
|
+
{te = p+1;}
|
|
425
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
426
|
+
{
|
|
427
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
428
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
429
|
+
}
|
|
430
|
+
else if (p > d->start) {
|
|
431
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
432
|
+
}
|
|
433
|
+
|
|
434
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
435
|
+
rb_ary_push(row, field);
|
|
436
|
+
}
|
|
437
|
+
|
|
438
|
+
if (RARRAY_LEN(row)) {
|
|
439
|
+
rb_yield(row);
|
|
440
|
+
}
|
|
441
|
+
}
|
|
442
|
+
goto st5;
|
|
443
|
+
tr13:
|
|
444
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
445
|
+
{
|
|
446
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
356
447
|
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
357
448
|
}
|
|
358
449
|
else if (p > d->start) {
|
|
@@ -367,65 +458,242 @@ tr9:
|
|
|
367
458
|
rb_yield(row);
|
|
368
459
|
}
|
|
369
460
|
}
|
|
370
|
-
#line
|
|
461
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
371
462
|
{te = p+1;}
|
|
372
|
-
goto
|
|
373
|
-
|
|
374
|
-
#line
|
|
463
|
+
goto st5;
|
|
464
|
+
tr19:
|
|
465
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
375
466
|
{
|
|
376
467
|
rb_ary_push(row, field);
|
|
377
468
|
field = Qnil;
|
|
378
469
|
}
|
|
379
|
-
#line
|
|
470
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
380
471
|
{te = p+1;}
|
|
381
|
-
goto
|
|
382
|
-
|
|
383
|
-
#line
|
|
472
|
+
goto st5;
|
|
473
|
+
tr20:
|
|
474
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
475
|
+
{
|
|
476
|
+
rb_ary_push(row, field);
|
|
477
|
+
field = Qnil;
|
|
478
|
+
}
|
|
479
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
480
|
+
{te = p+1;}
|
|
481
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
482
|
+
{
|
|
483
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
484
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
485
|
+
}
|
|
486
|
+
else if (p > d->start) {
|
|
487
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
488
|
+
}
|
|
489
|
+
|
|
490
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
491
|
+
rb_ary_push(row, field);
|
|
492
|
+
}
|
|
493
|
+
|
|
494
|
+
if (RARRAY_LEN(row)) {
|
|
495
|
+
rb_yield(row);
|
|
496
|
+
}
|
|
497
|
+
}
|
|
498
|
+
goto st5;
|
|
499
|
+
tr42:
|
|
500
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
384
501
|
{te = p;p--;}
|
|
385
|
-
goto
|
|
386
|
-
|
|
387
|
-
#line
|
|
502
|
+
goto st5;
|
|
503
|
+
tr43:
|
|
504
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
388
505
|
{
|
|
389
506
|
d->start = p;
|
|
507
|
+
|
|
508
|
+
if (len_row_sep) {
|
|
509
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
510
|
+
FREE;
|
|
511
|
+
|
|
512
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
513
|
+
}
|
|
514
|
+
}
|
|
515
|
+
else {
|
|
516
|
+
len_row_sep = p - mark_row_sep;
|
|
517
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
518
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
519
|
+
}
|
|
390
520
|
}
|
|
391
|
-
#line
|
|
392
|
-
{
|
|
393
|
-
|
|
394
|
-
|
|
521
|
+
#line 1 "NONE"
|
|
522
|
+
{ switch( act ) {
|
|
523
|
+
case 0:
|
|
524
|
+
{{goto st0;}}
|
|
525
|
+
break;
|
|
526
|
+
default:
|
|
527
|
+
{{p = ((te))-1;}}
|
|
528
|
+
break;
|
|
529
|
+
}
|
|
530
|
+
}
|
|
531
|
+
goto st5;
|
|
532
|
+
tr50:
|
|
533
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
534
|
+
{
|
|
535
|
+
d->start = p;
|
|
536
|
+
|
|
537
|
+
if (len_row_sep) {
|
|
538
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
539
|
+
FREE;
|
|
540
|
+
|
|
541
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
542
|
+
}
|
|
543
|
+
}
|
|
544
|
+
else {
|
|
545
|
+
len_row_sep = p - mark_row_sep;
|
|
546
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
547
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
548
|
+
}
|
|
549
|
+
}
|
|
550
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
551
|
+
{
|
|
552
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
553
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
554
|
+
}
|
|
555
|
+
else if (p > d->start) {
|
|
556
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
557
|
+
}
|
|
558
|
+
|
|
559
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
560
|
+
rb_ary_push(row, field);
|
|
561
|
+
}
|
|
562
|
+
|
|
563
|
+
if (RARRAY_LEN(row)) {
|
|
564
|
+
rb_yield(row);
|
|
565
|
+
}
|
|
566
|
+
}
|
|
567
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
568
|
+
{te = p+1;}
|
|
569
|
+
goto st5;
|
|
570
|
+
tr56:
|
|
571
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
572
|
+
{
|
|
573
|
+
rb_ary_push(row, field);
|
|
574
|
+
field = Qnil;
|
|
575
|
+
}
|
|
576
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
577
|
+
{te = p+1;}
|
|
578
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
579
|
+
{
|
|
580
|
+
d->start = p;
|
|
581
|
+
|
|
582
|
+
if (len_row_sep) {
|
|
583
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
584
|
+
FREE;
|
|
585
|
+
|
|
586
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
587
|
+
}
|
|
588
|
+
}
|
|
589
|
+
else {
|
|
590
|
+
len_row_sep = p - mark_row_sep;
|
|
591
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
592
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
593
|
+
}
|
|
594
|
+
}
|
|
595
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
596
|
+
{
|
|
597
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
598
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
599
|
+
}
|
|
600
|
+
else if (p > d->start) {
|
|
601
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
602
|
+
}
|
|
603
|
+
|
|
604
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
605
|
+
rb_ary_push(row, field);
|
|
606
|
+
}
|
|
607
|
+
|
|
608
|
+
if (RARRAY_LEN(row)) {
|
|
609
|
+
rb_yield(row);
|
|
610
|
+
}
|
|
611
|
+
}
|
|
612
|
+
goto st5;
|
|
613
|
+
st5:
|
|
395
614
|
#line 1 "NONE"
|
|
396
615
|
{ts = 0;}
|
|
397
616
|
#line 1 "NONE"
|
|
398
617
|
{act = 0;}
|
|
399
618
|
if ( ++p == pe )
|
|
400
|
-
goto
|
|
401
|
-
case
|
|
619
|
+
goto _test_eof5;
|
|
620
|
+
case 5:
|
|
402
621
|
#line 1 "NONE"
|
|
403
622
|
{ts = p;}
|
|
404
|
-
#line
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
623
|
+
#line 624 "ext/fastcsv/fastcsv.c"
|
|
624
|
+
_widec = (*p);
|
|
625
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
626
|
+
if (
|
|
627
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
628
|
+
(*p) == quote_char ) _widec += 256;
|
|
629
|
+
if (
|
|
630
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
631
|
+
(*p) == col_sep ) _widec += 512;
|
|
632
|
+
switch( _widec ) {
|
|
633
|
+
case 1280: goto tr33;
|
|
634
|
+
case 1290: goto tr3;
|
|
635
|
+
case 1293: goto tr4;
|
|
636
|
+
case 1536: goto tr35;
|
|
637
|
+
case 1546: goto tr36;
|
|
638
|
+
case 1549: goto tr37;
|
|
639
|
+
case 1792: goto tr7;
|
|
640
|
+
case 1802: goto tr8;
|
|
641
|
+
case 1805: goto tr9;
|
|
642
|
+
case 2048: goto tr39;
|
|
643
|
+
case 2058: goto tr40;
|
|
644
|
+
case 2061: goto tr41;
|
|
411
645
|
}
|
|
412
|
-
|
|
646
|
+
if ( _widec < 1408 ) {
|
|
647
|
+
if ( 1152 <= _widec && _widec <= 1407 )
|
|
648
|
+
goto st1;
|
|
649
|
+
} else if ( _widec > 1663 ) {
|
|
650
|
+
if ( _widec > 1919 ) {
|
|
651
|
+
if ( 1920 <= _widec && _widec <= 2175 )
|
|
652
|
+
goto tr38;
|
|
653
|
+
} else if ( _widec >= 1664 )
|
|
654
|
+
goto tr6;
|
|
655
|
+
} else
|
|
656
|
+
goto tr34;
|
|
657
|
+
goto st0;
|
|
658
|
+
st0:
|
|
659
|
+
cs = 0;
|
|
660
|
+
goto _out;
|
|
413
661
|
st1:
|
|
414
662
|
if ( ++p == pe )
|
|
415
663
|
goto _test_eof1;
|
|
416
664
|
case 1:
|
|
417
|
-
|
|
418
|
-
|
|
419
|
-
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
665
|
+
_widec = (*p);
|
|
666
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
667
|
+
if (
|
|
668
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
669
|
+
(*p) == quote_char ) _widec += 256;
|
|
670
|
+
if (
|
|
671
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
672
|
+
(*p) == col_sep ) _widec += 512;
|
|
673
|
+
switch( _widec ) {
|
|
674
|
+
case 1280: goto tr2;
|
|
675
|
+
case 1290: goto tr3;
|
|
676
|
+
case 1293: goto tr4;
|
|
677
|
+
case 1536: goto tr5;
|
|
678
|
+
case 1546: goto tr3;
|
|
679
|
+
case 1549: goto tr4;
|
|
680
|
+
case 1792: goto tr7;
|
|
681
|
+
case 1802: goto tr8;
|
|
682
|
+
case 1805: goto tr9;
|
|
683
|
+
case 2048: goto tr7;
|
|
684
|
+
case 2058: goto tr8;
|
|
685
|
+
case 2061: goto tr9;
|
|
423
686
|
}
|
|
424
|
-
|
|
687
|
+
if ( _widec > 1407 ) {
|
|
688
|
+
if ( 1664 <= _widec && _widec <= 2175 )
|
|
689
|
+
goto tr6;
|
|
690
|
+
} else if ( _widec >= 1152 )
|
|
691
|
+
goto st1;
|
|
692
|
+
goto tr0;
|
|
425
693
|
tr2:
|
|
426
694
|
#line 1 "NONE"
|
|
427
695
|
{te = p+1;}
|
|
428
|
-
#line
|
|
696
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
429
697
|
{
|
|
430
698
|
if (p == ts) {
|
|
431
699
|
// Unquoted empty fields are nil, not "", in Ruby.
|
|
@@ -436,9 +704,9 @@ tr2:
|
|
|
436
704
|
ENCODE;
|
|
437
705
|
}
|
|
438
706
|
}
|
|
439
|
-
#line
|
|
707
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
440
708
|
{
|
|
441
|
-
if (d->start == 0 || p == d->start) {
|
|
709
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
442
710
|
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
443
711
|
}
|
|
444
712
|
else if (p > d->start) {
|
|
@@ -453,24 +721,46 @@ tr2:
|
|
|
453
721
|
rb_yield(row);
|
|
454
722
|
}
|
|
455
723
|
}
|
|
456
|
-
#line
|
|
724
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
457
725
|
{act = 3;}
|
|
458
|
-
goto
|
|
459
|
-
|
|
726
|
+
goto st6;
|
|
727
|
+
st6:
|
|
460
728
|
if ( ++p == pe )
|
|
461
|
-
goto
|
|
462
|
-
case
|
|
463
|
-
#line
|
|
464
|
-
|
|
465
|
-
|
|
466
|
-
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
729
|
+
goto _test_eof6;
|
|
730
|
+
case 6:
|
|
731
|
+
#line 732 "ext/fastcsv/fastcsv.c"
|
|
732
|
+
_widec = (*p);
|
|
733
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
734
|
+
if (
|
|
735
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
736
|
+
(*p) == quote_char ) _widec += 256;
|
|
737
|
+
if (
|
|
738
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
739
|
+
(*p) == col_sep ) _widec += 512;
|
|
740
|
+
switch( _widec ) {
|
|
741
|
+
case 1280: goto tr2;
|
|
742
|
+
case 1290: goto tr3;
|
|
743
|
+
case 1293: goto tr4;
|
|
744
|
+
case 1536: goto tr5;
|
|
745
|
+
case 1546: goto tr3;
|
|
746
|
+
case 1549: goto tr4;
|
|
747
|
+
case 1792: goto tr7;
|
|
748
|
+
case 1802: goto tr8;
|
|
749
|
+
case 1805: goto tr9;
|
|
750
|
+
case 2048: goto tr7;
|
|
751
|
+
case 2058: goto tr8;
|
|
752
|
+
case 2061: goto tr9;
|
|
470
753
|
}
|
|
471
|
-
|
|
754
|
+
if ( _widec > 1407 ) {
|
|
755
|
+
if ( 1664 <= _widec && _widec <= 2175 )
|
|
756
|
+
goto tr6;
|
|
757
|
+
} else if ( _widec >= 1152 )
|
|
758
|
+
goto st1;
|
|
759
|
+
goto tr42;
|
|
472
760
|
tr3:
|
|
473
|
-
#line
|
|
761
|
+
#line 1 "NONE"
|
|
762
|
+
{te = p+1;}
|
|
763
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
474
764
|
{
|
|
475
765
|
if (p == ts) {
|
|
476
766
|
// Unquoted empty fields are nil, not "", in Ruby.
|
|
@@ -481,8 +771,9 @@ tr3:
|
|
|
481
771
|
ENCODE;
|
|
482
772
|
}
|
|
483
773
|
}
|
|
484
|
-
#line
|
|
774
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
485
775
|
{
|
|
776
|
+
mark_row_sep = p;
|
|
486
777
|
curline++;
|
|
487
778
|
|
|
488
779
|
if (d->start == 0 || p == d->start) {
|
|
@@ -500,10 +791,33 @@ tr3:
|
|
|
500
791
|
rb_yield(row);
|
|
501
792
|
row = rb_ary_new();
|
|
502
793
|
}
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
794
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
795
|
+
{act = 2;}
|
|
796
|
+
goto st7;
|
|
797
|
+
tr8:
|
|
798
|
+
#line 1 "NONE"
|
|
799
|
+
{te = p+1;}
|
|
800
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
506
801
|
{
|
|
802
|
+
if (p == ts) {
|
|
803
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
804
|
+
field = Qnil;
|
|
805
|
+
}
|
|
806
|
+
else if (p > ts) {
|
|
807
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
808
|
+
ENCODE;
|
|
809
|
+
}
|
|
810
|
+
}
|
|
811
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
812
|
+
{
|
|
813
|
+
rb_ary_push(row, field);
|
|
814
|
+
field = Qnil;
|
|
815
|
+
}
|
|
816
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
817
|
+
{act = 1;}
|
|
818
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
819
|
+
{
|
|
820
|
+
mark_row_sep = p;
|
|
507
821
|
curline++;
|
|
508
822
|
|
|
509
823
|
if (d->start == 0 || p == d->start) {
|
|
@@ -521,27 +835,13 @@ tr10:
|
|
|
521
835
|
rb_yield(row);
|
|
522
836
|
row = rb_ary_new();
|
|
523
837
|
}
|
|
524
|
-
goto
|
|
525
|
-
|
|
526
|
-
|
|
527
|
-
|
|
528
|
-
|
|
529
|
-
#line 530 "ext/fastcsv/fastcsv.c"
|
|
530
|
-
goto tr16;
|
|
531
|
-
tr4:
|
|
532
|
-
#line 46 "ext/fastcsv/fastcsv.rl"
|
|
533
|
-
{
|
|
534
|
-
if (p == ts) {
|
|
535
|
-
// Unquoted empty fields are nil, not "", in Ruby.
|
|
536
|
-
field = Qnil;
|
|
537
|
-
}
|
|
538
|
-
else if (p > ts) {
|
|
539
|
-
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
540
|
-
ENCODE;
|
|
541
|
-
}
|
|
542
|
-
}
|
|
543
|
-
#line 101 "ext/fastcsv/fastcsv.rl"
|
|
838
|
+
goto st7;
|
|
839
|
+
tr14:
|
|
840
|
+
#line 1 "NONE"
|
|
841
|
+
{te = p+1;}
|
|
842
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
544
843
|
{
|
|
844
|
+
mark_row_sep = p;
|
|
545
845
|
curline++;
|
|
546
846
|
|
|
547
847
|
if (d->start == 0 || p == d->start) {
|
|
@@ -559,10 +859,22 @@ tr4:
|
|
|
559
859
|
rb_yield(row);
|
|
560
860
|
row = rb_ary_new();
|
|
561
861
|
}
|
|
862
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
863
|
+
{act = 2;}
|
|
562
864
|
goto st7;
|
|
563
|
-
|
|
564
|
-
#line
|
|
865
|
+
tr21:
|
|
866
|
+
#line 1 "NONE"
|
|
867
|
+
{te = p+1;}
|
|
868
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
869
|
+
{
|
|
870
|
+
rb_ary_push(row, field);
|
|
871
|
+
field = Qnil;
|
|
872
|
+
}
|
|
873
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
874
|
+
{act = 1;}
|
|
875
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
565
876
|
{
|
|
877
|
+
mark_row_sep = p;
|
|
566
878
|
curline++;
|
|
567
879
|
|
|
568
880
|
if (d->start == 0 || p == d->start) {
|
|
@@ -581,18 +893,22 @@ tr11:
|
|
|
581
893
|
row = rb_ary_new();
|
|
582
894
|
}
|
|
583
895
|
goto st7;
|
|
896
|
+
tr44:
|
|
897
|
+
#line 1 "NONE"
|
|
898
|
+
{te = p+1;}
|
|
899
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
900
|
+
{act = 2;}
|
|
901
|
+
goto st7;
|
|
584
902
|
st7:
|
|
585
903
|
if ( ++p == pe )
|
|
586
904
|
goto _test_eof7;
|
|
587
905
|
case 7:
|
|
588
|
-
#line
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
goto tr16;
|
|
592
|
-
tr13:
|
|
906
|
+
#line 907 "ext/fastcsv/fastcsv.c"
|
|
907
|
+
goto tr43;
|
|
908
|
+
tr4:
|
|
593
909
|
#line 1 "NONE"
|
|
594
910
|
{te = p+1;}
|
|
595
|
-
#line
|
|
911
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
596
912
|
{
|
|
597
913
|
if (p == ts) {
|
|
598
914
|
// Unquoted empty fields are nil, not "", in Ruby.
|
|
@@ -603,59 +919,283 @@ tr13:
|
|
|
603
919
|
ENCODE;
|
|
604
920
|
}
|
|
605
921
|
}
|
|
606
|
-
#line
|
|
922
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
607
923
|
{
|
|
608
|
-
|
|
924
|
+
mark_row_sep = p;
|
|
925
|
+
curline++;
|
|
926
|
+
|
|
927
|
+
if (d->start == 0 || p == d->start) {
|
|
609
928
|
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
610
929
|
}
|
|
611
930
|
else if (p > d->start) {
|
|
612
931
|
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
613
932
|
}
|
|
614
933
|
|
|
615
|
-
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
934
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
616
935
|
rb_ary_push(row, field);
|
|
936
|
+
field = Qnil;
|
|
617
937
|
}
|
|
618
938
|
|
|
619
|
-
|
|
620
|
-
|
|
939
|
+
rb_yield(row);
|
|
940
|
+
row = rb_ary_new();
|
|
941
|
+
}
|
|
942
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
943
|
+
{act = 2;}
|
|
944
|
+
goto st8;
|
|
945
|
+
tr9:
|
|
946
|
+
#line 1 "NONE"
|
|
947
|
+
{te = p+1;}
|
|
948
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
949
|
+
{
|
|
950
|
+
if (p == ts) {
|
|
951
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
952
|
+
field = Qnil;
|
|
953
|
+
}
|
|
954
|
+
else if (p > ts) {
|
|
955
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
956
|
+
ENCODE;
|
|
621
957
|
}
|
|
622
958
|
}
|
|
623
|
-
#line
|
|
624
|
-
{
|
|
959
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
960
|
+
{
|
|
961
|
+
rb_ary_push(row, field);
|
|
962
|
+
field = Qnil;
|
|
963
|
+
}
|
|
964
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
965
|
+
{act = 1;}
|
|
966
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
967
|
+
{
|
|
968
|
+
mark_row_sep = p;
|
|
969
|
+
curline++;
|
|
970
|
+
|
|
971
|
+
if (d->start == 0 || p == d->start) {
|
|
972
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
973
|
+
}
|
|
974
|
+
else if (p > d->start) {
|
|
975
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
976
|
+
}
|
|
977
|
+
|
|
978
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
979
|
+
rb_ary_push(row, field);
|
|
980
|
+
field = Qnil;
|
|
981
|
+
}
|
|
982
|
+
|
|
983
|
+
rb_yield(row);
|
|
984
|
+
row = rb_ary_new();
|
|
985
|
+
}
|
|
986
|
+
goto st8;
|
|
987
|
+
tr15:
|
|
988
|
+
#line 1 "NONE"
|
|
989
|
+
{te = p+1;}
|
|
990
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
991
|
+
{
|
|
992
|
+
mark_row_sep = p;
|
|
993
|
+
curline++;
|
|
994
|
+
|
|
995
|
+
if (d->start == 0 || p == d->start) {
|
|
996
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
997
|
+
}
|
|
998
|
+
else if (p > d->start) {
|
|
999
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1000
|
+
}
|
|
1001
|
+
|
|
1002
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1003
|
+
rb_ary_push(row, field);
|
|
1004
|
+
field = Qnil;
|
|
1005
|
+
}
|
|
1006
|
+
|
|
1007
|
+
rb_yield(row);
|
|
1008
|
+
row = rb_ary_new();
|
|
1009
|
+
}
|
|
1010
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1011
|
+
{act = 2;}
|
|
1012
|
+
goto st8;
|
|
1013
|
+
tr22:
|
|
1014
|
+
#line 1 "NONE"
|
|
1015
|
+
{te = p+1;}
|
|
1016
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1017
|
+
{
|
|
1018
|
+
rb_ary_push(row, field);
|
|
1019
|
+
field = Qnil;
|
|
1020
|
+
}
|
|
1021
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1022
|
+
{act = 1;}
|
|
1023
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1024
|
+
{
|
|
1025
|
+
mark_row_sep = p;
|
|
1026
|
+
curline++;
|
|
1027
|
+
|
|
1028
|
+
if (d->start == 0 || p == d->start) {
|
|
1029
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1030
|
+
}
|
|
1031
|
+
else if (p > d->start) {
|
|
1032
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1033
|
+
}
|
|
1034
|
+
|
|
1035
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1036
|
+
rb_ary_push(row, field);
|
|
1037
|
+
field = Qnil;
|
|
1038
|
+
}
|
|
1039
|
+
|
|
1040
|
+
rb_yield(row);
|
|
1041
|
+
row = rb_ary_new();
|
|
1042
|
+
}
|
|
625
1043
|
goto st8;
|
|
626
1044
|
st8:
|
|
627
1045
|
if ( ++p == pe )
|
|
628
1046
|
goto _test_eof8;
|
|
629
1047
|
case 8:
|
|
630
|
-
#line
|
|
631
|
-
|
|
632
|
-
|
|
633
|
-
|
|
634
|
-
|
|
635
|
-
|
|
636
|
-
}
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
1048
|
+
#line 1049 "ext/fastcsv/fastcsv.c"
|
|
1049
|
+
if ( (*p) == 10 )
|
|
1050
|
+
goto tr44;
|
|
1051
|
+
goto tr43;
|
|
1052
|
+
tr33:
|
|
1053
|
+
#line 1 "NONE"
|
|
1054
|
+
{te = p+1;}
|
|
1055
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1056
|
+
{
|
|
1057
|
+
if (p == ts) {
|
|
1058
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1059
|
+
field = Qnil;
|
|
1060
|
+
}
|
|
1061
|
+
else if (p > ts) {
|
|
1062
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1063
|
+
ENCODE;
|
|
1064
|
+
}
|
|
1065
|
+
}
|
|
1066
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
1067
|
+
{
|
|
1068
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
1069
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1070
|
+
}
|
|
1071
|
+
else if (p > d->start) {
|
|
1072
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1073
|
+
}
|
|
1074
|
+
|
|
1075
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
1076
|
+
rb_ary_push(row, field);
|
|
1077
|
+
}
|
|
1078
|
+
|
|
1079
|
+
if (RARRAY_LEN(row)) {
|
|
1080
|
+
rb_yield(row);
|
|
1081
|
+
}
|
|
1082
|
+
}
|
|
1083
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
1084
|
+
{act = 3;}
|
|
1085
|
+
goto st9;
|
|
1086
|
+
st9:
|
|
1087
|
+
if ( ++p == pe )
|
|
1088
|
+
goto _test_eof9;
|
|
1089
|
+
case 9:
|
|
1090
|
+
#line 1091 "ext/fastcsv/fastcsv.c"
|
|
1091
|
+
_widec = (*p);
|
|
1092
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
1093
|
+
if (
|
|
1094
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1095
|
+
(*p) == quote_char ) _widec += 256;
|
|
1096
|
+
if (
|
|
1097
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
1098
|
+
(*p) == col_sep ) _widec += 512;
|
|
1099
|
+
if ( _widec < 1291 ) {
|
|
1100
|
+
if ( 1152 <= _widec && _widec <= 1289 )
|
|
1101
|
+
goto st1;
|
|
1102
|
+
} else if ( _widec > 1292 ) {
|
|
1103
|
+
if ( 1294 <= _widec && _widec <= 1407 )
|
|
1104
|
+
goto st1;
|
|
1105
|
+
} else
|
|
1106
|
+
goto st1;
|
|
1107
|
+
goto tr42;
|
|
1108
|
+
tr34:
|
|
1109
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
640
1110
|
{
|
|
641
1111
|
unclosed_line = curline;
|
|
642
1112
|
}
|
|
643
1113
|
goto st2;
|
|
1114
|
+
tr45:
|
|
1115
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1116
|
+
{
|
|
1117
|
+
d->start = p;
|
|
1118
|
+
|
|
1119
|
+
if (len_row_sep) {
|
|
1120
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1121
|
+
FREE;
|
|
1122
|
+
|
|
1123
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1124
|
+
}
|
|
1125
|
+
}
|
|
1126
|
+
else {
|
|
1127
|
+
len_row_sep = p - mark_row_sep;
|
|
1128
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1129
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1130
|
+
}
|
|
1131
|
+
}
|
|
1132
|
+
goto st2;
|
|
644
1133
|
st2:
|
|
645
1134
|
if ( ++p == pe )
|
|
646
1135
|
goto _test_eof2;
|
|
647
1136
|
case 2:
|
|
648
|
-
#line
|
|
649
|
-
|
|
650
|
-
|
|
651
|
-
|
|
1137
|
+
#line 1138 "ext/fastcsv/fastcsv.c"
|
|
1138
|
+
_widec = (*p);
|
|
1139
|
+
_widec = (short)(128 + ((*p) - -128));
|
|
1140
|
+
if (
|
|
1141
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1142
|
+
(*p) == quote_char ) _widec += 256;
|
|
1143
|
+
switch( _widec ) {
|
|
1144
|
+
case 522: goto tr12;
|
|
1145
|
+
case 525: goto tr12;
|
|
652
1146
|
}
|
|
653
|
-
|
|
654
|
-
|
|
655
|
-
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
1147
|
+
if ( _widec < 257 ) {
|
|
1148
|
+
if ( 128 <= _widec && _widec <= 255 )
|
|
1149
|
+
goto st2;
|
|
1150
|
+
} else if ( _widec > 383 ) {
|
|
1151
|
+
if ( 384 <= _widec && _widec <= 639 )
|
|
1152
|
+
goto tr11;
|
|
1153
|
+
} else
|
|
1154
|
+
goto st2;
|
|
1155
|
+
goto tr0;
|
|
1156
|
+
tr11:
|
|
1157
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
1158
|
+
{
|
|
1159
|
+
if (p == ts) {
|
|
1160
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
1161
|
+
ENCODE;
|
|
1162
|
+
}
|
|
1163
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
1164
|
+
else if (p > ts) {
|
|
1165
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
1166
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
1167
|
+
memcpy(copy, ts, p - ts);
|
|
1168
|
+
|
|
1169
|
+
char *reader = ts, *writer = copy;
|
|
1170
|
+
int escaped = 0;
|
|
1171
|
+
|
|
1172
|
+
while (p > reader) {
|
|
1173
|
+
if (*reader == quote_char && !escaped) {
|
|
1174
|
+
// Skip the escaping character.
|
|
1175
|
+
escaped = 1;
|
|
1176
|
+
}
|
|
1177
|
+
else {
|
|
1178
|
+
escaped = 0;
|
|
1179
|
+
*writer++ = *reader;
|
|
1180
|
+
}
|
|
1181
|
+
reader++;
|
|
1182
|
+
}
|
|
1183
|
+
|
|
1184
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
1185
|
+
ENCODE;
|
|
1186
|
+
|
|
1187
|
+
if (copy != NULL) {
|
|
1188
|
+
free(copy);
|
|
1189
|
+
}
|
|
1190
|
+
}
|
|
1191
|
+
}
|
|
1192
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
1193
|
+
{
|
|
1194
|
+
unclosed_line = 0;
|
|
1195
|
+
}
|
|
1196
|
+
goto st3;
|
|
1197
|
+
tr46:
|
|
1198
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
659
1199
|
{
|
|
660
1200
|
if (p == ts) {
|
|
661
1201
|
field = rb_enc_str_new("", 0, encoding);
|
|
@@ -690,53 +1230,1854 @@ tr8:
|
|
|
690
1230
|
}
|
|
691
1231
|
}
|
|
692
1232
|
}
|
|
693
|
-
#line
|
|
1233
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
694
1234
|
{
|
|
695
1235
|
unclosed_line = 0;
|
|
1236
|
+
}
|
|
1237
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1238
|
+
{
|
|
1239
|
+
d->start = p;
|
|
1240
|
+
|
|
1241
|
+
if (len_row_sep) {
|
|
1242
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1243
|
+
FREE;
|
|
1244
|
+
|
|
1245
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1246
|
+
}
|
|
1247
|
+
}
|
|
1248
|
+
else {
|
|
1249
|
+
len_row_sep = p - mark_row_sep;
|
|
1250
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1251
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1252
|
+
}
|
|
696
1253
|
}
|
|
697
1254
|
goto st3;
|
|
698
1255
|
st3:
|
|
699
1256
|
if ( ++p == pe )
|
|
700
1257
|
goto _test_eof3;
|
|
701
1258
|
case 3:
|
|
702
|
-
#line
|
|
703
|
-
|
|
704
|
-
|
|
705
|
-
|
|
706
|
-
|
|
707
|
-
|
|
708
|
-
|
|
709
|
-
|
|
710
|
-
|
|
1259
|
+
#line 1260 "ext/fastcsv/fastcsv.c"
|
|
1260
|
+
_widec = (*p);
|
|
1261
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
1262
|
+
if (
|
|
1263
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1264
|
+
(*p) == quote_char ) _widec += 256;
|
|
1265
|
+
if (
|
|
1266
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
1267
|
+
(*p) == col_sep ) _widec += 512;
|
|
1268
|
+
switch( _widec ) {
|
|
1269
|
+
case 1280: goto tr13;
|
|
1270
|
+
case 1290: goto tr14;
|
|
1271
|
+
case 1293: goto tr15;
|
|
1272
|
+
case 1536: goto tr16;
|
|
1273
|
+
case 1546: goto tr17;
|
|
1274
|
+
case 1549: goto tr18;
|
|
1275
|
+
case 1792: goto tr20;
|
|
1276
|
+
case 1802: goto tr21;
|
|
1277
|
+
case 1805: goto tr22;
|
|
1278
|
+
case 2048: goto tr24;
|
|
1279
|
+
case 2058: goto tr25;
|
|
1280
|
+
case 2061: goto tr26;
|
|
711
1281
|
}
|
|
712
|
-
|
|
713
|
-
|
|
714
|
-
|
|
715
|
-
|
|
716
|
-
|
|
717
|
-
|
|
718
|
-
|
|
719
|
-
|
|
1282
|
+
if ( _widec < 1664 ) {
|
|
1283
|
+
if ( 1408 <= _widec && _widec <= 1663 )
|
|
1284
|
+
goto st2;
|
|
1285
|
+
} else if ( _widec > 1919 ) {
|
|
1286
|
+
if ( 1920 <= _widec && _widec <= 2175 )
|
|
1287
|
+
goto tr23;
|
|
1288
|
+
} else
|
|
1289
|
+
goto tr19;
|
|
1290
|
+
goto tr0;
|
|
1291
|
+
tr16:
|
|
1292
|
+
#line 1 "NONE"
|
|
1293
|
+
{te = p+1;}
|
|
1294
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
1295
|
+
{
|
|
1296
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
1297
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1298
|
+
}
|
|
1299
|
+
else if (p > d->start) {
|
|
1300
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1301
|
+
}
|
|
1302
|
+
|
|
1303
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
1304
|
+
rb_ary_push(row, field);
|
|
1305
|
+
}
|
|
1306
|
+
|
|
1307
|
+
if (RARRAY_LEN(row)) {
|
|
1308
|
+
rb_yield(row);
|
|
1309
|
+
}
|
|
1310
|
+
}
|
|
1311
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
1312
|
+
{act = 3;}
|
|
1313
|
+
goto st10;
|
|
1314
|
+
tr23:
|
|
1315
|
+
#line 1 "NONE"
|
|
1316
|
+
{te = p+1;}
|
|
1317
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1318
|
+
{
|
|
1319
|
+
rb_ary_push(row, field);
|
|
1320
|
+
field = Qnil;
|
|
1321
|
+
}
|
|
1322
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1323
|
+
{act = 1;}
|
|
1324
|
+
goto st10;
|
|
1325
|
+
tr24:
|
|
1326
|
+
#line 1 "NONE"
|
|
1327
|
+
{te = p+1;}
|
|
1328
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1329
|
+
{
|
|
1330
|
+
rb_ary_push(row, field);
|
|
1331
|
+
field = Qnil;
|
|
1332
|
+
}
|
|
1333
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1334
|
+
{act = 1;}
|
|
1335
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
1336
|
+
{
|
|
1337
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
1338
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1339
|
+
}
|
|
1340
|
+
else if (p > d->start) {
|
|
1341
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1342
|
+
}
|
|
1343
|
+
|
|
1344
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
1345
|
+
rb_ary_push(row, field);
|
|
1346
|
+
}
|
|
1347
|
+
|
|
1348
|
+
if (RARRAY_LEN(row)) {
|
|
1349
|
+
rb_yield(row);
|
|
1350
|
+
}
|
|
1351
|
+
}
|
|
1352
|
+
goto st10;
|
|
1353
|
+
tr35:
|
|
1354
|
+
#line 1 "NONE"
|
|
1355
|
+
{te = p+1;}
|
|
1356
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1357
|
+
{
|
|
1358
|
+
unclosed_line = curline;
|
|
1359
|
+
}
|
|
1360
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1361
|
+
{
|
|
1362
|
+
if (p == ts) {
|
|
1363
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1364
|
+
field = Qnil;
|
|
1365
|
+
}
|
|
1366
|
+
else if (p > ts) {
|
|
1367
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1368
|
+
ENCODE;
|
|
1369
|
+
}
|
|
1370
|
+
}
|
|
1371
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
1372
|
+
{
|
|
1373
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
1374
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1375
|
+
}
|
|
1376
|
+
else if (p > d->start) {
|
|
1377
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1378
|
+
}
|
|
1379
|
+
|
|
1380
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
1381
|
+
rb_ary_push(row, field);
|
|
1382
|
+
}
|
|
1383
|
+
|
|
1384
|
+
if (RARRAY_LEN(row)) {
|
|
1385
|
+
rb_yield(row);
|
|
1386
|
+
}
|
|
1387
|
+
}
|
|
1388
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
1389
|
+
{act = 3;}
|
|
1390
|
+
goto st10;
|
|
1391
|
+
tr38:
|
|
1392
|
+
#line 1 "NONE"
|
|
1393
|
+
{te = p+1;}
|
|
1394
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1395
|
+
{
|
|
1396
|
+
if (p == ts) {
|
|
1397
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1398
|
+
field = Qnil;
|
|
1399
|
+
}
|
|
1400
|
+
else if (p > ts) {
|
|
1401
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1402
|
+
ENCODE;
|
|
1403
|
+
}
|
|
1404
|
+
}
|
|
1405
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1406
|
+
{
|
|
1407
|
+
unclosed_line = curline;
|
|
1408
|
+
}
|
|
1409
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1410
|
+
{
|
|
1411
|
+
rb_ary_push(row, field);
|
|
1412
|
+
field = Qnil;
|
|
1413
|
+
}
|
|
1414
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1415
|
+
{act = 1;}
|
|
1416
|
+
goto st10;
|
|
1417
|
+
tr39:
|
|
1418
|
+
#line 1 "NONE"
|
|
1419
|
+
{te = p+1;}
|
|
1420
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1421
|
+
{
|
|
1422
|
+
if (p == ts) {
|
|
1423
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1424
|
+
field = Qnil;
|
|
1425
|
+
}
|
|
1426
|
+
else if (p > ts) {
|
|
1427
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1428
|
+
ENCODE;
|
|
1429
|
+
}
|
|
1430
|
+
}
|
|
1431
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1432
|
+
{
|
|
1433
|
+
unclosed_line = curline;
|
|
1434
|
+
}
|
|
1435
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1436
|
+
{
|
|
1437
|
+
rb_ary_push(row, field);
|
|
1438
|
+
field = Qnil;
|
|
1439
|
+
}
|
|
1440
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1441
|
+
{act = 1;}
|
|
1442
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
1443
|
+
{
|
|
1444
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
1445
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1446
|
+
}
|
|
1447
|
+
else if (p > d->start) {
|
|
1448
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1449
|
+
}
|
|
1450
|
+
|
|
1451
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
1452
|
+
rb_ary_push(row, field);
|
|
1453
|
+
}
|
|
1454
|
+
|
|
1455
|
+
if (RARRAY_LEN(row)) {
|
|
1456
|
+
rb_yield(row);
|
|
1457
|
+
}
|
|
1458
|
+
}
|
|
1459
|
+
goto st10;
|
|
1460
|
+
tr55:
|
|
1461
|
+
#line 1 "NONE"
|
|
1462
|
+
{te = p+1;}
|
|
1463
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1464
|
+
{
|
|
1465
|
+
rb_ary_push(row, field);
|
|
1466
|
+
field = Qnil;
|
|
1467
|
+
}
|
|
1468
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1469
|
+
{act = 1;}
|
|
1470
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1471
|
+
{
|
|
1472
|
+
d->start = p;
|
|
1473
|
+
|
|
1474
|
+
if (len_row_sep) {
|
|
1475
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1476
|
+
FREE;
|
|
1477
|
+
|
|
1478
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1479
|
+
}
|
|
1480
|
+
}
|
|
1481
|
+
else {
|
|
1482
|
+
len_row_sep = p - mark_row_sep;
|
|
1483
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1484
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1485
|
+
}
|
|
1486
|
+
}
|
|
1487
|
+
goto st10;
|
|
1488
|
+
st10:
|
|
1489
|
+
if ( ++p == pe )
|
|
1490
|
+
goto _test_eof10;
|
|
1491
|
+
case 10:
|
|
1492
|
+
#line 1493 "ext/fastcsv/fastcsv.c"
|
|
1493
|
+
_widec = (*p);
|
|
1494
|
+
_widec = (short)(128 + ((*p) - -128));
|
|
1495
|
+
if (
|
|
1496
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1497
|
+
(*p) == quote_char ) _widec += 256;
|
|
1498
|
+
switch( _widec ) {
|
|
1499
|
+
case 522: goto tr12;
|
|
1500
|
+
case 525: goto tr12;
|
|
1501
|
+
}
|
|
1502
|
+
if ( _widec < 257 ) {
|
|
1503
|
+
if ( 128 <= _widec && _widec <= 255 )
|
|
1504
|
+
goto st2;
|
|
1505
|
+
} else if ( _widec > 383 ) {
|
|
1506
|
+
if ( 384 <= _widec && _widec <= 639 )
|
|
1507
|
+
goto tr11;
|
|
1508
|
+
} else
|
|
1509
|
+
goto st2;
|
|
1510
|
+
goto tr0;
|
|
1511
|
+
tr12:
|
|
1512
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
1513
|
+
{
|
|
1514
|
+
if (p == ts) {
|
|
1515
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
1516
|
+
ENCODE;
|
|
1517
|
+
}
|
|
1518
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
1519
|
+
else if (p > ts) {
|
|
1520
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
1521
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
1522
|
+
memcpy(copy, ts, p - ts);
|
|
1523
|
+
|
|
1524
|
+
char *reader = ts, *writer = copy;
|
|
1525
|
+
int escaped = 0;
|
|
1526
|
+
|
|
1527
|
+
while (p > reader) {
|
|
1528
|
+
if (*reader == quote_char && !escaped) {
|
|
1529
|
+
// Skip the escaping character.
|
|
1530
|
+
escaped = 1;
|
|
1531
|
+
}
|
|
1532
|
+
else {
|
|
1533
|
+
escaped = 0;
|
|
1534
|
+
*writer++ = *reader;
|
|
1535
|
+
}
|
|
1536
|
+
reader++;
|
|
1537
|
+
}
|
|
1538
|
+
|
|
1539
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
1540
|
+
ENCODE;
|
|
1541
|
+
|
|
1542
|
+
if (copy != NULL) {
|
|
1543
|
+
free(copy);
|
|
1544
|
+
}
|
|
1545
|
+
}
|
|
1546
|
+
}
|
|
1547
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
1548
|
+
{
|
|
1549
|
+
unclosed_line = 0;
|
|
1550
|
+
}
|
|
1551
|
+
goto st4;
|
|
1552
|
+
tr47:
|
|
1553
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
1554
|
+
{
|
|
1555
|
+
if (p == ts) {
|
|
1556
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
1557
|
+
ENCODE;
|
|
1558
|
+
}
|
|
1559
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
1560
|
+
else if (p > ts) {
|
|
1561
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
1562
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
1563
|
+
memcpy(copy, ts, p - ts);
|
|
1564
|
+
|
|
1565
|
+
char *reader = ts, *writer = copy;
|
|
1566
|
+
int escaped = 0;
|
|
1567
|
+
|
|
1568
|
+
while (p > reader) {
|
|
1569
|
+
if (*reader == quote_char && !escaped) {
|
|
1570
|
+
// Skip the escaping character.
|
|
1571
|
+
escaped = 1;
|
|
1572
|
+
}
|
|
1573
|
+
else {
|
|
1574
|
+
escaped = 0;
|
|
1575
|
+
*writer++ = *reader;
|
|
1576
|
+
}
|
|
1577
|
+
reader++;
|
|
1578
|
+
}
|
|
1579
|
+
|
|
1580
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
1581
|
+
ENCODE;
|
|
1582
|
+
|
|
1583
|
+
if (copy != NULL) {
|
|
1584
|
+
free(copy);
|
|
1585
|
+
}
|
|
1586
|
+
}
|
|
1587
|
+
}
|
|
1588
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
1589
|
+
{
|
|
1590
|
+
unclosed_line = 0;
|
|
1591
|
+
}
|
|
1592
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1593
|
+
{
|
|
1594
|
+
d->start = p;
|
|
1595
|
+
|
|
1596
|
+
if (len_row_sep) {
|
|
1597
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1598
|
+
FREE;
|
|
1599
|
+
|
|
1600
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1601
|
+
}
|
|
1602
|
+
}
|
|
1603
|
+
else {
|
|
1604
|
+
len_row_sep = p - mark_row_sep;
|
|
1605
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1606
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1607
|
+
}
|
|
1608
|
+
}
|
|
1609
|
+
goto st4;
|
|
1610
|
+
st4:
|
|
1611
|
+
if ( ++p == pe )
|
|
1612
|
+
goto _test_eof4;
|
|
1613
|
+
case 4:
|
|
1614
|
+
#line 1615 "ext/fastcsv/fastcsv.c"
|
|
1615
|
+
_widec = (*p);
|
|
1616
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
1617
|
+
if (
|
|
1618
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1619
|
+
(*p) == quote_char ) _widec += 256;
|
|
1620
|
+
if (
|
|
1621
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
1622
|
+
(*p) == col_sep ) _widec += 512;
|
|
1623
|
+
switch( _widec ) {
|
|
1624
|
+
case 1280: goto tr13;
|
|
1625
|
+
case 1290: goto tr17;
|
|
1626
|
+
case 1293: goto tr18;
|
|
1627
|
+
case 1536: goto tr27;
|
|
1628
|
+
case 1546: goto tr28;
|
|
1629
|
+
case 1549: goto tr28;
|
|
1630
|
+
case 1792: goto tr20;
|
|
1631
|
+
case 1802: goto tr25;
|
|
1632
|
+
case 1805: goto tr26;
|
|
1633
|
+
case 2048: goto tr30;
|
|
1634
|
+
case 2058: goto tr31;
|
|
1635
|
+
case 2061: goto tr31;
|
|
1636
|
+
}
|
|
1637
|
+
if ( _widec < 1408 ) {
|
|
1638
|
+
if ( 1152 <= _widec && _widec <= 1407 )
|
|
1639
|
+
goto st2;
|
|
1640
|
+
} else if ( _widec > 1663 ) {
|
|
1641
|
+
if ( _widec > 1919 ) {
|
|
1642
|
+
if ( 1920 <= _widec && _widec <= 2175 )
|
|
1643
|
+
goto tr29;
|
|
1644
|
+
} else if ( _widec >= 1664 )
|
|
1645
|
+
goto tr23;
|
|
1646
|
+
} else
|
|
1647
|
+
goto tr12;
|
|
1648
|
+
goto tr0;
|
|
1649
|
+
tr17:
|
|
1650
|
+
#line 1 "NONE"
|
|
1651
|
+
{te = p+1;}
|
|
1652
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1653
|
+
{
|
|
1654
|
+
mark_row_sep = p;
|
|
1655
|
+
curline++;
|
|
1656
|
+
|
|
1657
|
+
if (d->start == 0 || p == d->start) {
|
|
1658
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1659
|
+
}
|
|
1660
|
+
else if (p > d->start) {
|
|
1661
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1662
|
+
}
|
|
1663
|
+
|
|
1664
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1665
|
+
rb_ary_push(row, field);
|
|
1666
|
+
field = Qnil;
|
|
1667
|
+
}
|
|
1668
|
+
|
|
1669
|
+
rb_yield(row);
|
|
1670
|
+
row = rb_ary_new();
|
|
1671
|
+
}
|
|
1672
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1673
|
+
{act = 2;}
|
|
1674
|
+
goto st11;
|
|
1675
|
+
tr25:
|
|
1676
|
+
#line 1 "NONE"
|
|
1677
|
+
{te = p+1;}
|
|
1678
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1679
|
+
{
|
|
1680
|
+
rb_ary_push(row, field);
|
|
1681
|
+
field = Qnil;
|
|
1682
|
+
}
|
|
1683
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1684
|
+
{act = 1;}
|
|
1685
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1686
|
+
{
|
|
1687
|
+
mark_row_sep = p;
|
|
1688
|
+
curline++;
|
|
1689
|
+
|
|
1690
|
+
if (d->start == 0 || p == d->start) {
|
|
1691
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1692
|
+
}
|
|
1693
|
+
else if (p > d->start) {
|
|
1694
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1695
|
+
}
|
|
1696
|
+
|
|
1697
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1698
|
+
rb_ary_push(row, field);
|
|
1699
|
+
field = Qnil;
|
|
1700
|
+
}
|
|
1701
|
+
|
|
1702
|
+
rb_yield(row);
|
|
1703
|
+
row = rb_ary_new();
|
|
1704
|
+
}
|
|
1705
|
+
goto st11;
|
|
1706
|
+
tr36:
|
|
1707
|
+
#line 1 "NONE"
|
|
1708
|
+
{te = p+1;}
|
|
1709
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1710
|
+
{
|
|
1711
|
+
unclosed_line = curline;
|
|
1712
|
+
}
|
|
1713
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1714
|
+
{
|
|
1715
|
+
if (p == ts) {
|
|
1716
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1717
|
+
field = Qnil;
|
|
1718
|
+
}
|
|
1719
|
+
else if (p > ts) {
|
|
1720
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1721
|
+
ENCODE;
|
|
1722
|
+
}
|
|
1723
|
+
}
|
|
1724
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1725
|
+
{
|
|
1726
|
+
mark_row_sep = p;
|
|
1727
|
+
curline++;
|
|
1728
|
+
|
|
1729
|
+
if (d->start == 0 || p == d->start) {
|
|
1730
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1731
|
+
}
|
|
1732
|
+
else if (p > d->start) {
|
|
1733
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1734
|
+
}
|
|
1735
|
+
|
|
1736
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1737
|
+
rb_ary_push(row, field);
|
|
1738
|
+
field = Qnil;
|
|
1739
|
+
}
|
|
1740
|
+
|
|
1741
|
+
rb_yield(row);
|
|
1742
|
+
row = rb_ary_new();
|
|
1743
|
+
}
|
|
1744
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1745
|
+
{act = 2;}
|
|
1746
|
+
goto st11;
|
|
1747
|
+
tr40:
|
|
1748
|
+
#line 1 "NONE"
|
|
1749
|
+
{te = p+1;}
|
|
1750
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1751
|
+
{
|
|
1752
|
+
if (p == ts) {
|
|
1753
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1754
|
+
field = Qnil;
|
|
1755
|
+
}
|
|
1756
|
+
else if (p > ts) {
|
|
1757
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
1758
|
+
ENCODE;
|
|
1759
|
+
}
|
|
1760
|
+
}
|
|
1761
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1762
|
+
{
|
|
1763
|
+
unclosed_line = curline;
|
|
1764
|
+
}
|
|
1765
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1766
|
+
{
|
|
1767
|
+
rb_ary_push(row, field);
|
|
1768
|
+
field = Qnil;
|
|
1769
|
+
}
|
|
1770
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1771
|
+
{act = 1;}
|
|
1772
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1773
|
+
{
|
|
1774
|
+
mark_row_sep = p;
|
|
1775
|
+
curline++;
|
|
1776
|
+
|
|
1777
|
+
if (d->start == 0 || p == d->start) {
|
|
1778
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1779
|
+
}
|
|
1780
|
+
else if (p > d->start) {
|
|
1781
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1782
|
+
}
|
|
1783
|
+
|
|
1784
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1785
|
+
rb_ary_push(row, field);
|
|
1786
|
+
field = Qnil;
|
|
1787
|
+
}
|
|
1788
|
+
|
|
1789
|
+
rb_yield(row);
|
|
1790
|
+
row = rb_ary_new();
|
|
1791
|
+
}
|
|
1792
|
+
goto st11;
|
|
1793
|
+
tr48:
|
|
1794
|
+
#line 1 "NONE"
|
|
1795
|
+
{te = p+1;}
|
|
1796
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1797
|
+
{
|
|
1798
|
+
d->start = p;
|
|
1799
|
+
|
|
1800
|
+
if (len_row_sep) {
|
|
1801
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1802
|
+
FREE;
|
|
1803
|
+
|
|
1804
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1805
|
+
}
|
|
1806
|
+
}
|
|
1807
|
+
else {
|
|
1808
|
+
len_row_sep = p - mark_row_sep;
|
|
1809
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1810
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1811
|
+
}
|
|
1812
|
+
}
|
|
1813
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1814
|
+
{act = 2;}
|
|
1815
|
+
goto st11;
|
|
1816
|
+
tr51:
|
|
1817
|
+
#line 1 "NONE"
|
|
1818
|
+
{te = p+1;}
|
|
1819
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1820
|
+
{
|
|
1821
|
+
mark_row_sep = p;
|
|
1822
|
+
curline++;
|
|
1823
|
+
|
|
1824
|
+
if (d->start == 0 || p == d->start) {
|
|
1825
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1826
|
+
}
|
|
1827
|
+
else if (p > d->start) {
|
|
1828
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1829
|
+
}
|
|
1830
|
+
|
|
1831
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1832
|
+
rb_ary_push(row, field);
|
|
1833
|
+
field = Qnil;
|
|
1834
|
+
}
|
|
1835
|
+
|
|
1836
|
+
rb_yield(row);
|
|
1837
|
+
row = rb_ary_new();
|
|
1838
|
+
}
|
|
1839
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1840
|
+
{
|
|
1841
|
+
d->start = p;
|
|
1842
|
+
|
|
1843
|
+
if (len_row_sep) {
|
|
1844
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1845
|
+
FREE;
|
|
1846
|
+
|
|
1847
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1848
|
+
}
|
|
1849
|
+
}
|
|
1850
|
+
else {
|
|
1851
|
+
len_row_sep = p - mark_row_sep;
|
|
1852
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1853
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1854
|
+
}
|
|
1855
|
+
}
|
|
1856
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1857
|
+
{act = 2;}
|
|
1858
|
+
goto st11;
|
|
1859
|
+
tr57:
|
|
1860
|
+
#line 1 "NONE"
|
|
1861
|
+
{te = p+1;}
|
|
1862
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1863
|
+
{
|
|
1864
|
+
rb_ary_push(row, field);
|
|
1865
|
+
field = Qnil;
|
|
1866
|
+
}
|
|
1867
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1868
|
+
{act = 1;}
|
|
1869
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1870
|
+
{
|
|
1871
|
+
mark_row_sep = p;
|
|
1872
|
+
curline++;
|
|
1873
|
+
|
|
1874
|
+
if (d->start == 0 || p == d->start) {
|
|
1875
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1876
|
+
}
|
|
1877
|
+
else if (p > d->start) {
|
|
1878
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1879
|
+
}
|
|
1880
|
+
|
|
1881
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1882
|
+
rb_ary_push(row, field);
|
|
1883
|
+
field = Qnil;
|
|
1884
|
+
}
|
|
1885
|
+
|
|
1886
|
+
rb_yield(row);
|
|
1887
|
+
row = rb_ary_new();
|
|
1888
|
+
}
|
|
1889
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
1890
|
+
{
|
|
1891
|
+
d->start = p;
|
|
1892
|
+
|
|
1893
|
+
if (len_row_sep) {
|
|
1894
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
1895
|
+
FREE;
|
|
1896
|
+
|
|
1897
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
1898
|
+
}
|
|
1899
|
+
}
|
|
1900
|
+
else {
|
|
1901
|
+
len_row_sep = p - mark_row_sep;
|
|
1902
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
1903
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
1904
|
+
}
|
|
1905
|
+
}
|
|
1906
|
+
goto st11;
|
|
1907
|
+
st11:
|
|
1908
|
+
if ( ++p == pe )
|
|
1909
|
+
goto _test_eof11;
|
|
1910
|
+
case 11:
|
|
1911
|
+
#line 1912 "ext/fastcsv/fastcsv.c"
|
|
1912
|
+
_widec = (*p);
|
|
1913
|
+
_widec = (short)(128 + ((*p) - -128));
|
|
1914
|
+
if (
|
|
1915
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
1916
|
+
(*p) == quote_char ) _widec += 256;
|
|
1917
|
+
switch( _widec ) {
|
|
1918
|
+
case 256: goto tr43;
|
|
1919
|
+
case 522: goto tr47;
|
|
1920
|
+
case 525: goto tr47;
|
|
1921
|
+
}
|
|
1922
|
+
if ( _widec > 383 ) {
|
|
1923
|
+
if ( 384 <= _widec && _widec <= 639 )
|
|
1924
|
+
goto tr46;
|
|
1925
|
+
} else if ( _widec >= 128 )
|
|
1926
|
+
goto tr45;
|
|
1927
|
+
goto tr0;
|
|
1928
|
+
tr18:
|
|
1929
|
+
#line 1 "NONE"
|
|
1930
|
+
{te = p+1;}
|
|
1931
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1932
|
+
{
|
|
1933
|
+
mark_row_sep = p;
|
|
1934
|
+
curline++;
|
|
1935
|
+
|
|
1936
|
+
if (d->start == 0 || p == d->start) {
|
|
1937
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1938
|
+
}
|
|
1939
|
+
else if (p > d->start) {
|
|
1940
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1941
|
+
}
|
|
1942
|
+
|
|
1943
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1944
|
+
rb_ary_push(row, field);
|
|
1945
|
+
field = Qnil;
|
|
1946
|
+
}
|
|
1947
|
+
|
|
1948
|
+
rb_yield(row);
|
|
1949
|
+
row = rb_ary_new();
|
|
1950
|
+
}
|
|
1951
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
1952
|
+
{act = 2;}
|
|
1953
|
+
goto st12;
|
|
1954
|
+
tr26:
|
|
1955
|
+
#line 1 "NONE"
|
|
1956
|
+
{te = p+1;}
|
|
1957
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
1958
|
+
{
|
|
1959
|
+
rb_ary_push(row, field);
|
|
1960
|
+
field = Qnil;
|
|
1961
|
+
}
|
|
1962
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
1963
|
+
{act = 1;}
|
|
1964
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
1965
|
+
{
|
|
1966
|
+
mark_row_sep = p;
|
|
1967
|
+
curline++;
|
|
1968
|
+
|
|
1969
|
+
if (d->start == 0 || p == d->start) {
|
|
1970
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
1971
|
+
}
|
|
1972
|
+
else if (p > d->start) {
|
|
1973
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
1974
|
+
}
|
|
1975
|
+
|
|
1976
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
1977
|
+
rb_ary_push(row, field);
|
|
1978
|
+
field = Qnil;
|
|
1979
|
+
}
|
|
1980
|
+
|
|
1981
|
+
rb_yield(row);
|
|
1982
|
+
row = rb_ary_new();
|
|
1983
|
+
}
|
|
1984
|
+
goto st12;
|
|
1985
|
+
tr37:
|
|
1986
|
+
#line 1 "NONE"
|
|
1987
|
+
{te = p+1;}
|
|
1988
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
1989
|
+
{
|
|
1990
|
+
unclosed_line = curline;
|
|
1991
|
+
}
|
|
1992
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
1993
|
+
{
|
|
1994
|
+
if (p == ts) {
|
|
1995
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
1996
|
+
field = Qnil;
|
|
1997
|
+
}
|
|
1998
|
+
else if (p > ts) {
|
|
1999
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
2000
|
+
ENCODE;
|
|
2001
|
+
}
|
|
2002
|
+
}
|
|
2003
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2004
|
+
{
|
|
2005
|
+
mark_row_sep = p;
|
|
2006
|
+
curline++;
|
|
2007
|
+
|
|
2008
|
+
if (d->start == 0 || p == d->start) {
|
|
2009
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2010
|
+
}
|
|
2011
|
+
else if (p > d->start) {
|
|
2012
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2013
|
+
}
|
|
2014
|
+
|
|
2015
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2016
|
+
rb_ary_push(row, field);
|
|
2017
|
+
field = Qnil;
|
|
2018
|
+
}
|
|
2019
|
+
|
|
2020
|
+
rb_yield(row);
|
|
2021
|
+
row = rb_ary_new();
|
|
2022
|
+
}
|
|
2023
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
2024
|
+
{act = 2;}
|
|
2025
|
+
goto st12;
|
|
2026
|
+
tr41:
|
|
2027
|
+
#line 1 "NONE"
|
|
2028
|
+
{te = p+1;}
|
|
2029
|
+
#line 49 "ext/fastcsv/fastcsv.rl"
|
|
2030
|
+
{
|
|
2031
|
+
if (p == ts) {
|
|
2032
|
+
// Unquoted empty fields are nil, not "", in Ruby.
|
|
2033
|
+
field = Qnil;
|
|
2034
|
+
}
|
|
2035
|
+
else if (p > ts) {
|
|
2036
|
+
field = rb_enc_str_new(ts, p - ts, encoding);
|
|
2037
|
+
ENCODE;
|
|
2038
|
+
}
|
|
2039
|
+
}
|
|
2040
|
+
#line 41 "ext/fastcsv/fastcsv.rl"
|
|
2041
|
+
{
|
|
2042
|
+
unclosed_line = curline;
|
|
2043
|
+
}
|
|
2044
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2045
|
+
{
|
|
2046
|
+
rb_ary_push(row, field);
|
|
2047
|
+
field = Qnil;
|
|
2048
|
+
}
|
|
2049
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2050
|
+
{act = 1;}
|
|
2051
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2052
|
+
{
|
|
2053
|
+
mark_row_sep = p;
|
|
2054
|
+
curline++;
|
|
2055
|
+
|
|
2056
|
+
if (d->start == 0 || p == d->start) {
|
|
2057
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2058
|
+
}
|
|
2059
|
+
else if (p > d->start) {
|
|
2060
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2061
|
+
}
|
|
2062
|
+
|
|
2063
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2064
|
+
rb_ary_push(row, field);
|
|
2065
|
+
field = Qnil;
|
|
2066
|
+
}
|
|
2067
|
+
|
|
2068
|
+
rb_yield(row);
|
|
2069
|
+
row = rb_ary_new();
|
|
2070
|
+
}
|
|
2071
|
+
goto st12;
|
|
2072
|
+
tr52:
|
|
2073
|
+
#line 1 "NONE"
|
|
2074
|
+
{te = p+1;}
|
|
2075
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2076
|
+
{
|
|
2077
|
+
mark_row_sep = p;
|
|
2078
|
+
curline++;
|
|
2079
|
+
|
|
2080
|
+
if (d->start == 0 || p == d->start) {
|
|
2081
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2082
|
+
}
|
|
2083
|
+
else if (p > d->start) {
|
|
2084
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2085
|
+
}
|
|
2086
|
+
|
|
2087
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2088
|
+
rb_ary_push(row, field);
|
|
2089
|
+
field = Qnil;
|
|
2090
|
+
}
|
|
2091
|
+
|
|
2092
|
+
rb_yield(row);
|
|
2093
|
+
row = rb_ary_new();
|
|
2094
|
+
}
|
|
2095
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2096
|
+
{
|
|
2097
|
+
d->start = p;
|
|
2098
|
+
|
|
2099
|
+
if (len_row_sep) {
|
|
2100
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2101
|
+
FREE;
|
|
2102
|
+
|
|
2103
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2104
|
+
}
|
|
2105
|
+
}
|
|
2106
|
+
else {
|
|
2107
|
+
len_row_sep = p - mark_row_sep;
|
|
2108
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2109
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2110
|
+
}
|
|
2111
|
+
}
|
|
2112
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
2113
|
+
{act = 2;}
|
|
2114
|
+
goto st12;
|
|
2115
|
+
tr58:
|
|
2116
|
+
#line 1 "NONE"
|
|
2117
|
+
{te = p+1;}
|
|
2118
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2119
|
+
{
|
|
2120
|
+
rb_ary_push(row, field);
|
|
2121
|
+
field = Qnil;
|
|
2122
|
+
}
|
|
2123
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2124
|
+
{act = 1;}
|
|
2125
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2126
|
+
{
|
|
2127
|
+
mark_row_sep = p;
|
|
2128
|
+
curline++;
|
|
2129
|
+
|
|
2130
|
+
if (d->start == 0 || p == d->start) {
|
|
2131
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2132
|
+
}
|
|
2133
|
+
else if (p > d->start) {
|
|
2134
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2135
|
+
}
|
|
2136
|
+
|
|
2137
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2138
|
+
rb_ary_push(row, field);
|
|
2139
|
+
field = Qnil;
|
|
2140
|
+
}
|
|
2141
|
+
|
|
2142
|
+
rb_yield(row);
|
|
2143
|
+
row = rb_ary_new();
|
|
2144
|
+
}
|
|
2145
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2146
|
+
{
|
|
2147
|
+
d->start = p;
|
|
2148
|
+
|
|
2149
|
+
if (len_row_sep) {
|
|
2150
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2151
|
+
FREE;
|
|
2152
|
+
|
|
2153
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2154
|
+
}
|
|
2155
|
+
}
|
|
2156
|
+
else {
|
|
2157
|
+
len_row_sep = p - mark_row_sep;
|
|
2158
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2159
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2160
|
+
}
|
|
2161
|
+
}
|
|
2162
|
+
goto st12;
|
|
2163
|
+
st12:
|
|
2164
|
+
if ( ++p == pe )
|
|
2165
|
+
goto _test_eof12;
|
|
2166
|
+
case 12:
|
|
2167
|
+
#line 2168 "ext/fastcsv/fastcsv.c"
|
|
2168
|
+
_widec = (*p);
|
|
2169
|
+
_widec = (short)(128 + ((*p) - -128));
|
|
2170
|
+
if (
|
|
2171
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
2172
|
+
(*p) == quote_char ) _widec += 256;
|
|
2173
|
+
switch( _widec ) {
|
|
2174
|
+
case 256: goto tr43;
|
|
2175
|
+
case 266: goto tr48;
|
|
2176
|
+
case 522: goto tr49;
|
|
2177
|
+
case 525: goto tr47;
|
|
2178
|
+
}
|
|
2179
|
+
if ( _widec > 383 ) {
|
|
2180
|
+
if ( 384 <= _widec && _widec <= 639 )
|
|
2181
|
+
goto tr46;
|
|
2182
|
+
} else if ( _widec >= 128 )
|
|
2183
|
+
goto tr45;
|
|
2184
|
+
goto tr0;
|
|
2185
|
+
tr28:
|
|
2186
|
+
#line 1 "NONE"
|
|
2187
|
+
{te = p+1;}
|
|
2188
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2189
|
+
{
|
|
2190
|
+
if (p == ts) {
|
|
2191
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2192
|
+
ENCODE;
|
|
2193
|
+
}
|
|
2194
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2195
|
+
else if (p > ts) {
|
|
2196
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2197
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2198
|
+
memcpy(copy, ts, p - ts);
|
|
2199
|
+
|
|
2200
|
+
char *reader = ts, *writer = copy;
|
|
2201
|
+
int escaped = 0;
|
|
2202
|
+
|
|
2203
|
+
while (p > reader) {
|
|
2204
|
+
if (*reader == quote_char && !escaped) {
|
|
2205
|
+
// Skip the escaping character.
|
|
2206
|
+
escaped = 1;
|
|
2207
|
+
}
|
|
2208
|
+
else {
|
|
2209
|
+
escaped = 0;
|
|
2210
|
+
*writer++ = *reader;
|
|
2211
|
+
}
|
|
2212
|
+
reader++;
|
|
2213
|
+
}
|
|
2214
|
+
|
|
2215
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2216
|
+
ENCODE;
|
|
2217
|
+
|
|
2218
|
+
if (copy != NULL) {
|
|
2219
|
+
free(copy);
|
|
2220
|
+
}
|
|
2221
|
+
}
|
|
2222
|
+
}
|
|
2223
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2224
|
+
{
|
|
2225
|
+
unclosed_line = 0;
|
|
2226
|
+
}
|
|
2227
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2228
|
+
{
|
|
2229
|
+
mark_row_sep = p;
|
|
2230
|
+
curline++;
|
|
2231
|
+
|
|
2232
|
+
if (d->start == 0 || p == d->start) {
|
|
2233
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2234
|
+
}
|
|
2235
|
+
else if (p > d->start) {
|
|
2236
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2237
|
+
}
|
|
2238
|
+
|
|
2239
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2240
|
+
rb_ary_push(row, field);
|
|
2241
|
+
field = Qnil;
|
|
2242
|
+
}
|
|
2243
|
+
|
|
2244
|
+
rb_yield(row);
|
|
2245
|
+
row = rb_ary_new();
|
|
2246
|
+
}
|
|
2247
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
2248
|
+
{act = 2;}
|
|
2249
|
+
goto st13;
|
|
2250
|
+
tr31:
|
|
2251
|
+
#line 1 "NONE"
|
|
2252
|
+
{te = p+1;}
|
|
2253
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2254
|
+
{
|
|
2255
|
+
if (p == ts) {
|
|
2256
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2257
|
+
ENCODE;
|
|
2258
|
+
}
|
|
2259
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2260
|
+
else if (p > ts) {
|
|
2261
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2262
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2263
|
+
memcpy(copy, ts, p - ts);
|
|
2264
|
+
|
|
2265
|
+
char *reader = ts, *writer = copy;
|
|
2266
|
+
int escaped = 0;
|
|
2267
|
+
|
|
2268
|
+
while (p > reader) {
|
|
2269
|
+
if (*reader == quote_char && !escaped) {
|
|
2270
|
+
// Skip the escaping character.
|
|
2271
|
+
escaped = 1;
|
|
2272
|
+
}
|
|
2273
|
+
else {
|
|
2274
|
+
escaped = 0;
|
|
2275
|
+
*writer++ = *reader;
|
|
2276
|
+
}
|
|
2277
|
+
reader++;
|
|
2278
|
+
}
|
|
2279
|
+
|
|
2280
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2281
|
+
ENCODE;
|
|
2282
|
+
|
|
2283
|
+
if (copy != NULL) {
|
|
2284
|
+
free(copy);
|
|
2285
|
+
}
|
|
2286
|
+
}
|
|
2287
|
+
}
|
|
2288
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2289
|
+
{
|
|
2290
|
+
unclosed_line = 0;
|
|
2291
|
+
}
|
|
2292
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2293
|
+
{
|
|
2294
|
+
rb_ary_push(row, field);
|
|
2295
|
+
field = Qnil;
|
|
2296
|
+
}
|
|
2297
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2298
|
+
{act = 1;}
|
|
2299
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2300
|
+
{
|
|
2301
|
+
mark_row_sep = p;
|
|
2302
|
+
curline++;
|
|
2303
|
+
|
|
2304
|
+
if (d->start == 0 || p == d->start) {
|
|
2305
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2306
|
+
}
|
|
2307
|
+
else if (p > d->start) {
|
|
2308
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2309
|
+
}
|
|
2310
|
+
|
|
2311
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2312
|
+
rb_ary_push(row, field);
|
|
2313
|
+
field = Qnil;
|
|
2314
|
+
}
|
|
2315
|
+
|
|
2316
|
+
rb_yield(row);
|
|
2317
|
+
row = rb_ary_new();
|
|
2318
|
+
}
|
|
2319
|
+
goto st13;
|
|
2320
|
+
tr49:
|
|
2321
|
+
#line 1 "NONE"
|
|
2322
|
+
{te = p+1;}
|
|
2323
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2324
|
+
{
|
|
2325
|
+
if (p == ts) {
|
|
2326
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2327
|
+
ENCODE;
|
|
2328
|
+
}
|
|
2329
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2330
|
+
else if (p > ts) {
|
|
2331
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2332
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2333
|
+
memcpy(copy, ts, p - ts);
|
|
2334
|
+
|
|
2335
|
+
char *reader = ts, *writer = copy;
|
|
2336
|
+
int escaped = 0;
|
|
2337
|
+
|
|
2338
|
+
while (p > reader) {
|
|
2339
|
+
if (*reader == quote_char && !escaped) {
|
|
2340
|
+
// Skip the escaping character.
|
|
2341
|
+
escaped = 1;
|
|
2342
|
+
}
|
|
2343
|
+
else {
|
|
2344
|
+
escaped = 0;
|
|
2345
|
+
*writer++ = *reader;
|
|
2346
|
+
}
|
|
2347
|
+
reader++;
|
|
2348
|
+
}
|
|
2349
|
+
|
|
2350
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2351
|
+
ENCODE;
|
|
2352
|
+
|
|
2353
|
+
if (copy != NULL) {
|
|
2354
|
+
free(copy);
|
|
2355
|
+
}
|
|
2356
|
+
}
|
|
2357
|
+
}
|
|
2358
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2359
|
+
{
|
|
2360
|
+
unclosed_line = 0;
|
|
2361
|
+
}
|
|
2362
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2363
|
+
{
|
|
2364
|
+
d->start = p;
|
|
2365
|
+
|
|
2366
|
+
if (len_row_sep) {
|
|
2367
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2368
|
+
FREE;
|
|
2369
|
+
|
|
2370
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2371
|
+
}
|
|
2372
|
+
}
|
|
2373
|
+
else {
|
|
2374
|
+
len_row_sep = p - mark_row_sep;
|
|
2375
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2376
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2377
|
+
}
|
|
2378
|
+
}
|
|
2379
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
2380
|
+
{act = 2;}
|
|
2381
|
+
goto st13;
|
|
2382
|
+
tr54:
|
|
2383
|
+
#line 1 "NONE"
|
|
2384
|
+
{te = p+1;}
|
|
2385
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2386
|
+
{
|
|
2387
|
+
if (p == ts) {
|
|
2388
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2389
|
+
ENCODE;
|
|
2390
|
+
}
|
|
2391
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2392
|
+
else if (p > ts) {
|
|
2393
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2394
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2395
|
+
memcpy(copy, ts, p - ts);
|
|
2396
|
+
|
|
2397
|
+
char *reader = ts, *writer = copy;
|
|
2398
|
+
int escaped = 0;
|
|
2399
|
+
|
|
2400
|
+
while (p > reader) {
|
|
2401
|
+
if (*reader == quote_char && !escaped) {
|
|
2402
|
+
// Skip the escaping character.
|
|
2403
|
+
escaped = 1;
|
|
2404
|
+
}
|
|
2405
|
+
else {
|
|
2406
|
+
escaped = 0;
|
|
2407
|
+
*writer++ = *reader;
|
|
2408
|
+
}
|
|
2409
|
+
reader++;
|
|
2410
|
+
}
|
|
2411
|
+
|
|
2412
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2413
|
+
ENCODE;
|
|
2414
|
+
|
|
2415
|
+
if (copy != NULL) {
|
|
2416
|
+
free(copy);
|
|
2417
|
+
}
|
|
2418
|
+
}
|
|
2419
|
+
}
|
|
2420
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2421
|
+
{
|
|
2422
|
+
unclosed_line = 0;
|
|
2423
|
+
}
|
|
2424
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2425
|
+
{
|
|
2426
|
+
mark_row_sep = p;
|
|
2427
|
+
curline++;
|
|
2428
|
+
|
|
2429
|
+
if (d->start == 0 || p == d->start) {
|
|
2430
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2431
|
+
}
|
|
2432
|
+
else if (p > d->start) {
|
|
2433
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2434
|
+
}
|
|
2435
|
+
|
|
2436
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2437
|
+
rb_ary_push(row, field);
|
|
2438
|
+
field = Qnil;
|
|
2439
|
+
}
|
|
2440
|
+
|
|
2441
|
+
rb_yield(row);
|
|
2442
|
+
row = rb_ary_new();
|
|
2443
|
+
}
|
|
2444
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2445
|
+
{
|
|
2446
|
+
d->start = p;
|
|
2447
|
+
|
|
2448
|
+
if (len_row_sep) {
|
|
2449
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2450
|
+
FREE;
|
|
2451
|
+
|
|
2452
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2453
|
+
}
|
|
2454
|
+
}
|
|
2455
|
+
else {
|
|
2456
|
+
len_row_sep = p - mark_row_sep;
|
|
2457
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2458
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2459
|
+
}
|
|
2460
|
+
}
|
|
2461
|
+
#line 166 "ext/fastcsv/fastcsv.rl"
|
|
2462
|
+
{act = 2;}
|
|
2463
|
+
goto st13;
|
|
2464
|
+
tr61:
|
|
2465
|
+
#line 1 "NONE"
|
|
2466
|
+
{te = p+1;}
|
|
2467
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2468
|
+
{
|
|
2469
|
+
if (p == ts) {
|
|
2470
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2471
|
+
ENCODE;
|
|
2472
|
+
}
|
|
2473
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2474
|
+
else if (p > ts) {
|
|
2475
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2476
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2477
|
+
memcpy(copy, ts, p - ts);
|
|
2478
|
+
|
|
2479
|
+
char *reader = ts, *writer = copy;
|
|
2480
|
+
int escaped = 0;
|
|
2481
|
+
|
|
2482
|
+
while (p > reader) {
|
|
2483
|
+
if (*reader == quote_char && !escaped) {
|
|
2484
|
+
// Skip the escaping character.
|
|
2485
|
+
escaped = 1;
|
|
2486
|
+
}
|
|
2487
|
+
else {
|
|
2488
|
+
escaped = 0;
|
|
2489
|
+
*writer++ = *reader;
|
|
2490
|
+
}
|
|
2491
|
+
reader++;
|
|
2492
|
+
}
|
|
2493
|
+
|
|
2494
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2495
|
+
ENCODE;
|
|
2496
|
+
|
|
2497
|
+
if (copy != NULL) {
|
|
2498
|
+
free(copy);
|
|
2499
|
+
}
|
|
2500
|
+
}
|
|
2501
|
+
}
|
|
2502
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2503
|
+
{
|
|
2504
|
+
unclosed_line = 0;
|
|
2505
|
+
}
|
|
2506
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2507
|
+
{
|
|
2508
|
+
rb_ary_push(row, field);
|
|
2509
|
+
field = Qnil;
|
|
2510
|
+
}
|
|
2511
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2512
|
+
{act = 1;}
|
|
2513
|
+
#line 117 "ext/fastcsv/fastcsv.rl"
|
|
2514
|
+
{
|
|
2515
|
+
mark_row_sep = p;
|
|
2516
|
+
curline++;
|
|
2517
|
+
|
|
2518
|
+
if (d->start == 0 || p == d->start) {
|
|
2519
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2520
|
+
}
|
|
2521
|
+
else if (p > d->start) {
|
|
2522
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2523
|
+
}
|
|
2524
|
+
|
|
2525
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) { // same as new_field
|
|
2526
|
+
rb_ary_push(row, field);
|
|
2527
|
+
field = Qnil;
|
|
2528
|
+
}
|
|
2529
|
+
|
|
2530
|
+
rb_yield(row);
|
|
2531
|
+
row = rb_ary_new();
|
|
2532
|
+
}
|
|
2533
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2534
|
+
{
|
|
2535
|
+
d->start = p;
|
|
2536
|
+
|
|
2537
|
+
if (len_row_sep) {
|
|
2538
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2539
|
+
FREE;
|
|
2540
|
+
|
|
2541
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2542
|
+
}
|
|
2543
|
+
}
|
|
2544
|
+
else {
|
|
2545
|
+
len_row_sep = p - mark_row_sep;
|
|
2546
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2547
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2548
|
+
}
|
|
2549
|
+
}
|
|
2550
|
+
goto st13;
|
|
2551
|
+
st13:
|
|
2552
|
+
if ( ++p == pe )
|
|
2553
|
+
goto _test_eof13;
|
|
2554
|
+
case 13:
|
|
2555
|
+
#line 2556 "ext/fastcsv/fastcsv.c"
|
|
2556
|
+
_widec = (*p);
|
|
2557
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
2558
|
+
if (
|
|
2559
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
2560
|
+
(*p) == quote_char ) _widec += 256;
|
|
2561
|
+
if (
|
|
2562
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
2563
|
+
(*p) == col_sep ) _widec += 512;
|
|
2564
|
+
switch( _widec ) {
|
|
2565
|
+
case 1280: goto tr50;
|
|
2566
|
+
case 1290: goto tr51;
|
|
2567
|
+
case 1293: goto tr52;
|
|
2568
|
+
case 1536: goto tr53;
|
|
2569
|
+
case 1546: goto tr54;
|
|
2570
|
+
case 1549: goto tr54;
|
|
2571
|
+
case 1792: goto tr56;
|
|
2572
|
+
case 1802: goto tr57;
|
|
2573
|
+
case 1805: goto tr58;
|
|
2574
|
+
case 2048: goto tr60;
|
|
2575
|
+
case 2058: goto tr61;
|
|
2576
|
+
case 2061: goto tr61;
|
|
2577
|
+
}
|
|
2578
|
+
if ( _widec < 1408 ) {
|
|
2579
|
+
if ( 1152 <= _widec && _widec <= 1407 )
|
|
2580
|
+
goto tr45;
|
|
2581
|
+
} else if ( _widec > 1663 ) {
|
|
2582
|
+
if ( _widec > 1919 ) {
|
|
2583
|
+
if ( 1920 <= _widec && _widec <= 2175 )
|
|
2584
|
+
goto tr59;
|
|
2585
|
+
} else if ( _widec >= 1664 )
|
|
2586
|
+
goto tr55;
|
|
2587
|
+
} else
|
|
2588
|
+
goto tr47;
|
|
2589
|
+
goto tr0;
|
|
2590
|
+
tr27:
|
|
2591
|
+
#line 1 "NONE"
|
|
2592
|
+
{te = p+1;}
|
|
2593
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2594
|
+
{
|
|
2595
|
+
if (p == ts) {
|
|
2596
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2597
|
+
ENCODE;
|
|
2598
|
+
}
|
|
2599
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2600
|
+
else if (p > ts) {
|
|
2601
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2602
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2603
|
+
memcpy(copy, ts, p - ts);
|
|
2604
|
+
|
|
2605
|
+
char *reader = ts, *writer = copy;
|
|
2606
|
+
int escaped = 0;
|
|
2607
|
+
|
|
2608
|
+
while (p > reader) {
|
|
2609
|
+
if (*reader == quote_char && !escaped) {
|
|
2610
|
+
// Skip the escaping character.
|
|
2611
|
+
escaped = 1;
|
|
2612
|
+
}
|
|
2613
|
+
else {
|
|
2614
|
+
escaped = 0;
|
|
2615
|
+
*writer++ = *reader;
|
|
2616
|
+
}
|
|
2617
|
+
reader++;
|
|
2618
|
+
}
|
|
2619
|
+
|
|
2620
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2621
|
+
ENCODE;
|
|
2622
|
+
|
|
2623
|
+
if (copy != NULL) {
|
|
2624
|
+
free(copy);
|
|
2625
|
+
}
|
|
2626
|
+
}
|
|
2627
|
+
}
|
|
2628
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2629
|
+
{
|
|
2630
|
+
unclosed_line = 0;
|
|
2631
|
+
}
|
|
2632
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
2633
|
+
{
|
|
2634
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
2635
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2636
|
+
}
|
|
2637
|
+
else if (p > d->start) {
|
|
2638
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2639
|
+
}
|
|
2640
|
+
|
|
2641
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
2642
|
+
rb_ary_push(row, field);
|
|
2643
|
+
}
|
|
2644
|
+
|
|
2645
|
+
if (RARRAY_LEN(row)) {
|
|
2646
|
+
rb_yield(row);
|
|
2647
|
+
}
|
|
2648
|
+
}
|
|
2649
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
2650
|
+
{act = 3;}
|
|
2651
|
+
goto st14;
|
|
2652
|
+
tr29:
|
|
2653
|
+
#line 1 "NONE"
|
|
2654
|
+
{te = p+1;}
|
|
2655
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2656
|
+
{
|
|
2657
|
+
if (p == ts) {
|
|
2658
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2659
|
+
ENCODE;
|
|
2660
|
+
}
|
|
2661
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2662
|
+
else if (p > ts) {
|
|
2663
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2664
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2665
|
+
memcpy(copy, ts, p - ts);
|
|
2666
|
+
|
|
2667
|
+
char *reader = ts, *writer = copy;
|
|
2668
|
+
int escaped = 0;
|
|
2669
|
+
|
|
2670
|
+
while (p > reader) {
|
|
2671
|
+
if (*reader == quote_char && !escaped) {
|
|
2672
|
+
// Skip the escaping character.
|
|
2673
|
+
escaped = 1;
|
|
2674
|
+
}
|
|
2675
|
+
else {
|
|
2676
|
+
escaped = 0;
|
|
2677
|
+
*writer++ = *reader;
|
|
2678
|
+
}
|
|
2679
|
+
reader++;
|
|
2680
|
+
}
|
|
2681
|
+
|
|
2682
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2683
|
+
ENCODE;
|
|
2684
|
+
|
|
2685
|
+
if (copy != NULL) {
|
|
2686
|
+
free(copy);
|
|
2687
|
+
}
|
|
2688
|
+
}
|
|
2689
|
+
}
|
|
2690
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2691
|
+
{
|
|
2692
|
+
unclosed_line = 0;
|
|
2693
|
+
}
|
|
2694
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2695
|
+
{
|
|
2696
|
+
rb_ary_push(row, field);
|
|
2697
|
+
field = Qnil;
|
|
2698
|
+
}
|
|
2699
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2700
|
+
{act = 1;}
|
|
2701
|
+
goto st14;
|
|
2702
|
+
tr30:
|
|
2703
|
+
#line 1 "NONE"
|
|
2704
|
+
{te = p+1;}
|
|
2705
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2706
|
+
{
|
|
2707
|
+
if (p == ts) {
|
|
2708
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2709
|
+
ENCODE;
|
|
2710
|
+
}
|
|
2711
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2712
|
+
else if (p > ts) {
|
|
2713
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2714
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2715
|
+
memcpy(copy, ts, p - ts);
|
|
2716
|
+
|
|
2717
|
+
char *reader = ts, *writer = copy;
|
|
2718
|
+
int escaped = 0;
|
|
2719
|
+
|
|
2720
|
+
while (p > reader) {
|
|
2721
|
+
if (*reader == quote_char && !escaped) {
|
|
2722
|
+
// Skip the escaping character.
|
|
2723
|
+
escaped = 1;
|
|
2724
|
+
}
|
|
2725
|
+
else {
|
|
2726
|
+
escaped = 0;
|
|
2727
|
+
*writer++ = *reader;
|
|
2728
|
+
}
|
|
2729
|
+
reader++;
|
|
2730
|
+
}
|
|
2731
|
+
|
|
2732
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2733
|
+
ENCODE;
|
|
2734
|
+
|
|
2735
|
+
if (copy != NULL) {
|
|
2736
|
+
free(copy);
|
|
2737
|
+
}
|
|
2738
|
+
}
|
|
2739
|
+
}
|
|
2740
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2741
|
+
{
|
|
2742
|
+
unclosed_line = 0;
|
|
2743
|
+
}
|
|
2744
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2745
|
+
{
|
|
2746
|
+
rb_ary_push(row, field);
|
|
2747
|
+
field = Qnil;
|
|
2748
|
+
}
|
|
2749
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2750
|
+
{act = 1;}
|
|
2751
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
2752
|
+
{
|
|
2753
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
2754
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2755
|
+
}
|
|
2756
|
+
else if (p > d->start) {
|
|
2757
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2758
|
+
}
|
|
2759
|
+
|
|
2760
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
2761
|
+
rb_ary_push(row, field);
|
|
2762
|
+
}
|
|
2763
|
+
|
|
2764
|
+
if (RARRAY_LEN(row)) {
|
|
2765
|
+
rb_yield(row);
|
|
2766
|
+
}
|
|
2767
|
+
}
|
|
2768
|
+
goto st14;
|
|
2769
|
+
tr53:
|
|
2770
|
+
#line 1 "NONE"
|
|
2771
|
+
{te = p+1;}
|
|
2772
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2773
|
+
{
|
|
2774
|
+
if (p == ts) {
|
|
2775
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2776
|
+
ENCODE;
|
|
2777
|
+
}
|
|
2778
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2779
|
+
else if (p > ts) {
|
|
2780
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2781
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2782
|
+
memcpy(copy, ts, p - ts);
|
|
2783
|
+
|
|
2784
|
+
char *reader = ts, *writer = copy;
|
|
2785
|
+
int escaped = 0;
|
|
2786
|
+
|
|
2787
|
+
while (p > reader) {
|
|
2788
|
+
if (*reader == quote_char && !escaped) {
|
|
2789
|
+
// Skip the escaping character.
|
|
2790
|
+
escaped = 1;
|
|
2791
|
+
}
|
|
2792
|
+
else {
|
|
2793
|
+
escaped = 0;
|
|
2794
|
+
*writer++ = *reader;
|
|
2795
|
+
}
|
|
2796
|
+
reader++;
|
|
2797
|
+
}
|
|
2798
|
+
|
|
2799
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2800
|
+
ENCODE;
|
|
2801
|
+
|
|
2802
|
+
if (copy != NULL) {
|
|
2803
|
+
free(copy);
|
|
2804
|
+
}
|
|
2805
|
+
}
|
|
2806
|
+
}
|
|
2807
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2808
|
+
{
|
|
2809
|
+
unclosed_line = 0;
|
|
2810
|
+
}
|
|
2811
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2812
|
+
{
|
|
2813
|
+
d->start = p;
|
|
2814
|
+
|
|
2815
|
+
if (len_row_sep) {
|
|
2816
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2817
|
+
FREE;
|
|
2818
|
+
|
|
2819
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2820
|
+
}
|
|
2821
|
+
}
|
|
2822
|
+
else {
|
|
2823
|
+
len_row_sep = p - mark_row_sep;
|
|
2824
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2825
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2826
|
+
}
|
|
2827
|
+
}
|
|
2828
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
2829
|
+
{
|
|
2830
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
2831
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2832
|
+
}
|
|
2833
|
+
else if (p > d->start) {
|
|
2834
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2835
|
+
}
|
|
2836
|
+
|
|
2837
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
2838
|
+
rb_ary_push(row, field);
|
|
2839
|
+
}
|
|
2840
|
+
|
|
2841
|
+
if (RARRAY_LEN(row)) {
|
|
2842
|
+
rb_yield(row);
|
|
2843
|
+
}
|
|
2844
|
+
}
|
|
2845
|
+
#line 167 "ext/fastcsv/fastcsv.rl"
|
|
2846
|
+
{act = 3;}
|
|
2847
|
+
goto st14;
|
|
2848
|
+
tr59:
|
|
2849
|
+
#line 1 "NONE"
|
|
2850
|
+
{te = p+1;}
|
|
2851
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2852
|
+
{
|
|
2853
|
+
if (p == ts) {
|
|
2854
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2855
|
+
ENCODE;
|
|
2856
|
+
}
|
|
2857
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2858
|
+
else if (p > ts) {
|
|
2859
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2860
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2861
|
+
memcpy(copy, ts, p - ts);
|
|
2862
|
+
|
|
2863
|
+
char *reader = ts, *writer = copy;
|
|
2864
|
+
int escaped = 0;
|
|
2865
|
+
|
|
2866
|
+
while (p > reader) {
|
|
2867
|
+
if (*reader == quote_char && !escaped) {
|
|
2868
|
+
// Skip the escaping character.
|
|
2869
|
+
escaped = 1;
|
|
2870
|
+
}
|
|
2871
|
+
else {
|
|
2872
|
+
escaped = 0;
|
|
2873
|
+
*writer++ = *reader;
|
|
2874
|
+
}
|
|
2875
|
+
reader++;
|
|
2876
|
+
}
|
|
2877
|
+
|
|
2878
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2879
|
+
ENCODE;
|
|
2880
|
+
|
|
2881
|
+
if (copy != NULL) {
|
|
2882
|
+
free(copy);
|
|
2883
|
+
}
|
|
2884
|
+
}
|
|
2885
|
+
}
|
|
2886
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2887
|
+
{
|
|
2888
|
+
unclosed_line = 0;
|
|
2889
|
+
}
|
|
2890
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2891
|
+
{
|
|
2892
|
+
rb_ary_push(row, field);
|
|
2893
|
+
field = Qnil;
|
|
2894
|
+
}
|
|
2895
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2896
|
+
{act = 1;}
|
|
2897
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2898
|
+
{
|
|
2899
|
+
d->start = p;
|
|
2900
|
+
|
|
2901
|
+
if (len_row_sep) {
|
|
2902
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2903
|
+
FREE;
|
|
2904
|
+
|
|
2905
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2906
|
+
}
|
|
2907
|
+
}
|
|
2908
|
+
else {
|
|
2909
|
+
len_row_sep = p - mark_row_sep;
|
|
2910
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2911
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2912
|
+
}
|
|
2913
|
+
}
|
|
2914
|
+
goto st14;
|
|
2915
|
+
tr60:
|
|
2916
|
+
#line 1 "NONE"
|
|
2917
|
+
{te = p+1;}
|
|
2918
|
+
#line 60 "ext/fastcsv/fastcsv.rl"
|
|
2919
|
+
{
|
|
2920
|
+
if (p == ts) {
|
|
2921
|
+
field = rb_enc_str_new("", 0, encoding);
|
|
2922
|
+
ENCODE;
|
|
2923
|
+
}
|
|
2924
|
+
// @note If we add an action on '""', we can skip some steps if no '""' is found.
|
|
2925
|
+
else if (p > ts) {
|
|
2926
|
+
// Operating on ts in-place produces odd behavior, FYI.
|
|
2927
|
+
char *copy = ALLOC_N(char, p - ts);
|
|
2928
|
+
memcpy(copy, ts, p - ts);
|
|
2929
|
+
|
|
2930
|
+
char *reader = ts, *writer = copy;
|
|
2931
|
+
int escaped = 0;
|
|
2932
|
+
|
|
2933
|
+
while (p > reader) {
|
|
2934
|
+
if (*reader == quote_char && !escaped) {
|
|
2935
|
+
// Skip the escaping character.
|
|
2936
|
+
escaped = 1;
|
|
2937
|
+
}
|
|
2938
|
+
else {
|
|
2939
|
+
escaped = 0;
|
|
2940
|
+
*writer++ = *reader;
|
|
2941
|
+
}
|
|
2942
|
+
reader++;
|
|
2943
|
+
}
|
|
2944
|
+
|
|
2945
|
+
field = rb_enc_str_new(copy, writer - copy, encoding);
|
|
2946
|
+
ENCODE;
|
|
2947
|
+
|
|
2948
|
+
if (copy != NULL) {
|
|
2949
|
+
free(copy);
|
|
2950
|
+
}
|
|
2951
|
+
}
|
|
2952
|
+
}
|
|
2953
|
+
#line 45 "ext/fastcsv/fastcsv.rl"
|
|
2954
|
+
{
|
|
2955
|
+
unclosed_line = 0;
|
|
2956
|
+
}
|
|
2957
|
+
#line 95 "ext/fastcsv/fastcsv.rl"
|
|
2958
|
+
{
|
|
2959
|
+
rb_ary_push(row, field);
|
|
2960
|
+
field = Qnil;
|
|
2961
|
+
}
|
|
2962
|
+
#line 165 "ext/fastcsv/fastcsv.rl"
|
|
2963
|
+
{act = 1;}
|
|
2964
|
+
#line 100 "ext/fastcsv/fastcsv.rl"
|
|
2965
|
+
{
|
|
2966
|
+
d->start = p;
|
|
2967
|
+
|
|
2968
|
+
if (len_row_sep) {
|
|
2969
|
+
if (p - mark_row_sep != len_row_sep || row_sep[0] != *mark_row_sep || (len_row_sep == 2 && row_sep[1] != *(mark_row_sep + 1))) {
|
|
2970
|
+
FREE;
|
|
2971
|
+
|
|
2972
|
+
rb_raise(eError, "Unquoted fields do not allow \\r or \\n (line %d).", curline - 1);
|
|
2973
|
+
}
|
|
2974
|
+
}
|
|
2975
|
+
else {
|
|
2976
|
+
len_row_sep = p - mark_row_sep;
|
|
2977
|
+
row_sep = ALLOC_N(char, len_row_sep);
|
|
2978
|
+
memcpy(row_sep, mark_row_sep, len_row_sep);
|
|
2979
|
+
}
|
|
2980
|
+
}
|
|
2981
|
+
#line 137 "ext/fastcsv/fastcsv.rl"
|
|
2982
|
+
{
|
|
2983
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
2984
|
+
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
2985
|
+
}
|
|
2986
|
+
else if (p > d->start) {
|
|
2987
|
+
rb_ivar_set(self, s_row, rb_str_new(d->start, p - d->start));
|
|
2988
|
+
}
|
|
2989
|
+
|
|
2990
|
+
if (!NIL_P(field) || RARRAY_LEN(row)) {
|
|
2991
|
+
rb_ary_push(row, field);
|
|
2992
|
+
}
|
|
2993
|
+
|
|
2994
|
+
if (RARRAY_LEN(row)) {
|
|
2995
|
+
rb_yield(row);
|
|
2996
|
+
}
|
|
2997
|
+
}
|
|
2998
|
+
goto st14;
|
|
2999
|
+
st14:
|
|
3000
|
+
if ( ++p == pe )
|
|
3001
|
+
goto _test_eof14;
|
|
3002
|
+
case 14:
|
|
3003
|
+
#line 3004 "ext/fastcsv/fastcsv.c"
|
|
3004
|
+
_widec = (*p);
|
|
3005
|
+
_widec = (short)(1152 + ((*p) - -128));
|
|
3006
|
+
if (
|
|
3007
|
+
#line 155 "ext/fastcsv/fastcsv.rl"
|
|
3008
|
+
(*p) == quote_char ) _widec += 256;
|
|
3009
|
+
if (
|
|
3010
|
+
#line 156 "ext/fastcsv/fastcsv.rl"
|
|
3011
|
+
(*p) == col_sep ) _widec += 512;
|
|
3012
|
+
switch( _widec ) {
|
|
3013
|
+
case 1280: goto tr13;
|
|
3014
|
+
case 1290: goto tr17;
|
|
3015
|
+
case 1293: goto tr18;
|
|
3016
|
+
case 1536: goto tr27;
|
|
3017
|
+
case 1546: goto tr28;
|
|
3018
|
+
case 1549: goto tr28;
|
|
3019
|
+
case 1792: goto tr20;
|
|
3020
|
+
case 1802: goto tr25;
|
|
3021
|
+
case 1805: goto tr26;
|
|
3022
|
+
case 2048: goto tr30;
|
|
3023
|
+
case 2058: goto tr31;
|
|
3024
|
+
case 2061: goto tr31;
|
|
3025
|
+
}
|
|
3026
|
+
if ( _widec < 1408 ) {
|
|
3027
|
+
if ( 1152 <= _widec && _widec <= 1407 )
|
|
3028
|
+
goto st2;
|
|
3029
|
+
} else if ( _widec > 1663 ) {
|
|
3030
|
+
if ( _widec > 1919 ) {
|
|
3031
|
+
if ( 1920 <= _widec && _widec <= 2175 )
|
|
3032
|
+
goto tr29;
|
|
3033
|
+
} else if ( _widec >= 1664 )
|
|
3034
|
+
goto tr23;
|
|
3035
|
+
} else
|
|
3036
|
+
goto tr12;
|
|
3037
|
+
goto tr0;
|
|
3038
|
+
}
|
|
3039
|
+
_test_eof5: cs = 5; goto _test_eof;
|
|
3040
|
+
_test_eof1: cs = 1; goto _test_eof;
|
|
3041
|
+
_test_eof6: cs = 6; goto _test_eof;
|
|
3042
|
+
_test_eof7: cs = 7; goto _test_eof;
|
|
3043
|
+
_test_eof8: cs = 8; goto _test_eof;
|
|
3044
|
+
_test_eof9: cs = 9; goto _test_eof;
|
|
3045
|
+
_test_eof2: cs = 2; goto _test_eof;
|
|
3046
|
+
_test_eof3: cs = 3; goto _test_eof;
|
|
3047
|
+
_test_eof10: cs = 10; goto _test_eof;
|
|
3048
|
+
_test_eof4: cs = 4; goto _test_eof;
|
|
3049
|
+
_test_eof11: cs = 11; goto _test_eof;
|
|
3050
|
+
_test_eof12: cs = 12; goto _test_eof;
|
|
3051
|
+
_test_eof13: cs = 13; goto _test_eof;
|
|
3052
|
+
_test_eof14: cs = 14; goto _test_eof;
|
|
720
3053
|
|
|
721
3054
|
_test_eof: {}
|
|
722
3055
|
if ( p == eof )
|
|
723
3056
|
{
|
|
724
3057
|
switch ( cs ) {
|
|
725
3058
|
case 1: goto tr0;
|
|
726
|
-
case
|
|
727
|
-
case
|
|
728
|
-
case
|
|
729
|
-
case
|
|
3059
|
+
case 6: goto tr42;
|
|
3060
|
+
case 7: goto tr43;
|
|
3061
|
+
case 8: goto tr43;
|
|
3062
|
+
case 9: goto tr42;
|
|
3063
|
+
case 2: goto tr0;
|
|
3064
|
+
case 3: goto tr0;
|
|
3065
|
+
case 10: goto tr0;
|
|
3066
|
+
case 4: goto tr0;
|
|
3067
|
+
case 11: goto tr43;
|
|
3068
|
+
case 12: goto tr43;
|
|
3069
|
+
case 13: goto tr43;
|
|
3070
|
+
case 14: goto tr0;
|
|
730
3071
|
}
|
|
731
3072
|
}
|
|
732
3073
|
|
|
733
3074
|
_out: {}
|
|
734
3075
|
}
|
|
735
3076
|
|
|
736
|
-
#line
|
|
3077
|
+
#line 444 "ext/fastcsv/fastcsv.rl"
|
|
737
3078
|
|
|
738
3079
|
if (done && cs < raw_parse_first_final) {
|
|
739
|
-
if (d->start == 0 || p == d->start) {
|
|
3080
|
+
if (d->start == 0 || p == d->start) { // same as new_row
|
|
740
3081
|
rb_ivar_set(self, s_row, rb_str_new2(""));
|
|
741
3082
|
}
|
|
742
3083
|
else if (p > d->start) {
|