@ezs/basics 1.13.5 → 1.15.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +49 -0
- package/README.md +29 -36
- package/lib/csv-object.js +4 -3
- package/lib/csv-parse.js +6 -3
- package/lib/csv-string.js +2 -1
- package/lib/fetch.js +74 -0
- package/lib/json-parse.js +23 -11
- package/lib/obj-flatten.js +2 -1
- package/lib/skos-object.js +73 -0
- package/lib/url-connect.js +3 -0
- package/lib/url-fetch.js +4 -0
- package/lib/url-parse.js +5 -2
- package/lib/url-stream.js +4 -0
- package/lib/url-string.js +2 -1
- package/lib/utils.js +18 -0
- package/lib/xml-convert.js +3 -1
- package/lib/xml-parse.js +1 -1
- package/lib/xml-string.js +1 -1
- package/package.json +12 -11
package/CHANGELOG.md
CHANGED
|
@@ -3,6 +3,55 @@
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
See [Conventional Commits](https://conventionalcommits.org) for commit guidelines.
|
|
5
5
|
|
|
6
|
+
## [1.15.2](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.15.1...@ezs/basics@1.15.2) (2022-01-27)
|
|
7
|
+
|
|
8
|
+
**Note:** Version bump only for package @ezs/basics
|
|
9
|
+
|
|
10
|
+
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
|
|
14
|
+
## [1.15.1](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.15.0...@ezs/basics@1.15.1) (2021-10-05)
|
|
15
|
+
|
|
16
|
+
|
|
17
|
+
### Bug Fixes
|
|
18
|
+
|
|
19
|
+
* 🐛 fix wrong options + miss dep.x ([5b38d05](https://github.com/Inist-CNRS/ezs/commit/5b38d05199a9a49c73d264f4ddb9a45dd0e64c7e))
|
|
20
|
+
|
|
21
|
+
|
|
22
|
+
|
|
23
|
+
|
|
24
|
+
|
|
25
|
+
# [1.15.0](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.14.0...@ezs/basics@1.15.0) (2021-10-05)
|
|
26
|
+
|
|
27
|
+
|
|
28
|
+
### Features
|
|
29
|
+
|
|
30
|
+
* 🎸 introduce new parser ([66a408b](https://github.com/Inist-CNRS/ezs/commit/66a408b0be2f6f2374d7193df2576e2fa383d369))
|
|
31
|
+
|
|
32
|
+
|
|
33
|
+
|
|
34
|
+
|
|
35
|
+
|
|
36
|
+
# [1.14.0](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.13.6...@ezs/basics@1.14.0) (2021-08-27)
|
|
37
|
+
|
|
38
|
+
|
|
39
|
+
### Features
|
|
40
|
+
|
|
41
|
+
* 🎸 support http headers ([0d11b7c](https://github.com/Inist-CNRS/ezs/commit/0d11b7c46a44040c48aa0d0fcb7954611dd36658))
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
|
|
46
|
+
|
|
47
|
+
## [1.13.6](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.13.5...@ezs/basics@1.13.6) (2021-07-30)
|
|
48
|
+
|
|
49
|
+
**Note:** Version bump only for package @ezs/basics
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
|
|
53
|
+
|
|
54
|
+
|
|
6
55
|
## [1.13.5](https://github.com/Inist-CNRS/ezs/compare/@ezs/basics@1.13.4...@ezs/basics@1.13.5) (2021-07-16)
|
|
7
56
|
|
|
8
57
|
**Note:** Version bump only for package @ezs/basics
|
package/README.md
CHANGED
|
@@ -52,14 +52,13 @@ Returns **[Buffer](https://nodejs.org/api/buffer.html)**
|
|
|
52
52
|
|
|
53
53
|
### CSVObject
|
|
54
54
|
|
|
55
|
-
- **See: CSVParse
|
|
56
|
-
**
|
|
57
|
-
|
|
58
55
|
Take an `Array` of arrays and transform rows into objects.
|
|
59
56
|
|
|
60
57
|
Each row (Array) is tranformed into an object where keys are the values of
|
|
61
58
|
the first row.
|
|
62
59
|
|
|
60
|
+
See [CSVParse](#csvparse).
|
|
61
|
+
|
|
63
62
|
Input:
|
|
64
63
|
|
|
65
64
|
```json
|
|
@@ -95,7 +94,7 @@ Input:
|
|
|
95
94
|
```json
|
|
96
95
|
[
|
|
97
96
|
["a", "a", "b", "b", "b"],
|
|
98
|
-
[1, 2, 3, 4, 5]
|
|
97
|
+
[1, 2, 3, 4, 5]
|
|
99
98
|
]
|
|
100
99
|
```
|
|
101
100
|
|
|
@@ -107,7 +106,7 @@ Output:
|
|
|
107
106
|
"a2": 2,
|
|
108
107
|
"b1": 3,
|
|
109
108
|
"b2": 4,
|
|
110
|
-
"b3": 5
|
|
109
|
+
"b3": 5
|
|
111
110
|
}]
|
|
112
111
|
```
|
|
113
112
|
|
|
@@ -119,12 +118,12 @@ Returns **([Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/
|
|
|
119
118
|
|
|
120
119
|
### CSVParse
|
|
121
120
|
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
**
|
|
121
|
+
Take `String` and parse it as CSV to generate arrays.
|
|
122
|
+
|
|
123
|
+
See:
|
|
126
124
|
|
|
127
|
-
|
|
125
|
+
- [CSVObject](#csvobject)
|
|
126
|
+
- <https://github.com/Inist-CNRS/node-csv-string>
|
|
128
127
|
|
|
129
128
|
Input:
|
|
130
129
|
|
|
@@ -152,14 +151,13 @@ Returns **[Array](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Gl
|
|
|
152
151
|
|
|
153
152
|
### CSVString
|
|
154
153
|
|
|
155
|
-
- **See: CSVObject
|
|
156
|
-
**
|
|
157
|
-
|
|
158
154
|
Take an array of objects and transform row into a string where each field is
|
|
159
155
|
separated with a character.
|
|
160
156
|
|
|
161
157
|
The resulting string is CSV-compliant.
|
|
162
158
|
|
|
159
|
+
See [CSVObject](#csvobject)
|
|
160
|
+
|
|
163
161
|
Input:
|
|
164
162
|
|
|
165
163
|
```json
|
|
@@ -244,11 +242,10 @@ Returns **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
244
242
|
|
|
245
243
|
### JSONParse
|
|
246
244
|
|
|
247
|
-
- **See: <https://github.com/dominictarr/JSONStream>
|
|
248
|
-
**
|
|
249
|
-
|
|
250
245
|
Parse a `String` to JSON and generate objects.
|
|
251
246
|
|
|
247
|
+
See <https://github.com/dominictarr/JSONStream>
|
|
248
|
+
|
|
252
249
|
##### Example 1: with separator
|
|
253
250
|
|
|
254
251
|
Input:
|
|
@@ -287,6 +284,7 @@ Output:
|
|
|
287
284
|
#### Parameters
|
|
288
285
|
|
|
289
286
|
- `separator` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** to split at every JSONPath found (optional, default `"*"`)
|
|
287
|
+
- `legacy` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** use legacy or newer parser (separator should be different) (optional, default `true`)
|
|
290
288
|
|
|
291
289
|
Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)**
|
|
292
290
|
|
|
@@ -337,11 +335,10 @@ Returns **[Number](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
337
335
|
|
|
338
336
|
### OBJFlatten
|
|
339
337
|
|
|
340
|
-
- **See: <https://www.npmjs.com/package/flat>
|
|
341
|
-
**
|
|
342
|
-
|
|
343
338
|
Flatten an `Object` with a path delimiting character.
|
|
344
339
|
|
|
340
|
+
See <https://www.npmjs.com/package/flat>
|
|
341
|
+
|
|
345
342
|
Input:
|
|
346
343
|
|
|
347
344
|
```json
|
|
@@ -574,11 +571,6 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
574
571
|
|
|
575
572
|
### URLParse
|
|
576
573
|
|
|
577
|
-
- **See: URLString
|
|
578
|
-
**
|
|
579
|
-
- **See: <https://nodejs.org/api/url.html>
|
|
580
|
-
**
|
|
581
|
-
|
|
582
574
|
Take an URL `String`, parse it and return `Object`.
|
|
583
575
|
|
|
584
576
|
Fields of the returned object:
|
|
@@ -597,6 +589,11 @@ Fields of the returned object:
|
|
|
597
589
|
|
|
598
590
|
URLString statement convert such an object to a string.
|
|
599
591
|
|
|
592
|
+
See:
|
|
593
|
+
|
|
594
|
+
- [URLString](#urlstring)
|
|
595
|
+
- <https://nodejs.org/api/url.html>
|
|
596
|
+
|
|
600
597
|
Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/Object)**
|
|
601
598
|
|
|
602
599
|
### URLStream
|
|
@@ -668,18 +665,14 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
668
665
|
|
|
669
666
|
### URLString
|
|
670
667
|
|
|
671
|
-
- **See: URLParse
|
|
672
|
-
**
|
|
673
|
-
|
|
674
668
|
Take an `Object` representing an URL and stringify it.
|
|
675
669
|
|
|
670
|
+
See [URLParse](#urlparse)
|
|
671
|
+
|
|
676
672
|
Returns **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)**
|
|
677
673
|
|
|
678
674
|
### XMLConvert
|
|
679
675
|
|
|
680
|
-
- **See: <https://www.npmjs.com/package/xml-mapping>
|
|
681
|
-
**
|
|
682
|
-
|
|
683
676
|
Convert each chunk as XML String to JSON Object
|
|
684
677
|
|
|
685
678
|
##### Example 1: XML to JSON (default parameters)
|
|
@@ -742,6 +735,8 @@ Output:
|
|
|
742
735
|
]
|
|
743
736
|
```
|
|
744
737
|
|
|
738
|
+
See <https://www.npmjs.com/package/xml-mapping>
|
|
739
|
+
|
|
745
740
|
#### Parameters
|
|
746
741
|
|
|
747
742
|
- `invert` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** change conversion (JSON to XML) (optional, default `false`)
|
|
@@ -751,9 +746,6 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
751
746
|
|
|
752
747
|
### XMLParse
|
|
753
748
|
|
|
754
|
-
- **See: <https://www.npmjs.com/package/xml-splitter>
|
|
755
|
-
**
|
|
756
|
-
|
|
757
749
|
Take `String` as XML input, parse it and split it in multi document at each path found
|
|
758
750
|
|
|
759
751
|
Input:
|
|
@@ -775,6 +767,8 @@ Output:
|
|
|
775
767
|
["x", "y"]
|
|
776
768
|
```
|
|
777
769
|
|
|
770
|
+
See <https://www.npmjs.com/package/xml-splitter>
|
|
771
|
+
|
|
778
772
|
#### Parameters
|
|
779
773
|
|
|
780
774
|
- `separator` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** choose a character for flatten keys (optional, default `"/"`)
|
|
@@ -783,9 +777,6 @@ Returns **[Object](https://developer.mozilla.org/docs/Web/JavaScript/Reference/G
|
|
|
783
777
|
|
|
784
778
|
### XMLString
|
|
785
779
|
|
|
786
|
-
- **See: XMLParse
|
|
787
|
-
**
|
|
788
|
-
|
|
789
780
|
Transform an `Object` into an XML string.
|
|
790
781
|
|
|
791
782
|
Input:
|
|
@@ -802,6 +793,8 @@ Output:
|
|
|
802
793
|
]
|
|
803
794
|
```
|
|
804
795
|
|
|
796
|
+
See [XMLParse](#xmlparse)
|
|
797
|
+
|
|
805
798
|
#### Parameters
|
|
806
799
|
|
|
807
800
|
- `rootElement` **[String](https://developer.mozilla.org/docs/Web/JavaScript/Reference/Global_Objects/String)** Root element name for the tag which starts and close the feed (optional, default `"items"`)
|
package/lib/csv-object.js
CHANGED
|
@@ -45,6 +45,8 @@ function CSVObject(data, feed) {
|
|
|
45
45
|
* Each row (Array) is tranformed into an object where keys are the values of
|
|
46
46
|
* the first row.
|
|
47
47
|
*
|
|
48
|
+
* See {@link CSVParse}.
|
|
49
|
+
*
|
|
48
50
|
* Input:
|
|
49
51
|
*
|
|
50
52
|
* ```json
|
|
@@ -79,7 +81,7 @@ function CSVObject(data, feed) {
|
|
|
79
81
|
* ```json
|
|
80
82
|
* [
|
|
81
83
|
* ["a", "a", "b", "b", "b"],
|
|
82
|
-
* [1, 2, 3, 4, 5]
|
|
84
|
+
* [1, 2, 3, 4, 5]
|
|
83
85
|
* ]
|
|
84
86
|
* ```
|
|
85
87
|
*
|
|
@@ -91,12 +93,11 @@ function CSVObject(data, feed) {
|
|
|
91
93
|
* "a2": 2,
|
|
92
94
|
* "b1": 3,
|
|
93
95
|
* "b2": 4,
|
|
94
|
-
* "b3": 5
|
|
96
|
+
* "b3": 5
|
|
95
97
|
* }]
|
|
96
98
|
* ```
|
|
97
99
|
*
|
|
98
100
|
* @name CSVObject
|
|
99
|
-
* @see CSVParse
|
|
100
101
|
* @param {undefined} none
|
|
101
102
|
* @returns {Object|Object[]}
|
|
102
103
|
*/
|
package/lib/csv-parse.js
CHANGED
|
@@ -30,7 +30,12 @@ function CSVParse(data, feed) {
|
|
|
30
30
|
}
|
|
31
31
|
}
|
|
32
32
|
/**
|
|
33
|
-
* Take `String` and parse it as CSV to generate arrays
|
|
33
|
+
* Take `String` and parse it as CSV to generate arrays.
|
|
34
|
+
*
|
|
35
|
+
* See:
|
|
36
|
+
*
|
|
37
|
+
* - {@link CSVObject}
|
|
38
|
+
* - {@link https://github.com/Inist-CNRS/node-csv-string}
|
|
34
39
|
*
|
|
35
40
|
* Input:
|
|
36
41
|
*
|
|
@@ -53,8 +58,6 @@ function CSVParse(data, feed) {
|
|
|
53
58
|
* @param {String} [separator=auto] to indicate the CSV separator
|
|
54
59
|
* @param {String} [quote=auto] to indicate the CSV quote.
|
|
55
60
|
* @returns {Array<String[]>}
|
|
56
|
-
* @see https://github.com/Inist-CNRS/node-csv-string
|
|
57
|
-
* @see CSVObject
|
|
58
61
|
*/
|
|
59
62
|
|
|
60
63
|
|
package/lib/csv-string.js
CHANGED
|
@@ -47,6 +47,8 @@ function CSVString(data, feed) {
|
|
|
47
47
|
*
|
|
48
48
|
* The resulting string is CSV-compliant.
|
|
49
49
|
*
|
|
50
|
+
* See {@link CSVObject}
|
|
51
|
+
*
|
|
50
52
|
* Input:
|
|
51
53
|
*
|
|
52
54
|
* ```json
|
|
@@ -75,7 +77,6 @@ function CSVString(data, feed) {
|
|
|
75
77
|
* @param {String} [separator=";"] to indicate the CSV separator
|
|
76
78
|
* @param {Boolean} [header=true] first line contains key name
|
|
77
79
|
* @returns {String}
|
|
78
|
-
* @see CSVObject
|
|
79
80
|
*/
|
|
80
81
|
|
|
81
82
|
|
package/lib/fetch.js
ADDED
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
|
|
3
|
+
Object.defineProperty(exports, "__esModule", {
|
|
4
|
+
value: true
|
|
5
|
+
});
|
|
6
|
+
exports.default = fetch;
|
|
7
|
+
|
|
8
|
+
var _crossFetch = _interopRequireDefault(require("cross-fetch"));
|
|
9
|
+
|
|
10
|
+
var _proxyFromEnv = require("proxy-from-env");
|
|
11
|
+
|
|
12
|
+
var _http = _interopRequireDefault(require("http"));
|
|
13
|
+
|
|
14
|
+
var _https = _interopRequireDefault(require("https"));
|
|
15
|
+
|
|
16
|
+
var _betterHttpsProxyAgent = require("better-https-proxy-agent");
|
|
17
|
+
|
|
18
|
+
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
|
|
19
|
+
|
|
20
|
+
const DefaultOptions = {
|
|
21
|
+
keepAlive: true,
|
|
22
|
+
timeout: 1000,
|
|
23
|
+
keepAliveMsecs: 500,
|
|
24
|
+
maxSockets: 200,
|
|
25
|
+
maxFreeSockets: 5,
|
|
26
|
+
maxCachedSessions: 500
|
|
27
|
+
};
|
|
28
|
+
|
|
29
|
+
const chooseAgent = (parsedURL, options) => {
|
|
30
|
+
const proxyurl = (0, _proxyFromEnv.getProxyForUrl)(parsedURL.href);
|
|
31
|
+
|
|
32
|
+
if (proxyurl) {
|
|
33
|
+
const proxyRequestOptions = new URL(proxyurl);
|
|
34
|
+
return new _betterHttpsProxyAgent.Agent(options, proxyRequestOptions);
|
|
35
|
+
}
|
|
36
|
+
|
|
37
|
+
if (parsedURL.protocol === 'https:') {
|
|
38
|
+
return new _https.default.Agent(options);
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
return new _http.default.Agent(options);
|
|
42
|
+
};
|
|
43
|
+
|
|
44
|
+
function fetch(url, options) {
|
|
45
|
+
const opts = options || {};
|
|
46
|
+
const {
|
|
47
|
+
keepAlive,
|
|
48
|
+
timeout,
|
|
49
|
+
keepAliveMsecs,
|
|
50
|
+
maxSockets,
|
|
51
|
+
maxFreeSockets,
|
|
52
|
+
maxCachedSessions
|
|
53
|
+
} = { ...options,
|
|
54
|
+
...DefaultOptions
|
|
55
|
+
};
|
|
56
|
+
let agent = chooseAgent(new URL(url), {
|
|
57
|
+
keepAlive,
|
|
58
|
+
timeout,
|
|
59
|
+
keepAliveMsecs,
|
|
60
|
+
maxSockets,
|
|
61
|
+
maxFreeSockets,
|
|
62
|
+
maxCachedSessions
|
|
63
|
+
});
|
|
64
|
+
opts.agent = agent;
|
|
65
|
+
|
|
66
|
+
if (opts.signal) {
|
|
67
|
+
opts.signal.addEventListener('abort', () => {
|
|
68
|
+
agent.destroy();
|
|
69
|
+
agent = null;
|
|
70
|
+
});
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
return (0, _crossFetch.default)(url, options);
|
|
74
|
+
}
|
package/lib/json-parse.js
CHANGED
|
@@ -7,29 +7,41 @@ exports.default = void 0;
|
|
|
7
7
|
|
|
8
8
|
var _JSONStream = _interopRequireDefault(require("JSONStream"));
|
|
9
9
|
|
|
10
|
+
var _stream = require("stream");
|
|
11
|
+
|
|
12
|
+
var _yajsonStream = _interopRequireDefault(require("yajson-stream"));
|
|
13
|
+
|
|
10
14
|
var _streamWrite = _interopRequireDefault(require("stream-write"));
|
|
11
15
|
|
|
12
16
|
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
|
|
13
17
|
|
|
14
18
|
function JSONParse(data, feed) {
|
|
15
|
-
if (
|
|
19
|
+
if (this.isFirst()) {
|
|
20
|
+
const legacy = this.getParam('legacy', true);
|
|
16
21
|
const separator = this.getParam('separator', '*');
|
|
17
|
-
|
|
18
|
-
|
|
22
|
+
|
|
23
|
+
if (legacy) {
|
|
24
|
+
this.input = _JSONStream.default.parse(separator);
|
|
25
|
+
this.whenFinish = feed.flow(this.input);
|
|
26
|
+
} else {
|
|
27
|
+
this.input = new _stream.PassThrough();
|
|
28
|
+
const stream = this.input.pipe(this.ezs.toBuffer()).pipe((0, _yajsonStream.default)(separator)).pipe(this.ezs((d, f) => f.send(d ? d.value : d)));
|
|
29
|
+
this.whenFinish = feed.flow(stream);
|
|
30
|
+
}
|
|
19
31
|
}
|
|
20
32
|
|
|
21
|
-
if (
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
this.handle.end();
|
|
25
|
-
process.nextTick(() => {
|
|
26
|
-
feed.close();
|
|
27
|
-
});
|
|
33
|
+
if (this.isLast()) {
|
|
34
|
+
this.whenFinish.finally(() => feed.close());
|
|
35
|
+
return this.input.end();
|
|
28
36
|
}
|
|
37
|
+
|
|
38
|
+
(0, _streamWrite.default)(this.input, data, () => feed.end());
|
|
29
39
|
}
|
|
30
40
|
/**
|
|
31
41
|
* Parse a `String` to JSON and generate objects.
|
|
32
42
|
*
|
|
43
|
+
* See {@link https://github.com/dominictarr/JSONStream}
|
|
44
|
+
*
|
|
33
45
|
* #### Example 1: with separator
|
|
34
46
|
*
|
|
35
47
|
* Input:
|
|
@@ -67,8 +79,8 @@ function JSONParse(data, feed) {
|
|
|
67
79
|
*
|
|
68
80
|
* @name JSONParse
|
|
69
81
|
* @param {String} [separator="*"] to split at every JSONPath found
|
|
82
|
+
* @param {String} [legacy=true] use legacy or newer parser (separator should be different)
|
|
70
83
|
* @returns {Object}
|
|
71
|
-
* @see https://github.com/dominictarr/JSONStream
|
|
72
84
|
*/
|
|
73
85
|
|
|
74
86
|
|
package/lib/obj-flatten.js
CHANGED
|
@@ -14,6 +14,8 @@ function _interopRequireWildcard(obj) { if (obj && obj.__esModule) { return obj;
|
|
|
14
14
|
/**
|
|
15
15
|
* Flatten an `Object` with a path delimiting character.
|
|
16
16
|
*
|
|
17
|
+
* See {@link https://www.npmjs.com/package/flat}
|
|
18
|
+
*
|
|
17
19
|
* Input:
|
|
18
20
|
*
|
|
19
21
|
* ```json
|
|
@@ -36,7 +38,6 @@ function _interopRequireWildcard(obj) { if (obj && obj.__esModule) { return obj;
|
|
|
36
38
|
* @param {String} [separator="/"] choose a character to flatten keys
|
|
37
39
|
* @param {Boolean} [safe=false] preserve arrays and their contents,
|
|
38
40
|
* @returns {Object}
|
|
39
|
-
* @see https://www.npmjs.com/package/flat
|
|
40
41
|
*/
|
|
41
42
|
function OBJFlatten(data, feed) {
|
|
42
43
|
if (this.isLast()) {
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
|
|
3
|
+
Object.defineProperty(exports, "__esModule", {
|
|
4
|
+
value: true
|
|
5
|
+
});
|
|
6
|
+
exports.default = void 0;
|
|
7
|
+
|
|
8
|
+
function Concept(data, feed) {
|
|
9
|
+
const obj = {};
|
|
10
|
+
Object.keys(data).forEach(key => {
|
|
11
|
+
const newkey = key.replace('skos$', '');
|
|
12
|
+
|
|
13
|
+
if (Array.isArray(data[key])) {
|
|
14
|
+
data[key].filter(x => x.xml$lang).forEach(item => {
|
|
15
|
+
const localkey = newkey.concat('@').concat(item.xml$lang);
|
|
16
|
+
obj[localkey] = item.$t;
|
|
17
|
+
});
|
|
18
|
+
} else if (data[key].rdf$resource && !obj[newkey]) {
|
|
19
|
+
obj[newkey] = data[key].rdf$resource;
|
|
20
|
+
} else if (data[key].rdf$resource && obj[newkey]) {
|
|
21
|
+
obj[newkey] = [obj[newkey], data[key].rdf$resource];
|
|
22
|
+
} else if (data[key].$t && data[key].xml$lang) {
|
|
23
|
+
const localkey = newkey.concat('@').concat(data[key].xml$lang);
|
|
24
|
+
obj[localkey] = data[key].$t;
|
|
25
|
+
} else if (data[key].$t && Array.isArray(obj[newkey])) {
|
|
26
|
+
obj[newkey].push(data[key].$t);
|
|
27
|
+
} else if (data[key].$t && obj[newkey]) {
|
|
28
|
+
obj[newkey] = [obj[newkey], data[key].$t];
|
|
29
|
+
} else if (data[key].$t && !obj[newkey]) {
|
|
30
|
+
obj[newkey] = data[key].$t;
|
|
31
|
+
} else if (typeof data[key] === 'object') {
|
|
32
|
+
obj[newkey] = (this.getIndex().toString(36) + Math.random().toString(36).substr(2, 5)).toUpperCase();
|
|
33
|
+
let counter = 0;
|
|
34
|
+
Object.keys(data[key]).forEach(key2 => {
|
|
35
|
+
if (typeof data[key][key2] === 'object') {
|
|
36
|
+
data[key][key2].rdf$about = obj[newkey];
|
|
37
|
+
Concept.call(this, data[key][key2], feed);
|
|
38
|
+
counter += 1;
|
|
39
|
+
}
|
|
40
|
+
});
|
|
41
|
+
|
|
42
|
+
if (counter === 0) {
|
|
43
|
+
delete obj[newkey];
|
|
44
|
+
}
|
|
45
|
+
} else {
|
|
46
|
+
obj[newkey] = data[key];
|
|
47
|
+
}
|
|
48
|
+
});
|
|
49
|
+
feed.write(obj);
|
|
50
|
+
}
|
|
51
|
+
|
|
52
|
+
function SKOSObject(data, feed) {
|
|
53
|
+
if (this.isLast()) {
|
|
54
|
+
feed.close();
|
|
55
|
+
} else {
|
|
56
|
+
Concept.call(this, data, feed);
|
|
57
|
+
feed.end();
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
/**
|
|
61
|
+
* Take `Object` generated by XMLMapping & SKOS data and
|
|
62
|
+
* create a new basic object with only keys & values
|
|
63
|
+
*
|
|
64
|
+
* @name SKOSObject
|
|
65
|
+
* @param {undefined} none
|
|
66
|
+
* @returns {Object}
|
|
67
|
+
*/
|
|
68
|
+
|
|
69
|
+
|
|
70
|
+
var _default = {
|
|
71
|
+
SKOSObject
|
|
72
|
+
};
|
|
73
|
+
exports.default = _default;
|
package/lib/url-connect.js
CHANGED
|
@@ -15,6 +15,8 @@ var _nodeAbortController = _interopRequireDefault(require("node-abort-controller
|
|
|
15
15
|
|
|
16
16
|
var _fetchWithProxy = _interopRequireDefault(require("fetch-with-proxy"));
|
|
17
17
|
|
|
18
|
+
var _parseHeaders = _interopRequireDefault(require("parse-headers"));
|
|
19
|
+
|
|
18
20
|
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
|
|
19
21
|
|
|
20
22
|
/**
|
|
@@ -41,6 +43,7 @@ function URLConnect(data, feed) {
|
|
|
41
43
|
|
|
42
44
|
if (this.isFirst()) {
|
|
43
45
|
const timeout = Number(this.getParam('timeout')) || 1000;
|
|
46
|
+
const headers = (0, _parseHeaders.default)([].concat(this.getParam('header')).filter(Boolean).join('\n'));
|
|
44
47
|
const controller = new _nodeAbortController.default();
|
|
45
48
|
this.input = ezs.createStream(ezs.objectMode());
|
|
46
49
|
this.whenReady = (0, _fetchWithProxy.default)(url, {
|
package/lib/url-fetch.js
CHANGED
|
@@ -15,6 +15,8 @@ var _nodeAbortController = _interopRequireDefault(require("node-abort-controller
|
|
|
15
15
|
|
|
16
16
|
var _fetchWithProxy = _interopRequireDefault(require("fetch-with-proxy"));
|
|
17
17
|
|
|
18
|
+
var _parseHeaders = _interopRequireDefault(require("parse-headers"));
|
|
19
|
+
|
|
18
20
|
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
|
|
19
21
|
|
|
20
22
|
/**
|
|
@@ -43,12 +45,14 @@ async function URLFetch(data, feed) {
|
|
|
43
45
|
const json = Boolean(this.getParam('json', false));
|
|
44
46
|
const noerror = Boolean(this.getParam('noerror', false));
|
|
45
47
|
const timeout = Number(this.getParam('timeout')) || 1000;
|
|
48
|
+
const headers = (0, _parseHeaders.default)([].concat(this.getParam('header')).filter(Boolean).join('\n'));
|
|
46
49
|
const mimetype = String(this.getParam('mimetype', 'application/json'));
|
|
47
50
|
const controller = new _nodeAbortController.default();
|
|
48
51
|
const key = Array.isArray(path) ? path.shift() : path;
|
|
49
52
|
const body = (0, _lodash.default)(data, key);
|
|
50
53
|
const parameters = {
|
|
51
54
|
timeout,
|
|
55
|
+
headers,
|
|
52
56
|
signal: controller.signal
|
|
53
57
|
};
|
|
54
58
|
|
package/lib/url-parse.js
CHANGED
|
@@ -46,10 +46,13 @@ function URLParse(data, feed) {
|
|
|
46
46
|
*
|
|
47
47
|
* URLString statement convert such an object to a string.
|
|
48
48
|
*
|
|
49
|
+
* See:
|
|
50
|
+
*
|
|
51
|
+
* - {@link URLString}
|
|
52
|
+
* - {@link https://nodejs.org/api/url.html}
|
|
53
|
+
*
|
|
49
54
|
* @name URLParse
|
|
50
55
|
* @returns {Object}
|
|
51
|
-
* @see URLString
|
|
52
|
-
* @see https://nodejs.org/api/url.html
|
|
53
56
|
*/
|
|
54
57
|
|
|
55
58
|
|
package/lib/url-stream.js
CHANGED
|
@@ -15,6 +15,8 @@ var _JSONStream = _interopRequireDefault(require("JSONStream"));
|
|
|
15
15
|
|
|
16
16
|
var _fetchWithProxy = _interopRequireDefault(require("fetch-with-proxy"));
|
|
17
17
|
|
|
18
|
+
var _parseHeaders = _interopRequireDefault(require("parse-headers"));
|
|
19
|
+
|
|
18
20
|
function _interopRequireDefault(obj) { return obj && obj.__esModule ? obj : { default: obj }; }
|
|
19
21
|
|
|
20
22
|
/**
|
|
@@ -91,6 +93,7 @@ async function URLStream(data, feed) {
|
|
|
91
93
|
const path = this.getParam('path', '*');
|
|
92
94
|
const noerror = Boolean(this.getParam('noerror', false));
|
|
93
95
|
const timeout = Number(this.getParam('timeout')) || 1000;
|
|
96
|
+
const headers = (0, _parseHeaders.default)([].concat(this.getParam('header')).filter(Boolean).join('\n'));
|
|
94
97
|
const cURL = new _url.URL(url || data);
|
|
95
98
|
const controller = new _nodeAbortController.default();
|
|
96
99
|
|
|
@@ -100,6 +103,7 @@ async function URLStream(data, feed) {
|
|
|
100
103
|
|
|
101
104
|
try {
|
|
102
105
|
const response = await (0, _fetchWithProxy.default)(cURL.href, {
|
|
106
|
+
headers,
|
|
103
107
|
timeout,
|
|
104
108
|
signal: controller.signal
|
|
105
109
|
});
|
package/lib/url-string.js
CHANGED
package/lib/utils.js
ADDED
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
|
|
3
|
+
Object.defineProperty(exports, "__esModule", {
|
|
4
|
+
value: true
|
|
5
|
+
});
|
|
6
|
+
exports.default = exports.writeTo = void 0;
|
|
7
|
+
|
|
8
|
+
const writeTo = function writeTo(stream, data, cb) {
|
|
9
|
+
if (!stream.write(data)) {
|
|
10
|
+
stream.once('drain', cb);
|
|
11
|
+
} else {
|
|
12
|
+
process.nextTick(cb);
|
|
13
|
+
}
|
|
14
|
+
};
|
|
15
|
+
|
|
16
|
+
exports.writeTo = writeTo;
|
|
17
|
+
var _default = writeTo;
|
|
18
|
+
exports.default = _default;
|
package/lib/xml-convert.js
CHANGED
|
@@ -81,12 +81,14 @@ function XMLConvert(data, feed) {
|
|
|
81
81
|
* "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<x a=\"1\"/>",
|
|
82
82
|
* "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n<x a=\"2\"/>",
|
|
83
83
|
* ]
|
|
84
|
+
* ```
|
|
85
|
+
*
|
|
86
|
+
* See {@link https://www.npmjs.com/package/xml-mapping}
|
|
84
87
|
*
|
|
85
88
|
* @name XMLConvert
|
|
86
89
|
* @param {String} [invert=false] change conversion (JSON to XML)
|
|
87
90
|
* @param {String} [prologue=false] add XML prologue
|
|
88
91
|
* @returns {Object}
|
|
89
|
-
* @see https://www.npmjs.com/package/xml-mapping
|
|
90
92
|
*/
|
|
91
93
|
|
|
92
94
|
|
package/lib/xml-parse.js
CHANGED
|
@@ -55,10 +55,10 @@ function XMLParse(data, feed) {
|
|
|
55
55
|
* ["x", "y"]
|
|
56
56
|
* ```
|
|
57
57
|
*
|
|
58
|
+
* See {@link https://www.npmjs.com/package/xml-splitter}
|
|
58
59
|
* @name XMLParse
|
|
59
60
|
* @param {String} [separator="/"] choose a character for flatten keys
|
|
60
61
|
* @returns {Object}
|
|
61
|
-
* @see https://www.npmjs.com/package/xml-splitter
|
|
62
62
|
*/
|
|
63
63
|
|
|
64
64
|
|
package/lib/xml-string.js
CHANGED
|
@@ -50,13 +50,13 @@ function XMLString(data, feed) {
|
|
|
50
50
|
* ]
|
|
51
51
|
* ```
|
|
52
52
|
*
|
|
53
|
+
* See {@link XMLParse}
|
|
53
54
|
* @name XMLString
|
|
54
55
|
* @param {String} [rootElement="items"] Root element name for the tag which starts and close the feed
|
|
55
56
|
* @param {String} [contentElement="item"] Content element name for the tag which starts and closes each item
|
|
56
57
|
* @param {String} [rootNamespace] Namespace for the root tag (xmlns=)
|
|
57
58
|
* @param {Boolean} [prologue=false] Add XML prologue `<?xml`
|
|
58
59
|
* @returns {String}
|
|
59
|
-
* @see XMLParse
|
|
60
60
|
*/
|
|
61
61
|
|
|
62
62
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ezs/basics",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.15.2",
|
|
4
4
|
"description": "Basics statements for EZS",
|
|
5
5
|
"directories": {
|
|
6
6
|
"test": "test"
|
|
@@ -19,12 +19,12 @@
|
|
|
19
19
|
},
|
|
20
20
|
"homepage": "https://github.com/Inist-CNRS/ezs/tree/master/packages/basics#readme",
|
|
21
21
|
"dependencies": {
|
|
22
|
-
"JSONStream": "
|
|
22
|
+
"JSONStream": "1.3.5",
|
|
23
23
|
"better-https-proxy-agent": "1.0.8",
|
|
24
|
-
"csv-string": "
|
|
25
|
-
"debug": "4.
|
|
24
|
+
"csv-string": "3.2.0",
|
|
25
|
+
"debug": "4.3.2",
|
|
26
26
|
"fetch-with-proxy": "3.0.1",
|
|
27
|
-
"flat": "
|
|
27
|
+
"flat": "5.0.2",
|
|
28
28
|
"lodash.escaperegexp": "4.1.2",
|
|
29
29
|
"lodash.get": "4.4.2",
|
|
30
30
|
"lodash.mapkeys": "4.6.0",
|
|
@@ -33,11 +33,13 @@
|
|
|
33
33
|
"lodash.zipobject": "4.1.3",
|
|
34
34
|
"micromatch": "4.0.2",
|
|
35
35
|
"node-abort-controller": "1.1.0",
|
|
36
|
+
"parse-headers": "2.0.4",
|
|
36
37
|
"stream-write": "1.0.1",
|
|
37
38
|
"tmp-filepath": "2.0.0",
|
|
38
39
|
"unzipper": "0.10.11",
|
|
39
|
-
"xml-mapping": "
|
|
40
|
-
"xml-splitter": "
|
|
40
|
+
"xml-mapping": "1.7.1",
|
|
41
|
+
"xml-splitter": "1.2.1",
|
|
42
|
+
"yajson-stream": "1.3.3"
|
|
41
43
|
},
|
|
42
44
|
"main": "./lib/index.js",
|
|
43
45
|
"scripts": {
|
|
@@ -52,11 +54,10 @@
|
|
|
52
54
|
"access": "public"
|
|
53
55
|
},
|
|
54
56
|
"devDependencies": {
|
|
55
|
-
"
|
|
56
|
-
"pako": "~1.0.11"
|
|
57
|
+
"pako": "1.0.11"
|
|
57
58
|
},
|
|
58
59
|
"peerDependencies": {
|
|
59
|
-
"@ezs/core": "
|
|
60
|
+
"@ezs/core": "1.27.0"
|
|
60
61
|
},
|
|
61
|
-
"gitHead": "
|
|
62
|
+
"gitHead": "244fa62654a8651678bc3d2d0ee6ed46da2782b5"
|
|
62
63
|
}
|