jsonl-audit-kit 0.0.0-stage → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,42 @@
1
+ # Changelog
2
+
3
+ All notable changes to `jsonl-audit-kit` are documented here.
4
+
5
+ The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
6
+ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ---
9
+
10
+ ## [Unreleased]
11
+
12
+ ## [0.1.1] — Initial Release
13
+
14
+ ### Added
15
+
16
+ - **Streaming JSONL auditing** via `auditFile()` and `auditStream()` using
17
+ `node:readline` — files are never fully loaded into memory.
18
+ - **Per-line validation** with 1-based line numbers preserved throughout.
19
+ - **Configurable validation rules:**
20
+ - `requireObject` — require every record to be a JSON object.
21
+ - `requireArray` — require every record to be a JSON array.
22
+ - `requiredKeys` — require specific top-level keys to be present.
23
+ - `reportBlankLines` — optionally treat blank lines as errors.
24
+ - `maxLineBytes` — flag lines that exceed a byte-length limit.
25
+ - **Bounded diagnostics:** `maxErrors` controls how many errors are stored
26
+ while the full error count is always tracked separately.
27
+ - **Diagnostic types:** `INVALID_JSON`, `BLANK_LINE`, `NOT_AN_OBJECT`,
28
+ `NOT_AN_ARRAY`, `MISSING_KEY`, `LINE_TOO_LONG`.
29
+ - **CLI (`jsonl-audit`)** with:
30
+ - File path and stdin (`-`) input.
31
+ - Human-readable and `--json` machine-readable output.
32
+ - Exit codes: 0 (valid), 1 (invalid), 2 (operational error).
33
+ - `--require-object`, `--require-array`, `--required`, `--report-blank-lines`,
34
+ `--max-errors`, `--max-line-bytes` flags.
35
+ - `--help` / `-h` and `--version` / `-v`.
36
+ - **TypeScript declarations** in `types/index.d.ts`.
37
+ - **Zero runtime dependencies** — only Node.js built-in modules used.
38
+ - **Node.js ≥ 20** support.
39
+ - Full automated test suite using `node:test` and `node:assert`.
40
+ - API documentation, design-decisions document, and ecosystem research notes.
41
+ - Example JSONL files and runnable usage script.
42
+ - GitHub Actions CI workflow for Node.js 20, 22, and 24.
@@ -0,0 +1,71 @@
1
+ # Contributing to jsonl-audit-kit
2
+
3
+ Thank you for considering a contribution!
4
+
5
+ ---
6
+
7
+ ## Getting started
8
+
9
+ ```bash
10
+ git clone https://github.com/<your-username>/jsonl-audit-kit.git
11
+ cd jsonl-audit-kit
12
+ npm install # only devDependencies — there are no runtime dependencies
13
+ ```
14
+
15
+ ## Running the tests
16
+
17
+ ```bash
18
+ npm test
19
+ ```
20
+
21
+ The test suite uses Node.js's built-in test runner (`node:test`) and
22
+ requires no additional installation.
23
+
24
+ ## Running the examples
25
+
26
+ ```bash
27
+ node examples/usage.js
28
+ ```
29
+
30
+ ## Code style
31
+
32
+ - Plain JavaScript (ESM) only — do not add TypeScript source.
33
+ - One concern per module: keep `validator.js`, `diagnostics.js`, and
34
+ `utils/validate-options.js` focused on their single responsibility.
35
+ - All public functions must have JSDoc comments.
36
+
37
+ ## Adding a feature
38
+
39
+ 1. Open an issue describing the feature before implementing it.
40
+ 2. Write a test first (in `test/`) that describes the desired behaviour.
41
+ 3. Implement the feature.
42
+ 4. Run the full test suite and ensure all tests pass.
43
+ 5. Update `docs/api.md` and the relevant section of `README.md`.
44
+ 6. Add an entry to `CHANGELOG.md` under `[Unreleased]`.
45
+ 7. Open a pull request.
46
+
47
+ ## Reporting a bug
48
+
49
+ Please open a GitHub issue with:
50
+
51
+ - Node.js version (`node --version`)
52
+ - npm version (`npm --version`)
53
+ - A minimal reproduction (a small JSONL file and the command you ran)
54
+ - The actual output
55
+ - The expected output
56
+
57
+ ## Commit message convention
58
+
59
+ Use [Conventional Commits](https://www.conventionalcommits.org/):
60
+
61
+ - `feat:` new feature
62
+ - `fix:` bug fix
63
+ - `docs:` documentation only
64
+ - `test:` test only
65
+ - `refactor:` code change without feature or fix
66
+ - `chore:` tooling, CI
67
+
68
+ ## License
69
+
70
+ By contributing you agree that your contributions will be licensed under
71
+ the MIT License.
package/LICENSE ADDED
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2024 jsonl-audit-kit contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,3 +1,365 @@
1
- # Temporary Holding Version
1
+ # jsonl-audit-kit
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Audit JSONL (JSON Lines / NDJSON) files and identify invalid records before they enter your application, database, AI pipeline, or data-processing workflow.
4
+
5
+ ---
6
+
7
+ ## The problem
8
+
9
+ JSONL files are widely used for logs, datasets, AI training data, and event streams. Each line must be valid JSON, but files produced by flaky exporters, truncated downloads, or manual edits often contain broken records:
10
+
11
+ ```
12
+ {"id":1,"event":"created"}
13
+ {"id":2,"event":
14
+ {"id":3,"event":"deleted"}
15
+ ```
16
+
17
+ The second record is incomplete. Standard JSON parsers either crash on the first error or silently discard bad records. Neither behaviour tells you *which* lines are broken or *why*.
18
+
19
+ `jsonl-audit-kit` scans every line, collects per-line diagnostics, and exits with the right code for CI pipelines — without loading the entire file into memory.
20
+
21
+ ---
22
+
23
+ ## Features
24
+
25
+ - **Streaming** — reads files line-by-line using `node:readline`; works on files of any size
26
+ - **Per-line diagnostics** — precise line numbers and human-readable error messages
27
+ - **Configurable validation** — require objects, require specific keys, limit line size
28
+ - **Bounded error collection** — store up to N errors; always count all of them
29
+ - **Developer-friendly CLI** — human and `--json` machine-readable output
30
+ - **CI-friendly exit codes** — 0 valid, 1 invalid, 2 operational error
31
+ - **TypeScript declarations** included
32
+ - **Zero runtime dependencies** — only Node.js built-in modules
33
+
34
+ ---
35
+
36
+ ## Requirements
37
+
38
+ - Node.js **≥ 20.0.0**
39
+
40
+ ---
41
+
42
+ ## Installation
43
+
44
+ ```bash
45
+ npm install jsonl-audit-kit
46
+ ```
47
+
48
+ CLI only (global):
49
+
50
+ ```bash
51
+ npm install -g jsonl-audit-kit
52
+ ```
53
+
54
+ ---
55
+
56
+ ## Quick start
57
+
58
+ ### JavaScript API
59
+
60
+ ```js
61
+ import { auditFile } from 'jsonl-audit-kit';
62
+
63
+ const result = await auditFile('./events.jsonl', {
64
+ requireObject: true,
65
+ requiredKeys: ['id', 'event'],
66
+ });
67
+
68
+ if (!result.valid) {
69
+ console.error(`Found ${result.errorCount} error(s) in ${result.totalLines} line(s)`);
70
+
71
+ for (const err of result.errors) {
72
+ console.error(` Line ${err.line}: [${err.type}] ${err.message}`);
73
+ }
74
+
75
+ process.exit(1);
76
+ }
77
+
78
+ console.log('File is valid!');
79
+ ```
80
+
81
+ ### CLI
82
+
83
+ ```bash
84
+ # Basic audit
85
+ jsonl-audit events.jsonl
86
+
87
+ # Require objects with specific keys
88
+ jsonl-audit events.jsonl --require-object --required id,event
89
+
90
+ # Machine-readable output
91
+ jsonl-audit events.jsonl --json
92
+
93
+ # Read from stdin
94
+ cat events.jsonl | jsonl-audit -
95
+
96
+ # Limit stored errors (scanning always continues)
97
+ jsonl-audit events.jsonl --max-errors 50
98
+ ```
99
+
100
+ ---
101
+
102
+ ## JavaScript API
103
+
104
+ ### `auditFile(filePath, options?)`
105
+
106
+ Audit a JSONL file at the given path.
107
+
108
+ ```js
109
+ import { auditFile } from 'jsonl-audit-kit';
110
+
111
+ const result = await auditFile('./data.jsonl', {
112
+ requireObject: true,
113
+ requiredKeys: ['id', 'timestamp'],
114
+ maxErrors: 100,
115
+ });
116
+ ```
117
+
118
+ ### `auditStream(stream, options?)`
119
+
120
+ Audit any readable Node.js stream. Use this for `process.stdin`, HTTP responses, or custom sources.
121
+
122
+ ```js
123
+ import { auditStream } from 'jsonl-audit-kit';
124
+ import { createReadStream } from 'node:fs';
125
+
126
+ const stream = createReadStream('./data.jsonl', { encoding: 'utf8' });
127
+ const result = await auditStream(stream, { requireObject: true });
128
+ ```
129
+
130
+ ---
131
+
132
+ ## Configuration options
133
+
134
+ | Option | Type | Default | Description |
135
+ |--------|------|---------|-------------|
136
+ | `requireObject` | `boolean` | `false` | Every record must be a JSON object `{...}` |
137
+ | `requireArray` | `boolean` | `false` | Every record must be a JSON array `[...]` |
138
+ | `requiredKeys` | `string[]` | `[]` | Keys that must exist in every object record |
139
+ | `reportBlankLines` | `boolean` | `false` | Treat blank lines as errors |
140
+ | `maxErrors` | `number` | `100` | Maximum stored diagnostics |
141
+ | `maxLineBytes` | `number \| null` | `null` | Maximum line size in bytes |
142
+
143
+ `requireObject` and `requireArray` cannot both be `true`.
144
+
145
+ ---
146
+
147
+ ## Result object
148
+
149
+ ```js
150
+ {
151
+ valid: true, // false if any errors were found
152
+ totalLines: 1000, // total lines read (including blank lines)
153
+ blankLines: 5, // blank / whitespace-only lines
154
+ errorCount: 0, // total errors found (may exceed errors.length)
155
+ errors: [], // stored diagnostics (at most maxErrors)
156
+ truncated: false // true when errorCount > errors.length
157
+ }
158
+ ```
159
+
160
+ ---
161
+
162
+ ## Diagnostic format
163
+
164
+ Each diagnostic in `result.errors` looks like:
165
+
166
+ ```js
167
+ // Invalid JSON
168
+ { line: 42, type: 'INVALID_JSON', message: 'JSON parse error: Unexpected end of JSON input' }
169
+
170
+ // Not an object
171
+ { line: 7, type: 'NOT_AN_OBJECT', message: 'Expected a JSON object but got array' }
172
+
173
+ // Missing key
174
+ { line: 15, type: 'MISSING_KEY', message: 'Required key "id" is missing', key: 'id' }
175
+
176
+ // Line too long
177
+ { line: 3, type: 'LINE_TOO_LONG', message: 'Line is 2048 bytes, exceeds limit of 1024 bytes', length: 2048, max: 1024 }
178
+
179
+ // Blank line (only when reportBlankLines: true)
180
+ { line: 10, type: 'BLANK_LINE', message: 'Line is blank' }
181
+ ```
182
+
183
+ ### Diagnostic types
184
+
185
+ | Type | Meaning |
186
+ |------|---------|
187
+ | `INVALID_JSON` | Line is not valid JSON |
188
+ | `BLANK_LINE` | Line is blank (only when `reportBlankLines: true`) |
189
+ | `NOT_AN_OBJECT` | Parsed value is not a plain object (when `requireObject: true`) |
190
+ | `NOT_AN_ARRAY` | Parsed value is not an array (when `requireArray: true`) |
191
+ | `MISSING_KEY` | A required key is absent |
192
+ | `LINE_TOO_LONG` | Line exceeds `maxLineBytes` |
193
+
194
+ You can import `DiagnosticType` to avoid hardcoding strings:
195
+
196
+ ```js
197
+ import { auditFile, DiagnosticType } from 'jsonl-audit-kit';
198
+
199
+ const result = await auditFile('./data.jsonl');
200
+ const parseErrors = result.errors.filter(e => e.type === DiagnosticType.INVALID_JSON);
201
+ ```
202
+
203
+ ---
204
+
205
+ ## CLI usage
206
+
207
+ ```
208
+ jsonl-audit <file|-> [options]
209
+ ```
210
+
211
+ | Flag | Description |
212
+ |------|-------------|
213
+ | `--require-object` | Require every record to be a JSON object |
214
+ | `--require-array` | Require every record to be a JSON array |
215
+ | `--required k1,k2` | Required top-level key names (comma-separated) |
216
+ | `--report-blank-lines` | Treat blank lines as errors |
217
+ | `--max-errors n` | Maximum stored errors (default: 100) |
218
+ | `--max-line-bytes n` | Flag lines longer than n bytes |
219
+ | `--json` | Machine-readable JSON output |
220
+ | `--version` / `-v` | Print version |
221
+ | `--help` / `-h` | Show help |
222
+
223
+ ### Exit codes
224
+
225
+ | Code | Meaning |
226
+ |------|---------|
227
+ | `0` | All records valid |
228
+ | `1` | One or more records invalid |
229
+ | `2` | Operational error (file not found, bad arguments) |
230
+
231
+ ### Example output
232
+
233
+ **Human-readable (valid):**
234
+ ```
235
+ ✓ events.jsonl: valid — 1000 line(s) checked, 0 errors
236
+ ```
237
+
238
+ **Human-readable (invalid):**
239
+ ```
240
+ ✗ events.jsonl: INVALID — 3 error(s) in 1000 line(s)
241
+ Line 42: [INVALID_JSON] JSON parse error: Unexpected end of JSON input
242
+ Line 87: [MISSING_KEY] Required key "id" is missing [key: id]
243
+ Line 99: [NOT_AN_OBJECT] Expected a JSON object but got array
244
+ ```
245
+
246
+ **JSON output (`--json`):**
247
+ ```json
248
+ {
249
+ "source": "events.jsonl",
250
+ "valid": false,
251
+ "totalLines": 1000,
252
+ "blankLines": 0,
253
+ "errorCount": 3,
254
+ "errors": [
255
+ {
256
+ "line": 42,
257
+ "type": "INVALID_JSON",
258
+ "message": "JSON parse error: Unexpected end of JSON input"
259
+ }
260
+ ],
261
+ "truncated": false
262
+ }
263
+ ```
264
+
265
+ ---
266
+
267
+ ## Error handling
268
+
269
+ ```js
270
+ import { auditFile } from 'jsonl-audit-kit';
271
+
272
+ try {
273
+ const result = await auditFile('./data.jsonl', { requireObject: true });
274
+ // ...
275
+ } catch (err) {
276
+ if (err.code === 'ENOENT') {
277
+ console.error('File not found');
278
+ } else if (err.code === 'EACCES') {
279
+ console.error('Permission denied');
280
+ } else if (err.name === 'TypeError') {
281
+ console.error('Invalid options:', err.message);
282
+ } else {
283
+ console.error('Unexpected error:', err.message);
284
+ }
285
+ }
286
+ ```
287
+
288
+ ---
289
+
290
+ ## TypeScript
291
+
292
+ TypeScript declarations are included:
293
+
294
+ ```ts
295
+ import { auditFile, AuditResult, AuditOptions, Diagnostic, DiagnosticType } from 'jsonl-audit-kit';
296
+
297
+ const options: AuditOptions = {
298
+ requireObject: true,
299
+ requiredKeys: ['id', 'event'],
300
+ };
301
+
302
+ const result: AuditResult = await auditFile('./data.jsonl', options);
303
+ ```
304
+
305
+ ---
306
+
307
+ ## Node.js compatibility
308
+
309
+ | Node.js version | Supported |
310
+ |-----------------|-----------|
311
+ | 24.x | ✓ |
312
+ | 22.x | ✓ |
313
+ | 20.x | ✓ |
314
+ | 18.x and earlier | ✗ |
315
+
316
+ > **Note:** Node.js 18 is end-of-life. The built-in test runner used by this package became stable in Node.js 20.
317
+
318
+ ---
319
+
320
+ ## Limitations
321
+
322
+ - **Line buffer:** `node:readline` buffers one line at a time. A single line of several megabytes will occupy that memory. There is no per-line size enforcement by default; use `maxLineBytes` to guard against this.
323
+ - **UTF-8 only:** The streaming reader assumes UTF-8 input. Other encodings (Latin-1, UTF-16) must be transcoded before auditing.
324
+ - **No JSON Schema validation:** `requiredKeys` checks for key presence only. For full structural/type validation, pair `jsonl-audit-kit` with a schema validator such as [AJV](https://ajv.js.org/).
325
+ - **CommonJS consumers** must use dynamic `import()` since the package is ESM-only.
326
+
327
+ ---
328
+
329
+ ## Development setup
330
+
331
+ ```bash
332
+ git clone https://github.com/<your-username>/jsonl-audit-kit.git
333
+ cd jsonl-audit-kit
334
+ npm install
335
+ ```
336
+
337
+ ### Run tests
338
+
339
+ ```bash
340
+ npm test
341
+ ```
342
+
343
+ ### Syntax check
344
+
345
+ ```bash
346
+ npm run check
347
+ ```
348
+
349
+ ### Run examples
350
+
351
+ ```bash
352
+ node examples/usage.js
353
+ ```
354
+
355
+ ### Inspect package contents
356
+
357
+ ```bash
358
+ npm pack --dry-run
359
+ ```
360
+
361
+ ---
362
+
363
+ ## License
364
+
365
+ [MIT](./LICENSE)