n3.js-messages 0.1.0 → 0.1.2

package/README.md

@@ -1,5 +1,10 @@
 # n3.js-messages
 
+[![CI](https://github.com/pietercolpaert/n3.js-messages/actions/workflows/ci.yml/badge.svg)](https://github.com/pietercolpaert/n3.js-messages/actions/workflows/ci.yml)
+[![Coverage Status](https://coveralls.io/repos/github/pietercolpaert/n3.js-messages/badge.svg?branch=main)](https://coveralls.io/github/pietercolpaert/n3.js-messages?branch=main)
+[![npm version](https://img.shields.io/npm/v/n3.js-messages.svg)](https://www.npmjs.com/package/n3.js-messages)
+[![license](https://img.shields.io/npm/l/n3.js-messages.svg)](https://github.com/pietercolpaert/n3.js-messages/blob/main/LICENSE)
+
 [RDF Messages](https://w3c-cg.github.io/rsp/spec/messages) for [N3.js](https://github.com/rdfjs/N3.js):
 parse, stream and write message-delimited **Turtle, TriG, N-Triples and N-Quads**, with ordinary RDF/JS `Quad` objects.
 
@@ -22,7 +27,7 @@ it is **not a W3C Recommendation**, and may still change. This package implement
 npm install n3 n3.js-messages
 ```
 
-Requires Node.js ≥ 18 and `n3` ≥ 2.0.0 < 3. The package works with `import` and `require`, and ships TypeScript declarations.
+Requires Node.js ≥ 18 and `n3` ≥ 2.13.7 < 3. The package works with `import` and `require`, and ships TypeScript declarations.
 
 ## Parsing
 
@@ -98,7 +103,7 @@ await writer.end();
 const text = writeMessages(messages, { format: 'N-Quads' });  // synchronous, returns a string
 ```
 
-- Writes `VERSION "1.2-messages"` (`@version "1.2-messages" .` in Turtle/TriG) first, and a delimiter after each message:
+- Writes `VERSION "1.2-messages"` (`@version "1.2-messages".` in Turtle/TriG) first, and a delimiter after each message:
   `MESSAGE` in N-Triples and N-Quads, `@message .` in Turtle and TriG. A parser gets a message when it is written, and a trailing empty
   message survives a round trip.
 - Each message starts from a clean serializer state, so consecutive messages in the same named graph are valid. Prefixes (and `baseIRI` with
@@ -136,17 +141,20 @@ The 12 skipped cases do not apply to RDF syntax parsers and serializers: 7 test
 Jelly gRPC), and 5 are about NDJSON-LD, a different serialization. The run prints each skipped case with its reason.
 `npm run test:spec:earl` writes an EARL report.
 
-Beyond the suite there are 334 unit, regression, property and differential tests (100% statement, branch, function and line coverage),
+Beyond the suite there are 312 unit, regression, property and differential tests (100% statement, branch, function and line coverage),
 including every possible two-chunk split of the fixtures, deterministic randomized chunking, per-byte streaming, and the regressions of
 the review of N3.js [pull request 586](https://github.com/rdfjs/N3.js/pull/586), which proposed this feature for N3.js itself.
 
 ## Relationship to N3.js
 
-- `n3` is a **peer dependency** (`>=2.0.0 <3`), so that an application has one N3.js installation. It is tested against n3 2.0.0 (minimum) and
+- `n3` is a **peer dependency** (`>=2.13.7 <3`), so that an application has one N3.js installation. It is tested against n3 2.13.7 (minimum) and
   the latest 2.x, and against N3.js `main` in a scheduled, non-blocking job.
 - This is not a fork. The N3.js maintainers asked for RDF Messages to live in a separate package until the specification matures
   (they proposed it in the discussion of pull request 586, which first added the feature to N3.js itself); this package is that package.
-  Where N3.js lacks a hook, [IMPLEMENTATION-NOTES.md](IMPLEMENTATION-NOTES.md) says which, and how it is bridged with public API only.
+- N3.js parses the delimiters itself: the parser is created with `directives: ['message']`, and reports `MESSAGE` and `@message .`
+  through `onDirective`, in order with the quads. The writer is a small subclass of `N3.Writer` that announces the log with the
+  `version` option and ends every message with `_endStatement()` before it writes the delimiter.
+  Blank node scoping per message is done by this package.
 
 ## Performance
 
@@ -156,11 +164,11 @@ ratio-based guardrails against each other and compares with `perf/baseline.json`
 
 | | plain N3.js | n3.js-messages |
 | --- | --- | --- |
-| parse 100k quads, one message | 688k | 719k (`parseMessages`), 784k (stream, 64 KiB chunks) |
-| parse 10k messages × 5 quads | – | 588k (`parseMessages`), 813k (stream) |
-| stream, 16-byte chunks (10k quads) | – | 464k |
-| write 100k quads in one message (N-Quads) | – | 1.79M |
-| write 10k messages (N-Quads) | – | 1.18M (235k messages/s) |
+| parse 100k quads, one message | 649k | 691k (`parseMessages`), 735k (stream, 64 KiB chunks) |
+| parse 10k messages × 5 quads | – | 564k (`parseMessages`), 655k (stream) |
+| stream, 16-byte chunks (10k quads) | – | 324k |
+| write 100k quads in one message (N-Quads) | – | 1.34M |
+| write 10k messages (N-Quads) | – | 1.10M (221k messages/s) |
 
 Streaming 100k messages (17 MB) through `MessageStreamParser` grows the heap by 0.12 MB (after gc).
 
@@ -173,29 +181,30 @@ through both (`npm run perf`, then `npm run perf:compare`). Throughput in quads
 
 | Workload | n3.js-messages | rdf-parser-ts / rdf-writer-ts | Speed of n3.js-messages |
 | --- | --- | --- | --- |
-| Parse a string, one message, 100k quads | 719k | 696k | 1.03× |
-| Parse a string, 10k messages × 5 quads | 588k | 691k | 0.85× |
-| Parse a string, 10 messages × 10k quads | 680k | 745k | 0.91× |
-| Stream, 64 KiB chunks, 100k quads | 784k | 680k | 1.15× |
-| Stream, 64 KiB chunks, 10k messages | 813k | 549k | 1.48× |
-| Stream, 16-byte chunks, 10k quads | 464k | 254k | 1.83× |
-| Stream, 1-character chunks, 1k quads | 117k | 76k | 1.53× |
-| Write one message, 100k quads, N-Quads | 1,793k | 1,678k | 1.07× |
-| Write one message, 100k quads, TriG | 1,667k | 1,636k | 1.02× |
-| Write 10k messages, N-Quads | 1,177k | 1,540k | 0.76× |
-| Write 10k messages, TriG | 1,039k | 1,374k | 0.76× |
-
-- **Streaming is where this package is ahead**: 1.15× to 1.8×, most for tiny chunks. Our scanner and the N3.js lexer work incrementally; by its
+| Parse a string, one message, 100k quads | 691k | 551k | 1.25× |
+| Parse a string, 10k messages × 5 quads | 564k | 546k | 1.03× |
+| Parse a string, 10 messages × 10k quads | 732k | 596k | 1.23× |
+| Stream, 64 KiB chunks, 100k quads | 735k | 638k | 1.15× |
+| Stream, 64 KiB chunks, 10k messages | 655k | 511k | 1.28× |
+| Stream, 16-byte chunks, 10k quads | 324k | 198k | 1.64× |
+| Stream, 1-character chunks, 1k quads | 79k | 61k | 1.29× |
+| Write one message, 100k quads, N-Quads | 1,342k | 1,352k | 0.99× |
+| Write one message, 100k quads, TriG | 1,079k | 1,281k | 0.84× |
+| Write 10k messages, N-Quads | 1,103k | 689k | 1.60× |
+| Write 10k messages, TriG | 829k | 595k | 1.39× |
+
+- **Streaming is where this package is furthest ahead**: 1.15× to 1.64×, most for tiny chunks. The N3.js lexer works incrementally; by its
   documentation, rdf-parser-ts runs its parser over every complete statement prefix of the buffered input. I did not profile it to confirm the cause.
-- **Parsing a whole string is faster with rdf-parser-ts** (0.85× to 0.91× for many messages or large ones, parity for one message): `parseMessages` is
-  N3.js plus a scanner, so it is bounded by the speed of N3.js, which rdf-parser-ts also beats on ordinary documents (by 1.2×).
-- **Writing**: on par for one large message; rdf-writer-ts is faster for many small messages (0.76×), since this package creates an N3.js writer per message.
+- **Parsing a whole string** is faster here too (1.03× for many small messages, 1.23× to 1.25× for large ones): `parseMessages` is
+  N3.js with an extra directive, and N3.js itself is also ahead of rdf-parser-ts on an ordinary document (1.07×).
+- **Writing**: on par for one large N-Quads message (0.99×); rdf-writer-ts is faster for one large TriG message (0.84×); this package is faster for
+  many small messages (1.4× to 1.6×).
 - **Memory**: both stay flat while streaming 100k messages (17 MB): the heap grows by 0.12 MB here and
   0.08 MB with rdf-parser-ts.
 
 Beyond speed, the choice is mostly about the ecosystem. Choose n3.js-messages if you already use N3.js (one parser and one set of syntax quirks, `n3` as a
-shared peer dependency, Node.js 18+, messages as plain `Quad[]`, and a writer that starts every message from a clean state). Choose rdf-parser-ts and rdf-writer-ts if you do not need N3.js, or if parsing whole
-documents and writing many tiny messages are your bottleneck (they need Node.js 22+ and use `rdf-data-factory` by default).
+shared peer dependency, Node.js 18+, messages as plain `Quad[]`, and a writer that starts every message from a clean state). Choose rdf-parser-ts and rdf-writer-ts if you do not need N3.js, or if writing
+large TriG messages is your bottleneck (they need Node.js 22+ and use `rdf-data-factory` by default).
 These are one machine's numbers from a single process: run `npm run perf` on yours; method and caveats are in [perf/README.md](perf/README.md).
 
 ## Development

package/lib/MessageParserCore.js

@@ -6,16 +6,11 @@ Object.defineProperty(exports, "__esModule", {
 exports.default = void 0;
 var _events = require("events");
 var _n = require("n3");
-var _MessageScanner = require("./MessageScanner.js");
 var _formats = require("./formats.js");
 // **MessageParserCore** is the push-based engine behind all message parsers.
-// It feeds scanned text to the public N3.js parser and groups the resulting
-// quads into messages.
+// It feeds text to the N3.js parser, which recognizes the message delimiters
+// through its `directives` option, and groups the resulting quads into messages.
 
-// Unrecognizable by construction: a document cannot contain it by accident
-function createSentinel() {
-  return `rdf-messages-${Math.random().toString(36).slice(2)}${Math.random().toString(36).slice(2)}`;
-}
 class MessageParserCore {
   // `sink` receives `message(quads, index)`, `end(prefixes)`, `error(error)`
   // and optionally `quad(quad, messageIndex)`, `prefix(prefix, term)`, `version(label)`
@@ -38,40 +33,47 @@ class MessageParserCore {
       format: format.name,
       factory: this._factory,
       // Version labels are validated here, because N3.js does not know `-messages` labels
-      parseUnsupportedVersions: true
+      parseUnsupportedVersions: true,
+      // N3.js reports `MESSAGE` and `@message .` between statements through `onDirective`,
+      // in order with the quads, and rejects them anywhere else
+      directives: ['message']
     };
     if (options.baseIRI) n3Options.baseIRI = options.baseIRI;
     if (options.version) {
       this._checkVersion(options.version, true);
       n3Options.version = (0, _formats.baseVersion)(options.version);
     }
-    const sentinel = createSentinel();
-    this._sentinel = sentinel;
-    this._scanner = format.lineMode ? new _MessageScanner.LineScanner(sentinel) : new _MessageScanner.TurtleScanner(sentinel);
     this._input = new _events.EventEmitter();
-    const callbacks = {
+    this._lineMode = format.lineMode;
+    // The line of the current token, and of the last delimiter
+    this._line = 0;
+    this._delimiterLine = 0;
+    new _n.Parser(n3Options).parse(this._input, {
       onQuad: (error, quad, prefixes) => this._onQuad(error, quad, prefixes),
       onPrefix: (prefix, term) => this._failed || sink.prefix && sink.prefix(prefix, term),
-      onVersion: label => this._onVersion(label)
-    };
-    if (format.lineMode) callbacks.onComment = comment => this._onComment(comment);
-    new _n.Parser(n3Options).parse(this._input, callbacks);
+      onVersion: label => this._failed || this._announce(label),
+      onDirective: () => this._failed || this._delimiter(),
+      onToken: token => this._onToken(token)
+    });
   }
 
   // ### `write` accepts the next chunk of text
   write(text) {
-    if (!this._failed && !this._finished && text.length > 0) this._feed(this._scanner.push(text));
+    if (!this._failed && !this._finished && text.length > 0) this._input.emit('data', text);
   }
 
   // ### `end` signals the end of the input
   end() {
-    if (!this._failed && !this._finished) {
-      this._feed(this._scanner.end());
-      this._input.emit('end');
-    }
+    if (!this._failed && !this._finished) this._input.emit('end');
   }
-  _feed(text) {
-    if (text.length > 0) this._input.emit('data', text);
+
+  // Keeps track of lines; in N-Triples and N-Quads, a delimiter must be on its own line
+  _onToken(token) {
+    const line = token.line,
+      delimiter = token.type === 'MESSAGE';
+    if (this._lineMode && !this._failed && (delimiter ? line === this._line : line === this._delimiterLine && token.type !== 'eof')) this._fail(new Error(`Unexpected "MESSAGE" on line ${line}.`));
+    if (delimiter) this._delimiterLine = line;
+    this._line = line;
   }
   _onQuad(error, quad, prefixes) {
     if (this._failed || this._finished) return;
@@ -89,17 +91,6 @@ class MessageParserCore {
     this._current.push(quad);
     if (this._sink.quad) this._sink.quad(quad, this._index);
   }
-  _onVersion(label) {
-    if (this._failed) return;
-    if (label === this._sentinel) this._delimiter(this._scanner.delimiterLines.shift());else this._announce(label);
-  }
-
-  // Line mode reports delimiters and versions as comments
-  _onComment(comment) {
-    if (this._failed || !comment.startsWith(this._sentinel)) return;
-    const rest = comment.substring(this._sentinel.length);
-    if (rest[0] === 'M') this._delimiter(Number(rest.substring(1)));else this._announce(rest.substring(1));
-  }
   _announce(label) {
     const error = this._checkVersion(label);
     if (error) return this._fail(error);
@@ -117,9 +108,9 @@ class MessageParserCore {
     if (throwing) throw error;
     return error;
   }
-  _delimiter(line) {
+  _delimiter() {
     if (!this._messages) {
-      return this._fail(new Error(`Unexpected "MESSAGE" on line ${line}: the document did not announce ` + 'a version with the "-messages" suffix.'));
+      return this._fail(new Error(`Unexpected "MESSAGE" on line ${this._line}: the document did not announce ` + 'a version with the "-messages" suffix.'));
     }
     this._emitMessage();
   }

package/lib/MessageWriter.js

@@ -9,12 +9,20 @@ var _n = require("n3");
 var _formats = require("./formats.js");
 // **MessageWriter** writes RDF Message Logs (Turtle, TriG, N-Triples and N-Quads).
 //
-// Every message is written by a fresh N3.js writer, so no serialization state
-// (current graph, subject, predicate) leaks from one message into the next.
-// Every message is followed by a delimiter (`MESSAGE` in line formats, `@message .` in Turtle and TriG), so that consumers receive
-// a message as soon as it is written, and so that trailing empty messages survive
-// a round trip (a delimiter before the end of the input does not create a message).
+// One N3.js writer writes the whole log: its `version` option announces the log,
+// and after each message, `_endStatement` finishes the pending statement (and graph block)
+// before a delimiter is written (`MESSAGE` in line formats, `@message .` in Turtle and TriG).
+// That way, consumers receive a message as soon as it is written, no serialization state
+// (current graph, subject, predicate) leaks from one message into the next, and trailing
+// empty messages survive a round trip (a delimiter before the end of the input does not create a message).
 
+class LogWriter extends _n.Writer {
+  // ### `addDelimiter` ends the current message
+  addDelimiter() {
+    this._endStatement();
+    this._write(this._lineMode ? 'MESSAGE\n' : '@message .\n');
+  }
+}
 class MessageWriter {
   // `outputStream` is optional: without it, `end` returns the output as a string
   constructor(outputStream, options) {
@@ -30,15 +38,15 @@ class MessageWriter {
     this._pending = 0;
     this._waiting = [];
     this._error = null;
-    this._muted = false;
     this._started = false;
     this._ended = false;
     this._declared = Object.assign({}, options.prefixes);
     this._newPrefixes = {};
-    // The N3.js writers write to this sink, which they never end
+    // The N3.js writer writes to this sink, which it never ends
     this._sink = {
       write: (chunk, encoding, done) => this._write(chunk, done)
     };
+    this._writer = null;
   }
 
   // ### `addMessage` writes the quads of one message, followed by a delimiter.
@@ -106,8 +114,8 @@ class MessageWriter {
     done(error || null);
   }
   _write(chunk, done) {
-    if (this._muted || !this._stream) {
-      if (!this._muted) this._output += chunk;
+    if (!this._stream) {
+      this._output += chunk;
       return done && done();
     }
     this._pending++;
@@ -126,30 +134,24 @@ class MessageWriter {
     if (this._pending === 0) this._waiting.splice(0).forEach(callback => this._settle(callback));
   }
 
-  // Announces the RDF Messages version, which makes parsers expect delimiters
+  // Announces the RDF Messages version, which makes parsers expect delimiters,
+  // and declares the base and prefixes, which stay in effect for all messages
   _start() {
     if (!this._started) {
       this._started = true;
-      this._write(this._format.lineMode ? `VERSION "${this._version}"\n` : `@version "${this._version}" .\n`);
+      const options = this._options;
+      this._writer = new LogWriter(this._sink, {
+        format: this._format.name,
+        end: false,
+        version: this._version,
+        baseIRI: options.baseIRI,
+        writeBase: options.writeBase,
+        prefixes: this._declared
+      });
     }
   }
   _writeMessage(quads) {
-    const first = !this._messageWritten;
-    const options = {
-      format: this._format.name,
-      end: false
-    };
-    if (this._options.baseIRI) {
-      options.baseIRI = this._options.baseIRI;
-      // The base is declared by us, once, as it stays in effect (and older N3.js writers lack `writeBase`)
-      if (first && this._options.writeBase && !this._format.lineMode) this._write(`@base <${this._options.baseIRI}>.\n`);
-    }
-    // Prefixes and base are declared once, as they stay in effect
-    options.prefixes = this._declared;
-    this._muted = !first;
-    const writer = new _n.Writer(this._sink, options);
-    this._muted = false;
-    this._messageWritten = true;
+    const writer = this._writer;
     for (const prefix of Object.keys(this._newPrefixes)) {
       writer.addPrefix(prefix, this._newPrefixes[prefix]);
       this._declared[prefix] = this._newPrefixes[prefix];
@@ -157,8 +159,7 @@ class MessageWriter {
     this._newPrefixes = {};
     const record = error => error && (this._error = this._error || error);
     for (const quad of quads) writer.addQuad(quad.subject, quad.predicate, quad.object, quad.graph, record);
-    writer.end(record);
-    this._write(this._format.lineMode ? 'MESSAGE\n' : '@message .\n');
+    writer.addDelimiter();
   }
 }
 

package/lib/index.js

@@ -44,4 +44,5 @@ var _MessageStreamParser = _interopRequireDefault(require("./MessageStreamParser
 var _MessageWriter = _interopRequireWildcard(require("./MessageWriter.js"));
 var _MessageStreamWriter = _interopRequireDefault(require("./MessageStreamWriter.js"));
 function _interopRequireDefault(e) { return e && e.__esModule ? e : { default: e }; }
-function _interopRequireWildcard(e, t) { if ("function" == typeof WeakMap) var r = new WeakMap(), n = new WeakMap(); return (_interopRequireWildcard = function (e, t) { if (!t && e && e.__esModule) return e; var o, i, f = { __proto__: null, default: e }; if (null === e || "object" != typeof e && "function" != typeof e) return f; if (o = t ? n : r) { if (o.has(e)) return o.get(e); o.set(e, f); } for (const t in e) "default" !== t && {}.hasOwnProperty.call(e, t) && ((i = (o = Object.defineProperty) && Object.getOwnPropertyDescriptor(e, t)) && (i.get || i.set) ? o(f, t, i) : f[t] = e[t]); return f; })(e, t); }
+var cache;
+function _interopRequireWildcard(e, t) { if (!t && e && e.__esModule) return e; var r, c, a = Object.defineProperty, n = { __proto__: null, default: e }; if (Object(e) !== e) return n; if (cache || "function" != typeof WeakMap || (cache = new WeakMap()), cache) { if (cache.has(e)) return cache.get(e); cache.set(e, n); } for (c in e) "default" !== c && {}.hasOwnProperty.call(e, c) && ((r = a && Object.getOwnPropertyDescriptor(e, c)) && (r.get || r.set) ? a(n, c, r) : n[c] = e[c]); return n; }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "n3.js-messages",
-  "version": "0.1.0",
+  "version": "0.1.2",
   "description": "RDF Messages (message-delimited Turtle, TriG, N-Triples and N-Quads) for N3.js: message-aware parser, streaming parser and writer.",
   "author": "Pieter Colpaert",
   "license": "MIT",
@@ -37,7 +37,7 @@
     "index.d.ts"
   ],
   "peerDependencies": {
-    "n3": ">=2.0.0 <3"
+    "n3": ">=2.13.7 <3"
   },
   "repository": {
     "type": "git",
@@ -62,9 +62,9 @@
     "spec:update": "node scripts/update-spec.js"
   },
   "devDependencies": {
-    "@babel/cli": "^7.29.7",
-    "@babel/core": "^7.29.7",
-    "@babel/preset-env": "^7.29.7",
+    "@babel/cli": "^8.0.6",
+    "@babel/core": "^8.0.7",
+    "@babel/preset-env": "^8.0.7",
     "@types/n3": "^1.26.4",
     "@types/node": "^26.6.4",
     "babel-jest": "^30.5.2",
@@ -73,11 +73,11 @@
     "eslint-plugin-jest": "^29.16.6",
     "globals": "^17.13.0",
     "jest": "^30.5.2",
-    "n3": "^2.10.6",
+    "n3": "^2.13.8",
     "rdf-isomorphic": "^2.0.1",
     "rdf-parser-ts": "^0.4.0",
     "rdf-writer-ts": "^0.2.1",
-    "typescript": "^6.0.3"
+    "typescript": "^7.0.2"
   },
   "jest": {
     "testMatch": [

package/src/MessageParserCore.js

@@ -1,16 +1,10 @@
 // **MessageParserCore** is the push-based engine behind all message parsers.
-// It feeds scanned text to the public N3.js parser and groups the resulting
-// quads into messages.
+// It feeds text to the N3.js parser, which recognizes the message delimiters
+// through its `directives` option, and groups the resulting quads into messages.
 import { EventEmitter } from 'events';
 import { Parser, DataFactory } from 'n3';
-import { TurtleScanner, LineScanner } from './MessageScanner.js';
 import { resolveFormat, isMessagesVersion, baseVersion, isSupportedVersion } from './formats.js';
 
-// Unrecognizable by construction: a document cannot contain it by accident
-function createSentinel() {
-  return `rdf-messages-${Math.random().toString(36).slice(2)}${Math.random().toString(36).slice(2)}`;
-}
-
 export default class MessageParserCore {
   // `sink` receives `message(quads, index)`, `end(prefixes)`, `error(error)`
   // and optionally `quad(quad, messageIndex)`, `prefix(prefix, term)`, `version(label)`
@@ -36,6 +30,9 @@ export default class MessageParserCore {
       factory: this._factory,
       // Version labels are validated here, because N3.js does not know `-messages` labels
       parseUnsupportedVersions: true,
+      // N3.js reports `MESSAGE` and `@message .` between statements through `onDirective`,
+      // in order with the quads, and rejects them anywhere else
+      directives: ['message'],
     };
     if (options.baseIRI)
       n3Options.baseIRI = options.baseIRI;
@@ -44,37 +41,41 @@ export default class MessageParserCore {
       n3Options.version = baseVersion(options.version);
     }
 
-    const sentinel = createSentinel();
-    this._sentinel = sentinel;
-    this._scanner = format.lineMode ? new LineScanner(sentinel) : new TurtleScanner(sentinel);
     this._input = new EventEmitter();
-    const callbacks = {
+    this._lineMode = format.lineMode;
+    // The line of the current token, and of the last delimiter
+    this._line = 0;
+    this._delimiterLine = 0;
+    new Parser(n3Options).parse(this._input, {
       onQuad: (error, quad, prefixes) => this._onQuad(error, quad, prefixes),
       onPrefix: (prefix, term) => this._failed || sink.prefix && sink.prefix(prefix, term),
-      onVersion: label => this._onVersion(label),
-    };
-    if (format.lineMode)
-      callbacks.onComment = comment => this._onComment(comment);
-    new Parser(n3Options).parse(this._input, callbacks);
+      onVersion: label => this._failed || this._announce(label),
+      onDirective: () => this._failed || this._delimiter(),
+      onToken: token => this._onToken(token),
+    });
   }
 
   // ### `write` accepts the next chunk of text
   write(text) {
     if (!this._failed && !this._finished && text.length > 0)
-      this._feed(this._scanner.push(text));
+      this._input.emit('data', text);
   }
 
   // ### `end` signals the end of the input
   end() {
-    if (!this._failed && !this._finished) {
-      this._feed(this._scanner.end());
+    if (!this._failed && !this._finished)
       this._input.emit('end');
-    }
   }
 
-  _feed(text) {
-    if (text.length > 0)
-      this._input.emit('data', text);
+  // Keeps track of lines; in N-Triples and N-Quads, a delimiter must be on its own line
+  _onToken(token) {
+    const line = token.line, delimiter = token.type === 'MESSAGE';
+    if (this._lineMode && !this._failed &&
+        (delimiter ? line === this._line : line === this._delimiterLine && token.type !== 'eof'))
+      this._fail(new Error(`Unexpected "MESSAGE" on line ${line}.`));
+    if (delimiter)
+      this._delimiterLine = line;
+    this._line = line;
   }
 
   _onQuad(error, quad, prefixes) {
@@ -104,26 +105,6 @@ export default class MessageParserCore {
       this._sink.quad(quad, this._index);
   }
 
-  _onVersion(label) {
-    if (this._failed)
-      return;
-    if (label === this._sentinel)
-      this._delimiter(this._scanner.delimiterLines.shift());
-    else
-      this._announce(label);
-  }
-
-  // Line mode reports delimiters and versions as comments
-  _onComment(comment) {
-    if (this._failed || !comment.startsWith(this._sentinel))
-      return;
-    const rest = comment.substring(this._sentinel.length);
-    if (rest[0] === 'M')
-      this._delimiter(Number(rest.substring(1)));
-    else
-      this._announce(rest.substring(1));
-  }
-
   _announce(label) {
     const error = this._checkVersion(label);
     if (error)
@@ -147,9 +128,9 @@ export default class MessageParserCore {
     return error;
   }
 
-  _delimiter(line) {
+  _delimiter() {
     if (!this._messages) {
-      return this._fail(new Error(`Unexpected "MESSAGE" on line ${line}: the document did not announce ` +
+      return this._fail(new Error(`Unexpected "MESSAGE" on line ${this._line}: the document did not announce ` +
                                   'a version with the "-messages" suffix.'));
     }
     this._emitMessage();

package/src/MessageWriter.js

@@ -1,13 +1,22 @@
 // **MessageWriter** writes RDF Message Logs (Turtle, TriG, N-Triples and N-Quads).
 //
-// Every message is written by a fresh N3.js writer, so no serialization state
-// (current graph, subject, predicate) leaks from one message into the next.
-// Every message is followed by a delimiter (`MESSAGE` in line formats, `@message .` in Turtle and TriG), so that consumers receive
-// a message as soon as it is written, and so that trailing empty messages survive
-// a round trip (a delimiter before the end of the input does not create a message).
+// One N3.js writer writes the whole log: its `version` option announces the log,
+// and after each message, `_endStatement` finishes the pending statement (and graph block)
+// before a delimiter is written (`MESSAGE` in line formats, `@message .` in Turtle and TriG).
+// That way, consumers receive a message as soon as it is written, no serialization state
+// (current graph, subject, predicate) leaks from one message into the next, and trailing
+// empty messages survive a round trip (a delimiter before the end of the input does not create a message).
 import { Writer } from 'n3';
 import { resolveFormat, isMessagesVersion, baseVersion, isSupportedVersion } from './formats.js';
 
+class LogWriter extends Writer {
+  // ### `addDelimiter` ends the current message
+  addDelimiter() {
+    this._endStatement();
+    this._write(this._lineMode ? 'MESSAGE\n' : '@message .\n');
+  }
+}
+
 export default class MessageWriter {
   // `outputStream` is optional: without it, `end` returns the output as a string
   constructor(outputStream, options) {
@@ -26,13 +35,13 @@ export default class MessageWriter {
     this._pending = 0;
     this._waiting = [];
     this._error = null;
-    this._muted = false;
     this._started = false;
     this._ended = false;
     this._declared = Object.assign({}, options.prefixes);
     this._newPrefixes = {};
-    // The N3.js writers write to this sink, which they never end
+    // The N3.js writer writes to this sink, which it never ends
     this._sink = { write: (chunk, encoding, done) => this._write(chunk, done) };
+    this._writer = null;
   }
 
   // ### `addMessage` writes the quads of one message, followed by a delimiter.
@@ -117,9 +126,8 @@ export default class MessageWriter {
   }
 
   _write(chunk, done) {
-    if (this._muted || !this._stream) {
-      if (!this._muted)
-        this._output += chunk;
+    if (!this._stream) {
+      this._output += chunk;
       return done && done();
     }
     this._pending++;
@@ -140,29 +148,25 @@ export default class MessageWriter {
       this._waiting.splice(0).forEach(callback => this._settle(callback));
   }
 
-  // Announces the RDF Messages version, which makes parsers expect delimiters
+  // Announces the RDF Messages version, which makes parsers expect delimiters,
+  // and declares the base and prefixes, which stay in effect for all messages
   _start() {
     if (!this._started) {
       this._started = true;
-      this._write(this._format.lineMode ? `VERSION "${this._version}"\n` : `@version "${this._version}" .\n`);
+      const options = this._options;
+      this._writer = new LogWriter(this._sink, {
+        format: this._format.name,
+        end: false,
+        version: this._version,
+        baseIRI: options.baseIRI,
+        writeBase: options.writeBase,
+        prefixes: this._declared,
+      });
     }
   }
 
   _writeMessage(quads) {
-    const first = !this._messageWritten;
-    const options = { format: this._format.name, end: false };
-    if (this._options.baseIRI) {
-      options.baseIRI = this._options.baseIRI;
-      // The base is declared by us, once, as it stays in effect (and older N3.js writers lack `writeBase`)
-      if (first && this._options.writeBase && !this._format.lineMode)
-        this._write(`@base <${this._options.baseIRI}>.\n`);
-    }
-    // Prefixes and base are declared once, as they stay in effect
-    options.prefixes = this._declared;
-    this._muted = !first;
-    const writer = new Writer(this._sink, options);
-    this._muted = false;
-    this._messageWritten = true;
+    const writer = this._writer;
     for (const prefix of Object.keys(this._newPrefixes)) {
       writer.addPrefix(prefix, this._newPrefixes[prefix]);
       this._declared[prefix] = this._newPrefixes[prefix];
@@ -171,8 +175,7 @@ export default class MessageWriter {
     const record = error => error && (this._error = this._error || error);
     for (const quad of quads)
       writer.addQuad(quad.subject, quad.predicate, quad.object, quad.graph, record);
-    writer.end(record);
-    this._write(this._format.lineMode ? 'MESSAGE\n' : '@message .\n');
+    writer.addDelimiter();
   }
 }
 

package/src/MessageScanner.js

@@ -1,390 +0,0 @@
-// **MessageScanner** finds RDF Message delimiters in a text stream.
-//
-// N3.js has no extension point for new directives, and its lexer rejects
-// `MESSAGE` and `@message`. The scanners in this file therefore sit in front of
-// the N3.js lexer: they track just enough lexical context (IRIs, strings,
-// long strings, comments, words, braces) to recognize *real* delimiters, and
-// rewrite them into text that N3.js can lex and that is easy to recognize
-// again in N3.js' public parser callbacks:
-//
-// - Turtle and TriG: a delimiter becomes `@version "<sentinel>" .`.
-//   N3.js only accepts version directives at statement boundaries, so the
-//   N3.js parser itself rejects delimiters in other grammar positions, and
-//   reports the delimiter through its public `onVersion` callback, exactly in
-//   order with the emitted quads. The dot matters: the N3.js lexer reads an
-//   `@` after a literal as a language tag, so a literal that ends a SPARQL-style
-//   `VERSION "label"` directive would swallow a following `@prefix`.
-//   For the same reason, SPARQL-style `VERSION "label"` directives in the input
-//   are rewritten to `@version "label" .`.
-// - N-Triples and N-Quads: N3.js has no version directive in line mode.
-//   The grammar is line-based, so a delimiter or version line becomes the
-//   comment `#<sentinel>M<line>` / `#<sentinel>V<label>`, which is reported through
-//   the public `onComment` callback, again in order with the quads.
-//
-// Delimiters that are not recognized (inside strings, IRIs, comments, prefixed
-// names or graph blocks) are passed through unchanged. The scanners never
-// throw: whatever they do not rewrite is rejected by N3.js as a syntax error.
-
-const TOP = 0, IRI = 1, LT = 2, COMMENT = 3, STRING = 4, QUOTE1 = 5, QUOTE2 = 6, LONG = 7, CANDIDATE = 8;
-
-const MESSAGE = 'MESSAGE', AT_MESSAGE = '@message', VERSION = 'VERSION';
-
-function isWhitespace(code) {
-  return code === 32 || code === 9 || code === 10 || code === 13;
-}
-
-// Compares a character to a position of a keyword; `VERSION` is case-insensitive
-function matches(keyword, index, code) {
-  return keyword === VERSION ? String.fromCharCode(code).toUpperCase() === keyword[index] :
-    code === keyword.charCodeAt(index);
-}
-
-// Characters that may follow `MESSAGE`: whitespace, a comment, or an IRI
-function endsMessageKeyword(code) {
-  return isWhitespace(code) || code === 35 || code === 60;
-}
-
-// Characters that may follow `@message`: whitespace, a comment, or the dot
-function endsAtMessageKeyword(code) {
-  return isWhitespace(code) || code === 35 || code === 46;
-}
-
-// ## Turtle and TriG
-export class TurtleScanner {
-  constructor(sentinel) {
-    this._sentinel = sentinel;
-    this._state = TOP;
-    // Whether the next character can start a new token
-    this._tokenStart = true;
-    // Whether we are inside a bare word (prefixed name, number, keyword)
-    this._inWord = false;
-    this._wordEscape = false;
-    this._quote = 0;
-    this._escaped = false;
-    this._quoteRun = 0;
-    this._depth = 0;
-    this._candidate = '';
-    this._target = '';
-    this._candidateLine = 1;
-    this._beforeCandidate = 32;
-    this._versionPending = false;
-    this._line = 1;
-    this._afterCR = false;
-    this._started = false;
-    this._lastOut = 32;
-    // The line of every rewritten delimiter that was not yet consumed
-    this.delimiterLines = [];
-  }
-
-  // ### `push` scans a chunk of text and returns the text to hand to N3.js
-  push(text) {
-    let out = '';
-    let segmentStart = 0;
-    for (let i = 0; i < text.length; i++) {
-      const code = text.charCodeAt(i);
-      // Keep track of the line number (CRLF counts once, also across chunks)
-      if (code === 10) {
-        if (!this._afterCR)
-          this._line++;
-        this._afterCR = false;
-      }
-      else if (code === 13) {
-        this._line++;
-        this._afterCR = true;
-      }
-      else
-        this._afterCR = false;
-
-      // A state change may require the same character to be handled again
-      let again;
-      do {
-        again = false;
-        switch (this._state) {
-        case TOP:
-          if (this._scanTop(code)) {
-            // A possible delimiter starts here: hold it back until it is resolved
-            out += text.substring(segmentStart, i);
-            segmentStart = i + 1;
-            this._candidate = text[i];
-            this._candidateLine = this._line;
-            this._beforeCandidate = i > 0 ? text.charCodeAt(i - 1) : this._lastOut;
-            this._target = code === 77 ? MESSAGE : code === 64 ? AT_MESSAGE : VERSION;
-            this._state = CANDIDATE;
-          }
-          break;
-        case CANDIDATE: {
-          const index = this._candidate.length, target = this._target;
-          if (index < target.length && matches(target, index, code)) {
-            this._candidate += text[i];
-            segmentStart = i + 1;
-          }
-          else {
-            const complete = index === target.length &&
-              (target === AT_MESSAGE ? endsAtMessageKeyword(code) : endsMessageKeyword(code));
-            out += this._resolveCandidate(complete);
-            segmentStart = i;
-            again = true;
-          }
-          break;
-        }
-        case IRI:
-          if (code === 62)
-            this._endToken(true);
-          break;
-        case LT:
-          // `<<` starts a reified triple, everything else an IRI
-          if (code === 60)
-            this._endToken(true);
-          else {
-            this._state = IRI;
-            again = true;
-          }
-          break;
-        case COMMENT:
-          if (code === 10 || code === 13)
-            this._endToken(true);
-          break;
-        case STRING:
-          if (this._escaped)
-            this._escaped = false;
-          else if (code === 92)
-            this._escaped = true;
-          else if (code === this._quote) {
-            this._endToken(false);
-            if (this._versionPending) {
-              // Terminate a SPARQL-style version directive, which we turned into an `@version` directive
-              this._versionPending = false;
-              out += `${text.substring(segmentStart, i + 1)} .`;
-              segmentStart = i + 1;
-            }
-          }
-          break;
-        case QUOTE1:
-          // A second quote either starts a long string or ends an empty string
-          if (code === this._quote)
-            this._state = QUOTE2;
-          else {
-            this._state = STRING;
-            again = true;
-          }
-          break;
-        case QUOTE2:
-          if (code === this._quote) {
-            this._state = LONG;
-            this._escaped = false;
-            this._quoteRun = 0;
-          }
-          else {
-            this._endToken(false);
-            again = true;
-          }
-          break;
-        default: // LONG
-          if (this._escaped) {
-            this._escaped = false;
-            this._quoteRun = 0;
-          }
-          else if (code === 92) {
-            this._escaped = true;
-            this._quoteRun = 0;
-          }
-          else if (code === this._quote) {
-            if (++this._quoteRun === 3)
-              this._endToken(false);
-          }
-          else
-            this._quoteRun = 0;
-        }
-      } while (again);
-    }
-    out += text.substring(segmentStart);
-    return this._remember(out);
-  }
-
-  // ### `end` resolves any held-back text at the end of the input
-  end() {
-    let out = '';
-    if (this._state === CANDIDATE)
-      out = this._resolveCandidate(this._candidate.length === this._target.length);
-    return this._remember(out);
-  }
-
-  // ### `_scanTop` handles a character between tokens or inside a word.
-  // It returns true if the character may start a message delimiter.
-  _scanTop(code) {
-    // An escape inside a word makes the next character part of the word
-    if (this._wordEscape) {
-      this._wordEscape = false;
-      return false;
-    }
-    switch (code) {
-    case 32: case 9: case 10: case 13:
-      this._structural();
-      break;
-    case 0xFEFF:
-      // A byte order mark at the very start is not part of any token
-      if (this._started)
-        this._word();
-      break;
-    case 35: // #
-      this._state = COMMENT;
-      this._inWord = false;
-      break;
-    case 60: // <
-      this._state = LT;
-      this._inWord = false;
-      break;
-    case 34: case 39: // " '
-      this._state = QUOTE1;
-      this._quote = code;
-      this._inWord = false;
-      break;
-    case 123: // {
-      this._depth++;
-      this._structural();
-      break;
-    case 125: // }
-      this._depth = Math.max(0, this._depth - 1);
-      this._structural();
-      break;
-    case 40: case 41: case 91: case 93: case 44: case 59: // ( ) [ ] , ;
-      this._structural();
-      break;
-    case 46: // . ends a statement, unless it is part of a word
-      if (!this._inWord)
-        this._structural();
-      break;
-    case 92: // \
-      this._word();
-      this._wordEscape = true;
-      break;
-    case 77: case 64: case 86: case 118: // M @ V v
-      if (this._tokenStart) {
-        this._started = true;
-        return true;
-      }
-      this._word();
-      break;
-    default:
-      this._word();
-    }
-    this._started = true;
-    return false;
-  }
-
-  _structural() {
-    this._inWord = false;
-    this._tokenStart = true;
-  }
-
-  _word() {
-    this._inWord = true;
-    this._tokenStart = false;
-  }
-
-  // ### `_endToken` returns to the top level after an IRI, string or comment
-  _endToken(tokenStart) {
-    this._state = TOP;
-    this._inWord = false;
-    this._tokenStart = tokenStart;
-  }
-
-  // ### `_resolveCandidate` decides whether a held-back candidate is a delimiter
-  // and returns the text to emit for it
-  _resolveCandidate(complete) {
-    const candidate = this._candidate;
-    this._state = TOP;
-    this._candidate = '';
-    // A delimiter in a graph block is not rewritten, so that N3.js rejects it
-    if (!complete || (this._depth > 0 && this._target !== VERSION)) {
-      this._word();
-      return candidate;
-    }
-    this._structural();
-    const previous = this._beforeCandidate;
-    const space = isWhitespace(previous) || previous === 0xFEFF ? '' : ' ';
-    if (this._target === VERSION) {
-      this._versionPending = true;
-      return `${space}@version`;
-    }
-    this.delimiterLines.push(this._candidateLine);
-    return `${space}@version "${this._sentinel}"${candidate === MESSAGE ? ' .' : ''}`;
-  }
-
-  _remember(out) {
-    if (out.length > 0)
-      this._lastOut = out.charCodeAt(out.length - 1);
-    return out;
-  }
-}
-
-// ## N-Triples and N-Quads
-// Lines of the form `MESSAGE` or `VERSION "label"` are the only directives.
-// They cannot be confused with strings or IRIs, as those never span lines.
-export class LineScanner {
-  constructor(sentinel) {
-    this._sentinel = sentinel;
-    this._buffer = '';
-    this._line = 1;
-    this._messageLine = new RegExp(`^[ \\t]*${MESSAGE}[ \\t]*(?:#.*)?$`);
-    this._versionLine = /^[ \t]*VERSION[ \t]*(?:"([^"\\]*)"|'([^'\\]*)')[ \t]*(?:#.*)?$/i;
-  }
-
-  // ### `push` scans a chunk of text and returns the complete lines, rewritten
-  push(text) {
-    const data = this._buffer + text;
-    // A final carriage return waits for a possible line feed, so that CRLF counts as one line break
-    const limit = data.endsWith('\r') ? data.length - 1 : data.length;
-    const lastBreak = Math.max(data.lastIndexOf('\n', limit - 1), data.lastIndexOf('\r', limit - 1));
-    if (lastBreak < 0) {
-      this._buffer = data;
-      return '';
-    }
-    this._buffer = data.substring(lastBreak + 1);
-    return this._rewrite(data.substring(0, lastBreak + 1));
-  }
-
-  // ### `end` returns the last line, which has no line break
-  end() {
-    const rest = this._buffer;
-    this._buffer = '';
-    return rest.length > 0 ? this._rewrite(rest) : '';
-  }
-
-  _rewrite(block) {
-    let out = '';
-    let copyFrom = 0, lineStart = 0;
-    const breaks = /\r\n|\n|\r/g;
-    let match;
-    do {
-      match = breaks.exec(block);
-      const lineEnd = match ? match.index : block.length;
-      const replacement = this._rewriteLine(block, lineStart, lineEnd);
-      if (replacement !== null) {
-        out += block.substring(copyFrom, lineStart) + replacement;
-        copyFrom = lineEnd;
-      }
-      lineStart = match ? breaks.lastIndex : block.length + 1;
-      if (match)
-        this._line++;
-    } while (lineStart <= block.length);
-    return out + block.substring(copyFrom);
-  }
-
-  // Only lines that start with `MESSAGE` or `VERSION` need a closer look
-  _rewriteLine(block, start, end) {
-    let first = start;
-    while (first < end && (block[first] === ' ' || block[first] === '\t'))
-      first++;
-    const code = block.charCodeAt(first);
-    if (first === end || (code !== 77 && code !== 86 && code !== 118))
-      return null;
-    const line = block.substring(first, end);
-    if (this._messageLine.test(line))
-      return `#${this._sentinel}M${this._line}`;
-    const version = this._versionLine.exec(line);
-    if (version)
-      return `#${this._sentinel}V${version[1] === undefined ? version[2] : version[1]}`;
-    return null;
-  }
-}