soml-lang 0.0.1 → 0.0.2

package/readme.md

@@ -1,29 +1,34 @@
 # soml
 
-> The reference parser and canonical serializer for [SOML](../readme.md), a config format for humans
+> The reference parser, canonical serializer, and formatter for [SOML](../soml/readme.md), a config format for humans
 
 > [!NOTE]
-> The format is a draft. See the [specification](../spec.md).
+> The format is a draft. See the [specification](../soml/spec.md).
 
 - Strict: every rule in the specification is enforced, and every error says what is wrong and where
 - Exact: an int is a `bigint`, so every 64-bit int survives, and `3` and `3.0` stay different
 - Canonical: `stringify()` writes the one canonical form, so equal values give equal bytes
 - Safe: no prototype pollution, define semantics for every object member even with a frozen or modified `Object.prototype`, no stack overflow on deep input or on huge tokens, and `parse()` takes linear time on every input shape
 - Fast: parses about 1.7 times as fast as smol-toml and 7 times as fast as json5
-- Tested by a language-neutral [conformance suite](test/conformance) of more than 800 cases, plus property tests and fuzzing
+- Tested by a language-neutral [conformance suite](test/conformance) of almost 900 cases, plus property tests and fuzzing
 
 ## Install
 
 ```sh
-npm install soml
+npm install soml-lang
 ```
 
-Requires Node.js 26 or later, for `Temporal`.
+Requires Node.js 22 or later. Instants and durations use `Temporal`, which Node.js 26 and later have built in. On older versions, everything else works, but an instant, a duration, or a `Date` throws. To use them, load a [`Temporal` polyfill](https://github.com/fullcalendar/temporal-polyfill) before you import this package:
+
+```js
+import 'temporal-polyfill/global';
+import {parse} from 'soml-lang';
+```
 
 ## Usage
 
 ```js
-import {parse, stringify} from 'soml';
+import {parse, stringify} from 'soml-lang';
 
 const config = parse(`
 # The edge service.
@@ -111,13 +116,89 @@ parse('port: 8080', {integers: 'number'});
 //=> {port: 8080}
 ```
 
+### parseTree(text)
+
+Parse a document into a syntax tree, for tools such as linters and formatters. Returns a `Document` node, which also holds every token and comment.
+
+Throws the same [`ParseError`](#parseerror) as `parse()`, and a `TypeError` when `text` is not a string.
+
+```js
+import {parseTree} from 'soml-lang';
+
+const tree = parseTree('port: 8080 # The default');
+
+tree.body.members[0].value;
+//=> {type: 'Integer', value: 8080n, radix: 10, range: [6, 10], loc: {…}}
+
+tree.comments[0];
+//=> {type: 'Line', value: ' The default', range: [11, 24], loc: {…}}
+```
+
+Every node, token, and comment has a `range`, which is `[start, end]` as UTF-16 offsets into `text`, and a `loc`, which is `{start: {line, column}, end: {line, column}}`, with a 1-based line and a 0-based column in UTF-16 code units, as in [ESTree](https://github.com/estree/estree). A [`ParseError`](#parseerror) counts its column differently, for people to read, so use its `offset` to find the position in the tree.
+
+| Node | Fields |
+|---|---|
+| `Document` | `body`: an `Object` or an `Array`; `tokens`; `comments` |
+| `Object` | `members`; `braced`, which is `false` only for a top-level object without braces |
+| `Member` | `key`; `value` |
+| `Array` | `elements` |
+| `Key` | `segments`, more than one for a dotted key |
+| `KeySegment` | `value`, decoded; `style`: `'bare'`, `'literal'`, or `'escaped'` |
+| `String` | `value`, decoded; `style`: `'literal'` or `'escaped'`; `block` |
+| `Integer` | `value`, a `bigint`; `radix`: `2`, `8`, `10`, or `16` |
+| `Float` | `value`, including `Infinity` and `-Infinity` |
+| `Boolean` | `value` |
+| `Null` | |
+| `Instant` | `value`, a `Temporal.Instant` |
+| `Duration` | `value`, a `Temporal.Duration` |
+
+A token is `{type, value, range, loc}`, where `type` is `'Punctuator'` (for `{`, `}`, `[`, `]`, `:`, `,`, and `.`), `'BareKey'`, `'String'`, `'Integer'`, `'Float'`, `'Keyword'` (for `true`, `false`, and `null`), `'Instant'`, or `'Duration'`, and `value` is its source text. A comment is `{type, value, range, loc}`, where `type` is `'Line'` or `'Block'`, and `value` is the text without `#`, or without `/*` and `*/`.
+
+### visitorKeys
+
+The properties of each node type that hold its child nodes, in source order, for tools that walk the tree, such as an ESLint language plugin.
+
+```js
+import {visitorKeys} from 'soml-lang';
+
+visitorKeys.Member;
+//=> ['key', 'value']
+
+visitorKeys.Integer;
+//=> []
+```
+
+### format(text)
+
+Format a document. Returns the document with its layout normalized, ending with one line feed.
+
+The layout follows the [formatter](../soml/spec.md#formatting) in the specification: one tab per level, every member and item on its own line, a trailing comma after every member and item inside `{}` and `[]`, one space after `:`, and no trailing whitespace or runs of blank lines. Comments, member order, dotted keys, block strings, and the spelling of every value stay as they are, so the value never changes.
+
+Throws the same [`ParseError`](#parseerror) as `parse()`, and a `TypeError` when `text` is not a string.
+
+```js
+import {format} from 'soml-lang';
+
+format('pool: {min: 2, max: 16} # Connections');
+//=> 'pool: {\n\tmin: 2,\n\tmax: 16,\n} # Connections\n'
+```
+
+Where the specification does not say, the formatter decides:
+
+- A comment at the end of a line stays at the end of that line, after the inserted comma. A block comment that a value follows on the same line, as in `[1, /* note */ 2]`, stays in front of that value.
+- A value on the line after its `key:` moves up to that line, unless a comment comes between them.
+- A block string moves with its closing delimiter, and its lines are not otherwise changed.
+- There are no blank lines at the start of the file or directly inside brackets.
+
+In the rare comma-first style, a comment after a comma on its own line moves to its own line when the item before it already ends with a comment.
+
 ### stringify(value, options?)
 
 Serialize an object or an array to canonical form. Members are sorted by key, nesting is written with braces and tabs, and the output ends with one line feed. Comments, dotted keys, and block strings are never written.
 
 Besides the types above, a `Date` is accepted and written as an instant, a `Temporal.Duration` is accepted when it has no years, months, weeks, or days, because those are not a fixed length, and a member whose value is `undefined` is left out, as `JSON.stringify` does.
 
-Throws a `TypeError` for a value that cannot be represented: `NaN`, a function, a symbol, a class instance, `undefined` in an array, a circular reference, a non-collection at the top level, an invalid `Date`, a `Temporal.Duration` with years, months, weeks, or days, an object that only claims to be a `Date`, a `Temporal.Instant`, or a `Temporal.Duration`, and a string or key with a lone surrogate or a carriage return. Throws a `RangeError` for an int outside the 64-bit range, an instant outside the years 0001 to 9999, a duration outside the 64-bit range of nanoseconds, and nesting deeper than 1000 levels.
+Throws a `TypeError` for a value that cannot be represented: `NaN`, a function, a symbol, a class instance, `undefined` in an array, a circular reference, a non-collection at the top level, an invalid `Date`, a `Temporal.Duration` with years, months, weeks, or days, an object that only claims to be a `Date`, a `Temporal.Instant`, or a `Temporal.Duration`, and a string or key with a lone surrogate or a carriage return. Throws a `RangeError` for an int outside the 64-bit range, an instant outside the years 0001 to 9999, a duration outside the 64-bit range of nanoseconds, and nesting deeper than 100 levels.
 
 #### options
 
@@ -145,7 +226,7 @@ stringify({port: 8080}, {integers: 'number'});
 Thrown by `parse()`. Extends `SyntaxError`.
 
 ```js
-import {parse, ParseError} from 'soml';
+import {parse, ParseError} from 'soml-lang';
 
 try {
 	parse('name: api-gateway');
@@ -192,8 +273,7 @@ Up to three lines ending at the error, with a caret under the position. A long l
 
 ## Limits
 
-- **Nesting is limited to 1000 levels**, in `parse()` and in `stringify()`. A recursive parser would otherwise end with an engine stack overflow. A dotted key counts each segment.
-- **No formatter yet.** The specification defines a formatter that keeps comments, member order, and the author's spelling. This package only writes canonical form.
+- **Nesting is limited to 100 levels**, in `parse()` and in `stringify()`, as the spec requires: every document up to 100 levels is accepted, and every deeper one is rejected. The document's own collection is level 1, and a dotted key counts each segment.
 
 ## Conformance suite
 

package/index.d.ts

@@ -1,151 +0,0 @@
-/**
-A value in a document.
-
-| Format | JavaScript |
-|---|---|
-| string | `string` |
-| int | `bigint`, or `number` with `integers: 'number'` |
-| float | `number` |
-| bool | `boolean` |
-| null | `null` |
-| instant | `Temporal.Instant` |
-| duration | `Temporal.Duration`, in hours and smaller units |
-| array | `Array` |
-| object | plain `Object` |
-*/
-export type Value<Integer extends bigint | number = bigint> =
-	| string
-	| Integer
-	| number
-	| boolean
-	| null // eslint-disable-line @typescript-eslint/no-restricted-types -- The format has a null value, and `parse()` returns `null` for it.
-	| Temporal.Instant
-	| Temporal.Duration
-	| Value<Integer>[] // eslint-disable-line @typescript-eslint/array-type
-	| ObjectValue<Integer>;
-
-/**
-An object in a document.
-*/
-export type ObjectValue<Integer extends bigint | number = bigint> = {[key: string]: Value<Integer>};
-
-/**
-A whole document, which is always an object or an array.
-*/
-export type Document<Integer extends bigint | number = bigint> = ObjectValue<Integer> | Array<Value<Integer>>;
-
-export type ParseOptions = {
-	/**
-	How an int is represented.
-
-	- `'bigint'`: Every int is a `bigint`, and every float is a `number`, so `3` and `3.0` stay different, and every 64-bit int is exact. This is the only conforming mode.
-	- `'number'`: Every int is a `number`. An int outside `Number.MIN_SAFE_INTEGER` to `Number.MAX_SAFE_INTEGER` throws a `ParseError` rather than being rounded. `3` and `3.0` both become `3`, so `stringify()` cannot tell them apart afterwards.
-
-	@default 'bigint'
-	*/
-	readonly integers?: 'bigint' | 'number';
-};
-
-export type StringifyOptions = {
-	/**
-	How an int is represented in the value.
-
-	- `'bigint'`: A `bigint` is written as an int, and a `number` is always written as a float, so `8080` becomes `8080.0`.
-	- `'number'`: A `number` that is a safe integer is written as an int, and any other `number` as a float. A `bigint` is still written as an int.
-
-	@default 'bigint'
-	*/
-	readonly integers?: 'bigint' | 'number';
-};
-
-/**
-Parse a document.
-
-@param text - The document, as a string or as UTF-8 bytes.
-@returns The object or array the document contains.
-@throws {ParseError} When the document is not valid.
-@throws {TypeError} When `text` is not a string or a `Uint8Array`, or `options.integers` is not `'bigint'` or `'number'`.
-
-@example
-```
-import {parse} from 'soml';
-
-parse(`
-name: 'api-gateway'
-replicas: 3
-timeout: 30.0
-postgres.host: 'db.internal'
-`);
-//=> {name: 'api-gateway', replicas: 3n, timeout: 30, postgres: {host: 'db.internal'}}
-
-parse('replicas: 3', {integers: 'number'});
-//=> {replicas: 3}
-```
-*/
-export function parse(text: string | Uint8Array, options?: ParseOptions & {readonly integers?: 'bigint'}): Document;
-export function parse(text: string | Uint8Array, options: ParseOptions & {readonly integers: 'number'}): Document<number>;
-export function parse(text: string | Uint8Array, options?: ParseOptions): Document | Document<number>;
-
-/**
-Serialize an object or an array to canonical form.
-
-Canonical form is unique for a value: two equal values produce the same bytes, so the output can be hashed, signed, or compared. Members are sorted by key, and comments, dotted keys, and block strings are never written.
-
-A `bigint` is written as an int and a `number` as a float, so `{port: 8080}` becomes `port: 8080.0`. Use `8080n`, or the `integers: 'number'` option.
-
-A `Date` is accepted and written as an instant. A member whose value is `undefined` is left out.
-
-@param value - A plain object or an array.
-@returns The canonical form, ending with one line feed.
-@throws {TypeError} When the value contains something that cannot be represented, such as `NaN`, a function, a class instance, an invalid `Date`, a `Temporal.Duration` with years, months, weeks, or days, a circular reference, or a string or key with a lone surrogate or a carriage return. Also when `options.integers` is not `'bigint'` or `'number'`.
-@throws {RangeError} When an int is outside the 64-bit range, an instant is outside the years 0001 to 9999, a duration is outside the 64-bit range of nanoseconds, or the value is nested more than 1000 levels deep.
-
-@example
-```
-import {stringify} from 'soml';
-
-stringify({name: 'api-gateway', replicas: 3n, timeout: 30});
-//=> "name: 'api-gateway'\nreplicas: 3\ntimeout: 30.0\n"
-```
-*/
-// `object` rather than `Record<string, unknown>`, so that a value typed with an interface is accepted too. The value is checked at runtime.
-export function stringify(value: object, options?: StringifyOptions): string; // eslint-disable-line @typescript-eslint/no-restricted-types
-
-/**
-Thrown by `parse()` when the input is not a valid document.
-
-The `message` includes the position and a code frame. Use `reason` for the message alone.
-*/
-export class ParseError extends SyntaxError {
-	/**
-	Only `parse()` creates a `ParseError`.
-	*/
-	private constructor();
-
-	readonly name: 'ParseError';
-
-	/**
-	What is wrong, without the position.
-	*/
-	readonly reason: string;
-
-	/**
-	The 1-based line of the error.
-	*/
-	readonly line: number;
-
-	/**
-	The 1-based column of the error, counted in Unicode code points.
-	*/
-	readonly column: number;
-
-	/**
-	The 0-based UTF-16 index of the error in the decoded text.
-	*/
-	readonly offset: number;
-
-	/**
-	Up to three lines ending at the error, with a caret under the position. A long line is clipped around the position.
-	*/
-	readonly codeFrame: string;
-}

package/index.js

@@ -1,3 +0,0 @@
-export {parse} from './source/parse.js';
-export {stringify} from './source/stringify.js';
-export {ParseError} from './source/error.js';

package/source/error.js

@@ -1,130 +0,0 @@
-const CONTEXT_LINES = 2;
-const MAX_LINE_WIDTH = 100;
-
-// With the `v` flag, a surrogate in the class matches only when it is unpaired.
-const UNPRINTABLE = /[\p{Bidi_Control}\u{D800}-\u{DFFF}[\p{Control}--\t]]/gv;
-
-/**
-Thrown by `parse()` when the input is not a valid document.
-*/
-export class ParseError extends SyntaxError {
-	name = 'ParseError';
-
-	constructor(reason, source, offset) {
-		const sanitizedReason = sanitize(reason);
-		const {line, column} = locate(source, offset);
-		const codeFrame = createCodeFrame(source, offset, line);
-		super(`${sanitizedReason} at line ${line}, column ${column}\n\n${codeFrame}`);
-		this.reason = sanitizedReason;
-		this.line = line;
-		this.column = column;
-		this.offset = offset;
-		this.codeFrame = codeFrame;
-	}
-}
-
-/*
-Line and column are 1-based, and the column counts code points, so an emoji is one column.
-*/
-function locate(source, offset) {
-	let line = 1;
-	let lineStart = 0;
-
-	for (let index = source.indexOf('\n'); index !== -1 && index < offset; index = source.indexOf('\n', index + 1)) {
-		line++;
-		lineStart = index + 1;
-	}
-
-	return {line, column: countCodePoints(source.slice(lineStart, offset)) + 1};
-}
-
-function countCodePoints(string) {
-	let count = 0;
-
-	for (let index = 0; index < string.length; index++) {
-		const code = string.charCodeAt(index);
-
-		// A high surrogate followed by a low one is one code point.
-		if (code >= 0xD8_00 && code <= 0xDB_FF) {
-			const next = string.charCodeAt(index + 1);
-
-			if (next >= 0xDC_00 && next <= 0xDF_FF) {
-				index++;
-			}
-		}
-
-		count++;
-	}
-
-	return count;
-}
-
-function createCodeFrame(source, offset, line) {
-	// The error line and up to two lines before it, found by scanning back from the error, so that an error late in a long document does not split the whole document.
-	const lineStarts = [offset === 0 ? 0 : source.lastIndexOf('\n', offset - 1) + 1];
-
-	while (lineStarts.length <= CONTEXT_LINES && lineStarts[0] > 0) {
-		const searchFrom = lineStarts[0] - 2;
-		lineStarts.unshift(searchFrom < 0 ? 0 : source.lastIndexOf('\n', searchFrom) + 1);
-	}
-
-	const firstLine = line - lineStarts.length + 1;
-	const gutterWidth = String(line).length;
-	const output = [];
-
-	for (const [index, lineStart] of lineStarts.entries()) {
-		const number = firstLine + index;
-		const isErrorLine = number === line;
-		const lineEnd = source.indexOf('\n', lineStart);
-		const {text, pointerOffset} = clip(sanitize(source.slice(lineStart, lineEnd === -1 ? undefined : lineEnd)), isErrorLine ? offset - lineStart : 0);
-		const gutter = `${isErrorLine ? '>' : ' '} ${String(number).padStart(gutterWidth)} |`;
-		output.push(text === '' ? gutter : `${gutter} ${text}`);
-
-		if (isErrorLine) {
-			// Keep the tabs from the line, so the caret lines up whatever the tab width is.
-			const padding = text.slice(0, pointerOffset).replaceAll(/[^\t]/gv, ' ');
-			output.push(`  ${' '.repeat(gutterWidth)} | ${padding}^`);
-		}
-	}
-
-	return output.join('\n');
-}
-
-/*
-The rejected text can hold characters that a terminal acts on rather than shows: control characters such as an escape or a carriage return, bidirectional controls, and lone surrogates. Each becomes U+FFFD, which is also one code unit, so the caret still lines up. Tabs are kept.
-*/
-function sanitize(text) {
-	return text.replaceAll(UNPRINTABLE, '\u{FFFD}');
-}
-
-/*
-A long line, such as a minified document, is cut to a window around the pointer. The cut never splits a surrogate pair.
-*/
-function clip(text, pointerOffset) {
-	if (text.length <= MAX_LINE_WIDTH) {
-		return {text, pointerOffset};
-	}
-
-	let start = Math.max(0, Math.min(pointerOffset - (MAX_LINE_WIDTH / 2), text.length - MAX_LINE_WIDTH));
-	let end = start + MAX_LINE_WIDTH;
-
-	if (isLowSurrogate(text.charCodeAt(start))) {
-		start++;
-	}
-
-	if (isLowSurrogate(text.charCodeAt(end))) {
-		end--;
-	}
-
-	const prefix = start > 0 ? '…' : '';
-	const suffix = end < text.length ? '…' : '';
-
-	return {
-		text: `${prefix}${text.slice(start, end)}${suffix}`,
-		pointerOffset: pointerOffset - start + prefix.length,
-	};
-}
-
-function isLowSurrogate(code) {
-	return code >= 0xDC_00 && code <= 0xDF_FF;
-}