@op-ted/efx-syntax-highlighter 0.1.0 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,41 +1,77 @@
1
- # efx-syntax-highlighter
2
-
3
- A minimal JavaScript package providing Lezer-based parsing and CodeMirror 6 language support for the efx language.
4
-
5
- ## Features
6
- - Lezer parser for efx
7
- - CodeMirror 6 language extension
8
- - Custom highlight theme
9
-
10
- ## Usage
11
- Install via npm:
12
- ```bash
13
- npm install @op-ted/efx-syntax-highlighter
14
- ```
15
-
16
- Import and use in CodeMirror 6:
17
- ```js
18
- import { efx, efxHighlightStyle } from '@op-ted/efx-syntax-highlighter';
19
-
20
- const editor = new EditorView({
21
- extensions: [efx(), efxHighlightStyle]
22
- });
23
- ```
24
-
25
- ## Build & Test
26
- To build the parser:
27
- ```bash
28
- npm run build
29
- ```
30
- To run tests:
31
- ```bash
32
- npm test
33
- ```
34
-
35
- ## Publishing
36
- - Only the `dist/` output is published
37
- - Do not include grammar or test files in the package
38
- - Use `npm publish` to release
39
-
40
- ## License
41
- EUPL 2.0
1
+ # efx-syntax-highlighter
2
+
3
+ EFX support for [CodeMirror 6](https://codemirror.net/) editors: syntax highlighting and completions for the EFX language of the eForms SDK 2. It reads the three kinds of EFX text: template files (view templates), rules files (validation rules) and single expressions.
4
+
5
+ The parser is a [Lezer](https://lezer.codemirror.net/) grammar that follows the SDK's ANTLR grammar (`efx-grammar/Efx.g4` and `EfxLexer.g4`) rule for rule. Its words and the completion list are generated from the SDK and the EFX Toolkit at the commits pinned in `package.json` (`config`), and a test compares its tokens with those of the ANTLR lexer over the SDK's view templates and rules files. What each release changes is in [CHANGELOG.md](CHANGELOG.md).
6
+
7
+ ## Install
8
+
9
+ ```bash
10
+ npm install @op-ted/efx-syntax-highlighter
11
+ ```
12
+
13
+ The package is an ES module. It needs the CodeMirror packages it lists as dependencies; a bundler resolves them, or an import map in the page, which must then name their own dependencies too (`demo/index.html` has one).
14
+
15
+ ## Use
16
+
17
+ ```js
18
+ import { autocompletion } from '@codemirror/autocomplete';
19
+ import { EditorState } from '@codemirror/state';
20
+ import { EditorView } from '@codemirror/view';
21
+ import { efx, efxSymbols, efxTheme } from '@op-ted/efx-syntax-highlighter';
22
+
23
+ const view = new EditorView({
24
+ parent: document.getElementById('editor'),
25
+ state: EditorState.create({
26
+ doc: text, // the EFX text to edit
27
+ extensions: [
28
+ efx('template'),
29
+ efxSymbols.of({ fields: ['BT-01-notice', 'BT-26(m)-Procedure'], nodes: ['ND-Root', 'ND-Lot'] }),
30
+ autocompletion(),
31
+ efxTheme,
32
+ ],
33
+ }),
34
+ });
35
+ ```
36
+
37
+ The package offers four things:
38
+
39
+ - **`efx(kind)`** gives an editor what it needs to read one kind of EFX text: its parser, the EFX colours and the completions. The kind is `'template'` for a template file, `'rules'` for a rules file, or `'expression'` for a single expression with its context, such as `WITH ND-Root COMPUTE BT-01-notice is present`. Any other kind is refused with a `RangeError`. `efx(kind).language` is the CodeMirror language, for an application that needs it on its own.
40
+ - **`efxSymbols`** gives the editor the field and node ids it offers: `efxSymbols.of({ fields, nodes })`, each a list of ids, for example those of the SDK's `fields.json`. The ids of several such extensions add up.
41
+ - **`efxHighlightStyle`** is the EFX colours. `efx(kind)` already uses them as the editor's default colours; an application that gives its editor its own colours with `syntaxHighlighting(...)` replaces them.
42
+ - **`efxTheme`** is a light look for the editor itself (white background, black cursor). Add it if you want it.
43
+
44
+ ### Completions
45
+
46
+ The application turns completion on, with `autocompletion()` from `@codemirror/autocomplete` (or the `basicSetup` of the `codemirror` package, which includes it). While you type, the editor offers the words the grammar can take at the cursor:
47
+
48
+ - at the start of an expression, the functions as calls with their required parameters (`contains(haystack, needle)`), with their signatures beside them, and the keywords;
49
+ - after a field and a colon, the field's properties (`:rawValue`, `:publicationDate`, ...);
50
+ - after `is`, `not`, `empty`, `present` and `unique`;
51
+ - in a label block, the asset types and label types; after the pipe of an expression block, the format options;
52
+ - at the start of a line, the words of a template line or of a rule;
53
+ - the variables the cursor can use, and the field and node ids given through `efxSymbols`, where the grammar takes one.
54
+
55
+ In template text any word can go, so there the list opens only when asked for (Ctrl-Space). Nothing is offered inside a comment or a string.
56
+
57
+ Known limits: after the label shorthand `#{name|` no field ids are offered; after a `/` the `@` of an attribute is not offered; after a syntax error nothing is offered until the error is fixed.
58
+
59
+ ## Develop
60
+
61
+ ```bash
62
+ npm run build # generates the parser from grammar/efx.grammar and writes the package to dist/
63
+ npm test # the package, tree, conformance, style and completion tests, on the build
64
+ npm run demo # serves a demo page at http://localhost:8080/demo/ (after npm run build)
65
+ npm run site # gathers the demo page and what it loads into site/ (after npm run build)
66
+ npm run sync # copies the words, the corpus, the ANTLR oracle and the completion list again
67
+ ```
68
+
69
+ `npm run sync` reads the SDK and the toolkit from GitHub at the commits pinned in `package.json`, or pins new ones first: `npm run sync -- <owner>/<repository>@<commit> <owner>/<repository>@<commit>` (the SDK, then the toolkit). It needs Node 22.13 or later and network access; behind a proxy, set `NODE_USE_ENV_PROXY=1`. A `GITHUB_TOKEN`, when set, is sent to the GitHub API only. Build and test need neither network nor Java. The Check workflow builds, tests and syncs every push, and fails if a generated file differs from the committed one. The Pages workflow publishes `site/` to the repository's GitHub Pages site on every push to `develop`.
70
+
71
+ ## Publishing
72
+
73
+ Only `dist/` is published, with the README and the licence. Pushing a tag of the form `x.y.z` runs the Publish workflow, which builds, tests and publishes to npm, with provenance, the version written in `package.json`: the tag does not set the version, so tag the commit that sets it, with the same number (for example `0.3.0`).
74
+
75
+ ## License
76
+
77
+ [EUPL 2.0](LICENSE)
@@ -0,0 +1,51 @@
1
+ // The toolkit's completion list (src/generated/completions.js) by word: the option for a word the
2
+ // editor offers. A function is offered as a call where an expression can start, which is where the
3
+ // parser can take `(`: its word, then the required parameters of its first signature as fields to
4
+ // fill in, with all its signatures as information. Elsewhere its word is offered bare, as `not` and
5
+ // `empty` after `is`. The list writes a property with its colon; its word is offered after the
6
+ // colon. A word the list does not have is offered as a keyword.
7
+
8
+ import { snippetCompletion } from '@codemirror/autocomplete';
9
+
10
+ // The type of option of each kind of item, which picks its icon.
11
+ const TYPES = { FUNCTION: 'function', KEYWORD: 'keyword', PROPERTY: 'property' };
12
+
13
+ export class CompletionItems {
14
+ #byWord = new Map();
15
+
16
+ constructor(items) {
17
+ for (const item of items) {
18
+ const word = item.label.replace(/^:/, '');
19
+ this.#byWord.set(word, [...(this.#byWord.get(word) ?? []), item]);
20
+ }
21
+ }
22
+
23
+ /** The option for a word, as a call when an expression can start where it goes. */
24
+ optionFor(word, expressionStarts) {
25
+ const items = this.#byWord.get(word) ?? [];
26
+ const functions = items.filter(({ kind }) => kind === 'FUNCTION');
27
+ if (expressionStarts && functions.length > 0) {
28
+ return CompletionItems.#call(word, functions);
29
+ }
30
+ const item = items.find(({ kind }) => kind !== 'FUNCTION');
31
+ return { label: word, type: TYPES[item?.kind ?? 'KEYWORD'], detail: item?.dataType };
32
+ }
33
+
34
+ static #call(word, functions) {
35
+ const [first] = functions;
36
+ const fields = first.parameters.filter(({ optional }) => !optional).map(({ name }) => `\${${name}}`);
37
+ return snippetCompletion(`${word}(${fields.join(', ')})`, {
38
+ label: word,
39
+ type: TYPES.FUNCTION,
40
+ detail: first.dataType,
41
+ info: functions.map((item) => CompletionItems.#signature(item)).join('\n'),
42
+ });
43
+ }
44
+
45
+ /** A signature as the toolkit's list writes it: `label(name TYPE, name TYPE?, name TYPE...) -> TYPE`. */
46
+ static #signature({ label, parameters, dataType }) {
47
+ const mark = ({ optional, variadic }) => (optional ? '?' : variadic ? '...' : '');
48
+ const written = parameters.map((parameter) => `${parameter.name} ${parameter.dataType}${mark(parameter)}`);
49
+ return `${label}(${written.join(', ')}) -> ${dataType}`;
50
+ }
51
+ }
@@ -0,0 +1,86 @@
1
+ // What the editor offers at the cursor, a completion source of @codemirror/autocomplete for the
2
+ // language of one kind of EFX text: the words the lexer reads there as tokens the parser can take
3
+ // (ExpectedTokens), as the toolkit's completion list describes them (CompletionItems); the field and
4
+ // node ids the application gives (efxSymbols) where the parser takes one; and the variables of the
5
+ // text (Variables) where it takes a `$`. The options replace the word before the cursor: a name, a
6
+ // field id with its part in parentheses (BT-26(m)-Lot) or a variable with its `$`. Nothing is offered
7
+ // inside a comment or a string. Template text takes any word, so where it can go the options are
8
+ // offered only when asked for, not while typing.
9
+
10
+ import { syntaxTree } from '@codemirror/language';
11
+ import * as terms from '../efx-parser.terms.js';
12
+ import { efxSymbols } from '../efxSymbols.js';
13
+ import { completionItems } from '../generated/completions.js';
14
+ import { CompletionItems } from './CompletionItems.js';
15
+ import { ExpectedTokens } from './ExpectedTokens.js';
16
+ import { Variables } from './Variables.js';
17
+
18
+ // The word before the cursor: a name, which starts with a letter as the lexer's Identifier does, or a
19
+ // field id with its part in parentheses; with the `$` of a variable or without. A number or a minus
20
+ // starts no word, so typing one opens no list and an option never replaces it.
21
+ const NAME = String.raw`[A-Za-z_][\w-]*`;
22
+ const FIELD_ID_WITH_PART = String.raw`(?:BT|OPP|OPT|OPA)-\d+\((?:BT-\d*|[a-z])?\)?[\w-]*`;
23
+ const WORD = new RegExp(String.raw`\$?(?:${FIELD_ID_WITH_PART}|${NAME})?`);
24
+ const WHOLE_WORD = new RegExp(`^${WORD.source}$`);
25
+
26
+ // The tokens that decide the options besides the words: an expression can start, a field id, a
27
+ // node id, a variable, template text.
28
+ const ASKED = [terms.OpenParenthesis, terms.FieldId, terms.NodeId, terms.VariablePrefix, terms.FreeText];
29
+
30
+ // The nodes inside which nothing is offered.
31
+ const QUIET = new Set(['Comment', 'StringLiteral']);
32
+
33
+ const ITEMS = new CompletionItems(completionItems);
34
+
35
+ export class EfxCompletions {
36
+ #parser;
37
+
38
+ /** The completions of the language a parser reads. */
39
+ constructor(parser) {
40
+ this.#parser = parser;
41
+ }
42
+
43
+ /** The options for the word before the cursor of a CompletionContext, or null when there are none to offer. */
44
+ complete(context) {
45
+ const word = context.matchBefore(WORD);
46
+ if ((word.from === word.to && !context.explicit) || EfxCompletions.#isQuiet(context.state, word.from)) {
47
+ return null;
48
+ }
49
+ const expected = new ExpectedTokens(this.#parser, context.state, word.from, ASKED);
50
+ if (expected.has(terms.FreeText) && !context.explicit) {
51
+ return null;
52
+ }
53
+ const options = [
54
+ ...EfxCompletions.#wordOptions(expected),
55
+ ...EfxCompletions.#symbolOptions(expected, context.state),
56
+ ...EfxCompletions.#variableOptions(expected, context.state, word.from),
57
+ ];
58
+ return { from: word.from, options, validFor: WHOLE_WORD };
59
+ }
60
+
61
+ static #isQuiet(state, pos) {
62
+ const node = syntaxTree(state).resolveInner(pos, -1);
63
+ return QUIET.has(node.name) && node.from < pos;
64
+ }
65
+
66
+ static #wordOptions(expected) {
67
+ const expressionStarts = expected.has(terms.OpenParenthesis);
68
+ return expected.words().map((word) => ITEMS.optionFor(word, expressionStarts));
69
+ }
70
+
71
+ static #symbolOptions(expected, state) {
72
+ const { fields, nodes } = state.facet(efxSymbols);
73
+ const options = (ids, type, detail) => ids.map((id) => ({ label: id, type, detail }));
74
+ return [
75
+ ...(expected.has(terms.FieldId) ? options(fields, 'constant', 'field') : []),
76
+ ...(expected.has(terms.NodeId) ? options(nodes, 'namespace', 'node') : []),
77
+ ];
78
+ }
79
+
80
+ static #variableOptions(expected, state, pos) {
81
+ if (!expected.has(terms.VariablePrefix)) {
82
+ return [];
83
+ }
84
+ return new Variables(state).usableAt(pos).map((name) => ({ label: name, type: 'variable' }));
85
+ }
86
+ }
@@ -0,0 +1,64 @@
1
+ // What the parser can take at a position of an editor's text: the tokens it is asked about and the
2
+ // words the lexer reads there as tokens it can take. The text before the position is parsed as the
3
+ // editor parses it, reusing the editor's tree, with the tokenizer of the lexer's modes wrapped:
4
+ // where a parse stack stands at the position, the wrapper asks it which tokens it can take
5
+ // (Stack.canShift) and finds the lexer's mode there, as the tokenizer does. The parse is strict, so
6
+ // every stack asked is one of the parser's own readings of the text and never one of its recoveries
7
+ // from an error: after an error the parser takes nothing.
8
+ //
9
+ // The parser calls the wrapper only where it can take a token of the lexer's modes or a skip token.
10
+ // In a label block, where nothing is skipped, it can stand where it takes only the grammar's own
11
+ // tokens: after the label type of `#{name|`, where it takes a field id, it takes nothing asked.
12
+
13
+ import { syntaxTree } from '@codemirror/language';
14
+ import { TreeFragment } from '@lezer/common';
15
+ import { ExternalTokenizer } from '@lezer/lr';
16
+ import { lexerModeAt, lexerModes, readToken } from '../lexer/LexerModes.js';
17
+
18
+ export class ExpectedTokens {
19
+ // One for each time a parse stack at the position is tokenized: the words its lexer mode reads
20
+ // as tokens it can take, and the tokens asked about that it can take.
21
+ #readings = [];
22
+
23
+ /** What a language's parser can take at a position of an editor state, asked about some tokens. */
24
+ constructor(parser, state, pos, tokens) {
25
+ const probe = new ExternalTokenizer((input, stack) => {
26
+ if (input.pos === pos) {
27
+ this.#readings.push(ExpectedTokens.#reading(stack, tokens));
28
+ }
29
+ readToken(input, stack);
30
+ }, { contextual: true });
31
+ const parse = parser.configure({ strict: true, tokenizers: [{ from: lexerModes, to: probe }] })
32
+ .startParse(state.doc.sliceString(0, pos), ExpectedTokens.#fragments(state, pos));
33
+ try {
34
+ while (parse.advance() === null) {
35
+ // On to the end of the text, or to the first error.
36
+ }
37
+ } catch (error) {
38
+ if (!(error instanceof SyntaxError)) {
39
+ throw error;
40
+ }
41
+ }
42
+ }
43
+
44
+ /** Whether the parser can take the token there. */
45
+ has(token) {
46
+ return this.#readings.some(({ tokens }) => tokens.has(token));
47
+ }
48
+
49
+ /** The words the lexer reads there as tokens the parser can take, each once. */
50
+ words() {
51
+ return [...new Set(this.#readings.flatMap(({ words }) => words))];
52
+ }
53
+
54
+ static #reading(stack, tokens) {
55
+ const words = lexerModeAt(stack).words().filter(([, token]) => stack.canShift(token)).map(([word]) => word);
56
+ return { words, tokens: new Set(tokens.filter((token) => stack.canShift(token))) };
57
+ }
58
+
59
+ /** The editor's tree before the position, for the parse to reuse. */
60
+ static #fragments(state, pos) {
61
+ const cut = { fromA: pos, toA: state.doc.length, fromB: pos, toB: pos };
62
+ return TreeFragment.applyChanges(TreeFragment.addTree(syntaxTree(state)), [cut]);
63
+ }
64
+ }
@@ -0,0 +1,60 @@
1
+ // The variables declared in an editor's text, read from its tree, and those a position can use. The
2
+ // variable of an iterator is used in the expression that iterates, and a parameter in the
3
+ // declaration that lists it, after their own declaration. Any other variable, declared with LET, in
4
+ // a context or as a dictionary, is taken as usable everywhere: the editor does not follow how far
5
+ // each of those reaches.
6
+
7
+ import { syntaxTree } from '@codemirror/language';
8
+
9
+ const ITERATIONS = ['QuantifiedExpression', 'IterationExpression'];
10
+ const WITH_PARAMETERS = ['FunctionDeclaration', 'DynamicFunctionDeclaration', 'TemplateDeclaration', 'SingleExpression'];
11
+
12
+ // The nodes that declare a variable, each with the nodes where its variable is used; none: everywhere.
13
+ const DECLARATIONS = new Map([
14
+ ['TypedVariableInitializer', []],
15
+ ['DynamicVariableInitializer', []],
16
+ ['ContextVariableInitializer', []],
17
+ ['DictionaryDeclaration', []],
18
+ ['ValueIterator', ITERATIONS],
19
+ ['ContextIterator', ITERATIONS],
20
+ ['ParameterDeclaration', WITH_PARAMETERS],
21
+ ]);
22
+
23
+ export class Variables {
24
+ // Each declared variable: its name with its `$`, and the part of the text that can use it.
25
+ #variables = [];
26
+
27
+ constructor(state) {
28
+ syntaxTree(state).iterate({
29
+ enter: ({ name, node }) => {
30
+ if (DECLARATIONS.has(name)) {
31
+ this.#declare(node, DECLARATIONS.get(name), state.doc);
32
+ }
33
+ },
34
+ });
35
+ }
36
+
37
+ /** The names of the variables usable at a position, each once, with their `$`. */
38
+ usableAt(pos) {
39
+ const usable = ({ from, to }) => from <= pos && pos <= to;
40
+ return [...new Set(this.#variables.filter(usable).map(({ name }) => name))];
41
+ }
42
+
43
+ #declare(declaration, users, doc) {
44
+ const name = declaration.getChild('VariablePrefix')?.nextSibling;
45
+ const usedIn = users.length === 0 ? { from: 0, to: doc.length } : Variables.#usedIn(declaration, users);
46
+ if (name?.name === 'Identifier' && usedIn !== null) {
47
+ this.#variables.push({ name: `$${doc.sliceString(name.from, name.to)}`, ...usedIn });
48
+ }
49
+ }
50
+
51
+ /** From the end of a declaration to the end of the nearest node above it of the given names; null when none is. */
52
+ static #usedIn(declaration, users) {
53
+ for (let node = declaration.parent; node !== null; node = node.parent) {
54
+ if (users.includes(node.name)) {
55
+ return { from: declaration.to, to: node.to };
56
+ }
57
+ }
58
+ return null;
59
+ }
60
+ }