yukigo-mini-parser 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/tutorial.md ADDED
@@ -0,0 +1,753 @@
1
+ # Quickstart: How to make a Yukigo parser
2
+
3
+ In this tutorial we will cover some basics on making a parser compatible with the yukigo analyzer.
4
+
5
+ By the end of this tutorial, you will be able to:
6
+ - Set up a nearley-based parser with TypeScript
7
+ - Design a lexer using `moo`
8
+ - Write grammar rules with proper operator precedence and associativity
9
+ - Use postprocessors to build a Yukigo-compatible AST
10
+ - Implement support for variables, functions, lists, conditionals, and loops
11
+ - Write and run unit tests for your parser
12
+
13
+ First, let's check what yukigo expects from a parser.
14
+
15
+ ```ts
16
+ interface YukigoParser {
17
+ errors?: string[];
18
+ parse: (code: string) => AST;
19
+ }
20
+ ```
21
+
22
+ Every parser needs to expose a `parse` method and can also include an errors array. The internal checks or functionality isn't important to Yukigo. For example, if the language is typed, we could add a type checker step before returning the parsed AST.
23
+
24
+ For this tutorial we will be implementing a parser for a subset of [mini-lang](https://github.com/mini-lang/mini-lang). We will cover expressions, statements, control flow, functions, etc.
25
+
26
+ We will be using [nearley.js](https://nearley.js.org/) as our parser generator, but feel free to use any tool you find convenient.
27
+
28
+ Let's start a new Typescript project and install all the packages we need.
29
+
30
+ ```sh
31
+ > mkdir yukigo-mini-parser
32
+ > npm init -y
33
+ > npm i -D typescript ts-node @types/node @types/chai @types/mocha chai mocha
34
+ > npm i -g nearley
35
+ > npm i nearley moo @types/moo moo-ignore yukigo-core
36
+ > mkdir src
37
+ > touch src/index.ts
38
+ > tsc --init
39
+ ```
40
+
41
+ For the `tsconfig.json` file this will be the configuration:
42
+
43
+ ```json
44
+ {
45
+ "compilerOptions": {
46
+ "target": "ES2024",
47
+ "module": "NodeNext",
48
+ "moduleResolution": "NodeNext",
49
+ "lib": ["ES2024"],
50
+ "outDir": "./dist",
51
+ "declaration": true,
52
+ "esModuleInterop": true,
53
+ "allowSyntheticDefaultImports": true,
54
+ "sourceMap": true,
55
+ "forceConsistentCasingInFileNames": true,
56
+ "skipLibCheck": true
57
+ },
58
+ "include": ["src"],
59
+ "exclude": ["dist/**/*"]
60
+ }
61
+ ```
62
+
63
+ And in `package.json` we will have this
64
+ ```json
65
+ {
66
+ "name": "yukigo-mini-parser",
67
+ "version": "1.0.0",
68
+ "description": "",
69
+ "scripts": {
70
+ "build": "nearleyc src/grammar.ne -o src/grammar.ts && tsc",
71
+ "test": "npm run build && mocha"
72
+ },
73
+ "type": "module",
74
+ "main": "dist/index.js",
75
+ "types": "dist/index.d.ts",
76
+ "exports": {
77
+ ".": "./dist/index.js",
78
+ "./package.json": "./package.json"
79
+ },
80
+ "keywords": [],
81
+ "author": "",
82
+ "license": "ISC",
83
+ "devDependencies": {
84
+ "@types/chai": "^5.2.2",
85
+ "@types/mocha": "^10.0.10",
86
+ "@types/node": "^24.7.0",
87
+ "chai": "^6.2.0",
88
+ "mocha": "^11.7.4",
89
+ "ts-node": "^10.9.2",
90
+ "typescript": "^5.9.2"
91
+ },
92
+ "dependencies": {
93
+ "@types/moo": "^0.5.10",
94
+ "moo": "^0.5.2",
95
+ "moo-ignore": "^2.5.3",
96
+ "nearley": "^2.20.1",
97
+ "yukigo-core": "file:../yukigo-core"
98
+ }
99
+ }
100
+ ```
101
+
102
+ So let's start our parser as a class that implements `YukigoParser`.
103
+ In our `src/index.ts`
104
+
105
+ ```ts
106
+ import { YukigoParser } from "yukigo-core";
107
+
108
+ export class YukigoMiniParser implements YukigoParser {
109
+ public errors: string[] = [];
110
+
111
+ public parse(code: string) {
112
+ return [];
113
+ }
114
+ }
115
+ ```
116
+
117
+ Great! We have a base to work on. Now let's work on the lexer, this is the component reponsible for lexical tokenization, the process where a string is converted to meaningful tokens.
118
+
119
+ Let's create a `src/lexer.ts` file and define the meaningful tokens in our language:
120
+ ```ts
121
+ import moo from "moo";
122
+ import { makeLexer } from "moo-ignore";
123
+
124
+ const keywords = []
125
+
126
+ export const MiniLexerConfig = {
127
+ EOF: "*__EOF__*",
128
+ wildcard: "_",
129
+ WS: /[ \t]+/,
130
+ comment: /--.*?$|{-[\s\S]*?-}/,
131
+ number:
132
+ /0[xX][0-9a-fA-F]+|0[bB][01]+|0[oO][0-7]+|(?:\d*\.\d+|\d+)(?:[eE][+-]?\d+)?/,
133
+ char: /'(?:\\['\\bfnrtv0]|\\u[0-9a-fA-F]{4}|[^'\\\n\r])?'/,
134
+ string: /"(?:\\["\\bfnrtv0]|\\u[0-9a-fA-F]{4}|[^"\\\n\r])*"/,
135
+ bool: {
136
+ match: ["True", "False"],
137
+ },
138
+ semicolon: ";",
139
+ assign: ":=",
140
+ variable: {
141
+ match: /[a-z_][a-zA-Z0-9_']*/,
142
+ type: moo.keywords({
143
+ keyword: keywords,
144
+ }),
145
+ },
146
+ NL: { match: /\r?\n/, lineBreaks: true },
147
+ };
148
+
149
+ export const MiniLexer = makeLexer(MiniLexerConfig, [], {
150
+ eof: true,
151
+ });
152
+ ```
153
+ As you can see, we have defined primitive tokens like `number`, `string`, `char`, `bool`. We also defined how a `variable` looks like and some `keywords`. The tokens for whitespace (`WS`), newline (`NL`), end-of-file (`EOF`) will be useful when designing our grammar.
154
+
155
+ > We need to define every token that our lexer should expect. That's why we defined `assign` and `semicolon`
156
+
157
+ Now let's start with the grammar file.
158
+ Make a `src/grammar.ne` file and add this boilerplate for now
159
+
160
+ ```nearley
161
+ @{%
162
+ import { MiniLexer } from "./lexer.js"
163
+ %}
164
+
165
+ @preprocessor typescript
166
+ @lexer MiniLexer
167
+
168
+ program -> %WS {% (d) => d %}
169
+
170
+ _ -> %WS:* {% d => null %}
171
+ __ -> %WS:+ {% d => null %}
172
+ ```
173
+
174
+ > `_` and `__` are rules to match zero-or-more and one-or-more whitespaces.
175
+
176
+ > nearley.js allows us to use [EBNF](https://en.wikipedia.org/wiki/Extended_Backus%E2%80%93Naur_form) operators `:+`, `:*`, `:?`
177
+
178
+
179
+ Let's start small by supporting variable assignment, we want to be able to assign and declare variables like this
180
+ ```
181
+ int x := 10;
182
+ ```
183
+ So we need to add multiple things first. As we see, the assignment statement is composed of a `type` a `variable` and an optional `expression` (we want to support `int x;` also)
184
+
185
+ Let's modify our `program` rule and add a `statement` rule that we can later expand
186
+ ```ne
187
+ program -> statement:+ _ %EOF
188
+
189
+ statement -> assignment _ ";" _
190
+
191
+ assignment -> type __ variable (_ ":=" _ expression):?
192
+ ```
193
+
194
+ Let's continue with the expressions. An expression is a syntactic notation that can be evaluated to get its value. So we will need to add some more rules to support arithmetic binary operations and primitive values.
195
+
196
+ ```ne
197
+ # Below our current code
198
+
199
+ expression -> addition
200
+
201
+ addition ->
202
+ addition _ "+" _ multiplication
203
+ | addition _ "-" _ multiplication
204
+ | multiplication
205
+
206
+ multiplication ->
207
+ multiplication _ "*" _ primary
208
+ | multiplication _ "/" _ primary
209
+ | primary
210
+
211
+ primary ->
212
+ variable
213
+ | "(" _ expression _ ")"
214
+ | primitive
215
+
216
+ # ...
217
+
218
+ primitive ->
219
+ %number
220
+ | variable
221
+ | %char
222
+ | %string
223
+ | %bool
224
+
225
+ variable -> %variable {% (d) => new SymbolPrimitive(d[0].value) %}
226
+ ```
227
+ As you may have notice reading these new rules, they are recursive. This allows to have expressions like `1 + 1 + 1` which parse like `(1 + 1) + 1`. Because the recursive [non-terminal](https://en.wikipedia.org/wiki/Terminal_and_nonterminal_symbols) appears as the leftmost symbol in the rule, we can say that the rule is left recursive which help us build the left [associativity](https://en.wikipedia.org/wiki/Operator_associativity)
228
+
229
+ > If we define these rules with right associativity we would have errors in the evaluation of operations like `5 − 3 − 2` where the parser would parse as `5 − (3 − 2)` which later would wrongly evaluate to `5 − 1 = 4` instead of `2 − 2 = 0`
230
+
231
+ Also notice that we defined a `primary` rule that serves the purpose of being the base case for recursion and the highest precedence expressions
232
+
233
+ Good! Finally for this first statement we will implement a simple type rule that for now it's enough.
234
+
235
+ ```ne
236
+ # ...
237
+ type -> variable
238
+
239
+ variable -> %variable
240
+ ```
241
+
242
+ Now let's add the post processing of these rules. We want our parser to produce certain output after matching a rule. For that we can use a syntax that nearley provides. Let's use the `variable` rule for example
243
+
244
+ ```ne
245
+ variable -> %variable {% (d) => ... %}
246
+ ```
247
+
248
+ We can define JavaScript/TypeScript code inside {%%} after a rule. These are called [postprocessors](https://nearley.js.org/docs/grammar#postprocessors) and the `d` argument is an array with the symbols matched.
249
+
250
+ ```ne
251
+ variable -> %variable {% (d) => new SymbolPrimitive(d[0].value) %}
252
+ ```
253
+
254
+ We access the `%variable` symbol with `d[0]` and then it's value from the `moo` lexer token. But wait... What is `SymbolPrimitive`?
255
+
256
+ Yukigo provides a collection of AST nodes to build your parser quicker and yukigo compatible. `SymbolPrimitive` is the node that represents symbols like variables.
257
+
258
+ > You can check the Yukigo's AST reference here: TODO DOCS
259
+
260
+ The output that the rule returns will be available for other rules that use the non-terminal. For example:
261
+
262
+ ```ne
263
+ type -> variable {% (d) => new SimpleType(d[0], []) %}
264
+ ```
265
+
266
+ We do not need to instantiate another `SymbolPrimitive` we just access it by it's position in the rule. Now let's use the available yukigo's nodes to process all of our rules.
267
+
268
+ ```ne
269
+ @{%
270
+ import { MiniLexer } from "./lexer.js"
271
+ import {
272
+ SimpleType,
273
+ Assignment,
274
+ Variable,
275
+ ArithmeticBinaryOperation,
276
+ SymbolPrimitive,
277
+ NumberPrimitive,
278
+ BooleanPrimitive,
279
+ StringPrimitive,
280
+ CharPrimitive,
281
+ NilPrimitive
282
+ } from "yukigo-core"
283
+ %}
284
+
285
+ @preprocessor typescript
286
+ @lexer MiniLexer
287
+
288
+ program -> statement:+ _ %EOF {% (d) => d[0].flat(Infinity) %}
289
+
290
+ statement -> assignment _ ";" _ {% (d) => d[0] %}
291
+
292
+ assignment -> type __ variable (_ ":=" _ expression):? {% (d) => new Variable(d[2], d[3] ? d[3][3] : new NilPrimitive(null), d[0]) %}
293
+
294
+ expression -> addition {% (d) => d[0] %}
295
+
296
+ addition ->
297
+ addition _ "+" _ multiplication {% (d) => new ArithmeticBinaryOperation("Plus", d[0], d[4]) %}
298
+ | addition _ "-" _ multiplication {% (d) => new ArithmeticBinaryOperation("Minus", d[0], d[4]) %}
299
+ | multiplication {% (d) => d[0] %}
300
+
301
+ multiplication ->
302
+ multiplication _ "*" _ primary {% (d) => new ArithmeticBinaryOperation("Multiply", d[0], d[4]) %}
303
+ | multiplication _ "/" _ primary {% (d) => new ArithmeticBinaryOperation("Divide", d[0], d[4]) %}
304
+ | primary {% (d) => d[0] %}
305
+
306
+ primary ->
307
+ variable {% (d) => d[0] %}
308
+ | "(" _ expression _ ")" {% (d) => d[2] %}
309
+ | primitive {% (d) => d[0] %}
310
+
311
+ primitive ->
312
+ %number {% (d) => new NumberPrimitive(Number(d[0].value)) %}
313
+ | %char {% (d) => new CharPrimitive(d[0].value) %}
314
+ | variable {% (d) => d[0] %}
315
+ | %string {% (d) => new StringPrimitive(d[0].value) %}
316
+ | %bool {% (d) => new BooleanPrimitive(d[0].value) %}
317
+
318
+ type -> variable {% (d) => new SimpleType(d[0].value, []) %}
319
+
320
+ variable -> %variable {% (d) => new SymbolPrimitive(d[0].value) %}
321
+
322
+ _ -> %WS:* {% d => null %}
323
+ __ -> %WS:+ {% d => null %}
324
+ ```
325
+
326
+ > As you may have notice, in the `program` rule we use `.flat(Infinity)` this is because some rules (like `function_statement`) return multiple top-level declarations. We flatten the result to produce a flat list of statements.
327
+
328
+ Excellent! We need to load the compiled grammar into our `YukigoParser` class, where we will also add some error handling
329
+ ```ts
330
+ import { YukigoParser } from "yukigo-core";
331
+ import nearley from "nearley";
332
+ import grammar from "./grammar.js";
333
+
334
+ export class YukigoMiniParser implements YukigoParser {
335
+ public errors: string[] = [];
336
+
337
+ public parse(code: string) {
338
+ const parser = new nearley.Parser(nearley.Grammar.fromCompiled(grammar));
339
+ try {
340
+ parser.feed(code);
341
+ parser.finish();
342
+ } catch (error) {
343
+ const token = error.token;
344
+ const message = `Unexpected '${token.type}' token '${token.value}' at line ${token.line} col ${token.col}.`;
345
+ this.errors.push(message)
346
+ }
347
+ const results = parser.results
348
+ if(results.length > 1)
349
+ throw Error(`Ambiguous grammar. The parser generated ${results} ASTs`)
350
+
351
+ return results[0]
352
+ }
353
+ }
354
+ ```
355
+
356
+ > We need to ensure our grammar only produces one AST, nearley returns all possible ASTs so we need to throw in case that the parser returns more than one.
357
+
358
+ Let's use `mocha` and `chai` to write a test in `tests/parser.spec.ts`
359
+
360
+ ```ts
361
+ import { assert } from "chai";
362
+ import {
363
+ NilPrimitive,
364
+ SimpleType,
365
+ SymbolPrimitive,
366
+ Variable,
367
+ YukigoParser,
368
+ } from "yukigo-core";
369
+ import { YukigoMiniParser } from "../src/index.js";
370
+
371
+ describe("Parser Tests", () => {
372
+ let parser: YukigoParser;
373
+ beforeEach(() => {
374
+ parser = new YukigoMiniParser();
375
+ });
376
+
377
+ it("should parse assignment", () => {
378
+ const code = `int n; int result;`;
379
+ assert.deepEqual(parser.parse(code), [
380
+ new Variable(
381
+ new SymbolPrimitive("n"),
382
+ new NilPrimitive(null),
383
+ new SimpleType("int", [])
384
+ ),
385
+ new Variable(
386
+ new SymbolPrimitive("result"),
387
+ new NilPrimitive(null),
388
+ new SimpleType("int", [])
389
+ ),
390
+ ]);
391
+ });
392
+ });
393
+ ```
394
+
395
+ ```
396
+ Parser Tests
397
+ ✔ should parse assignment
398
+ ```
399
+
400
+ Excellent! We have our first feature implemented with the test running
401
+
402
+ Now let's build some more advaced features
403
+
404
+ ## Functions
405
+
406
+ We want to add support for functions like this
407
+ ```
408
+ int add(int x, int y) {
409
+ int result := x + y;
410
+ return result;
411
+ };
412
+ int three := add(1, 2);
413
+ ```
414
+ We see that the function `add` is a `Procedure` with one `Equation` that has two `VariablePattern` and an `UnguardedBody` with a `Sequence` of two statements: `Variable` and `Return`.
415
+
416
+ So let's start with the rule for the function statement.
417
+
418
+ ```
419
+ # ...
420
+ function_statement -> type __ variable ("(" _ param_list:? _ ")" _ "{" _ body _ "}") {% (d) => {
421
+ const paramTypeList = []
422
+ const patternList = []
423
+ if(d[3][2]) {
424
+ for(const [paramType, paramPattern] of d[3][2]) {
425
+ paramTypeList.push(paramType);
426
+ patternList.push(paramPattern);
427
+ }
428
+ }
429
+ const signatureType = new ParameterizedType(paramTypeList, d[0], [])
430
+
431
+ const signature = new TypeSignature(d[2], signatureType);
432
+ const procedure = new Procedure(d[2], [new Equation(patternList, d[3][8])])
433
+ return [signature, procedure]
434
+ }%}
435
+
436
+ param_list -> param (_ "," _ param):* {% d => [d[0], ...d[1].map(x => x[3])] %}
437
+
438
+ param -> type __ variable {% d => [d[0], new VariablePattern(d[2])] %}
439
+
440
+ body -> statement:* {% (d) => new UnguardedBody(new Sequence(d[0])) %}
441
+ # ...
442
+ ```
443
+
444
+ > Notice that, we also add a `TypeSignature` node which represents the signature of a function. In `paramTypeList` we collect the types of each argument to later add it to the `inputs` of the `ParameterizedType`.
445
+
446
+ Also let's add a return statement
447
+
448
+ ```
449
+ return_statement -> "return" _ expression {% (d) => new Return(d[2]) %}
450
+ ```
451
+
452
+ And finally let's add them to our statement rule
453
+ ```
454
+ statement -> (assignment | function_statement | return_statement) _ ";" _ {% (d) => d[0][0] %}
455
+ ```
456
+
457
+ Finally let's add a test that validates this behaviour
458
+
459
+ ```ts
460
+ it("should parse function declaration", () => {
461
+ const code = `int add(int x, int y) {
462
+ int result := x + y;
463
+ return result;
464
+ };`;
465
+ assert.deepEqual(parser.parse(code), [
466
+ new TypeSignature(
467
+ new SymbolPrimitive("add"),
468
+ new ParameterizedType(
469
+ [new SimpleType("int", []), new SimpleType("int", [])],
470
+ new SimpleType("int", []),
471
+ []
472
+ )
473
+ ),
474
+ new Procedure(new SymbolPrimitive("add"), [
475
+ new Equation(
476
+ [
477
+ new VariablePattern(new SymbolPrimitive("x")),
478
+ new VariablePattern(new SymbolPrimitive("y")),
479
+ ],
480
+ new UnguardedBody(
481
+ new Sequence([
482
+ new Variable(
483
+ new SymbolPrimitive("result"),
484
+ new ArithmeticBinaryOperation(
485
+ "Plus",
486
+ new SymbolPrimitive("x"),
487
+ new SymbolPrimitive("y")
488
+ ),
489
+ new SimpleType("int", [])
490
+ ),
491
+ new Return(new SymbolPrimitive("result")),
492
+ ])
493
+ )
494
+ ),
495
+ ]),
496
+ ]);
497
+ });
498
+ ```
499
+
500
+ Hopefully that will give us
501
+ ```
502
+ Parser Tests
503
+ ✔ should parse assignment
504
+ ✔ should parse function declaration
505
+ ```
506
+ It's a pretty simple workflow, if you like you could even make the tests first so you already have expectations setted for your grammar.
507
+
508
+ ## Collection Primitive
509
+
510
+ We are missing a key primitive though.
511
+
512
+ ```
513
+ int[] numberList := [1, 2, 3 + 4];
514
+ ```
515
+
516
+ This is not hard to implement. We need to define a primary rule `list` and use `ListPrimitive` node from `yukigo-core`. Let's add it
517
+
518
+ First, let's think about the test. We expect the example above to parse as a `Variable` with expression `ListPrimitive` and type `ListType` which has int for it's elements. We might think of something like this:
519
+
520
+ ```ts
521
+ it("should parse list primitive", () => {
522
+ const code = `int[] numberList := [1, 2, 3 + 4];`;
523
+ assert.deepEqual(parser.parse(code), [
524
+ new Variable(
525
+ new SymbolPrimitive("numberList"),
526
+ new ListPrimitive([
527
+ new NumberPrimitive(1),
528
+ new NumberPrimitive(2),
529
+ new ArithmeticBinaryOperation(
530
+ "Plus",
531
+ new NumberPrimitive(3),
532
+ new NumberPrimitive(4)
533
+ ),
534
+ ]),
535
+ new ListType(new SimpleType("int", []), [])
536
+ ),
537
+ ]);
538
+ });
539
+ ```
540
+ Let's modify our type rule to support this new list type
541
+ ```
542
+ type ->
543
+ variable {% (d) => new SimpleType(d[0].value, []) %}
544
+ | type "[" "]" {% (d) => new ListType(d[0], []) %}
545
+ ```
546
+ Good! Now for the expression we have something like
547
+ ```
548
+ list_primitive -> "[" _ expression_list _ "]" {% (d) => new ListPrimitive(d[2]) %}
549
+
550
+ expression_list -> expression _ ("," _ expression _):* {% (d) => ([d[0], ...d[2].map(x => x[2])]) %}
551
+ ```
552
+ So lastly we add the `list_primitive` rule to the `primitive` rule
553
+ ```
554
+ primitive ->
555
+ %number {% (d) => new NumberPrimitive(Number(d[0].value)) %}
556
+ | %char {% (d) => new CharPrimitive(d[0].value) %}
557
+ | %string {% (d) => new StringPrimitive(d[0].value) %}
558
+ | %bool {% (d) => new BooleanPrimitive(d[0].value) %}
559
+ | variable {% (d) => d[0] %}
560
+ | list_primitive {% d => d[0] %}
561
+ ```
562
+
563
+ Let's run the tests and check if we got it right
564
+ ```
565
+ Parser Tests
566
+ ✔ should parse assignment
567
+ ✔ should parse function declaration
568
+ ✔ should parse list primitive
569
+ ```
570
+
571
+
572
+ ## Control Flow: If & While statements
573
+
574
+ Finally, we want our language to have if statements and while loops. So we will need to implement some rules that produce `If` and `While` nodes
575
+
576
+ We expect something like this for if statements
577
+ ```ts
578
+ it("should parse if statement", () => {
579
+ const code = `if(a != b) { c := a + b; } else { c := a * 2; };`;
580
+ assert.deepEqual(parser.parse(code), [
581
+ new If(
582
+ new ComparisonOperation(
583
+ "NotEqual",
584
+ new SymbolPrimitive("a"),
585
+ new SymbolPrimitive("b")
586
+ ),
587
+ new Sequence([
588
+ new Assignment(
589
+ new SymbolPrimitive("c"),
590
+ new ArithmeticBinaryOperation(
591
+ "Plus",
592
+ new SymbolPrimitive("a"),
593
+ new SymbolPrimitive("b")
594
+ )
595
+ ),
596
+ ]),
597
+ new Sequence([
598
+ new Assignment(
599
+ new SymbolPrimitive("c"),
600
+ new ArithmeticBinaryOperation(
601
+ "Multiply",
602
+ new SymbolPrimitive("a"),
603
+ new NumberPrimitive(2)
604
+ )
605
+ ),
606
+ ]),
607
+ ),
608
+ ]);
609
+ });
610
+ });
611
+ ```
612
+
613
+ First, let's add the `!=`, `==`, `*`, `if`, and `else` tokens to our lexer.
614
+ ```ts
615
+ // ...
616
+ const keywords = [
617
+ "return",
618
+ "if",
619
+ "else"
620
+ ]
621
+
622
+ export const MiniLexerConfig = {
623
+ // other tokens
624
+ semicolon: ";",
625
+ assign: ":=",
626
+ notEqual: "!=",
627
+ equal: "==",
628
+ // other tokens
629
+ operator: /\+|\*/,
630
+ variable: {
631
+ match: /[a-z_][a-zA-Z0-9_']*/,
632
+ type: moo.keywords({
633
+ keyword: keywords,
634
+ }),
635
+ },
636
+ NL: { match: /\r?\n/, lineBreaks: true },
637
+ };
638
+ // ...
639
+ ```
640
+
641
+ Now, let's define the production for the if statement
642
+
643
+ ```
644
+ if_statement -> "if" _ condition _ statement_list _ "else" _ statement_list {% d => new If(d[2], d[4], d[8]) %}
645
+
646
+ condition -> "(" _ expression _ ")" {% (d) => d[2] %}
647
+
648
+ statement_list -> "{" _ statement:* _ "}" {% d => new Sequence(d[2]) %}
649
+ ```
650
+
651
+ Good, now we are missing the rule that produces `ComparisonOperation` nodes. Let's modify the operations section to add comparison **before** arithmetic operations
652
+
653
+ ```
654
+ expression -> comparison {% (d) => d[0] %}
655
+
656
+ comparison ->
657
+ addition _ comparison_operator _ addition {% (d) => new ComparisonOperation(d[2], d[0], d[4]) %}
658
+ | addition {% (d) => d[0] %}
659
+
660
+ addition ->
661
+ addition _ "+" _ multiplication {% (d) => new ArithmeticBinaryOperation("Plus", d[0], d[4]) %}
662
+ | addition _ "-" _ multiplication {% (d) => new ArithmeticBinaryOperation("Minus", d[0], d[4]) %}
663
+ | multiplication {% (d) => d[0] %}
664
+
665
+ multiplication ->
666
+ multiplication _ "*" _ primary {% (d) => new ArithmeticBinaryOperation("Multiply", d[0], d[4]) %}
667
+ | multiplication _ "/" _ primary {% (d) => new ArithmeticBinaryOperation("Divide", d[0], d[4]) %}
668
+ | primary {% (d) => d[0] %}
669
+
670
+ # the rest of the grammar
671
+
672
+ comparison_operator ->
673
+ %equal {% d => "Equal" %}
674
+ | %notEqual {% d => "NotEqual" %}
675
+
676
+ ```
677
+
678
+ The `comparison_operator` rule helps us assign semantic value to the operation by the operator we received from the lexer.
679
+
680
+ Also we are missing assignment statements so we can add in our statement rule
681
+
682
+ ```
683
+ assignment -> variable _ ":=" _ expression {% (d) => new Assignment(d[0], d[4]) %}
684
+ ```
685
+
686
+ And that should work, let's now implement `while` loops which have a `condition` and a `body` to parse something like this
687
+
688
+ ```ts
689
+ it("should parse while loop statement", () => {
690
+ const code = `while(a < 10) { a := a + 1; };`;
691
+ assert.deepEqual(parser.parse(code), [
692
+ new While(
693
+ new ComparisonOperation(
694
+ "LessThan",
695
+ new SymbolPrimitive("a"),
696
+ new NumberPrimitive(10)
697
+ ),
698
+ new Sequence([
699
+ new Assignment(
700
+ new SymbolPrimitive("a"),
701
+ new ArithmeticBinaryOperation(
702
+ "Plus",
703
+ new SymbolPrimitive("a"),
704
+ new NumberPrimitive(1)
705
+ )
706
+ ),
707
+ ]),
708
+ ),
709
+ ]);
710
+ });
711
+ ```
712
+
713
+ In the lexer we need to add
714
+ ```ts
715
+ export const MiniLexerConfig = {
716
+ // other tokens
717
+ gte: ">=",
718
+ gt: ">",
719
+ lte: "<=",
720
+ lt: "<",
721
+ // other tokens
722
+ };
723
+ ```
724
+
725
+ > Notice the order in which we defined these tokens. If the `gt` token were to be before the `gte` token, then the lexer would return the token `gt` even if it read `>=`. This is called [maximal munch](https://en.wikipedia.org/wiki/Maximal_munch) or longest match principle.
726
+
727
+ Let's update our `comparison_operator` rule with these new operators
728
+
729
+ ```
730
+ comparison_operator ->
731
+ %equal {% d => "Equal" %}
732
+ | %notEqual {% d => "NotEqual" %}
733
+ | %lt {% d => "LessThan" %}
734
+ | %lte {% d => "LessOrEqualThan" %}
735
+ | %gt {% d => "GreaterThan" %}
736
+ | %gte {% d => "GreaterOrEqualThan" %}
737
+ ```
738
+
739
+ And now define the `while_statement` rule reusing the condition and body from the `if_statement` rule
740
+
741
+ ```
742
+ while_statement -> "while" _ condition _ statement_list {% d => new While(d[2], d[4]) %}
743
+ ```
744
+
745
+ Finally we should have all tests working
746
+ ```
747
+ Parser Tests
748
+ ✔ should parse assignment
749
+ ✔ should parse function declaration
750
+ ✔ should parse list primitive
751
+ ✔ should parse if statement
752
+ ✔ should parse while loop statement
753
+ ```