@algosail/lang 0.2.9

package/README.md

@@ -0,0 +1,28 @@
+# @algosail/lang
+
+Sail language tooling. Re-exports parser and typecheck.
+
+## Exports
+
+```javascript
+import { createParser, typecheck } from '@algosail/lang'
+```
+
+## Docs
+
+- [FFI-GUIDE.md](docs/FFI-GUIDE.md) — JSDoc format for FFI modules
+- [ARCHITECTURE.md](docs/ARCHITECTURE.md) — Pipeline, packages, dependencies
+- [TESTING.md](docs/TESTING.md) — Brittle setup, integration tests
+- [CHANGELOG.md](docs/CHANGELOG.md) — Version history
+- [RELEASE.md](docs/RELEASE.md) — Release checklist
+
+## Packages
+
+| Package | Description |
+|---------|-------------|
+| @algosail/parser | Parse Sail + JS → symbol table |
+| @algosail/typecheck | Type checking |
+| @algosail/compiler | Sail → JS (CLI: `sail-compile`) |
+| @algosail/builtins | Builtin words |
+| @algosail/tree-sitter | Grammar |
+| @algosail/lsp | LSP server (hover, completion, go-to-definition) |

package/cli/typecheck.js

@@ -0,0 +1,39 @@
+#!/usr/bin/env node
+import { createParser } from '@algosail/parser'
+import { typecheck } from '@algosail/typecheck'
+import { readFile } from 'node:fs/promises'
+import { pathToFileURL } from 'node:url'
+import { resolve } from 'node:path'
+
+const filePath = process.argv[2]
+
+if (!filePath) {
+  console.error('Usage: sail-typecheck <path-to-file.sail>')
+  process.exit(1)
+}
+
+const absPath = resolve(process.cwd(), filePath)
+const uri = pathToFileURL(absPath).href
+
+let text
+try {
+  text = await readFile(absPath, 'utf8')
+} catch (err) {
+  console.error(`sail-typecheck: cannot read ${filePath}: ${err.message}`)
+  process.exit(1)
+}
+
+const parser = await createParser()
+const result = await parser.parseSail(uri, text)
+const parseErrors = result.errors ?? []
+const typeErrors = typecheck(result)
+const errors = [...parseErrors.map((e) => ({ ...e, message: e.type === 'error' ? `Parse error: unexpected ${JSON.stringify(e.text)}` : `Parse error: missing ${JSON.stringify(e.text)}` })), ...typeErrors]
+
+if (errors.length > 0) {
+  for (const err of errors) {
+    const { row, column } = err.startPosition ?? {}
+    const loc = row != null ? `:${row + 1}:${(column ?? 0) + 1}` : ''
+    console.error(`${filePath}${loc}: ${err.message}`)
+  }
+  process.exit(1)
+}

package/docs/ARCHITECTURE.md

@@ -0,0 +1,50 @@
+# Sail Architecture
+
+## Pipeline
+
+```
+Sail source (.sail)  →  [Parser]  →  Symbol Table  →  [Typecheck]  →  [Compiler]  →  JavaScript
+       ↑                    ↑              ↑
+FFI (.js)  →  [parseJsDoc] ─┘              └── modules, groups, tags, words
+```
+
+1. **Parser** (`@algosail/parser`): Builds symbol table from Sail and JS sources
+2. **Typecheck** (`@algosail/typecheck`): Validates words and tags against signatures
+3. **Compiler** (`@algosail/compiler`): Emits JS from validated symbol table
+
+## Packages
+
+| Package | Role |
+|---------|------|
+| `@algosail/tree-sitter` | Grammar (grammar.js), WASM lexer/parser for Sail |
+| `@algosail/parser` | buildSymbolTable, parseJsDoc, Sail + JS → AST |
+| `@algosail/builtins` | Builtin words (DUP, SWAP, DIP, MATCH, …) with sigs |
+| `@algosail/shared` | getStepSignature, getStepStackEffect (used by typecheck, compiler) |
+| `@algosail/typecheck` | checkWord, checkTag, stack-effect validation |
+| `@algosail/compiler` | compile(), emitWord, emitStep → JS |
+| `@algosail/lsp` | LSP server (validate, completion, hover, go-to-definition) |
+| `@algosail/lang` | Re-exports typecheck, createParser; integration tests |
+
+## Data Flow
+
+- **Symbol table**: `{ modules, groups, tags, words, errors, imports }`
+- **Word**: `{ name, sig, signature, body, doc }` — body is steps (builtin_word, word_ref, quotation, tag, …)
+- **Typecheck** uses `builtins` for builtin signatures; FFI words come from `parseJsDoc` (modules)
+- **Compiler** uses builtins for emit logic; FFI calls go through `emitStep` (CALL to imported module)
+
+## Dependencies
+
+```
+tree-sitter  (grammar)
+    ↓
+parser  (tree-sitter, tree-sitter-javascript, web-tree-sitter)
+    ↓
+builtins  (no deps)
+    ↓
+typecheck  (builtins)
+compiler   (parser, typecheck, builtins)
+lsp        (parser, typecheck, builtins)
+lang       (parser, typecheck, compiler)
+```
+
+**Note:** `tree-sitter` uses regex `/[A-Z][A-Z0-9_]*/` for `builtin_word` — decoupled from builtins (Phase 2.2 done).

package/docs/CHANGELOG.md

@@ -0,0 +1,18 @@
+# Changelog
+
+## [Unreleased]
+
+### Added
+- Integration tests (parse → typecheck → compile → run) in `lang/test/integration.test.js`
+- `npm test` in @algosail/lang
+
+### Changed
+- Documentation: compiler README (source maps, error format, CLI options)
+- Documentation: LSP README (hover, definition)
+- Documentation: TESTING.md (integration tests section)
+
+## [0.2.8] - previous
+- @algosail/lang: sail-typecheck CLI, re-exports typecheck + createParser
+- @algosail/compiler: source maps, JSDoc, async/await, error hints
+- @algosail/lsp: hover, go-to-definition
+- @algosail/typecheck: effects (addEffect, removeEffect)

package/docs/FFI-GUIDE.md

@@ -0,0 +1,65 @@
+# Sail FFI: JSDoc Format
+
+Sail reads type information from JavaScript/JS modules via JSDoc. This document describes the format.
+
+## Overview
+
+FFI modules are JS files with `/** ... @sail ... */` blocks. The parser (`@algosail/parser`) uses `parseJsDoc` to extract groups, tags, and words. Compiled Sail output must emit the same format so it can be used as FFI by other Sail code and JS consumers.
+
+## Word (exported function)
+
+A JSDoc block **immediately before** an `export function` declaration defines a Sail word. The function name becomes the word name when called as `~Module/wordName`.
+
+```javascript
+/**
+ * @sail
+ * @of ( * -- Int ) ( Convert a raw value to an integer )
+ */
+export function of(a) {
+  return parseInt(a, 10)
+}
+```
+
+**Format after `@sail`:**
+- `@functionName ( stack_sig ) ( description )`
+- `stack_sig`: `inputs -- outputs`, e.g. `( * -- Int )`, `( a b -- b a )`
+- `*` = raw value (untyped literal)
+- Types: `Int`, `Str`, `Bool`, `[a]` (list), `( a -- b )` (quotation), `&Group{Int}` (tagged union)
+
+**Full word form** (when the block starts with `@wordName`):
+```
+@wordName ( sig ) ( doc )
+```
+The sail parser treats everything after `@sail` as Sail source and parses it with the Sail grammar (word_def, sig, etc.).
+
+## Groups and Tags
+
+Standalone JSDoc blocks (not before export) can define groups and tags:
+
+```javascript
+/**
+ * @sail
+ * &Maybe a
+ *     #Just a
+ *     #Nothing
+ */
+```
+
+Parsed as Sail: `&Maybe a` (group with type param), `#Just a` and `#Nothing` (tags).
+
+## Extraction Rules
+
+- Comment must start with `/**` and contain `@sail`
+- Content after `@sail` is parsed as Sail source (groups, tags, words)
+- For export pairs: `comment` + `export function name` → word `name` with body from JSDoc
+- If sailText starts with `@wordName` (e.g. `@of`), it's used as-is; else prefixed with `@functionName`
+
+## References
+
+- Parser: `parser/lib/jsDoc.js` — `parseJsDoc`, `findExportSailPairs`, `extractSailBlock`
+- Examples: `zed/test/int.js`, `zed/test/buffer.js`
+- Builtins use a different format (JS objects with `sig`, `signature`, `docs`) — see `builtins/index.js`
+
+## Buffer (Bare/Node)
+
+Для Bare + Hyperswarm + RocksDB: `zed/test/buffer.js` — FFI-модуль с `from`, `alloc`, `concat`.

package/docs/RELEASE.md

@@ -0,0 +1,36 @@
+# Release Checklist
+
+## Pre-release
+
+1. Run all tests:
+   ```bash
+   cd parser && npm test
+   cd ../typecheck && npm test
+   cd ../compiler && npm test
+   cd ../lang && npm test
+   ```
+
+2. Ensure `npm link` for local dev (see TESTING.md)
+
+## Version Bumps (for publish)
+
+Suggested order (dependencies first):
+
+- `@algosail/builtins` — bump if builtin sigs changed
+- `@algosail/shared` — bump if API changed
+- `@algosail/typecheck` — bump if API or behaviour changed
+- `@algosail/parser` — bump if API changed
+- `@algosail/compiler` — bump for source maps, JSDoc, async, error format
+- `@algosail/lsp` — bump for hover, definition
+- `@algosail/lang` — bump last; update deps to new versions
+
+## Publish
+
+```bash
+cd <package> && npm publish --access public
+```
+
+## Post-release
+
+- Update CHANGELOG.md with version and date
+- Tag release if using git tags

package/docs/TESTING.md

@@ -0,0 +1,86 @@
+# Sail Testing
+
+## Framework: Brittle
+
+Sail packages use [brittle](https://github.com/holepunchto/brittle) for tests.
+
+### Setup
+
+```json
+{
+  "scripts": {
+    "test": "brittle \"test/**/*.test.js\""
+  },
+  "devDependencies": {
+    "brittle": "^3.19.1"
+  }
+}
+```
+
+### Test File Pattern
+
+- Location: `test/**/*.test.js`
+- Import: `import test from 'brittle'`
+
+### Example
+
+```javascript
+import test from 'brittle'
+import { checkWord } from '../lib/word/checkWord.js'
+
+test('word step applies callee signature', (t) => {
+  const table = baseTable()
+  table.words.toInt = { ... }
+  const word = { ... }
+  t.alike(checkWord(word, table), [])
+})
+
+test('DUP builtin applies sig', (t) => {
+  // ...
+  t.alike(checkWord(word, table), [])
+})
+```
+
+### Assertions
+
+- `t.ok(value)` — truthy
+- `t.absent(value)` — falsy
+- `t.alike(actual, expected)` — deep equality
+- `t.is(actual, expected)` — strict equality
+
+### Running Tests
+
+```bash
+# In each package
+cd typecheck && npm test
+cd parser && npm test
+cd compiler && npm test
+```
+
+### Local Package Linking (npm link)
+
+Для локальной разработки связывай пакеты через `npm link`:
+
+```bash
+# 1. В пакете-источнике (shared, builtins и т.д.)
+cd shared && npm link
+
+# 2. В пакете-потребителе (compiler, typecheck и т.д.)
+cd ../compiler && npm link @algosail/shared
+cd ../typecheck && npm link @algosail/shared
+```
+
+Проверка: `ls -la node_modules/@algosail/shared` должен быть symlink.
+
+### Integration Tests
+
+Full pipeline (parse → typecheck → compile → run) is tested in `lang/test/integration.test.js`:
+
+```bash
+cd lang && npm test
+```
+
+Tests:
+- Compile a simple program (builtins only), run via dynamic import, assert result
+- Compile program with FFI (async_main.sail), run, assert result
+- Invalid program (stack underflow) produces errors

package/index.js

@@ -0,0 +1,2 @@
+export { typecheck } from '@algosail/typecheck'
+export { createParser } from '@algosail/parser'

package/package.json

@@ -0,0 +1,21 @@
+{
+  "name": "@algosail/lang",
+  "version": "0.2.9",
+  "type": "module",
+  "main": "index.js",
+  "bin": {
+    "sail-typecheck": "cli/typecheck.js"
+  },
+  "scripts": {
+    "test": "brittle \"test/**/*.test.js\""
+  },
+  "devDependencies": {
+    "brittle": "^3.19.1"
+  },
+  "dependencies": {
+    "@algosail/builtins": "^0.0.6",
+    "@algosail/compiler": "^0.0.2",
+    "@algosail/parser": "^0.1.1",
+    "@algosail/typecheck": "^0.1.1"
+  }
+}

package/test/integration.test.js

@@ -0,0 +1,61 @@
+/**
+ * Integration tests: parse → typecheck → compile → run
+ */
+
+import test from 'brittle'
+import { compile } from '@algosail/compiler'
+import { writeFile, unlink, readFile } from 'node:fs/promises'
+import { tmpdir } from 'node:os'
+import { join, dirname } from 'node:path'
+import { pathToFileURL, fileURLToPath } from 'node:url'
+import { randomUUID } from 'node:crypto'
+
+const __dirname = dirname(fileURLToPath(import.meta.url))
+
+test('full pipeline: compile simple program and run', async (t) => {
+  const source = `
+@main ( Int -- Int )
+  DUP NIP
+`
+  const uri = 'file:///test/integration.sail'
+  const { js, errors } = await compile(uri, source)
+
+  t.is(errors.length, 0, `expected no errors, got: ${JSON.stringify(errors)}`)
+  t.ok(js)
+  t.ok(js.includes('export function main'))
+
+  const tmpFile = join(tmpdir(), `sail-integration-${randomUUID()}.mjs`)
+  await writeFile(tmpFile, js, 'utf8')
+  try {
+    const mod = await import(pathToFileURL(tmpFile).href)
+    const result = mod.main(42)
+    t.is(result, 42, 'main(42) should return 42 (DUP NIP)')
+  } finally {
+    await unlink(tmpFile)
+  }
+})
+
+test('full pipeline: compile program with FFI', async (t) => {
+  const fixturesDir = join(__dirname, '..', '..', 'compiler', 'test', 'fixtures')
+  const uri = pathToFileURL(join(fixturesDir, 'async_main.sail')).href
+  const source = await readFile(join(fixturesDir, 'async_main.sail'), 'utf8')
+
+  const { js, errors } = await compile(uri, source, { outPath: fixturesDir })
+
+  t.is(errors.length, 0, `expected no errors, got: ${JSON.stringify(errors)}`)
+  t.ok(js, 'should emit JS')
+  t.ok(js.includes('export function main') || js.includes('export async function main'), 'should export main')
+  t.ok(/Async\.incAsync|incAsync/.test(js), 'should call FFI word')
+})
+
+test('full pipeline: typecheck rejects invalid program', async (t) => {
+  const source = `
+@main ( Int -- Int )
+  DUP DROP POP
+`
+  const uri = 'file:///test/invalid.sail'
+  const { js, errors } = await compile(uri, source)
+
+  t.ok(errors.length > 0, 'should have typecheck/stack errors')
+  t.ok(!js || errors.length > 0)
+})