miki-template 1.2.0

package/README.md

@@ -0,0 +1,304 @@
+# miki-template
+![npm version](https://img.shields.io/npm/v/miki-template.svg) ![CI](https://github.com/your-repo/miki-template/workflows/ci.yml/badge.svg)
+A robust, production-ready template engine that brings **Django's template language** features and syntax to Node.js and Express, fully compliant with modern JavaScript (ES6+), CommonJS, and **ESM** (`import`) support.
+
+---
+
+## 🚀 Features
+
+- **Full Syntax Parity**: Supports variables, dotted lookups, filters (`|`), and block tags (`{% %}`).
+- **Template Inheritance**: Multi-level inheritance with `extends`, block overrides, and `{{ block.super }}` support.
+- **Express Integration**: Simple, zero-config integration via `app.engine()`.
+- **ESM & CommonJS**: Works seamlessly with both `import` and `require` syntax.
+- **Security by Default**: Auto-escaping enabled by default with a `SafeString` wrapper.
+- **CSRF & CSP Support**: Native tags for `{% csrf_token %}` and `{% csp_nonce_attr %}` to keep apps secure out-of-the-box.
+- **Block Partials**: Render a single block from a compiled template via `compiled.renderBlock('block_name')`.
+- **Async Rendering**: Support for async filters/tags with `asyncRender()`.
+- **Extensible API**: Easy registration for custom tags and filters.
+- **No Unsafe Code Execution**: Evaluates expressions securely without using `eval()`.
+- **Editor Support**: First-class syntax highlighting and snippets for VS Code, Sublime Text, Atom, and TextMate-compatible editors.
+
+### VS Code
+
+#### Option A: Install the official extension (recommended)
+
+Search for **miki-template** in the VS Code Marketplace, or install from the command line:
+
+```bash
+code --install-extension miki-template
+```
+
+#### Option B: Manual install from this repo
+
+1. Copy the `syntaxes/` and `snippets/` folders from this repo.
+2. In VS Code, run **Preferences: Configure File Associations** and associate `*.miki` with `miki-template`.
+3. Or add a workspace-level `.vscode/settings.json`:
+
+```json
+{
+  "files.associations": {
+    "*.miki": "miki-template"
+  }
+}
+```
+
+### Sublime Text / Atom / TextMate
+
+Drop the `syntaxes/miki-template.tmLanguage.json` file into your editor’s `Packages/User/` folder and associate it with the `.miki` extension.
+
+---
+
+## 📚 Documentation
+
+- [Installation](docs/installation.md)
+- [Usage Guide](docs/usage.md)
+- [Tags Reference](docs/tags.md)
+- [Filters Reference](docs/filters.md)
+- [Security](docs/security.md)
+- [API Reference](docs/api.md)
+- [Partial Definitions](docs/partialdef.md)
+- [Contributing](docs/advanced_usage.md)
+
+---
+
+## 📦 Installation
+
+```bash
+npm install miki-template
+```
+
+---
+
+## 🛠️ Quick Start
+
+### CommonJS (require)
+
+```javascript
+const { render, compile, __express, SafeString, markSafe } = require('miki-template');
+
+const template = 'Hello {{ user.name|title }}! Roles: {{ user.roles|join:", " }}';
+const context = {
+  user: {
+    name: 'miki coder',
+    roles: ['admin', 'developer']
+  }
+};
+
+const result = render(template, context);
+console.log(result); // Output: "Hello Miki Coder! Roles: admin, developer"
+```
+
+### ES Modules (import)
+
+```javascript
+// Named imports
+import { render, compile, __express, SafeString, markSafe } from 'miki-template';
+
+// Or default import (gets all exports)
+import miki from 'miki-template';
+const { render: mikiRender } = miki;
+
+const template = 'Hello {{ user.name|title }}!';
+const result = render(template, { user: { name: 'world' } });
+console.log(result); // Output: "Hello World!"
+```
+
+> **Note:** For ESM in Node.js, either name your files `.mjs` or add `"type": "module"` to your `package.json`.
+
+### Express Integration
+
+**CommonJS:**
+```javascript
+const express = require('express');
+const { __express: renderDtpl } = require('miki-template');
+
+const app = express();
+
+// Register both .html and .miki extensions
+app.engine('html', renderDtpl);
+app.engine('miki', renderDtpl);
+app.set('view engine', 'miki');
+app.set('views', './views');
+
+app.get('/', (req, res) => {
+  res.render('home', {
+    title: 'Django Templates in Node!',
+    items: ['Apple', 'Banana', 'Orange']
+  });
+});
+
+app.listen(3000, () => console.log('App listening on port 3000'));
+```
+
+**ESM:**
+```javascript
+import express from 'express';
+import { __express as renderDtpl } from 'miki-template';
+
+const app = express();
+app.engine('html', renderDtpl);
+app.set('view engine', 'html');
+app.set('views', './views');
+
+app.get('/', (req, res) => {
+  res.render('home', {
+    title: 'Django Templates with ESM!',
+    items: ['Apple', 'Banana', 'Orange']
+  });
+});
+
+app.listen(3000, () => console.log('App listening on port 3000'));
+```
+
+---
+
+## 📖 Template Syntax & Parity
+
+### Variables & Dotted Lookups
+Resolve properties dynamically on nested objects or arrays. If the resolved value is a callable/function, it is automatically executed with zero arguments.
+```html
+{{ user.profile.name }}
+{{ items.0 }} <!-- Array indexing -->
+{{ user.getFullName }} <!-- Function resolution -->
+```
+
+### Built-in Filters
+Apply filters using pipes (`|`). Arguments are passed after a colon (`:`).
+- **Text**: `upper`, `lower`, `title`, `capfirst`, `slugify`, `wordcount`, `striptags`, `linebreaks`, `linebreaksbr`, `truncatewords:N`, `truncatechars:N`.
+- **HTML**: `safe`, `escape`.
+- **List**: `length`, `join:","`, `slice:"start:end"`, `dictsort:"key"`, `dictsortreversed:"key"`.
+- **Default**: `default:"fallback"`, `default_if_none:"fallback"`.
+- **Date/Time**: `date:"Y-m-d"`, `time:"H:i"`, `timesince`, `timeuntil`.
+- **Numeric**: `add:5`, `divisibleby:2`, `floatformat:2`.
+- **Misc**: `yesno:"yes,no,maybe"`, `pluralize:"suffix"`, `filesizeformat`.
+
+### Built-in Control Tags
+- **if / elif / else / endif**: Supports conditional expressions with operators: `==`, `!=`, `<`, `<=`, `>`, `>=`, `in`, `not in`, `and`, `or`, `not`.
+  ```html
+  {% if user.role == 'admin' or user.is_staff %}
+    <p>Access Granted</p>
+  {% elif user.age >= 18 %}
+    <p>Standard Access</p>
+  {% else %}
+    <p>Access Denied</p>
+  {% endif %}
+  ```
+- **for / empty / endfor**: Loop over arrays and objects. Injects `forloop` meta tracking.
+  ```html
+  {% for item in items %}
+    <li>{{ forloop.counter }}: {{ item }}</li>
+  {% empty %}
+    <li>No items found</li>
+  {% endfor %}
+  ```
+- **with / endwith**: Scopes localized variables.
+  ```html
+  {% with user.profile.address as addr %}
+    <p>{{ addr.city }}, {{ addr.zip }}</p>
+  {% endwith %}
+  ```
+- **cycle**: Cycle through values sequentially.
+  ```html
+  {% for row in rows %}
+    <tr class="{% cycle 'row-odd' 'row-even' %}">...</tr>
+  {% endfor %}
+  ```
+- **autoescape on/off**: Control auto-escaping block behavior.
+- **verbatim / endverbatim**: Treat raw text inside literally.
+- **comment / endcomment**: Block comment ignored during parse.
+
+### Security Tags
+- **csrf_token**: Automatically outputs a hidden input carrying the CSRF token from the context variable `csrf_token`.
+  ```html
+  <form method="post">
+    {% csrf_token %}
+    ...
+  </form>
+  ```
+- **csp_nonce_attr**: Dynamically outputs `nonce="value"` if the variable `csp_nonce` is in the context.
+  ```html
+  <script {% csp_nonce_attr %} src="app.js"></script>
+  ```
+
+### Inheritance & Block Rendering
+Inherit structure from parent templates.
+- `base.html`:
+  ```html
+  <html>
+  <body>
+    {% block content %}Default Content{% endblock %}
+  </body>
+  </html>
+  ```
+- `child.html`:
+  ```html
+  {% extends "base.html" %}
+  {% block content %}
+    <h1>Child Content</h1>
+    {{ block.super }} <!-- Renders parent's default content -->
+  {% endblock %}
+  ```
+
+#### Rendering a Block-Level Partial (Django 5.1+ / HTMX Style)
+You can compile a template and choose to render *only a specific block* (useful for AJAX or HTMX requests):
+```javascript
+const compiled = compile(childTemplateStr, { views: './templates' });
+const partialHtml = compiled.renderBlock('content', context);
+console.log(partialHtml); // Output: "<h1>Child Content</h1> Default Content"
+```
+
+---
+
+## 🔧 Extensibility API
+
+### Register a Custom Filter
+
+**CommonJS:**
+```javascript
+const { registerFilter } = require('miki-template');
+
+registerFilter('reverse', (val) => {
+  return String(val).split('').reverse().join('');
+});
+```
+
+**ESM:**
+```javascript
+import { registerFilter } from 'miki-template';
+
+registerFilter('reverse', (val) => {
+  return String(val).split('').reverse().join('');
+});
+```
+
+### Register a Custom Tag
+
+**CommonJS:**
+```javascript
+const { registerTag } = require('miki-template');
+
+// Custom tag parser returning an AST Node
+registerTag('hello', (tagContent, parser) => {
+  return {
+    render: (context) => 'Hello World!'
+  };
+});
+```
+
+**ESM:**
+```javascript
+import { registerTag } from 'miki-template';
+
+registerTag('hello', (tagContent, parser) => {
+  return {
+    render: (context) => 'Hello World!'
+  };
+});
+```
+
+---
+
+## 🔒 Security
+- **HTML Auto-escaping**: Enabled by default to guard against Cross-Site Scripting (XSS).
+- **SafeString Wrapper**: Explicitly bypass escaping using the `|safe` filter or marking variables via `markSafe(val)`.
+- **No eval() Execution**: Parser evaluates logic statements securely using standard tokens mapping.

package/ROADMAP.md

@@ -0,0 +1,40 @@
+# Project Roadmap - Django-Style Template Engine
+
+Our release and packaging timeline.
+
+---
+
+## 🏁 Phase 1 - Core Delivery (Day 1-2)
+- [x] High-performance tokenizing Lexer.
+- [x] AST parser mapping nested blocks.
+- [x] Context Stack Manager supporting dotted lookup and function calls.
+- [x] HTML Auto-escaping using `he` library and `SafeString` wrappers.
+- [x] Complete suite of built-in filters (25+ filters).
+- [x] Core control flow tags (`if`, `for`, `with`, `cycle`, `autoescape`, `comment`, `verbatim`).
+- [x] Dynamic inheritance (`extends`, `block`, `include` with parameter scope).
+- [x] Express layout adapter (`__express`).
+- [x] Complete test suite verification.
+
+---
+
+## 🚀 Phase 2 - Advanced Performance & Ecosystem (Next Month)
+- [x] **CI/CD Pipeline** – linting, testing, and automated npm publishing via GitHub Actions.
+- [x] **AST Caching** – cache compiled ASTs for static templates (implemented).
+- [x] **Custom Tag Helpers** – simplify registration of custom block tags (implemented via helpers).
+- [x] **Async Render Options** – support async filters/tags for DB lookups (implemented).
+- [x] **Extended Date Syntax** – integrate `date-fns` for full date formatting parity (implemented).
+- [x] **Advanced Usage Docs** – examples covering partialdef, block rendering, and API (added).
+- [x] **Performance Benchmarks** – baseline measurements and optimization guide (added).
+
+---
+
+## 📦 Phase 3 - Ecosystem & Tooling (Q4)
+- [ ] VS Code Extension: Syntax highlighting for `.dtpl` or `.html` Django templates.
+- [ ] Web Playground: An interactive sandbox to experiment with the engine.
+- [ ] CLI Compiler: Render templates from the command line.
+
+---
+
+## 📚 Documentation
+- [x] Comprehensive docs in `docs/` covering installation, usage, tags, filters, partialdef, security, and API.
+- [x] Advanced examples and performance guide (added).

package/benchmarks/report.json

@@ -0,0 +1,17 @@
+[
+  {
+    "name": "small",
+    "syncAvgMs": "0.04",
+    "asyncAvgMs": "0.03"
+  },
+  {
+    "name": "medium",
+    "syncAvgMs": "0.00",
+    "asyncAvgMs": "0.01"
+  },
+  {
+    "name": "large",
+    "syncAvgMs": "0.00",
+    "asyncAvgMs": "0.01"
+  }
+]

package/benchmarks/run.js

@@ -0,0 +1,49 @@
+// benchmarks/run.js
+// Simple benchmark for sync vs async rendering
+const { compile, asyncRender } = require('../src');
+const fs = require('fs');
+const path = require('path');
+const { performance } = require('perf_hooks');
+
+function loadTemplate(name) {
+  const filePath = path.join(__dirname, 'templates', `${name}.dtpl`);
+  return fs.readFileSync(filePath, 'utf8');
+}
+
+function benchRender(name, iterations = 20) {
+  const tmplStr = loadTemplate(name);
+  const compiled = compile(tmplStr);
+  // warm up cache
+  compiled.render({});
+  compiled.asyncRender({});
+
+  const syncTimes = [];
+  const asyncTimes = [];
+  for (let i = 0; i < iterations; i++) {
+    const t0 = performance.now();
+    compiled.render({});
+    syncTimes.push(performance.now() - t0);
+
+    const t1 = performance.now();
+    asyncRender(tmplStr, {});
+    asyncTimes.push(performance.now() - t1);
+  }
+  const avg = arr => arr.reduce((a,b)=>a+b,0)/arr.length;
+  return {
+    name,
+    syncAvgMs: avg(syncTimes).toFixed(2),
+    asyncAvgMs: avg(asyncTimes).toFixed(2)
+  };
+}
+
+function main() {
+  const results = [];
+  ['small','medium','large'].forEach(name => {
+    results.push(benchRender(name));
+  });
+  console.log('Benchmark results:', results);
+  const outPath = path.join(__dirname, 'report.json');
+  fs.writeFileSync(outPath, JSON.stringify(results, null, 2));
+}
+
+main();

package/benchmarks/templates/large.dtpl

@@ -0,0 +1,7 @@
+// large.dtpl
+{% for i in range(0, 1000) %}
+<div class="item">
+  <h2>Item {{ i }}</h2>
+  <p>{{ "Lorem ipsum dolor sit amet, consectetur adipiscing elit. " | repeat:5 }}</p>
+</div>
+{% endfor %}

package/benchmarks/templates/medium.dtpl

@@ -0,0 +1,3 @@
+// medium.dtpl
+{% for i in range(0, 100) %}Item {{ i }}
+{% endfor %}

package/benchmarks/templates/small.dtpl

@@ -0,0 +1,7 @@
+// small.dtpl
+Hello {{ name|upper }}!
+{% if showList %}
+<ul>
+{% for item in items %}<li>{{ item }}</li>{% endfor %}
+</ul>
+{% endif %}

package/context/component.md

@@ -0,0 +1,109 @@
+# Component Hierarchy — Django-Style Template Engine (Node.js/Express)
+
+## 📖 Purpose
+This document defines the **modules, responsibilities, and relationships** for the template engine. It ensures clarity in implementation and maintainability.
+
+---
+
+## 🧩 Core Components
+
+### 1. **Lexer**
+- **Responsibility**: Tokenize template strings into `TEXT`, `VAR`, `BLOCK`.
+- **Inputs**: Raw template string.
+- **Outputs**: Token stream.
+- **Dependencies**: None.
+
+### 2. **Parser**
+- **Responsibility**: Convert tokens into an AST (Abstract Syntax Tree).
+- **Inputs**: Token stream.
+- **Outputs**: AST nodes (`Text`, `Var`, `If`, `For`, `Block`, etc.).
+- **Dependencies**: Lexer.
+
+### 3. **Renderer**
+- **Responsibility**: Walk AST, evaluate expressions, apply filters, render output.
+- **Inputs**: AST + context.
+- **Outputs**: Final HTML string.
+- **Dependencies**: Parser, Filter Registry, Tag Handlers.
+
+---
+
+## 🔧 Supporting Modules
+
+### 4. **Filter Registry**
+- **Responsibility**: Store and apply filters.
+- **Built-in Filters**: `upper`, `lower`, `date`, `truncatechars`, `safe`, `escape`, etc.
+- **Extensibility**: Developers can register custom filters.
+- **Dependencies**: Renderer.
+
+### 5. **Tag Handlers**
+- **Responsibility**: Implement logic for each tag.
+- **Control Flow**: `if`, `elif`, `else`, `for`, `empty`, `with`, `cycle`.
+- **Inheritance**: `extends`, `block`, `include`.
+- **Utilities**: `url`, `static`, `regroup`, `spaceless`, `comment`, `verbatim`.
+- **Security**: `autoescape`, `csrf_token`, `csp_nonce_attr`.
+- **Dependencies**: Renderer, Context.
+
+### 6. **Inheritance System**
+- **Responsibility**: Manage parent/child templates, block overrides.
+- **Mechanism**: Block registry + AST merging.
+- **Dependencies**: Parser, Renderer.
+
+### 7. **Context Manager**
+- **Responsibility**: Provide variables to templates.
+- **Features**: Dotted lookups (`user.name`), querystring injection, context processors.
+- **Dependencies**: Renderer.
+
+---
+
+## 🔐 Security Components
+
+### 8. **Escaping Engine**
+- **Responsibility**: Autoescape HTML by default.
+- **Features**: `safe` filter disables escaping, `escape` forces escaping.
+- **Dependencies**: Renderer, Filter Registry.
+
+---
+
+## 📦 Integration Layer
+
+### 9. **Express Adapter**
+- **Responsibility**: Integrate engine with Express.
+- **API**: `app.engine('dtpl', renderFile)`.
+- **Dependencies**: Renderer, Context Manager.
+
+---
+
+## 🛠️ Extensibility Components
+
+### 10. **Custom Tag API**
+- **Responsibility**: Allow developers to register new tags.
+- **Mechanism**: Tag registry with handler functions.
+
+### 11. **Custom Filter API**
+- **Responsibility**: Allow developers to register new filters.
+- **Mechanism**: Filter registry extension.
+
+### 12. **Context Processors**
+- **Responsibility**: Inject global variables (e.g., `user`, `request`).
+- **Mechanism**: Middleware-like hooks.
+
+---
+
+## 📋 Relationships Diagram (Textual)
+
+- **Lexer → Parser → Renderer**
+- **Renderer → Filter Registry + Tag Handlers + Context Manager**
+- **Tag Handlers → Inheritance System + Escaping Engine**
+- **Express Adapter → Renderer**
+- **Custom APIs → Filter Registry + Tag Handlers**
+- **Context Processors → Context Manager**
+
+---
+
+## 🎯 Deliverables
+- Modular codebase with clear separation of concerns.
+- Each tag/filter implemented as independent handler.
+- Extensible APIs for developers.
+- Secure rendering pipeline with autoescape.
+- Seamless Express integration.
+

package/context/prd.md

@@ -0,0 +1,131 @@
+# Django-Style Template Engine for Node.js/Express
+
+## 📖 Overview
+A custom template engine for Express that replicates **Django’s template language** in Node.js. It supports variables, filters, tags, inheritance, partials, escaping, and extensibility — providing full parity with Django templates while integrating seamlessly into Express.
+
+---
+
+## 🎯 Goals
+- Full Django template feature parity in Node.js.
+- Simple Express integration via `app.engine`.
+- Extensible API for custom tags, filters, and context processors.
+- Secure by default (autoescape enabled).
+- Production-ready performance (AST caching, error handling).
+
+---
+
+## 🧩 Core Architecture
+1. **Lexer** → Tokenizes template into `TEXT`, `VAR`, `BLOCK`.
+2. **Parser** → Builds AST nodes for tags/filters.
+3. **Renderer** → Walks AST, evaluates expressions, applies filters.
+4. **Filter Registry** → Built-in + custom filters.
+5. **Tag Handlers** → Functions for each tag (`if`, `for`, `block`, etc.).
+6. **Inheritance System** → Block registry + parent merging.
+7. **Express Integration** → `app.engine('dtpl', renderFile)`.
+
+---
+
+## 🏷️ Built-in Template Tags
+
+| Tag | Purpose | Node.js Implementation |
+|-----|---------|------------------------|
+| **autoescape** | Toggle HTML escaping. | Maintain `context.autoescape` flag. |
+| **block** | Define overridable content. | Store block AST in registry. |
+| **extends** | Template inheritance. | Load parent file, merge blocks. |
+| **include** | Insert partial template. | Load file, render with context. |
+| **if/elif/else** | Conditional rendering. | Evaluate JS expression safely. |
+| **for/empty** | Loop over iterable. | Inject `forloop` vars (`counter`, `first`, `last`). |
+| **with** | Assign temporary variable. | Extend context object. |
+| **cycle** | Alternate values. | Maintain cycle state per loop. |
+| **comment** | Ignore enclosed content. | Strip from AST. |
+| **verbatim** | Raw output. | Treat enclosed text as literal. |
+| **csrf_token** | CSRF protection. | Insert token from context. |
+| **csp_nonce_attr** | CSP nonce attribute. | Render `nonce="value"`. |
+| **url** | Generate route URL. | Integrate with Express router. |
+| **static** | Reference static files. | Map to Express static dir. |
+| **regroup** | Group list by attribute. | Use JS `reduce`. |
+| **spaceless** | Strip whitespace. | Regex replace in output. |
+
+---
+
+## 🔧 Built-in Filters
+
+| Filter | Purpose | Node.js Implementation |
+|--------|---------|------------------------|
+| **Text** | `upper`, `lower`, `title`, `capfirst`, `truncatewords`, `truncatechars`, `wordcount`, `linebreaks`, `linebreaksbr`, `striptags`, `slugify` | JS string methods, regex, libraries. |
+| **HTML** | `safe`, `escape` | Use `he` for escaping; safe flag. |
+| **List** | `length`, `join`, `slice`, `dictsort`, `dictsortreversed` | JS array methods. |
+| **Date/Time** | `date`, `time`, `timesince`, `timeuntil` | Wrap JS `Date` with `Intl.DateTimeFormat`. |
+| **Numeric** | `add`, `divisibleby`, `floatformat` | Arithmetic ops; `toFixed`. |
+| **Default** | `default`, `default_if_none` | Fallback values. |
+| **Misc** | `pluralize`, `yesno`, `filesizeformat` | String transformations; filesize units. |
+
+---
+
+## 🔐 Escaping & Security
+- Autoescape enabled by default.
+- `safe` filter disables escaping.
+- `escape` filter forces escaping.
+- Use `he` library for HTML entity encoding.
+
+---
+
+## 🛠️ Extensibility
+- **Custom filters**: `filters[name] = fn`.
+- **Custom tags**: Register handler functions in tag registry.
+- **Context processors**: Middleware to inject globals (e.g., `user`, `request`).
+
+---
+
+## 📋 Development Roadmap
+
+### Phase 1 — Core Foundations
+- Lexer, parser, renderer.
+- Express integration.
+
+### Phase 2 — Variables & Filters
+- Variables with dotted lookups.
+- Full filter system (text, list, date, numeric, misc).
+- Custom filter API.
+
+### Phase 3 — Control Flow Tags
+- `if/elif/else`, `for/empty`, `with`, `cycle`, `comment`, `verbatim`.
+
+### Phase 4 — Template Inheritance
+- `extends`, `block`, nested blocks.
+- `include` for partials.
+
+### Phase 5 — Utilities & Helpers
+- `static`, `url`, `regroup`, `spaceless`.
+
+### Phase 6 — Security
+- Autoescape, safe strings, escape filter.
+
+### Phase 7 — Extensibility
+- Custom tags/filters.
+- Context processors.
+
+### Phase 8 — Advanced Features
+- Querystring/context passing.
+- Template partials with context overrides.
+- Error handling.
+- Performance optimizations (AST caching).
+
+### Phase 9 — Developer Experience
+- Documentation.
+- Testing suite.
+- Packaging as npm module.
+- Example apps.
+
+---
+
+## 🎯 Final Deliverables
+- **Engine core**: Lexer, parser, renderer.
+- **Filter library**: All Django filters.
+- **Tag handlers**: All Django tags.
+- **Inheritance system**: Full block/extends support.
+- **Express integration**: `app.engine('dtpl', ...)`.
+- **Extensibility API**: Custom tags/filters/context processors.
+- **Security**: Autoescape + safe strings.
+- **Docs & tests**: Full coverage.
+