md2wiki 1.0.0

package/README.md

@@ -0,0 +1,180 @@
+<div align="center">
+  <br />
+  <img src="public/logo.svg" alt="md2wiki logo" width="160" />
+  <br />
+  <h1>🔄 MD2WIKI</h1>
+  <p><strong>Real-Time Markdown to MediaWiki Wikitext Compiler</strong></p>
+  <p><i>A premium, high-density Next.js utility for Wikipedia editors featuring SUL OAuth 2.0 Sandbox publishing and CORS-free repository README importing.</i></p>
+
+  <br />
+
+  <div align="center">
+    <img src="https://img.shields.io/badge/Stack-Next.js%2015%20%7C%20Tailwind%20v4-emerald?style=for-the-badge" alt="Stack" />
+    <img src="https://img.shields.io/badge/License-MIT-blue?style=for-the-badge" alt="License" />
+    <img src="https://img.shields.io/badge/Platform-Wikimedia%20Toolforge-violet?style=for-the-badge" alt="Platform" />
+  </div>
+
+  <br />
+  <div align="center">
+    <img src="assets/preview.png" alt="Showcase Preview" width="600" style="border-radius: 12px; box-shadow: 0 8px 24px rgba(0, 0, 0, 0.25);" />
+  </div>
+  <br />
+</div>
+
+---
+
+## 🧭 Navigation
+
+* [📐 The Project Vision](#-the-project-vision)
+* [✨ Key Features](#-key-features)
+* [🚀 Quick Start (Local Setup)](#-quick-start-local-setup)
+* [🔒 OAuth 2.0 Credentials Setup](#-oauth-20-credentials-setup)
+* [📦 Deployment to Toolforge](#-deployment-to-toolforge)
+* [📂 Project Structure](#-project-structure)
+* [🧪 Running Unit Tests](#-running-unit-tests)
+
+---
+
+## 📐 The Project Vision
+
+Writing high-quality drafts or documentation offline in Markdown is standard practice for modern developers. However, publishing these drafts on Wikipedia talk pages, user sandboxes, or Meta-Wiki documentation requires manually converting markdown format to MediaWiki Wikitext markup. 
+
+**md2wiki** solves this pain point. It serves as:
+1. **An NPM-ready parser module** (`src/core/parser.ts`) that compiles Markdown to wikitext with smart rules.
+2. **A SUL-integrated Toolforge service** that allows editors to paste raw Markdown (or fetch direct GitHub/GitLab READMEs) and publish the compiled Wikitext straight to their Wikipedia Sandbox with a single click.
+
+---
+
+## ✨ Key Features
+
+<div align="center">
+  <table border="0" cellspacing="0" cellpadding="20">
+    <tr>
+      <td width="300" valign="top" style="border: 1px solid #333; border-radius: 15px; background: rgba(255,255,255,0.02); padding: 15px;">
+        <h3>⚡ Real-Time Compiler</h3>
+        <p>Instant, client-side translation of Markdown headers, text styling, tables, code blocks, lists, and links without network lag.</p>
+      </td>
+      <td width="300" valign="top" style="border: 1px solid #333; border-radius: 15px; background: rgba(255,255,255,0.02); padding: 15px;">
+        <h3>🛡️ Smart Heading Shift</h3>
+        <p>Detects Markdown H1s and auto-shifts headings to avoid duplicate H1 page titles on Wikipedia, conforming strictly to style guidelines.</p>
+      </td>
+    </tr>
+    <tr>
+      <td width="300" valign="top" style="border: 1px solid #333; border-radius: 15px; background: rgba(255,255,255,0.02); padding: 15px;">
+        <h3>🔗 Repository Importer</h3>
+        <p>Paste any GitHub or GitLab repository link, and our CORS-free proxy endpoint fetches, parses, and translates the README file.</p>
+      </td>
+      <td width="300" valign="top" style="border: 1px solid #333; border-radius: 15px; background: rgba(255,255,255,0.02); padding: 15px;">
+        <h3>📝 Sandbox SUL Publisher</h3>
+        <p>Log in securely via Wikimedia SUL OAuth 2.0 and save wikitext directly to <code>User:&lt;Username&gt;/sandbox</code> on any language Wikipedia.</p>
+      </td>
+    </tr>
+  </table>
+</div>
+
+---
+
+## 🚀 Quick Start (Local Setup)
+
+Follow these steps to run the application on your local machine:
+
+### 1. Clone the repository
+```bash
+git clone https://github.com/your-username/md2wiki.git
+cd md2wiki
+```
+
+### 2. Install dependencies
+```bash
+npm install
+```
+
+### 3. Set up environment variables
+Create a `.env` file in the root directory (the application will run in **Mock SUL Mode** automatically if credentials are left blank):
+```env
+WIKIMEDIA_CLIENT_ID=your_oauth_consumer_token
+WIKIMEDIA_CLIENT_SECRET=your_oauth_secret_token
+SESSION_SECRET=a_long_secure_session_encryption_key
+WIKIMEDIA_REDIRECT_URI=http://localhost:8000/api/auth/callback
+```
+
+### 4. Start the development server
+```bash
+npm run dev
+```
+Open **[http://localhost:8000](http://localhost:8000)** in your browser.
+
+---
+
+## 🔒 OAuth 2.0 Credentials Setup
+
+To write to Wikipedia Sandboxes on behalf of users, you must register the tool:
+
+1. Log in to [Meta-Wiki](https://meta.wikimedia.org/).
+2. Navigate to [Special:OAuthConsumerRegistration/propose/oauth2](https://meta.wikimedia.org/wiki/Special:OAuthConsumerRegistration/propose/oauth2).
+3. Set the parameters:
+   * **OAuth callback URL**: `https://md2wiki.toolforge.org/api/auth/callback` (or `http://localhost:8000/api/auth/callback` for dev).
+   * **Grants**: Select **Basic rights** (`basic`), **Edit existing pages** (`edit`), and **Create, edit, and move pages** (`writepage` / `create`).
+   * **Allowed pages**: Leave blank to allow editing of all pages (required since sandbox pages reside under user namespaces).
+4. Submit and record your **Consumer Token** and **Secret Token**.
+
+---
+
+## 📦 Deployment to Toolforge
+
+Wikimedia Toolforge supports hosting Node.js Next.js web applications using Kubernetes.
+
+### 1. Log in to Toolforge Bastion
+```bash
+ssh <username>@login.toolforge.org
+```
+
+### 2. Initialize the Node webservice
+Ensure your project files are cloned in your Toolforge home directory `$HOME/www/js/`:
+```bash
+git clone https://github.com/your-username/md2wiki.git www/js
+cd www/js
+npm install
+```
+
+### 3. Create your production `.env` file
+```env
+WIKIMEDIA_CLIENT_ID=your_production_consumer_token
+WIKIMEDIA_CLIENT_SECRET=your_production_secret_token
+SESSION_SECRET=another_long_production_secret
+WIKIMEDIA_REDIRECT_URI=https://md2wiki.toolforge.org/api/auth/callback
+NODE_ENV=production
+PORT=8000
+```
+
+### 4. Build and start the service
+Build the Next.js assets:
+```bash
+npm run build
+```
+
+Configure Toolforge to start the Next.js production server:
+```bash
+toolforge webservice node18 start
+```
+Toolforge automatically routes traffic to port `8000`. Next.js will serve the client pages and API routes at `https://md2wiki.toolforge.org/`.
+
+---
+
+## 📂 Project Structure
+
+* `src/core/`: The core conversion engine. Contains the TS parser and test files.
+* `src/app/`: The Next.js App Router containing:
+  * `api/`: API Route Handlers for SUL OAuth redirects, proxy readmes, and sandbox edits.
+  * `utils/`: File helper wrappers (session encryptor, raw README URL resolver).
+  * `globals.css` / `layout.tsx` / `page.tsx`: The styling baseline and frontend editor layout.
+* `toolinfo.json`: Toolhub scraper metadata.
+
+---
+
+## 🧪 Running Unit Tests
+
+We use **Vitest** for running our unit tests. To verify the Markdown parser rules:
+```bash
+npm run test
+```

package/dist/parser.d.ts

@@ -0,0 +1,9 @@
+export interface ConverterOptions {
+    shiftHeadings?: boolean;
+    wikiLinks?: boolean;
+}
+/**
+ * Converts Markdown string into MediaWiki Wikitext.
+ * Includes smart heading alignment, link conversion, table rendering, and nested list conversion.
+ */
+export declare function markdownToWikitext(markdown: string, options?: ConverterOptions): string;

package/dist/parser.js

@@ -0,0 +1,228 @@
+import { marked } from 'marked';
+/**
+ * Converts Markdown string into MediaWiki Wikitext.
+ * Includes smart heading alignment, link conversion, table rendering, and nested list conversion.
+ */
+export function markdownToWikitext(markdown, options = {}) {
+    if (!markdown)
+        return '';
+    const tokens = marked.lexer(markdown);
+    // Smart Heading Normalization
+    // If shiftHeadings is undefined (auto), check if there is an H1 heading (depth === 1) in the token tree.
+    let shouldShift = options.shiftHeadings;
+    if (shouldShift === undefined) {
+        shouldShift = hasH1(tokens);
+    }
+    const shiftAmount = shouldShift ? 1 : 0;
+    function hasH1(tokenList) {
+        for (const token of tokenList) {
+            if (token.type === 'heading' && token.depth === 1) {
+                return true;
+            }
+            if (token.tokens && hasH1(token.tokens)) {
+                return true;
+            }
+            if (token.items) {
+                for (const item of token.items) {
+                    if (item.tokens && hasH1(item.tokens)) {
+                        return true;
+                    }
+                }
+            }
+        }
+        return false;
+    }
+    function walk(tokenList, listPrefix = '') {
+        if (!tokenList)
+            return '';
+        let out = '';
+        for (const token of tokenList) {
+            switch (token.type) {
+                case 'space':
+                    out += token.raw;
+                    break;
+                case 'heading': {
+                    const depth = Math.min(6, token.depth + shiftAmount);
+                    const eq = '='.repeat(depth);
+                    const content = walk(token.tokens).trim();
+                    out += `\n${eq} ${content} ${eq}\n`;
+                    break;
+                }
+                case 'paragraph': {
+                    const content = walk(token.tokens);
+                    out += `\n${content}\n`;
+                    break;
+                }
+                case 'text': {
+                    if (token.tokens) {
+                        out += walk(token.tokens);
+                    }
+                    else {
+                        out += token.text;
+                    }
+                    break;
+                }
+                case 'strong': {
+                    out += `'''${walk(token.tokens)}'''`;
+                    break;
+                }
+                case 'em': {
+                    out += `''${walk(token.tokens)}''`;
+                    break;
+                }
+                case 'codespan': {
+                    out += `<code>${token.text}</code>`;
+                    break;
+                }
+                case 'code': {
+                    const lang = token.lang ? ` lang="${token.lang}"` : '';
+                    out += `\n<syntaxhighlight${lang}>\n${token.text}\n</syntaxhighlight>\n`;
+                    break;
+                }
+                case 'hr': {
+                    out += '\n----\n';
+                    break;
+                }
+                case 'br': {
+                    out += '<br />';
+                    break;
+                }
+                case 'del': {
+                    out += `<s>${walk(token.tokens)}</s>`;
+                    break;
+                }
+                case 'blockquote': {
+                    const content = walk(token.tokens).trim();
+                    out += `\n<blockquote>\n${content}\n</blockquote>\n`;
+                    break;
+                }
+                case 'link': {
+                    const href = token.href;
+                    const text = walk(token.tokens).trim() || token.text;
+                    // Check if Wikipedia URL conversion is enabled
+                    let isWikiLink = false;
+                    if (options.wikiLinks !== false) {
+                        const wpMatch = href.match(/^https?:\/\/([a-z-]+)\.wikipedia\.org\/wiki\/([^#?]+)/);
+                        if (wpMatch) {
+                            const pageName = decodeURIComponent(wpMatch[2]).replace(/_/g, ' ');
+                            if (text === pageName || text === href || text === `https://${wpMatch[1]}.wikipedia.org/wiki/${wpMatch[2]}`) {
+                                out += `[[${pageName}]]`;
+                            }
+                            else {
+                                out += `[[${pageName}|${text}]]`;
+                            }
+                            isWikiLink = true;
+                        }
+                    }
+                    if (!isWikiLink) {
+                        if (href.startsWith('#')) {
+                            out += `[[#${href.slice(1)}|${text}]]`;
+                        }
+                        else {
+                            if (text) {
+                                out += `[${href} ${text}]`;
+                            }
+                            else {
+                                out += `[${href}]`;
+                            }
+                        }
+                    }
+                    break;
+                }
+                case 'image': {
+                    const href = token.href;
+                    const alt = token.text || '';
+                    const parts = href.split('/');
+                    const filename = parts[parts.length - 1] || href;
+                    if (alt) {
+                        out += `[[File:${filename}|thumb|alt=${alt}|${alt}]]`;
+                    }
+                    else {
+                        out += `[[File:${filename}|thumb]]`;
+                    }
+                    break;
+                }
+                case 'list': {
+                    out += '\n' + walkList(token, listPrefix) + '\n';
+                    break;
+                }
+                case 'table': {
+                    out += '\n' + walkTable(token) + '\n';
+                    break;
+                }
+                case 'html': {
+                    out += token.text;
+                    break;
+                }
+                case 'escape': {
+                    out += token.text;
+                    break;
+                }
+                default: {
+                    out += token.raw || '';
+                    break;
+                }
+            }
+        }
+        return out;
+    }
+    function walkList(listToken, currentPrefix) {
+        const symbol = listToken.ordered ? '#' : '*';
+        const itemPrefix = currentPrefix + symbol;
+        let result = '';
+        for (const item of listToken.items) {
+            const inlineTokens = [];
+            const childLists = [];
+            for (const token of item.tokens) {
+                if (token.type === 'list') {
+                    childLists.push(token);
+                }
+                else {
+                    inlineTokens.push(token);
+                }
+            }
+            // Render the item's main content
+            const content = walk(inlineTokens).trim();
+            result += `${itemPrefix} ${content}\n`;
+            // Render child lists
+            for (const childList of childLists) {
+                result += walkList(childList, itemPrefix);
+            }
+        }
+        return result;
+    }
+    function walkTable(tableToken) {
+        let wikitext = '{| class="wikitable"\n';
+        // Render Headers
+        if (tableToken.header && tableToken.header.length > 0) {
+            wikitext += '! ';
+            const headers = tableToken.header.map((cell, idx) => {
+                const align = tableToken.align[idx];
+                const alignAttr = align && align !== 'center' ? `align="${align}" | ` : '';
+                return `${alignAttr}${walk(cell.tokens).trim()}`;
+            });
+            wikitext += headers.join(' !! ') + '\n';
+        }
+        // Render Rows
+        if (tableToken.rows) {
+            for (const row of tableToken.rows) {
+                wikitext += '|-\n';
+                wikitext += '| ';
+                const cells = row.map((cell, idx) => {
+                    const align = tableToken.align[idx];
+                    const alignAttr = align && align !== 'center' ? `align="${align}" | ` : '';
+                    return `${alignAttr}${walk(cell.tokens).trim()}`;
+                });
+                wikitext += cells.join(' || ') + '\n';
+            }
+        }
+        wikitext += '|}';
+        return wikitext;
+    }
+    let result = walk(tokens);
+    // Clean up excessive newlines
+    result = result
+        .replace(/\n{3,}/g, '\n\n')
+        .trim();
+    return result;
+}

package/package.json

@@ -0,0 +1,42 @@
+{
+  "name": "md2wiki",
+  "version": "1.0.0",
+  "description": "Convert Markdown to MediaWiki Wikitext with SUL sandbox publishing.",
+  "type": "module",
+  "main": "./dist/parser.js",
+  "types": "./dist/parser.d.ts",
+  "files": [
+    "dist"
+  ],
+  "scripts": {
+    "dev": "next dev -p 8000",
+    "build": "next build",
+    "build:lib": "tsc -p tsconfig.lib.json",
+    "start": "next start -p 8000",
+    "test": "vitest run",
+    "test:watch": "vitest",
+    "parse": "tsx src/core/cli.ts",
+    "prepublishOnly": "npm run build:lib"
+  },
+  "dependencies": {
+    "@tailwindcss/postcss": "^4.3.1",
+    "dotenv": "^16.4.5",
+    "marked": "^12.0.1",
+    "next": "^16.2.9",
+    "postcss": "^8.5.15",
+    "react": "^19.2.7",
+    "react-dom": "^19.2.7"
+  },
+  "devDependencies": {
+    "@tabler/icons-react": "^3.1.0",
+    "@tailwindcss/vite": "^4.0.0",
+    "@types/node": "^20.11.24",
+    "@types/react": "^19.0.0",
+    "@types/react-dom": "^19.0.0",
+    "tailwindcss": "^4.0.0",
+    "tsx": "^4.7.1",
+    "typescript": "^5.3.3",
+    "vitest": "^1.3.1"
+  },
+  "license": "MIT"
+}