xlsform2lstsv 0.2.3 → 0.4.0

package/README.md

@@ -1,4 +1,5 @@
 [![npm version](https://img.shields.io/npm/v/xlsform2lstsv)](https://www.npmjs.com/package/xlsform2lstsv)
+[![AI-Assisted](https://img.shields.io/badge/AI--assisted-Claude%20Code-blueviolet?logo=anthropic&logoColor=white)](./AI_DISCLOSURE.md)
 
 
 # xlsform2lstsv
@@ -9,15 +10,109 @@ Convert XLSForm surveys to LimeSurvey TSV format.
 
 - This package is still WIP and not all features of xlsform have been implemented and verified.
 - While importing is tested in an automated fashion (see `scripts/test-compatibility-safe.ts`), this only verifies whether all questions were successfully imported, but not if e.g. validation and relevance expressions were transformed correctly. To be safe, always use the "Survey logic view" in the LimeSurvey GUI.
-- If you want question and choice names to be the same in LimeSurvey, make them <=5 chars (this is a LimeSurvey requiremtn)
+- To keep question and choice names unchanged after conversion, use short alphanumeric IDs (≤ 20 chars for questions, ≤ 5 chars for choices) without underscores or hyphens.
 
 
+## Installation
+
+```bash
+npm install xlsform2lstsv
+```
+
+## Quick Start
+
+The XFormParser provides direct XLS/XLSX file support:
+
+```typescript
+import { XFormParser } from 'xlsform2lstsv';
+
+// Parse XLS/XLSX file and convert to TSV
+const tsv = await XFormParser.convertXLSFileToTSV('path/to/survey.xlsx');
+
+// Or parse XLS/XLSX data directly
+const xlsxData = fs.readFileSync('path/to/survey.xlsx');
+const tsv = await XFormParser.convertXLSDataToTSV(xlsxData);
+```
+
+**Methods:**
+- `convertXLSFileToTSV(filePath, config)`: Direct conversion from file
+- `convertXLSDataToTSV(data, config)`: Direct conversion from buffer
+- `parseXLSFile(filePath)`: Parse to structured arrays
+- `parseXLSData(data)`: Parse buffer to structured arrays
+
+### Using Arrays 
+
+A different entry point accepts XLSForm data as JavaScript arrays:
+
+```typescript
+import { XLSFormToTSVConverter } from 'xlsform2lstsv';
+
+const converter = new XLSFormToTSVConverter();
+const tsv = converter.convert(surveyData, choicesData, settingsData);
+```
+
+**Parameters:**
+- `surveyData`: Array of survey rows (questions, groups, etc.)
+- `choicesData`: Array of choice/option data  
+- `settingsData`: Array of survey settings
+
+**Returns:** TSV string suitable for LimeSurvey import
+
+
+## Configuration
+
+Both `XLSFormParser` and `XLSFormToTSVConverter` accept an optional config object:
+
+```typescript
+const tsv = await XLSFormParser.convertXLSFileToTSV('survey.xlsx', {
+  handleRepeats: 'error',
+  debugLogging: true,
+  convertWelcomeNote: false,
+  defaults: { language: 'de', surveyTitle: 'My Survey' },
+});
+```
+
+| Option | Type | Default | Description |
+|---|---|---|---|
+| `convertWelcomeNote` | `boolean` | `true` | Promote a `note` named `welcome` to LimeSurvey's survey welcome text. |
+| `convertEndNote` | `boolean` | `true` | Promote a `note` named `end` to LimeSurvey's survey end text. |
+| `convertOtherPattern` | `boolean` | `true` | Auto-detect the `_other` question pattern and set `other=Y`. |
+| `convertMarkdown` | `boolean` | `true` | Parse labels/hints as Markdown and convert to HTML. |
+
 ## Implemented features
 
 - Question Types and Choices (see `src/processors/TypeMapper.ts` for how this library maps XLSForm types to LimeSurvey types)
   - everything but the types specified in `UNIMPLEMENTED_TYPES` in `src/xlsformConverter.ts`
   - record types ❌ (start, end, today, device_id, username, phonenumber, email)
 
+- **"Other" Option Handling** ✅
+  - **Explicit `or_other` modifier**: Add `or_other` to question type (e.g., `select_one colors or_other`) to enable the "other" option
+  - **Automatic pattern detection**: The converter automatically detects when you have:
+    - A main question (single or multiple choice)
+    - A follow-up question with the same name + `_other` suffix
+    - The follow-up question has relevance targeting the "other" option of the main question
+    - When this pattern is detected, the "other" choice is removed from the choices list and `other=Y` is set on the main question
+  - **Example pattern**:
+    ```
+    # Main question
+    type: select_one colors
+    name: favorite_color
+    label: What is your favorite color?
+    
+    # Choices (including "other")
+    list_name: colors
+    name: red, label: Red
+    name: blue, label: Blue
+    name: other, label: Other
+    
+    # Follow-up question for "other" specification
+    type: text
+    name: favorite_color_other  # Same name + "_other" suffix
+    label: Please specify your favorite color
+    relevant: ${favorite_color} = 'other'  # Targets the "other" option
+    ```
+  - **Result**: The "other" choice is automatically removed and `other=Y` is set on the main question
+
 - Settings sheet 
   - -> LS Survey Global Parameters (only name of survey)  ✅
   - -> Survey Language-Specific Parameters (default language is first row, other rows are extracted from label translations)  ✅
@@ -46,7 +141,7 @@ Convert XLSForm surveys to LimeSurvey TSV format.
   - `multiline` on text questions → LimeSurvey type `T` (Long free text) ✅
   - `likert` on select_one → kept as `L` (no LimeSurvey visual equivalent) ✅
   - `label`/`list-nolabel` → LimeSurvey matrix question type `F` ✅
-  - `field-list` on groups → silently ignored (format=A already shows everything on one page) ✅
+  - `field-list` on groups → each group becomes a separate page when `style=pages` is set (`format=G`); silently ignored otherwise ✅
   - Other appearances (e.g. `minimal`, `compact`, `horizontal`) trigger a warning and are ignored
 - Additional columns ❌
 - guidance_hint ❌
@@ -55,57 +150,17 @@ Convert XLSForm surveys to LimeSurvey TSV format.
 
 XLSForm and LimeSurvey differ in how they model surveys. Some information is lost or transformed during conversion, and some defaults are applied:
 
-- **Survey format**: The output defaults to "All in one" mode (`format=A`), displaying all groups and questions on a single page.
+- **Survey format**: The output defaults to "All in one" mode (`format=A`), displaying all groups and questions on a single page. If the settings sheet has `style=pages`, the format is set to `G` (group by group), so each group with `appearance=field-list` becomes a separate page — matching XLSForm's multi-page behaviour.
 - **Nested groups**: LimeSurvey does not support nested groups. Parent-only groups (containing only child groups, no direct questions) are flattened — their label becomes a note question (type X) in the first child group.
-- **Field name truncation**: LimeSurvey limits question codes to 20 characters and answer codes to 5 characters. Longer names are truncated (underscores removed first, then cut to length).
+- **Field name sanitization**: LimeSurvey only allows alphanumeric question codes (max 20 characters) and answer codes (max 5 characters). Underscores and hyphens are stripped, then names are truncated to fit. If two fields end up with the same sanitized name, a numeric suffix is appended to the later one (e.g. `fieldname1`). **Recommendation:** to avoid renaming, use short IDs (≤ 20 chars for questions, ≤ 5 chars for choices) without underscores or hyphens — these will pass through unchanged.
 - **Record/metadata types**: XLSForm `start`, `end`, `today`, `deviceid` etc. are silently skipped — LimeSurvey handles these internally.
-- **Appearances**: Most XLSForm `appearance` values have no LimeSurvey equivalent and are ignored (a warning is logged). Supported appearances: `multiline` on text questions maps to type `T` (Long free text); `likert` on select_one is accepted silently (stays type `L`); `label`/`list-nolabel` is converted to LimeSurvey's matrix question type (`F`); `field-list` on groups is a no-op since format=A already shows everything on one page.
+- **Reserved note names — `welcome` and `end`**: A `note` question with `name=welcome` is promoted to the LimeSurvey survey welcome text (`surveyls_welcometext`) instead of appearing as a question. A `note` with `name=end` is promoted to the end text (`surveyls_endtext`). Both support multilingual labels. If either note is the sole content of a group, that wrapping group is silently suppressed (no group row is emitted). If the group also contains other questions, it is kept and the note is still promoted.
+- **Appearances**: Most XLSForm `appearance` values have no LimeSurvey equivalent and are ignored (a warning is logged). Supported appearances: `multiline` on text questions maps to type `T` (Long free text); `likert` on select_one is accepted silently (stays type `L`); `label`/`list-nolabel` is converted to LimeSurvey's matrix question type (`F`); `field-list` on groups is silently ignored in `format=A` mode, or becomes a page boundary in `format=G` mode (when `style=pages` is set).
 - **Multilingual row ordering**: Rows are grouped by language within each group (all base-language rows first, then translations) to work around a LimeSurvey TSV importer bug that resets question ordering counters on translation rows.
 
-## Installation
+- **Lime survey** soft mandatory doesnt work only mandatory or not
 
-```bash
-npm install xlsform2lstsv
-```
-
-## Quick Start
 
-The XFormParser provides direct XLS/XLSX file support:
-
-```typescript
-import { XFormParser } from 'xlsform2lstsv';
-
-// Parse XLS/XLSX file and convert to TSV
-const tsv = await XFormParser.convertXLSFileToTSV('path/to/survey.xlsx');
-
-// Or parse XLS/XLSX data directly
-const xlsxData = fs.readFileSync('path/to/survey.xlsx');
-const tsv = await XFormParser.convertXLSDataToTSV(xlsxData);
-```
-
-**Methods:**
-- `convertXLSFileToTSV(filePath, config)`: Direct conversion from file
-- `convertXLSDataToTSV(data, config)`: Direct conversion from buffer
-- `parseXLSFile(filePath)`: Parse to structured arrays
-- `parseXLSData(data)`: Parse buffer to structured arrays
-
-### Using Arrays 
-
-A different entry point accepts XLSForm data as JavaScript arrays:
-
-```typescript
-import { XLSFormToTSVConverter } from 'xlsform2lstsv';
-
-const converter = new XLSFormToTSVConverter();
-const tsv = converter.convert(surveyData, choicesData, settingsData);
-```
-
-**Parameters:**
-- `surveyData`: Array of survey rows (questions, groups, etc.)
-- `choicesData`: Array of choice/option data  
-- `settingsData`: Array of survey settings
-
-**Returns:** TSV string suitable for LimeSurvey import
 
 ## Development Setup
 
@@ -180,6 +235,11 @@ To test specific versions, set the `SPECIFIC_VERSIONS` environment variable:
 SPECIFIC_VERSIONS="6.16.4,6.17.0" npm run test-compatibility
 ```
 
+To test with current specified version:
+
+```bash
+npm run test:integration
+```
 
 ### Commit Message Format
 

package/dist/config/types.js

@@ -4,6 +4,10 @@
 export const defaultConfig = {
     handleRepeats: 'warn',
     debugLogging: false,
+    convertWelcomeNote: true,
+    convertEndNote: true,
+    convertOtherPattern: true,
+    convertMarkdown: true,
     defaults: {
         language: 'en',
         groupName: 'Questions',

package/dist/converters/xpathTranspiler.js

@@ -129,14 +129,21 @@ function transpile(node, ctx) {
                     return `endsWith(${transpile(node.args[0], ctx)}, ${transpile(node.args[1], ctx)})`;
                 }
                 break;
+            case 'normalize-space':
+                if (node.args?.length === 1) {
+                    return `trim(${transpile(node.args[0], ctx)})`;
+                }
+                break;
             case 'not':
                 if (node.args?.length === 1) {
                     return `!(${transpile(node.args[0], ctx)})`;
                 }
                 break;
             case 'if':
+                // Use if() function instead of ternary (? :) because EM's ternary parser
+                // can misinterpret colons inside string literals (e.g. '2026-03-CHW: Chancenwerk')
                 if (node.args?.length === 3) {
-                    return `(${transpile(node.args[0], ctx)} ? ${transpile(node.args[1], ctx)} : ${transpile(node.args[2], ctx)})`;
+                    return `if(${transpile(node.args[0], ctx)}, ${transpile(node.args[1], ctx)}, ${transpile(node.args[2], ctx)})`;
                 }
                 break;
             case 'today':
@@ -166,7 +173,9 @@ function transpile(node, ctx) {
                     const rawValue = rightNode.value;
                     const rewritten = lookupAnswerCode(fieldName, rawValue);
                     if (rewritten !== rawValue) {
-                        return `${fieldName} == "${rewritten}"`;
+                        // Use truncated field name if available
+                        const truncatedFieldName = ctx?.getTruncatedFieldName ? ctx.getTruncatedFieldName(fieldName) : fieldName;
+                        return `${truncatedFieldName} == '${rewritten}'`;
                     }
                 }
                 return `${transpile(leftNode, ctx)} == ${transpile(rightNode, ctx)}`;
@@ -179,7 +188,9 @@ function transpile(node, ctx) {
                     const rawValue = rightNode.value;
                     const rewritten = lookupAnswerCode(fieldName, rawValue);
                     if (rewritten !== rawValue) {
-                        return `${fieldName} != "${rewritten}"`;
+                        // Use truncated field name if available
+                        const truncatedFieldName = ctx?.getTruncatedFieldName ? ctx.getTruncatedFieldName(fieldName) : fieldName;
+                        return `${truncatedFieldName} != '${rewritten}'`;
                     }
                 }
                 return `${transpile(leftNode, ctx)} != ${transpile(rightNode, ctx)}`;
@@ -222,7 +233,10 @@ function transpile(node, ctx) {
     if (node.steps && node.steps.length > 0) {
         const step = node.steps[0];
         if (step.name) {
-            return sanitizeName(step.name);
+            const fieldName = sanitizeName(step.name);
+            // Use truncated field name if available in context
+            const truncatedFieldName = ctx?.getTruncatedFieldName ? ctx.getTruncatedFieldName(fieldName) : fieldName;
+            return truncatedFieldName;
         }
         // Handle self reference (.)
         if (step.axis === 'self') {
@@ -428,7 +442,7 @@ export async function convertRelevance(xpathExpr, ctx) {
         .replace(/\bOR\b/gi, 'or');
     const result = await xpathToLimeSurvey(normalizedXPath, ctx);
     // Handle edge case: selected() with just {field} (without $)
-    if (result && result.includes('selected(')) {
+    if (result && typeof result === 'string' && result.includes('selected(')) {
         return result.replace(/selected\s*\(\s*\{(\w+)\}\s*,\s*["']([^'"]+)["']\s*\)/g, (_match, fieldName, value) => {
             return `(${sanitizeName(fieldName)}="${value}")`;
         });

package/dist/generateFixtures.js

@@ -31,7 +31,7 @@ function cleanOutputDirectory(dir) {
         }
     }
 }
-async function generateTSVFromFixture(fixturePath, outputPath) {
+async function generateTSVFromFixture(fixturePath, outputPath, config = {}) {
     console.log(`Processing: ${path.basename(fixturePath)}`);
     // Read fixture
     const fixtureContent = fs.readFileSync(fixturePath, 'utf-8');
@@ -39,7 +39,7 @@ async function generateTSVFromFixture(fixturePath, outputPath) {
     // Convert to TSV with configuration that removes underscores but doesn't truncate field names
     // This matches LimeSurvey's behavior (removes underscores but allows longer field names)
     // Answer codes are already limited to 5 chars in the fixtures
-    const converter = new XLSFormToTSVConverter({});
+    const converter = new XLSFormToTSVConverter(config);
     const tsv = await converter.convert(fixture.survey, fixture.choices, fixture.settings);
     // Write output
     fs.writeFileSync(outputPath, tsv, 'utf-8');
@@ -96,6 +96,24 @@ async function main() {
         }
         console.log('');
     }
+    // Generate settings variant: same fixture with all conversion settings disabled
+    const settingsFixturePath = path.join(FIXTURES_DIR, 'settings_survey.json');
+    if (fs.existsSync(settingsFixturePath)) {
+        const variantPath = path.join(OUTPUT_DIR, 'settings_survey_disabled.tsv');
+        try {
+            await generateTSVFromFixture(settingsFixturePath, variantPath, {
+                convertWelcomeNote: false,
+                convertEndNote: false,
+                convertOtherPattern: false,
+                convertMarkdown: false,
+            });
+            jsonSuccessCount++;
+        }
+        catch (error) {
+            console.error('  ✗ Error processing settings_survey_disabled:', error);
+        }
+        console.log('');
+    }
     // Generate TSV for each XLSX fixture
     // XLSX files may contain unimplemented types (e.g. range) — failures are logged but not fatal
     let xlsxSuccessCount = 0;

package/dist/processors/FieldSanitizer.js

@@ -1,9 +1,72 @@
 import { sanitizeFieldName } from '../utils/helpers.js';
+const MAX_FIELD_LENGTH = 20;
 export class FieldSanitizer {
-    constructor() { }
+    constructor() {
+        /** Set of unique sanitized names already assigned */
+        this.usedNames = new Set();
+        /**
+         * Map from stripped name (underscores/hyphens removed, NOT truncated)
+         * to the unique sanitized name (truncated + deduplicated).
+         * Used by the transpiler to resolve variable references.
+         */
+        this.strippedToUnique = new Map();
+    }
+    /**
+     * Basic sanitization: remove underscores/hyphens and truncate to 20 chars.
+     * Does NOT check for duplicates. Use sanitizeNameUnique for that.
+     */
     sanitizeName(name) {
         return sanitizeFieldName(name);
     }
+    /**
+     * Sanitize a field name and ensure it is unique among all previously
+     * registered names. If a collision is detected after sanitization,
+     * a numeric suffix is appended (e.g. "fieldname1").
+     */
+    sanitizeNameUnique(name) {
+        const stripped = name.replace(/[_-]/g, '');
+        let truncated = stripped.length > MAX_FIELD_LENGTH
+            ? stripped.substring(0, MAX_FIELD_LENGTH)
+            : stripped;
+        if (!this.usedNames.has(truncated)) {
+            this.usedNames.add(truncated);
+            this.strippedToUnique.set(stripped, truncated);
+            return truncated;
+        }
+        // Collision detected — append a numeric suffix
+        let counter = 1;
+        let candidate;
+        do {
+            const suffix = String(counter);
+            candidate = truncated.substring(0, MAX_FIELD_LENGTH - suffix.length) + suffix;
+            counter++;
+        } while (this.usedNames.has(candidate));
+        this.usedNames.add(candidate);
+        this.strippedToUnique.set(stripped, candidate);
+        console.warn(`Field name "${name}" collides with an existing name after sanitization; renamed to "${candidate}"`);
+        return candidate;
+    }
+    /**
+     * Resolve a stripped field name (underscores already removed, not truncated)
+     * to its unique sanitized name. Falls back to simple truncation if the name
+     * was never registered.
+     */
+    resolveStrippedName(strippedName) {
+        const mapped = this.strippedToUnique.get(strippedName);
+        if (mapped)
+            return mapped;
+        // Fallback: truncate like normal (name was never registered)
+        return strippedName.length > MAX_FIELD_LENGTH
+            ? strippedName.substring(0, MAX_FIELD_LENGTH)
+            : strippedName;
+    }
+    /**
+     * Clear all registered names. Must be called at the start of each conversion.
+     */
+    resetNames() {
+        this.usedNames.clear();
+        this.strippedToUnique.clear();
+    }
     sanitizeAnswerCode(code) {
         // Answer codes in LimeSurvey have a 5-character limit
         let result = code;

package/dist/processors/TSVGenerator.js

@@ -20,11 +20,12 @@ export class TSVGenerator {
             'mandatory',
             'other',
             'default',
-            'same_default'
+            'same_default',
+            'hidden'
         ];
         const lines = [headers.join('\t')];
         for (const row of this.rows) {
-            const values = headers.map((h) => this.escapeForTSV(row[h] || ''));
+            const values = headers.map((h) => this.escapeForTSV(row[h] ?? ''));
             lines.push(values.join('\t'));
         }
         return lines.join('\n');

package/dist/utils/helpers.js

@@ -23,6 +23,31 @@ export function deepMerge(target, ...sources) {
 function isObject(item) {
     return item !== null && typeof item === 'object' && !Array.isArray(item);
 }
+/**
+ * Deduplicate a list of names by appending numeric suffixes on collision.
+ * Returns a new array with unique names, preserving order.
+ */
+export function deduplicateNames(names, maxLength) {
+    const result = [...names];
+    const used = new Set();
+    for (let i = 0; i < result.length; i++) {
+        if (!result[i])
+            continue;
+        let name = result[i];
+        if (used.has(name)) {
+            let counter = 1;
+            let candidate;
+            do {
+                const suffix = String(counter);
+                candidate = name.substring(0, maxLength - suffix.length) + suffix;
+                counter++;
+            } while (used.has(candidate));
+            result[i] = candidate;
+        }
+        used.add(result[i]);
+    }
+    return result;
+}
 /**
  * Sanitize field names for LimeSurvey compatibility
  * Removes underscores/hyphens and truncates to 20 characters (LimeSurvey question title limit)

package/dist/utils/markdownRenderer.js

@@ -0,0 +1,20 @@
+import { marked } from 'marked';
+/**
+ * Convert a markdown string to HTML for use in LimeSurvey text fields.
+ *
+ * - Block content (multiple paragraphs, lists, etc.) is returned as full HTML.
+ * - Single-paragraph content has its outer <p>…</p> stripped so that short
+ *   labels remain inline strings rather than block elements.
+ * - Empty / non-string input is returned as-is.
+ */
+export function markdownToHtml(text) {
+    if (!text)
+        return text;
+    const html = marked.parse(text).trim();
+    // Strip the wrapping <p>…</p> only when the output is a single paragraph
+    // (i.e. exactly one <p> tag). Multi-paragraph output keeps its structure.
+    if (html.startsWith('<p>') && html.endsWith('</p>') && (html.match(/<p>/g) || []).length === 1) {
+        return html.slice(3, -4);
+    }
+    return html;
+}