i18ntk 3.2.0 → 4.0.0

package/CHANGELOG.md

@@ -3,9 +3,69 @@
 All notable changes to this project will be documented in this file.
 
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
-and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [4.0.0] - 2026-05-21
 
-## [3.2.0] - 2026-05-16
+### Added
+- **Sizing Expansion Prediction**: `i18ntk-sizing` now supports `--predict-expansion` flag that computes per-key character-count expansion ratios across languages and classifies them into Safe/Warning/Critical risk tiers for UI layout planning. Includes a built-in language-pair expansion reference table (EN→DE 35%, EN→RU 50%, EN→JA -40%, etc.).
+- **Watch Hot Reload**: `utils/watch-locales.js` rewritten as an EventEmitter-compatible watcher with debouncing (300ms default) and SHA-256 hash tracking to skip no-change saves. Returns a callable watcher object with `change`, `add`, `unlink`, `error` events and `stop()`.
+- **Usage Dead Key Detection**: `i18ntk-usage` adds `--cleanup` and `--dry-run-delete` flags that identify unused translation keys with confidence scores (0.0–1.0) factoring dynamic access patterns, comment references, and file recency. Produces a `.dead-keys.json` report for safe review before deletion.
+- **Validator Key Naming Convention**: `i18ntk-validate` adds `--enforce-key-style` flag and `keyStyle` config setting supporting `dot.notation`, `snake_case`, `camelCase`, `kebab-case`, and `flat` conventions. Reports all violating keys with suggested canonical forms.
+- **Scanner Multi-Language Detection**: `i18ntk-scanner` adds `--source-language` flag with character-class profiles for 12+ languages (English, German, French, Spanish, Japanese, Chinese, Russian, Korean, Arabic, Hindi, etc.). Language-specific stopword lists and key generation with transliteration for non-Latin scripts.
+- **Backup Incremental Mode**: `i18ntk-backup` adds `--incremental` flag producing differential backups with SHA-256 file hashing, parent-chain metadata, and chained restore (oldest full → incremental diffs in order). `verify` validates the hash chain. Chain depth capped at 10.
+- **Runtime Lazy Loading**: `runtime/index.js` adds `lazy: true` option to `initRuntime()`. Defers locale file loading until the first key access, guided by an auto-generated key-to-file manifest. Falls back to eager loading if manifest is missing. Manifest capped at 100KB with path containment validation.
+- **Protection Context-Aware Rules**: `utils/translate/protection.js` extends the protection config schema to support context rules (`after:word`, `before:word`, `standalone`, `surrounded:left,right`). Plain string terms remain fully backward compatible. Total context rules capped at 100.
+
+### Fixed
+- `i18ntk/runtime` `initRuntime()` now returns independent runtime instances with separate language, fallback language, base directory, and cache state. Later `initRuntime()` calls no longer overwrite earlier returned runtimes or the module-level compatibility singleton.
+- Manager option 7 ("Fix placeholder translations") now interpolates fixer status values correctly instead of printing raw `{languages}`, `{sourceDir}`, `{skipped}`, or `{totalIssues}` placeholders.
+
+### Changed
+- `watchLocales()` now returns a callable watcher object with EventEmitter methods instead of only a bare `stop` function. Existing `const stop = watchLocales(...); stop();` usage remains supported. The returned object fires `change`, `add`, `unlink`, `error` events. If a callback function is passed as the second argument, it is auto-subscribed to `change` and `add` for backward compatibility.
+- **BREAKING**: `i18ntk-sizing` JSON reports now include `expansionPredictions` at the top level when `--predict-expansion` is used. This field is additive — existing report fields are preserved.
+
+### Security
+- All new features reuse existing `SecurityUtils.safe*` wrappers for file I/O, path validation, and input sanitization.
+- Watch module validates all directory paths against project root with containment checks and caps at 50 directories.
+- Runtime lazy-loading manifest entries validated for path containment and size-limited to 100KB.
+- Protection context rules parsed from constrained DSL — never accepts raw user-controlled regex from config.
+- Backup incremental chain depth capped at 10 to prevent resource exhaustion; hash chain verified before restore.
+
+## [3.3.0] - 2026-05-20
+
+### Changed
+- Auto Translate now supports `--provider google|deepl|libretranslate`; DeepL uses `DEEPL_API_KEY`, while LibreTranslate supports `LIBRETRANSLATE_URL` and optional `LIBRETRANSLATE_API_KEY`.
+- Auto Translate provider networking now keeps HTTPS, host allowlist, response-size, private-network, and redacted security logging protections in place for additional providers.
+
+### Fixed
+- `i18ntk-complete` now fills missing target-language keys from the English source value with a language prefix such as `[DE] Home` instead of writing `NOT_TRANSLATED`; this works for both `locales/en/*.json` and monolith `locales/en.json` layouts.
+
+### Security
+- Eliminated all 21 dynamic `require()` calls flagged by Socket.dev: 20 `require(path.join(__dirname, ...))` patterns in `i18ntk-js.js`, `i18ntk-py.js`, `i18ntk-java.js`, `i18ntk-php.js`, and `i18ntk-go.js` converted to static string literal requires.
+- Added `SecurityUtils.validatePath()` gate around the remaining dynamic `require()` in `i18ntk-translate.js` `loadCustomTranslateFn`.
+- Created `utils/translate/safe-network.js` — a secure HTTPS wrapper with URL host/path allowlist validation, response size limits (100KB), suspicious query parameter detection, and security event logging. All outbound network access now flows through this validated layer.
+- Replaced direct `https.get` call in `utils/translate/api.js` with `safeHttpGet` from the safe-network wrapper.
+
+### Docs
+- README.md updated for v3.3.0 Auto Translate providers and secure provider operations.
+- SECURITY.md updated with Socket.dev analysis disclaimer and guidance on expected alerts for a CLI/i18n toolkit.
+- CHANGELOG.md and `package.json` versionInfo updated for v3.3.0.
+
+### Socket.dev Analysis Disclaimer
+
+This package is a developer CLI and runtime helper that performs file I/O, network access (translation provider APIs on user request), and environment variable access. As such, Socket.dev will flag the following alerts that are **expected and by design**:
+
+| Alert | Why it's expected |
+|---|---|
+| Network access | Only contacts configured translation providers via HTTPS when user invokes auto-translate. All outbound calls flow through `safe-network.js` with host/path allowlist validation, response size limits, private-network blocking, and redacted security event logging. No telemetry, no unexpected outbound calls. |
+| Environment variable access | Centralized through `env-manager.js` with a strict allowlist. Blocks `SECRET`, `PASSWORD`, `KEY`, `TOKEN`, `AWS_*`, `NPM_*`, and 15+ other patterns. |
+| Filesystem access | Reads/writes only project locale files and reports within validated paths. All FS operations gated by `SecurityUtils.validatePath`. |
+| URL strings | Hardcoded default provider URLs for Google, DeepL, and LibreTranslate used only for auto-translation. No external resource loading. |
+
+The v3.3.0 release resolves the actionable dynamic-require alert by eliminating all 21 instances.
+
+## [3.2.0] - 2026-05-16
 
 ### Security
 - **CRITICAL**: Fixed invalid `crypto.createCipherGCM`/`createDecipherGCM` API calls in `admin-pin.js` — replaced with `crypto.createCipheriv`/`createDecipheriv`.

package/README.md

@@ -1,6 +1,6 @@
-# i18ntk v3.2.0
+# i18ntk v4.0.0
 
-Zero-dependency internationalization toolkit for setup, scanning, analysis, validation, usage tracking, translation completion, automatic JSON locale translation, reporting, and runtime translation loading.
+A i18n toolkit - A zero-dependency internationalization toolkit for setup, scanning, analysis, validation, usage tracking, translation completion, automatic JSON locale translation, reporting, and runtime translation loading.
 
 ![i18ntk Logo](https://raw.githubusercontent.com/vladnoskv/i18ntk/main/docs/screenshots/i18ntk-logo-public.PNG)
 
@@ -9,7 +9,7 @@ Zero-dependency internationalization toolkit for setup, scanning, analysis, vali
 [![node](https://img.shields.io/badge/node-%3E%3D16-339933)](https://nodejs.org)
 [![dependencies](https://img.shields.io/badge/dependencies-0-success)](https://www.npmjs.com/package/i18ntk)
 [![license](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
-[![socket](https://socket.dev/api/badge/npm/package/i18ntk/3.2.0)](https://socket.dev/npm/package/i18ntk/overview/3.2.0)
+[![socket](https://socket.dev/api/badge/npm/package/i18ntk/4.0.0)](https://socket.dev/npm/package/i18ntk/overview/4.0.0)
 
 ## Install
 
@@ -30,14 +30,19 @@ Requirements:
 - npm `>=8.0.0`
 - No runtime dependencies
 
-## What's New in 3.2.0
+## What's New in 4.0.0
 
-- **SECURITY**: Fixed 4 critical runtime-crash bugs (invalid crypto APIs, missing imports) across admin-pin.js, security-config.js, and scripts/security-check.js.
-- **SECURITY**: Removed encryption key stored alongside ciphertext in admin-pin.js; encryption key is now derived via HKDF.
-- **SECURITY**: Enforced HTTPS-only for Google Translate API requests; fixed http.get timeout for Node.js <16.14 compatibility.
-- **SECURITY**: Added path validation to backup restore/verify operations; locked down FileManagementService PIN verification stubs.
+- **SIZING**: `--predict-expansion` flag computes per-key expansion ratios across languages with Safe/Warning/Critical risk tiers for UI layout planning.
+- **WATCH**: `watchLocales()` now returns an EventEmitter-compatible watcher with debounced `change`/`add`/`unlink`/`error` events and SHA-256 hash tracking.
+- **USAGE**: `--cleanup` and `--dry-run-delete` flags identify dead translation keys with confidence scores.
+- **VALIDATOR**: `--enforce-key-style` enforces dot.notation, snake_case, camelCase, kebab-case, or flat naming conventions.
+- **SCANNER**: `--source-language` supports multi-language hardcoded text detection with 12+ language profiles.
+- **BACKUP**: `--incremental` flag creates differential backups with SHA-256 hashing and chained restores.
+- **RUNTIME**: `lazy: true` option defers locale file loading until first key access for lower memory usage.
+- **PROTECTION**: Context-aware rules (`after:word`, `before:word`, `standalone`, `surrounded:left,right`) for precise term masking.
+- **FIX**: `initRuntime()` now returns independent instances with isolated language and cache state.
 
-See [CHANGELOG.md](./CHANGELOG.md) and [docs/migration-guide-v3.2.0.md](./docs/migration-guide-v3.2.0.md) for release details.
+See [CHANGELOG.md](./CHANGELOG.md) for more release details.
 
 ## Quick Start
 
@@ -45,7 +50,7 @@ Initialize a project:
 
 ```bash
 i18ntk
-# or
+# or with explicit command
 i18ntk --command=init
 ```
 
@@ -111,8 +116,8 @@ i18ntk-fixer
 i18ntk-backup
 i18ntk-translate
 ```
-
-Note: manager route `i18ntk --command=backup` is disabled in current builds. Use `i18ntk-backup` directly for backup operations.
+`n
+Note: manager route `i18ntk --command=backup` is disabled in current builds. Use `i18ntk-backup` (or legacy `i18ntk-backup`) directly for backup operations.
 
 ## Common Options
 
@@ -150,6 +155,21 @@ i18ntk-translate locales/en/common.json fr --dry-run --report-stdout
 i18ntk-translate locales/en es --source-dir locales/en --files "*.json" --no-confirm --preserve-placeholders
 ```
 
+Provider examples:
+
+```bash
+export DEEPL_API_KEY="your-deepl-api-key"
+i18ntk-translate locales/en/common.json de --provider deepl --no-confirm --preserve-placeholders
+
+export LIBRETRANSLATE_URL="https://libretranslate.com/translate"
+export LIBRETRANSLATE_API_KEY="optional-api-key"
+i18ntk-translate locales/en/common.json es --provider libretranslate --no-confirm --preserve-placeholders
+```
+
+`google` remains the default provider. You can also set `I18NTK_TRANSLATE_PROVIDER=deepl` or `I18NTK_TRANSLATE_PROVIDER=libretranslate`.
+
+Provider requests are HTTPS-only and response-size limited, and security logs redact provider query strings and response bodies. DeepL is pinned to official DeepL hosts by default; set `I18NTK_ALLOW_CUSTOM_TRANSLATE_HOSTS=1` only for a trusted DeepL-compatible proxy. Custom LibreTranslate URLs are blocked for localhost/private IP ranges unless `I18NTK_ALLOW_PRIVATE_TRANSLATE_URLS=1` is set for trusted local testing. Keep provider API keys in environment variables or a secret manager.
+
 The manager flow asks for:
 
 - source locale directory, either the folder with JSON files or a locale root such as `./locales`
@@ -199,7 +219,12 @@ Example `i18ntk-auto-translate.json`:
 ```json
 {
   "version": 1,
-  "terms": ["BrandName", "PRODUCT_CODE", "API"],
+  "terms": [
+    "BrandName",
+    "PRODUCT_CODE",
+    { "value": "OK", "context": "after:Click|Press|Tap" },
+    { "value": "API", "context": "standalone" }
+  ],
   "keys": ["app.brandName", "legal.companyName", "product.*.symbol"],
   "values": ["BrandName Ltd", "support@example.com"],
   "patterns": ["[A-Z]{2,}-\\d+"]
@@ -207,6 +232,8 @@ Example `i18ntk-auto-translate.json`:
 ```
 
 - `terms` are masked before translation and restored exactly afterward.
+  - **Plain strings**: masked everywhere (backward compatible).
+  - **Context objects**: masked only in specific contexts (`after:word`, `before:word`, `standalone`, `surrounded:left,right`).
 - `keys` are exact key paths or `*` wildcard paths copied unchanged.
 - `values` are exact source values copied unchanged.
 - `patterns` are JavaScript regex strings for advanced protected substrings.
@@ -257,6 +284,136 @@ i18ntk-sizing --source-dir ./locales --detailed --output-dir ./i18ntk-reports
 
 Use `--detailed` to print per-file rows in the terminal.
 
+### Expansion Prediction (New in 4.0.0)
+
+Predict UI layout overflow risk by analyzing per-key character-count expansion across languages:
+
+```bash
+i18ntk-sizing --source-dir ./locales --predict-expansion --output-report
+```
+
+Expansion ratios are classified into risk tiers:
+
+- **Safe** (<30% expansion): no UI impact expected
+- **Warning** (30–50%): may overflow in tight layouts — test on target languages
+- **Critical** (>50%): high risk of truncation — review UI element sizing
+
+The report includes a built-in language-pair expansion reference table (EN→DE +35%, EN→RU +50%, EN→JA −40%, etc.) and lists the top-30 most-expanded keys.
+
+## Scanner: Multi-Language Detection (New in 4.0.0)
+
+`i18ntk-scanner` now supports detecting hardcoded text in multiple source languages beyond English:
+
+```bash
+i18ntk-scanner --source-dir ./src --source-language de
+i18ntk-scanner --source-dir ./src --source-language ja --output-report
+```
+
+Supported language profiles (12+): English, German, French, Spanish, Japanese, Chinese, Russian, Korean, Arabic, Hindi, and more. Each profile includes language-specific character ranges, stopword lists for false-positive filtering, and transliteration rules for key generation.
+
+## Usage: Dead Key Detection (New in 4.0.0)
+
+`i18ntk-usage` can identify translation keys that are defined but never referenced in source code:
+
+```bash
+i18ntk-usage --source-dir ./src --i18n-dir ./locales --cleanup
+i18ntk-usage --source-dir ./src --i18n-dir ./locales --cleanup --dry-run-delete
+```
+
+Each dead key receives a confidence score (0.0–1.0) factoring:
+- Dynamic key patterns (e.g., `` t(`prefix.${dynamic}`) ``) — lower score
+- Key appears in source code comments or JSDoc — medium score
+- Parent file recently modified (<30 days) — medium score
+- No references found anywhere — high score (>0.8)
+
+The `--dry-run-delete` flag writes a `.dead-keys.json` report for review before any destructive action.
+
+## Validator: Key Naming Conventions (New in 4.0.0)
+
+Enforce consistent translation key naming across your project:
+
+```bash
+i18ntk-validate --enforce-key-style
+```
+
+Configure the expected style in `.i18ntk-config`:
+
+```json
+{
+  "keyStyle": "dot.notation"
+}
+```
+
+Supported styles: `dot.notation`, `snake_case`, `camelCase`, `kebab-case`, `flat`. Violations are reported as warnings with suggested canonical forms.
+
+## Watch: Hot Reload (New in 4.0.0)
+
+`utils/watch-locales.js` now provides debounced file watching with EventEmitter support:
+
+```js
+const watchLocales = require('i18ntk/utils/watch-locales');
+const watcher = watchLocales('./locales');
+
+watcher.on('change', (filePath) => {
+  console.log('Locale changed:', filePath);
+});
+
+watcher.on('add', (filePath) => {
+  console.log('Locale added:', filePath);
+});
+
+// Later:
+watcher.stop();
+```
+
+Features: 300ms debounce (configurable), SHA-256 hash tracking to skip no-change saves, and a maximum of 50 watched directories.
+
+### Migration
+
+The `watchLocales` return value gained EventEmitter methods in v4.0.0. Existing stop-function usage still works:
+
+```js
+const stop = watchLocales('./locales', onChange);
+```
+
+Can be updated to:
+
+```js
+const watcher = watchLocales('./locales');
+watcher.on('change', onChange);
+watcher.stop();
+```
+
+Passing a callback as the second argument is still supported — it auto-subscribes to `change` and `add` events.
+
+## Backup: Incremental Mode (New in 4.0.0)
+
+Create differential backups that only include changed files:
+
+```bash
+i18ntk-backup create ./locales --incremental
+```
+
+Incremental backups store SHA-256 hashes per file and a parent-chain reference. Restoring an incremental backup automatically chains from the oldest full backup through each incremental diff in order. Chain depth is capped at 10 increments. Use `verify` to validate the hash chain.
+
+## Runtime: Lazy Loading (New in 4.0.0)
+
+Reduce memory usage by deferring locale file loads until first key access:
+
+```js
+const runtime = require('i18ntk/runtime');
+
+const i18n = runtime.initRuntime({
+  baseDir: './locales',
+  language: 'en',
+  lazy: true
+});
+
+console.log(i18n.t('common.hello')); // loads common.json on first access
+```
+
+When `lazy: true`, the runtime builds a key-to-file manifest on first access and loads individual files on demand. Files are loaded once and cached. If the manifest is missing or incomplete, the runtime falls back to full eager loading for that language. Manifest size is capped at 100KB with path containment validation.
+
 ## Runtime API
 
 Use `i18ntk/runtime` when an application needs to read locale JSON files at runtime.
@@ -264,7 +421,7 @@ Use `i18ntk/runtime` when an application needs to read locale JSON files at runt
 ```js
 const runtime = require('i18ntk/runtime');
 
-runtime.initRuntime({
+const i18n = runtime.initRuntime({
   baseDir: './locales',
   language: 'en',
   fallbackLanguage: 'en',
@@ -272,11 +429,11 @@ runtime.initRuntime({
   preload: true
 });
 
-console.log(runtime.t('common.hello'));
-runtime.setLanguage('fr');
-console.log(runtime.getLanguage());
-console.log(runtime.getAvailableLanguages());
-runtime.refresh('fr');
+console.log(i18n.t('common.hello'));
+i18n.setLanguage('fr');
+console.log(i18n.getLanguage());
+console.log(i18n.getAvailableLanguages());
+i18n.refresh('fr');
 ```
 
 See [docs/runtime.md](./docs/runtime.md) for runtime details.
@@ -289,7 +446,7 @@ Example:
 
 ```json
 {
-  "version": "3.2.0",
+  "version": "4.0.0",
   "sourceDir": "./locales",
   "i18nDir": "./locales",
   "outputDir": "./i18ntk-reports",
@@ -350,8 +507,6 @@ The public package manifest includes `readmeFilename: "README.md"`, and the rele
 - [Migration Guide v3.2.0](./docs/migration-guide-v3.2.0.md)
 - [Migration Guide v3.1.1](./docs/migration-guide-v3.1.1.md)
 - [Migration Guide v3.0.0](./docs/migration-guide-v3.0.0.md)
-- [Migration Guide v2.6.0](./docs/migration-guide-v2.6.0.md)
-- [Migration Guide v2.5.1](./docs/migration-guide-v2.5.1.md)
 
 ## Security
 

package/SECURITY.md

@@ -1,10 +1,10 @@
 # Security Policy
 
-## Supported Versions
-
-The supported production line is `2.5.x`.
-
-Versions earlier than `2.5.0` are not recommended for production use because later releases include package, filesystem, environment, and admin-auth hardening.
+## Supported Versions
+
+The supported production line is `4.x`.
+
+Versions earlier than `4.0.0` are not recommended for production use because later releases include Auto Translate provider hardening, dynamic-require elimination, path-validation hardening, lazy loading with manifest validation, and incremental backup hash-chain verification.
 
 ## Security Model
 
@@ -14,8 +14,8 @@ Security priorities:
 
 - zero runtime dependencies
 - no install-time lifecycle commands in the public package manifest
-- no shipped local setup state, admin PINs, backups, reports, logs, credentials, or generated artifacts
-- centralized environment-variable access through `utils/env-manager.js`
+- no shipped local setup state, admin PINs, backup files, reports, logs, credentials, or generated artifacts
+- centralized environment-variable access through `utils/env-manager.js` with a strict allowlist
 - path containment checks based on resolved paths and `path.relative()`
 - timing-safe comparison for authentication hashes or tokens
 - silent-by-default logging for production-like contexts
@@ -24,6 +24,26 @@ Security priorities:
 
 The npm package uses a stripped public manifest. It must not contain install-time lifecycle commands, dependency fields, local setup state, or development tooling.
 
+## Socket.dev Analysis Disclaimer
+
+Socket.dev scans the published npm package and may flag the following alerts. These are **expected behaviors** for a developer CLI/i18n toolkit and are mitigated as described:
+
+| Alert | Design Rationale | Mitigation |
+|---|---|---|
+| **Network access** | Contacts configured translation providers via HTTPS only when user explicitly invokes auto-translate. No telemetry, no beaconing, no background network activity. | `utils/translate/safe-network.js` enforces HTTPS, host/path allowlists, response-size limits, private-network blocking, and redacted security logging for Google, DeepL, and LibreTranslate provider requests. |
+| **Environment variable access** | Reads env vars for logging level, output directory, UI language, and project paths. Required for CLI configuration. | `utils/env-manager.js` uses a fixed allowlist (28 vars). Blocks `SECRET`, `PASSWORD`, `KEY`, `TOKEN`, `AWS_*`, `GITHUB_*`, `NPM_*`, `NODE_*`, `PATH`, `HOME`, `USER`, `SHELL`, and 8 more patterns. |
+| **Filesystem access** | Reads/writes locale JSON files, config files, and reports within the user's project. Core function of the toolkit. | All FS operations are gated by `SecurityUtils.validatePath()` with path-traversal detection, symlink resolution, and base-path containment checks. |
+| **URL strings** | Contains hardcoded translation provider endpoint URLs for Google, DeepL, and LibreTranslate defaults. | Only used when user invokes `i18ntk-translate`. Custom provider URLs remain HTTPS-only and are validated before use. |
+
+The v3.3.0 release **resolved** the previously actionable Socket.dev alert:
+- **Dynamic require** — all 21 instances eliminated (20 converted to static string literals, 1 gated with `SecurityUtils.validatePath`).
+
+The v4.0.0 release adds the following security hardening:
+- **Watch module**: all watched directories validated against project root with containment checks; capped at 50 directories.
+- **Runtime lazy loading**: key-to-file manifest entries validated for path containment; manifest size capped at 100KB.
+- **Incremental backups**: hash-chain verification before restore; chain depth capped at 10 increments.
+- **Protection context rules**: DSL-parsed context rules — never raw user-controlled regex from config; bounded at 200 chars per rule, 100 rules total.
+
 ## Reporting Vulnerabilities
 
 Do not report security vulnerabilities in public GitHub issues.
@@ -46,7 +66,7 @@ Security reports are reviewed privately first. Confirmed issues should receive:
 
 ## User Guidance
 
-- Keep i18ntk updated to `2.5.0` or newer.
+- Keep i18ntk updated to `4.0.0` or newer.
 - Do not commit `.i18ntk-config`, admin PIN files, backup directories, generated reports, logs, npm credentials, or secret material.
 - Run i18ntk only in project directories you trust.
 - Review generated translation changes before committing them.