i18ntk 3.0.0 → 3.1.1

package/CHANGELOG.md

@@ -5,6 +5,47 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [3.1.1] - 2026-05-07
+
+### Added
+- **Auto Translate protection file workflow**: Added user-editable `i18ntk-auto-translate.json` support for protected terms, key paths, exact values, and regex patterns.
+- **Public package README guard**: Public package staging now verifies `README.md` is included and non-empty before publish.
+
+### Changed
+- Updated README and release documentation for the current Auto Translate protection workflow and public package contents.
+- Removed project-specific hardcoded validation examples so users configure their own brand and domain terms.
+
+### Fixed
+- Removed provider-shaped fake secret fixtures from tests to avoid GitHub push protection false positives.
+- Ensured public package metadata includes `readmeFilename: "README.md"` so npm can render the package README.
+
+## [3.1.0] - 2026-05-07
+
+### Added
+- **Placeholder-preserve translation mode**: Translates text segments around dynamic placeholders and reinserts the original tokens exactly.
+- **Auto Translate beta settings**: Added settings for placeholder mode, concurrency, batch size, progress interval, retry count, retry delay, timeout, dry-run preview, report output, and BOM output.
+- **Large-file tuning flags**: Added `--batch-size` and `--progress-interval` to `i18ntk-translate`.
+- **Short-lived placeholder manifest**: Mirrors placeholder maps to an OS temp file during processing and removes it after each file completes.
+- **Validation content-risk helper**: Added structured detection for URLs, email addresses, secret-like values, and likely untranslated English content.
+- **Sizing file-set analysis**: Added per-language file counts, per-file sizing statistics, and missing/extra file comparison across locale folders.
+
+### Changed
+- Automated and manager Auto Translate flows now default to placeholder `preserve` mode instead of skipping placeholder-bearing strings.
+- `i18ntk-translate` can now be imported and run in-process by other package modules.
+- Source JSON reads tolerate UTF-8 BOM-prefixed files.
+- Validation warnings now report specific issue types and reasons instead of the vague `Risky content` message.
+- English-content validation now reports an English percentage and only warns above a 10% threshold with at least three detected English words.
+- Sizing reports now include folder-level file counts and per-file key/character breakdowns for each language.
+
+### Fixed
+- Fixed false-positive validation warnings for normal product copy terms.
+- Fixed validator handling so ordinary explanatory uses of words like `token` or `secret` are not treated as leaked credentials.
+- Fixed distorted `i18ntk-sizing` table output by rendering aligned columns from measured values instead of fixed localized spacing.
+- Fixed sizing language comparison output so it uses analyzed languages and the configured source language baseline.
+
+### Security
+- Removed production `child_process` usage from `main/manage/commands/TranslateCommand.js` by replacing the spawned CLI process with an in-process translator call.
+
 ## [3.0.0] - 2026-05-05
 
 ### Added
@@ -63,22 +104,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Scripts
 - Fixed `scripts/build-public-package.js` and `scripts/reset-release-state.js` `npm_execpath` fallback for missing env var.
-- Fixed `scripts/lint-locales.js` BOM handling and try-catch for `fs.readdirSync`.
-
-## [2.5.1] - 2026-04-29
-
-### Security
-- Fixed `AdminAuth.verifyPin()` to fail closed when admin config is missing, disabled, or malformed instead of returning success.
-- Fixed auth-required checks to fail closed when settings require admin PIN protection but the admin config is unusable.
-- Normalized admin session expiry handling by storing both `expires` and `expiresAt` and cleaning up both formats consistently.
-
-### Added
-- Added regression tests for admin PIN fail-closed behavior and session expiry cleanup.
-
-### Changed
-- Documented the public npm package staging flow introduced after `2.5.0`.
-
-## [2.5.0] - 2026-04-29
+- Fixed `scripts/lint-locales.js` BOM handling and try-catch for `fs.readdirSync`.
+
+## [2.5.1] - 2026-04-29
+
+### Security
+- Fixed `AdminAuth.verifyPin()` to fail closed when admin config is missing, disabled, or malformed instead of returning success.
+- Fixed auth-required checks to fail closed when settings require admin PIN protection but the admin config is unusable.
+- Normalized admin session expiry handling by storing both `expires` and `expiresAt` and cleaning up both formats consistently.
+
+### Added
+- Added regression tests for admin PIN fail-closed behavior and session expiry cleanup.
+
+### Changed
+- Documented the public npm package staging flow introduced after `2.5.0`.
+
+## [2.5.0] - 2026-04-29
 
 ### Security
 - Centralized environment-variable access behind the `utils/env-manager.js` allowlist.

package/README.md

@@ -1,6 +1,6 @@
-# i18ntk v3.0.0
+# i18ntk v3.1.1
 
-Zero-dependency internationalization toolkit for setup, scanning, analysis, validation, usage tracking, translation completion, automatic locale translation, and runtime translation loading.
+Zero-dependency internationalization toolkit for setup, scanning, analysis, validation, usage tracking, translation completion, automatic JSON locale translation, reporting, and runtime translation loading.
 
 ![i18ntk Logo](https://raw.githubusercontent.com/vladnoskv/i18ntk/main/docs/screenshots/i18ntk-logo-public.PNG)
 
@@ -9,47 +9,7 @@ Zero-dependency internationalization toolkit for setup, scanning, analysis, vali
 [![node](https://img.shields.io/badge/node-%3E%3D16-339933)](https://nodejs.org)
 [![dependencies](https://img.shields.io/badge/dependencies-0-success)](https://www.npmjs.com/package/i18ntk)
 [![license](https://img.shields.io/badge/license-MIT-yellow.svg)](LICENSE)
-[![socket](https://socket.dev/api/badge/npm/package/i18ntk/3.0.0)](https://socket.dev/npm/package/i18ntk/overview/3.0.0)
-
-## Upgrade Notice
-
-Versions earlier than `3.0.0` may contain known stability and security issues.
-They are considered unsupported for production use. Upgrade to `3.0.0` or newer.
-
-## v3.0.0 - Auto Translate Release
-
-v3.0.0 adds automatic JSON locale translation through the management menu and the standalone `i18ntk-translate` command. Highlights:
-
-- **Auto Translate (Beta)**: Translate one or more source JSON files into one or more target languages from menu option 14.
-- **Standalone CLI**: Use `i18ntk-translate <source-file> <target-lang>` for direct automation and batch translation.
-- **Dry-run preview**: Review translated/skipped counts before writing target files.
-- **Placeholder protection**: Detect and preserve placeholders such as `{name}`, `{{count}}`, `%s`, `%d`, `:id`, `%{name}`, and `${value}`.
-- **Post-translation report**: Print or write translated and skipped key counts.
-- **Zero dependencies**: Translation support uses built-in Node.js modules and the free Google Translate endpoint.
-
-For the full detailed changelog, see [CHANGELOG.md](./CHANGELOG.md). For migration notes, see [docs/migration-guide-v3.0.0.md](./docs/migration-guide-v3.0.0.md).
-
-## What i18ntk Does
-
-- Zero runtime dependencies
-- Interactive and non-interactive project setup
-- Translation completeness analysis and usage tracking
-- Validation, sizing, and summary reporting
-- Missing-key completion and fixer workflows
-- Automatic translation of JSON locale files
-- Runtime translation helpers for application code
-- Support for JS/TS, React, Vue, Angular, and generic projects
-
-## Getting Started
-
-1. Install the package.
-2. Run `i18ntk` or `i18ntk --command=init` to initialize the project.
-3. Confirm the source language and locale directories.
-4. Run `i18ntk --command=analyze` or `i18ntk --command=validate` to inspect translation coverage.
-5. Use `i18ntk --command=complete` to fill missing keys when needed.
-6. Use `i18ntk --command=translate` or menu option 14 to auto-translate source JSON files.
-
-The full onboarding flow is documented in [docs/getting-started.md](docs/getting-started.md).
+[![socket](https://socket.dev/api/badge/npm/package/i18ntk/3.1.1)](https://socket.dev/npm/package/i18ntk/overview/3.1.1)
 
 ## Install
 
@@ -64,11 +24,29 @@ npm install --save-dev i18ntk
 npx i18ntk --help
 ```
 
-## Setup
+Requirements:
 
-The toolkit stores project configuration in `.i18ntk-config` at the project root.
+- Node.js `>=16.0.0`
+- npm `>=8.0.0`
+- No runtime dependencies
 
-Recommended setup flow:
+## What's New in 3.1.1
+
+- Auto Translate can translate strings that contain placeholders by translating text around the placeholders and reinserting the original tokens.
+- Auto Translate supports user-editable protection rules in `i18ntk-auto-translate.json` for brand names, product terms, exact values, key paths, and regex patterns.
+- The manager Auto Translate flow runs in-process, avoiding production `child_process` usage for that command.
+- The target-language prompt supports `all` to translate into every configured target language while excluding the source language.
+- Source-directory prompts are clearer and accept absolute paths or project-relative paths.
+- Validation warnings now distinguish URLs, email addresses, secret-like values, and likely untranslated English content.
+- Sizing reports now include per-language file counts, file-set mismatches, and per-file key/character statistics.
+- Internal UI locale coverage is enforced against the English UI locale.
+- Public package staging verifies `README.md` is present before publish.
+
+See [CHANGELOG.md](./CHANGELOG.md) and [docs/migration-guide-v3.1.1.md](./docs/migration-guide-v3.1.1.md) for release details.
+
+## Quick Start
+
+Initialize a project:
 
 ```bash
 i18ntk
@@ -76,24 +54,41 @@ i18ntk
 i18ntk --command=init
 ```
 
-During setup, you can define:
+Run common checks:
+
+```bash
+i18ntk --command=analyze
+i18ntk --command=validate
+i18ntk --command=usage
+i18ntk --command=sizing
+i18ntk --command=summary
+```
+
+Complete or fix translation files:
 
-- source directory
-- source language
-- UI language
-- framework preference
-- output directory
-- backup behavior
+```bash
+i18ntk --command=complete
+i18ntk-fixer --help
+```
 
-If you run in CI or a non-interactive shell, use:
+Auto-translate locale JSON:
 
 ```bash
-i18ntk --command=init --no-prompt
+i18ntk --command=translate
+# or
+i18ntk-translate locales/en/common.json de --report-stdout
 ```
 
-## Daily Use
+The full onboarding guide is in [docs/getting-started.md](./docs/getting-started.md).
+
+## Main Commands
+
+Primary CLI:
 
 ```bash
+i18ntk
+i18ntk --help
+i18ntk --command=init
 i18ntk --command=analyze
 i18ntk --command=validate
 i18ntk --command=usage
@@ -102,9 +97,10 @@ i18ntk --command=sizing
 i18ntk --command=complete
 i18ntk --command=translate
 i18ntk --command=summary
+i18ntk --command=debug
 ```
 
-Standalone commands are also available:
+Standalone executables:
 
 ```bash
 i18ntk-init
@@ -121,10 +117,11 @@ i18ntk-backup
 i18ntk-translate
 ```
 
-Note: `i18ntk --command=backup` in the manager flow is disabled in current builds.
-Use the standalone `i18ntk-backup` executable when backup operations are required.
+Note: manager route `i18ntk --command=backup` is disabled in current builds. Use `i18ntk-backup` directly for backup operations.
 
-## Common Flags
+## Common Options
+
+Most commands support:
 
 - `--source-dir <path>`
 - `--i18n-dir <path>`
@@ -135,15 +132,6 @@ Use the standalone `i18ntk-backup` executable when backup operations are require
 - `--dry-run`
 - `--help`
 
-Auto Translate also supports:
-
-- `--source-lang <code>`
-- `--files <pattern>`
-- `--skip-placeholders`
-- `--send-placeholders`
-- `--report-file <path>`
-- `--report-stdout`
-
 Example:
 
 ```bash
@@ -152,7 +140,7 @@ i18ntk --command=analyze --source-dir=./src --i18n-dir=./locales --output-dir=./
 
 ## Auto Translate
 
-Interactive menu flow:
+Interactive manager flow:
 
 ```bash
 i18ntk
@@ -164,16 +152,117 @@ Direct CLI examples:
 ```bash
 i18ntk-translate locales/en/common.json de
 i18ntk-translate locales/en/common.json fr --dry-run --report-stdout
-i18ntk-translate locales/en es --files "*.json" --no-confirm --skip-placeholders
+i18ntk-translate locales/en es --source-dir locales/en --files "*.json" --no-confirm --preserve-placeholders
 ```
 
-The manager flow accepts comma- or space-separated target language codes, previews the first target language with a dry run, asks for confirmation, then writes translated files under matching target-language directories such as `locales/de/common.json`.
+The manager flow asks for:
+
+- source locale directory
+- source language code
+- one or more target languages, or `all`
+- one JSON file or all JSON files in the source directory
+
+Before writing files, the manager can run a dry-run preview. After confirmation it writes translated files under sibling target-language folders, for example:
+
+```text
+locales/en/common.json
+locales/de/common.json
+locales/fr/common.json
+```
+
+### Placeholder Handling
+
+Auto Translate detects common placeholders such as:
+
+- `{name}`
+- `{{count}}`
+- `%s`
+- `%d`
+- `:id`
+- `%{name}`
+- `${value}`
+
+Useful flags:
 
-See [docs/auto-translate.md](docs/auto-translate.md) for full usage details.
+- `--preserve-placeholders`: translate text around placeholders and reinsert original tokens
+- `--skip-placeholders`: copy placeholder-bearing strings unchanged
+- `--send-placeholders`: send placeholder-bearing strings through translation after masking
+- `--custom-regex <regex>`: add project-specific placeholder detection
+
+### Protected Terms and Keys
+
+Auto Translate can create and use a project-local protection file:
+
+```bash
+i18ntk-translate locales/en/common.json de --create-protection-file --protection-file ./i18ntk-auto-translate.json
+```
+
+Example `i18ntk-auto-translate.json`:
+
+```json
+{
+  "version": 1,
+  "terms": ["BrandName", "PRODUCT_CODE", "API"],
+  "keys": ["app.brandName", "legal.companyName", "product.*.symbol"],
+  "values": ["BrandName Ltd", "support@example.com"],
+  "patterns": ["[A-Z]{2,}-\\d+"]
+}
+```
+
+- `terms` are masked before translation and restored exactly afterward.
+- `keys` are exact key paths or `*` wildcard paths copied unchanged.
+- `values` are exact source values copied unchanged.
+- `patterns` are JavaScript regex strings for advanced protected substrings.
+
+Useful flags:
+
+- `--protection-file <path>`
+- `--create-protection-file`
+- `--no-protection`
+
+Open Settings and choose `Auto Translate Beta` to edit defaults for placeholder mode, concurrency, batch size, retry settings, report output, BOM output, protection file path, first-run setup prompt, and update prompt.
+
+See [docs/auto-translate.md](./docs/auto-translate.md) for the full Auto Translate guide.
+
+## Validation
+
+Validation checks locale structure, completeness, placeholders, and content risks.
+
+In 3.1.1, warning types are more specific:
+
+- `Potential risky content`: URL, email address, or secret-like value
+- `Possible untranslated English content`: target-language value appears to contain too much English
+
+English-content warnings include:
+
+- detected English percentage
+- configured threshold
+- matched word count
+- sample matched words
+
+Tune warnings in `.i18ntk-config`:
+
+```json
+{
+  "englishContentThresholdPercent": 10,
+  "allowedEnglishTerms": ["BrandName", "PRODUCT_CODE"]
+}
+```
+
+## Sizing Analysis
+
+`i18ntk-sizing` reports translation file sizes, key counts, average value length, and file-set mismatches across language folders.
+
+```bash
+i18ntk-sizing --source-dir ./locales --format table
+i18ntk-sizing --source-dir ./locales --detailed --output-dir ./i18ntk-reports
+```
+
+Use `--detailed` to print per-file rows in the terminal.
 
 ## Runtime API
 
-Use `i18ntk/runtime` when your application needs to read locale JSON files at runtime.
+Use `i18ntk/runtime` when an application needs to read locale JSON files at runtime.
 
 ```js
 const runtime = require('i18ntk/runtime');
@@ -193,53 +282,92 @@ console.log(runtime.getAvailableLanguages());
 runtime.refresh('fr');
 ```
 
-For a deeper walkthrough, see [docs/runtime.md](docs/runtime.md).
+See [docs/runtime.md](./docs/runtime.md) for runtime details.
 
 ## Configuration
 
-Example `.i18ntk-config`:
+i18ntk uses a project-local `.i18ntk-config` file.
+
+Example:
 
 ```json
 {
-  "version": "3.0.0",
+  "version": "3.1.1",
   "sourceDir": "./locales",
   "i18nDir": "./locales",
   "outputDir": "./i18ntk-reports",
   "sourceLanguage": "en",
   "defaultLanguages": ["de", "es", "fr", "ru"],
+  "englishContentThresholdPercent": 10,
+  "allowedEnglishTerms": ["BrandName", "PRODUCT_CODE"],
+  "autoTranslate": {
+    "placeholderMode": "preserve",
+    "concurrency": 6,
+    "batchSize": 100,
+    "progressInterval": 25,
+    "retryCount": 3,
+    "retryDelay": 1000,
+    "timeout": 15000,
+    "dryRunFirst": true,
+    "reportStdout": true,
+    "bom": false,
+    "protectionEnabled": true,
+    "protectionFile": "./i18ntk-auto-translate.json",
+    "promptProtectionSetup": true,
+    "promptProtectionUpdate": true
+  },
   "setup": {
     "completed": true
   }
 }
 ```
 
-See [docs/api/CONFIGURATION.md](docs/api/CONFIGURATION.md) for the full configuration model.
+See [docs/api/CONFIGURATION.md](./docs/api/CONFIGURATION.md) for the full configuration model.
 
-## Docs
+## Public Package Contents
 
-- [Documentation Index](https://github.com/vladnoskv/i18ntk/blob/main/docs/README.md)
-- [Getting Started](https://github.com/vladnoskv/i18ntk/blob/main/docs/getting-started.md)
-- [API Reference](https://github.com/vladnoskv/i18ntk/blob/main/docs/api/API_REFERENCE.md)
-- [Configuration Guide](https://github.com/vladnoskv/i18ntk/blob/main/docs/api/CONFIGURATION.md)
-- [Runtime API Guide](https://github.com/vladnoskv/i18ntk/blob/main/docs/runtime.md)
-- [Auto Translate Guide](https://github.com/vladnoskv/i18ntk/blob/main/docs/auto-translate.md)
-- [Scanner Guide](https://github.com/vladnoskv/i18ntk/blob/main/docs/scanner-guide.md)
-- [Environment Variables](https://github.com/vladnoskv/i18ntk/blob/main/docs/environment-variables.md)
-- [Migration Guide v3.0.0](https://github.com/vladnoskv/i18ntk/blob/main/docs/migration-guide-v3.0.0.md)
-- [Migration Guide v2.6.0](https://github.com/vladnoskv/i18ntk/blob/main/docs/migration-guide-v2.6.0.md)
-- [Migration Guide v2.5.1](https://github.com/vladnoskv/i18ntk/blob/main/docs/migration-guide-v2.5.1.md)
+The public package intentionally ships runtime and CLI files only. The publish staging script excludes development-only content such as tests, scripts, docs, release staging folders, local config files, and generated protection files.
 
-## Community
+The package includes:
+
+- CLI entry points under `main/`
+- manager commands and services
+- runtime API files under `runtime/`
+- settings UI files required at runtime
+- bundled internal UI locales
+- shared utilities required by the shipped commands
+- `README.md`, `CHANGELOG.md`, `LICENSE`, and policy files
 
-- [Contributing](CONTRIBUTING.md)
-- [Code of Conduct](CODE_OF_CONDUCT.md)
-- [Security Policy](SECURITY.md)
-- [Funding](FUNDING.md)
+The public package manifest includes `readmeFilename: "README.md"`, and the release staging script fails if `README.md` is missing or empty.
 
-## Code of Conduct
+## Documentation
+
+- [Documentation Index](./docs/README.md)
+- [Getting Started](./docs/getting-started.md)
+- [API Reference](./docs/api/API_REFERENCE.md)
+- [Configuration Guide](./docs/api/CONFIGURATION.md)
+- [Runtime API Guide](./docs/runtime.md)
+- [Auto Translate Guide](./docs/auto-translate.md)
+- [Scanner Guide](./docs/scanner-guide.md)
+- [Environment Variables](./docs/environment-variables.md)
+- [Migration Guide v3.1.1](./docs/migration-guide-v3.1.1.md)
+- [Migration Guide v3.0.0](./docs/migration-guide-v3.0.0.md)
+- [Migration Guide v2.6.0](./docs/migration-guide-v2.6.0.md)
+- [Migration Guide v2.5.1](./docs/migration-guide-v2.5.1.md)
+
+## Security
+
+- No API key is required for the default Auto Translate flow.
+- Do not store secrets in locale files, `.i18ntk-config`, or protection files.
+- Project-specific brand/product terms should be configured by the user, not hardcoded into the package.
+- Report security issues using [SECURITY.md](./SECURITY.md).
+
+## Community
 
-We are committed to providing a friendly, safe and welcoming environment for all. Please read and respect our [Code of Conduct](CODE_OF_CONDUCT.md).
+- [Contributing](./CONTRIBUTING.md)
+- [Code of Conduct](./CODE_OF_CONDUCT.md)
+- [Funding](./FUNDING.md)
 
 ## License
 
-This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.
+MIT. See [LICENSE](./LICENSE).