@photostructure/fs-metadata 0.5.0 → 0.5.1

package/C++_REVIEW_TODO.md

@@ -243,7 +243,7 @@ This document outlines a comprehensive review of all C++ files in the fs-metadat
 3. **Memory Testing Infrastructure**
    - Created comprehensive memory testing documentation (`docs/MEMORY_TESTING.md`)
    - Improved ASan configuration with proper suppressions
-   - Created standalone test runner (`scripts/run-asan.sh`)
+   - Created standalone test runner (`scripts/sanitizers-test.sh`)
    - Updated `scripts/check-memory.mjs` with better ASan support
    - All memory tests integrated into CI/CD
 

package/CHANGELOG.md

@@ -14,7 +14,7 @@ Fixed for any bug fixes.
 Security in case of vulnerabilities.
 -->
 
-## [0.5.0] - 2025-06-01
+## [0.6.0] - 2025-06-09
 
 ### Added
 
@@ -22,6 +22,10 @@ Security in case of vulnerabilities.
 - Thread safety tests for Windows platform
 - Platform-specific build scripts with automatic OS detection
 - Clang-tidy integration for C++ code analysis
+- Worker thread helper for volume metadata operations
+- NAPI_VERSION=9 definition for improved compatibility
+- Dynamic test timeout configuration based on environment (CI, platform, architecture)
+- Prebuild script for Linux GLIBC compatibility in Docker environments
 
 ### Breaking
 
@@ -32,6 +36,9 @@ Security in case of vulnerabilities.
 - Simplified ESM/CJS dual module support with unified build configuration
 - Enhanced test coverage with additional error handling and edge case tests
 - Updated all imports to use `node:` prefix for built-in modules
+- Reorganized build and test scripts for better clarity
+- Improved memory test workflow for cross-platform compatibility
+- Renamed native module target from `node_fs_meta` to `fs_metadata` for consistency
 
 ### Fixed
 
@@ -40,7 +47,9 @@ Security in case of vulnerabilities.
 - Fixed buffer allocation issues in Windows networking and volume operations
 - Enhanced resource management with better validation for empty mount points
 - Made `SystemPathPatternsDefault` values visible in TypeScript typings
-- Improved test reliability across different CI environments
+- Added Napi::HandleScope to OnOK and OnError methods for proper scope management
+- Removed unnecessary std::move operations in worker implementations
+- Resolved CI test reliability issues across different environments, particularly Alpine ARM64 emulation timeouts
 
 ## [0.4.0] - 2025-01-09
 

package/CLAUDE.md

@@ -15,92 +15,13 @@ This is @photostructure/fs-metadata - a cross-platform native Node.js module for
 - Timeout handling for unresponsive network volumes
 - ESM and CJS module support
 - Full TypeScript type definitions
+- Worker threads support with proper context isolation
 
 ### Platform-Specific Notes
 - **Linux**: Optional GIO/GVfs mount support via Gnome libraries
 - **Windows**: Uses separate threads per mountpoint for health checks to handle blocked system calls
 - **System Volumes**: Each platform handles system volumes differently; the library uses heuristics to identify them
 
-## Common Commands
-
-### Build and Development
-```bash
-# Install dependencies and build native modules
-npm install
-
-# Configure platform-specific build settings
-npm run configure
-
-# Build native bindings
-npm run node-gyp-rebuild
-
-# Create prebuilds for distribution  
-npm run prebuild
-
-# Bundle TypeScript to dist/
-npm run bundle
-```
-
-### Testing
-```bash
-# Run all tests with coverage (includes memory tests on Linux)
-npm run tests
-
-# Run CommonJS tests
-npm test cjs
-
-# Run ESM tests
-npm test esm
-
-# Test memory leaks (JavaScript)
-npm run test:memory
-
-# Run valgrind memory analysis (Linux only)
-npm run test:valgrind
-
-# Run comprehensive memory tests (JavaScript + valgrind on Linux)
-npm run tests:memory
-
-# Run AddressSanitizer tests (Linux only)
-npm run asan
-
-# Run a specific test file (no coverage)
-npm test volume_metadata
-```
-
-### Code Quality
-```bash
-# Run ESLint
-npm run lint
-
-# Fix ESLint issues
-npm run lint:fix
-
-# Format code (all formats)
-npm run fmt
-
-# Format C++ code only
-npm run fmt:cpp
-
-# Format TypeScript only
-npm run fmt:ts
-
-# Type checking
-npm run compile
-```
-
-### Pre-commit
-```bash
-# Full precommit check (fmt, clean, prebuild, tests)
-npm run precommit
-```
-
-### Documentation
-```bash
-# Generate API documentation
-npm run docs
-```
-
 ## Architecture Overview
 
 ### Core Structure
@@ -122,23 +43,29 @@ npm run docs
 - Provides prebuilt binaries via `prebuildify` for common platforms
 - Supports both ESM and CJS module formats
 
+### Cross-Module Compatibility
+- **Directory Path Resolution**: Use `_dirname()` from `./dirname` instead of `__dirname`
+  - Works in both CommonJS and ESM contexts
+  - Handles Jest test environments correctly
+  - Example: `const dir = _dirname()` instead of `const dir = __dirname`
+
 ### Testing Strategy
 - Jest for both CJS and ESM test suites
 - Platform-specific test expectations handled via `isWindows`, `isMacOS`, `isLinux` helpers
 - Memory leak testing with garbage collection monitoring
 - Coverage thresholds: 80% for all metrics
 
-### Memory Testing
-- JavaScript memory tests: `npm run test:memory` - uses GC and heap monitoring
-- Valgrind integration: `npm run test:valgrind` - runs on Linux only, checks for memory leaks
-- AddressSanitizer: `npm run asan` - runs on Linux only, detects memory errors and leaks (~2x faster than Valgrind)
-- Comprehensive memory tests: `npm run tests:memory` - runs all memory tests appropriate for the platform
-- Automated test runners: `scripts/valgrind-test.mjs` and `scripts/run-asan.sh`
-- CI/CD includes both valgrind and ASAN tests via `.github/workflows/memory-tests.yml`
-- Cross-platform memory check script: `scripts/check-memory.mjs` handles platform differences
-- Suppression files: `.valgrind.supp` (Valgrind), `.lsan-suppressions.txt` (LeakSanitizer)
-- Memory tests are integrated into `npm run tests` pipeline on Linux
-- See `docs/MEMORY_TESTING.md` for detailed memory testing guide
+### Testing File System Metadata
+- **Important**: File system metadata like `available` space, `used` space, and other dynamic properties change continuously as other processes run on the machine
+- Tests should **never** expect exact equality for these values between multiple calls
+- **Do not** make range assertions (e.g., `available > 0` or `used < size`) because:
+  - Files can be created or deleted between calls (potentially gigabytes)
+  - The changes can be dramatic and unpredictable
+- Instead, for dynamic metadata:
+  - Only verify the value exists and has the correct type
+  - Use `typeof result.available === 'number'` rather than range checks
+  - Focus on testing static properties (e.g., `size`, `mountFrom`, `fstype`) for exact equality
+  - Consider using snapshot testing only for stable properties
 
 ### Timeout Handling
 - Default timeout for volume operations to handle unresponsive network mounts
@@ -166,4 +93,254 @@ console.dir({ volumeMetadata });
 // Check if a file is hidden
 import { isHidden } from "@photostructure/fs-metadata";
 const hidden = await isHidden("/path/to/file");
-```
+```
+
+## CI/CD Test Reliability Guidelines
+
+Based on analysis of recent test failures, here are critical patterns to avoid flaky tests:
+
+### 1. Benchmark and Performance Tests
+- **Problem**: "Cannot log after tests are done" errors in worker_threads.test.ts
+- **Solution**: 
+  - Always await all async operations before test completion
+  - Use proper test lifecycle hooks (afterEach/afterAll) for cleanup
+  - Avoid console.log in async contexts without proper synchronization
+  - Consider using test.concurrent with explicit done() callbacks
+
+### 2. Alpine Linux ARM64 Issues
+- **Problem**: Tests timeout on emulated Alpine ARM64 environments
+- **Solution**:
+  - Skip process-spawning tests on Alpine ARM64 (`if (isAlpine && isARM64)`)
+  - Use increased timeout multipliers (20x) for emulated environments
+  - Detect emulation via `/proc/cpuinfo` or environment checks
+  - Consider separate test suites for native vs emulated environments
+
+### 3. Worker Thread Management
+- **Problem**: Race conditions in concurrent worker operations
+- **Solution**:
+  - Implement proper worker pool management with size limits
+  - Use Promise.allSettled() instead of Promise.all() for parallel operations
+  - Add explicit cleanup in test teardown to terminate all workers
+  - Set reasonable concurrency limits based on environment (CPU cores)
+
+### 4. Timeout Test Reliability
+- **Problem**: Timeout tests fail due to timing precision issues
+- **Solution**:
+  - Never use exact timing assertions (e.g., expect 100ms)
+  - Use ranges with adequate margins (e.g., 90-110ms)
+  - Account for CI environment variability (slower machines)
+  - Consider mocking timers for deterministic behavior
+
+### 5. File System Operations
+- **Problem**: ENOENT errors for test directories, permission issues
+- **Solution**:
+  - Always use unique temporary directories per test
+  - Clean up test artifacts in afterEach hooks
+  - Check directory existence before operations
+  - Handle platform-specific path separators
+
+### 6. Memory and Resource Leaks
+- **Problem**: Tests don't properly clean up resources
+- **Solution**:
+  - Explicitly close all file handles, network connections
+  - Use try-finally blocks for resource cleanup
+  - Monitor memory usage in long-running tests
+  - Implement proper garbage collection triggers
+
+### 7. Platform-Specific Failures
+- **Problem**: Different behavior across Windows/macOS/Linux
+- **Solution**:
+  - Use platform detection helpers consistently
+  - Skip platform-specific tests appropriately
+  - Account for filesystem differences (case sensitivity, path formats)
+  - Test with platform-specific CI matrices
+
+### 8. Jest Configuration
+- **Problem**: Tests interfere with each other
+- **Solution**:
+  - Use `--runInBand` for tests with shared resources
+  - Clear module cache between tests when needed
+  - Isolate tests that spawn processes
+  - Configure proper test timeouts per environment
+
+### Async Cleanup Anti-Patterns
+
+**IMPORTANT**: The following approaches are NOT valid solutions for async cleanup issues:
+
+```javascript
+// BAD: Arbitrary timeouts in tests
+await new Promise((resolve) => setTimeout(resolve, 100));
+
+// BAD: Forcing garbage collection
+if (global.gc) {
+  global.gc();
+}
+
+// BAD: Adding setImmediate in afterAll to "fix" hanging tests
+afterAll(async () => {
+  await new Promise((resolve) => setImmediate(resolve));
+});
+```
+
+**Why these are problematic:**
+
+1. **Arbitrary timeouts** are race conditions waiting to happen. They might work on fast machines but fail on slower CI runners.
+2. **Forcing GC** should never be required for correct behavior. If your code depends on GC for correctness, it has a fundamental design flaw.
+3. **setImmediate/nextTick delays** in cleanup hooks don't fix the root cause - they just paper over the real issue.
+4. These approaches mask the real problem instead of fixing it.
+
+**Note**: This is different from legitimate uses of timeouts, such as:
+
+- Waiting for time to pass to test timestamp changes
+- Rate limiting or throttling tests
+- Testing timeout behavior itself
+
+The anti-pattern is using timeouts or GC to "fix" async cleanup issues.
+
+**What to do instead:**
+
+1. Find the actual resource that's keeping the process alive (use `--detectOpenHandles`)
+2. Ensure all database connections are properly closed
+3. Ensure all file handles are closed
+4. Cancel or await all pending async operations
+5. Use proper resource management patterns (RAII, try-finally, using statements)
+
+### Windows-Compatible Directory Cleanup
+
+**IMPORTANT**: Never use `fs.rmSync()` or `fs.rm()` without proper Windows retry logic for directory cleanup in tests.
+
+**Problem**: On Windows, file handles and directory locks can remain active longer than on Unix systems, causing `EBUSY` errors during cleanup.
+
+**Proper Solution**: Use `fsp.rm()` (async) with retry options:
+
+```typescript
+await fsp.rm(tempDir, {
+  recursive: true,
+  force: true,
+  maxRetries: process.platform === "win32" ? 3 : 1,
+  retryDelay: process.platform === "win32" ? 100 : 0,
+});
+```
+
+**Best Practice**: Use existing test utilities that handle Windows-compatible cleanup patterns. Don't manually clean up temp directories - let the test framework handle it with proper retry logic.
+
+### Adaptive Timeout Testing
+
+**Problem**: Fixed timeouts don't account for varying CI environment performance.
+
+**Root Causes**:
+
+- Alpine Linux (musl libc) is 2x slower than glibc
+- ARM64 emulation on x64 runners is 5x slower
+- Windows process operations are 4x slower
+- macOS VMs are 4x slower
+- CI environments have resource constraints
+
+**Solutions**:
+
+```typescript
+// DON'T: Use fixed timeouts
+test("my test", async () => {
+  // Test code
+}, 10000);
+
+// DO: Use adaptive timeouts based on environment
+import { getTestTimeout } from "./test-utils/test-timeout-config";
+
+test(
+  "my test",
+  async () => {
+    // Test code
+  },
+  getTestTimeout(10000),
+);
+
+// DO: Account for platform timing differences
+const timingMultiplier = process.platform === "win32" ? 4 :
+                        process.platform === "darwin" ? 4 :
+                        process.env.CI ? 2 : 1;
+```
+
+### Multi-Process Test Synchronization
+
+**Problem**: Multi-process tests failing due to race conditions between process startup and test assertions.
+
+**Root Cause**: The timing between process startup and resource acquisition varies significantly by platform.
+
+**Solutions**:
+
+```typescript
+// DON'T: Assume immediate process readiness
+const proc = spawn(nodeCmd, [script]);
+const result = await waitForProcessResult(proc);
+expect(result).toBe("expected_outcome"); // May fail due to timing
+
+// DO: Use explicit synchronization signals
+const script = `
+  // Setup code
+  console.log("READY");  // Signal readiness
+  // Main test logic
+  console.log("RESULT:" + outcome);
+`;
+
+const proc = spawn(process.execPath, ["-e", script]);
+await waitForOutput(proc, "READY"); // Wait for process to be ready
+const result = await waitForOutput(proc, "RESULT:");
+expect(result.split(":")[1]).toBe("expected_outcome");
+```
+
+### Wait-for-Condition Pattern
+
+**Problem**: Tests failing because they don't wait for asynchronous conditions to be met.
+
+**Solution**: Implement robust condition waiting with platform-aware timing:
+
+```typescript
+async function waitForCondition(
+  check: () => boolean | Promise<boolean>,
+  options: {
+    maxAttempts?: number;
+    delay?: number;
+    timeoutMs?: number;
+  } = {}
+) {
+  const {
+    maxAttempts = 50,
+    delay = 100,
+    timeoutMs = 30000
+  } = options;
+  
+  const startTime = Date.now();
+  
+  for (let i = 0; i < maxAttempts; i++) {
+    if (Date.now() - startTime > timeoutMs) {
+      throw new Error(`Condition not met within ${timeoutMs}ms`);
+    }
+    
+    if (await check()) return true;
+    await new Promise((resolve) => setTimeout(resolve, delay));
+  }
+  
+  return false;
+}
+
+// Usage example
+await waitForCondition(
+  () => fs.existsSync(expectedFile),
+  { timeoutMs: getTestTimeout(10000) }
+);
+```
+
+### Best Practices Summary
+1. **Always clean up**: Resources, timers, workers, file handles
+2. **Never assume timing**: Use ranges and adaptive timeouts, not exact values
+3. **Isolate tests**: Each test should be independent
+4. **Platform awareness**: Skip tests that can't work on certain platforms
+5. **Proper async handling**: Always await or return promises
+6. **Resource limits**: Don't spawn unlimited workers/processes
+7. **Environment detection**: Adjust behavior for CI vs local
+8. **Deterministic tests**: Mock external dependencies when possible
+9. **Explicit synchronization**: Use signals for multi-process coordination
+10. **Robust waiting**: Use condition-based waiting instead of arbitrary timeouts
+11. **Windows compatibility**: Use retry logic for file operations on Windows
+12. **Anti-pattern awareness**: Avoid masking problems with timeouts or forced GC

package/CONTRIBUTING.md

@@ -54,18 +54,59 @@ into account both Windows and POSIX systems.
 **Why**: Windows Command Prompt/PowerShell parses the entire command line before execution, including the Unix-specific parts that would never run on Windows. Even though constructs like `node scripts/is-platform.mjs win32 || <unix-command>` would exit early on Windows, the shell still tries to parse the syntax after `||`.
 
 **Solution**: For platform-specific npm scripts that use shell operators:
+
 - Create a wrapper Node.js script that handles platform detection internally (see `scripts/clang-tidy.mjs`)
 - The wrapper can use `process.platform` or `os.platform()` to detect Windows and exit early
 - Unix-specific commands can then be spawned using `child_process.spawn()` with `sh -c`
 
 **Example**: The `clang-tidy` npm script was moved from:
+
 ```json
 "clang-tidy": "node scripts/is-platform.mjs win32 || (npm run configure && bear -- npm run node-gyp-rebuild && find src -name '*.cpp' -o -name '*.h' | grep -E '\\.(cpp|h)$' | grep -v -E '(windows|darwin)/' | xargs clang-tidy)"
 ```
 
 To:
+
 ```json
 "clang-tidy": "node scripts/clang-tidy.mjs"
 ```
 
 Where the script handles platform detection and command execution internally.
+
+## npm Script Naming Conventions
+
+This project follows consistent naming patterns for npm scripts to improve discoverability and maintainability:
+
+### Action:Target Format
+
+Scripts follow an `action:target` pattern where:
+
+- **action**: The operation being performed (`build`, `clean`, `lint`, `test`)
+- **target**: What the action operates on (`native`, `ts`, `dist`)
+
+Examples:
+
+- `build:native` - Build native C++ code
+- `lint:ts` - Lint TypeScript code
+- `clean:dist` - Clean distribution files
+
+### Parallel Execution
+
+Actions that have multiple targets can be run in parallel using wildcards:
+
+- `npm run clean` runs all `clean:*` scripts
+- `npm run lint` runs all `lint:*` scripts
+- Uses `run-p` from npm-run-all for parallel execution
+
+### Special Namespaces
+
+- **memory:\*** - Memory testing scripts that should not run automatically with `test:*`
+  - `memory:test` - Comprehensive memory testing suite
+  - Run with `ENABLE_ASAN=1` to include sanitizer tests
+
+### Naming Guidelines
+
+- Use explicit names to avoid ambiguity (e.g., `setup:native` instead of just `setup`)
+- Group related scripts by action prefix for easy wildcard execution
+- Avoid names that could cause npm lifecycle conflicts (e.g., `prebuild` vs `build`)
+- Use descriptive suffixes that clearly indicate the target or purpose

package/README.md

@@ -16,7 +16,8 @@ Built and supported by [PhotoStructure](https://photostructure.com).
 - Cross-platform support:
   - Windows 10+ (x64)
   - macOS 14+
-  - Ubuntu 22+ (x64, arm64) (with Gnome GIO/`GVfs` mount support where available)
+  - Debian 11/Ubuntu 20.04+ (x64, arm64) (with Gnome GIO/`GVfs` mount support where available)
+  - Alpine 3.21+ (x64, arm64)
 
 - [List all mounted volumes/drives](https://photostructure.github.io/fs-metadata/functions/getVolumeMountPoints.html)
 
@@ -41,6 +42,24 @@ Built and supported by [PhotoStructure](https://photostructure.com).
 
 - Comprehensive test coverage
 
+## Supported Platforms
+
+Prebuilt binaries are provided for the following platforms:
+
+| Platform | Architecture | Node.js | Minimum OS Version |
+|----------|--------------|---------|-------------------|
+| Windows | x64 | 20+ | Windows 10 |
+| macOS | x64 | 20+ | macOS 14 (Sonoma) |
+| macOS | arm64 | 20+ | macOS 14 (Sonoma) |
+| Linux (glibc) | x64 | 20+ | Debian 11 (Bullseye), Ubuntu 20.04, GLIBC 2.31+ |
+| Linux (glibc) | arm64 | 20+ | Debian 11 (Bullseye), Ubuntu 20.04, GLIBC 2.31+ |
+| Linux (musl) | x64 | 20+ | Alpine 3.21 |
+| Linux (musl) | arm64 | 20+ | Alpine 3.21 |
+
+Notes:
+- Linux binaries require GLIBC 2.31+ (Debian 11 Bullseye or newer). **Note**: this means the `node:20` docker image is **not** supported, due to the last several major versions of `node-gyp-build` requiring both a newer python and newer C++ version.
+- Electron is supported via [Node-API](https://nodejs.org/api/n-api.html) compatibility
+
 ## Installation
 
 ```bash

package/WINDOWS_DEBUG_GUIDE.md

@@ -0,0 +1,89 @@
+# Windows and Alpine Linux Debugging Guide for debuglog.test.ts
+
+## Issue Summary
+The debuglog.test.ts file was failing on:
+1. **Windows CI**: "Converting circular structure to JSON" error in Jest's message passing system when `execFileSync` threw errors with circular references
+2. **Alpine Linux**: `ETIMEDOUT` errors when spawning child processes with `npx`
+
+## Changes Made
+
+1. **Replaced execFileSync with spawnSync**
+   - `spawnSync` provides better process control and doesn't throw errors with circular references
+   - Added `windowsHide: true` to prevent console windows from appearing on Windows
+   - Added `shell: true` on Windows only for proper npx execution
+   - Increased timeout to 30 seconds for slower environments like Alpine Linux containers
+
+2. **Enhanced Error Handling**
+   - Extract only serializable properties from errors (message, code, status, stderr, stdout)
+   - Added detailed logging for Windows-specific debugging
+   - Prevent circular reference errors from reaching Jest's worker communication
+
+3. **Improved Child Process Scripts**
+   - Added uncaughtException and unhandledRejection handlers
+   - Use `process.stdout.write` instead of `console.log` for cleaner output
+   - Added detailed error logging with stack traces to stderr
+
+## Manual Testing on Windows
+
+To test these changes on your Windows test box:
+
+1. **Run the specific test file:**
+   ```powershell
+   npm test -- src/debuglog.test.ts
+   ```
+
+2. **Run with verbose output to see debugging info:**
+   ```powershell
+   npm test -- src/debuglog.test.ts --verbose
+   ```
+
+3. **If tests fail, check for:**
+   - Console output showing "Windows child process error" with detailed info
+   - Error messages, status codes, and stderr output
+   - Whether `npx tsx` is available and working correctly
+
+4. **To test individual child scripts directly:**
+   ```powershell
+   # Test debuglog-child.ts
+   $env:NODE_DEBUG="fs-metadata"
+   npx tsx src/test-utils/debuglog-child.ts
+   
+   # Test debuglog-enabled-child.ts
+   $env:NODE_DEBUG="fs-metadata"
+   npx tsx src/test-utils/debuglog-enabled-child.ts
+   ```
+
+5. **Check for hanging processes:**
+   - The tests now have a 5-second timeout
+   - If a test hangs, it will fail with a timeout error
+   - Check Task Manager for any orphaned node.exe processes
+
+## Debugging Tips
+
+1. **Enable Node.js debugging:**
+   ```powershell
+   $env:NODE_OPTIONS="--trace-warnings"
+   npm test -- src/debuglog.test.ts
+   ```
+
+2. **Check npx/tsx availability:**
+   ```powershell
+   npx --version
+   npx tsx --version
+   ```
+
+3. **If npx tsx fails, try direct execution:**
+   ```powershell
+   node_modules/.bin/tsx src/test-utils/debuglog-child.ts
+   ```
+
+## Expected Behavior
+
+- All 10 tests in debuglog.test.ts should pass
+- No "Converting circular structure to JSON" errors
+- Child processes should exit cleanly without hanging
+- Detailed error information should be logged on failures
+
+## Rollback Plan
+
+If these changes cause new issues, the previous approach using execFileSync can be restored by reverting this commit. The key difference is the switch from execFileSync to spawnSync and the enhanced error handling.

package/binding.gyp

@@ -4,7 +4,7 @@
   },
   "targets": [
     {
-      "target_name": "node_fs_meta",
+      "target_name": "fs_metadata",
       "sources": [
         "src/binding.cpp"
       ],
@@ -16,7 +16,8 @@
         "<!(node -p \"require('node-addon-api').gyp\")"
       ],
       "defines": [
-        "NAPI_CPP_EXCEPTIONS"
+        "NAPI_CPP_EXCEPTIONS",
+        "NAPI_VERSION=9"
       ],
       "conditions": [
         [