@photostructure/fs-metadata 0.6.0 → 0.7.0

package/doc/windows-build.md

@@ -0,0 +1,226 @@
+# Windows Build Issues and Solutions
+
+## Overview
+
+This document summarizes the Windows-specific build issues encountered with fs-metadata, particularly the "No Target Architecture" error from the Windows SDK, and various approaches to resolving it.
+
+## The Core Issue
+
+When building on Windows with node-gyp/prebuildify, we encounter:
+
+```
+C:\Program Files (x86)\Windows Kits\10\Include\10.0.19041.0\um\winnt.h(173,1): fatal error C1189: #error:  "No Target Architecture"
+```
+
+This error occurs because:
+
+1. The Windows SDK headers require architecture macros (`_M_X64`, `_WIN64`, etc.) to be defined before including `<windows.h>`
+2. Node-gyp doesn't always properly pass these defines when using prebuildify
+3. The `target_arch` variable in binding.gyp conditions might not be evaluated correctly
+
+## Why This Happens
+
+### Root Cause
+
+- Prebuildify calls node-gyp with `--arch` (previously `--target-arch`)
+- The architecture should default to `os.arch()` (which correctly returns `x64` on Windows)
+- However, the architecture defines aren't being set properly in the MSVC project generated by node-gyp
+
+### Why Other Projects Don't Have This Issue
+
+Projects like node-sqlite3 don't typically encounter this because:
+
+1. They don't define architecture-specific macros in binding.gyp (these should come from the compiler)
+2. They may not be using prebuildify, or are using different build configurations
+3. They might be including Windows headers in a different order
+
+## Attempted Solutions
+
+### 1. Architecture Defines in binding.gyp (Not Sufficient)
+
+Attempted to add architecture defines conditionally:
+
+```json
+"conditions": [
+  [
+    "target_arch=='x64'",
+    {
+      "defines": [
+        "_M_X64",
+        "_WIN64"
+      ]
+    }
+  ]
+]
+```
+
+**Result**: Doesn't reliably fix prebuildify builds
+
+### 2. Windows Header Wrapper (Failed)
+
+Created `windows_headers.h` to force architecture defines:
+
+```cpp
+#ifndef _M_X64
+  #define _M_X64 1
+  #define _AMD64_ 1
+  #define _WIN64 1
+#endif
+#include <windows.h>
+```
+
+**Result**: Didn't resolve the issue for all files
+
+### 3. Direct Source File Defines (Don't Use - Breaks ARM64)
+
+Adding architecture defines at the top of each Windows source file:
+
+```cpp
+// Force architecture defines for Windows SDK
+#ifndef _M_X64
+  #define _M_X64 1
+#endif
+#ifndef _AMD64_
+  #define _AMD64_ 1
+#endif
+#ifndef _WIN64
+  #define _WIN64 1
+#endif
+```
+
+**Result**: This hardcodes x64 defines and would break ARM64 builds
+
+### 4. CL Environment Variable (Current Solution)
+
+Created `scripts/prebuildify-wrapper.ts` that sets the `CL` environment variable:
+
+```typescript
+if (currentArch === "x64") {
+  env.CL = "/D_M_X64 /D_WIN64 /D_AMD64_";
+} else if (currentArch === "arm64") {
+  env.CL = "/D_M_ARM64 /D_WIN64";
+}
+```
+
+**Result**: Works correctly for both x64 and ARM64 architectures
+
+## Current Status
+
+### What Works
+
+- Building with the prebuildify wrapper that sets CL environment variable
+- This solution properly handles both x64 and ARM64 architectures
+
+### What Doesn't Work
+
+- Relying on node-gyp/prebuildify to properly evaluate binding.gyp conditions
+- Direct source file defines (would break ARM64 builds)
+- windows_compat.h wrapper (can't distinguish architectures at compile time)
+
+### Debug Builds Removed
+
+- Debug builds have been removed from the project as they provided no value
+- Windows debug CRT builds cannot be loaded by Node.js due to missing runtime dependencies
+- JavaScript-based memory testing provides adequate leak detection
+
+## Other Important Findings
+
+### Node.js Version Compatibility
+
+- Jest 30 doesn't support Node.js 23
+- CI/CD workflows use Node.js 20, 22, and 24
+- Removed Node.js 23 from test matrices
+
+### Related Issues
+
+1. **Memory Test Failures**: Initially appeared as "unsupported engine" warnings
+2. **Architecture Detection**: `process.arch` correctly returns `x64` on Windows
+3. **Prebuildify Defaults**: Should use `os.arch()` and `os.platform()` but something in the chain breaks
+
+## Potential Future Solutions
+
+### 1. Wrapper Script for Prebuildify
+
+Create a TypeScript script to explicitly pass architecture:
+
+```typescript
+import { arch, platform } from "os";
+import { spawn } from "child_process";
+
+spawn(
+  "prebuildify",
+  [
+    "--napi",
+    "--tag-libc",
+    "--strip",
+    "--arch",
+    arch(),
+    "--platform",
+    platform(),
+  ],
+  { stdio: "inherit" },
+);
+```
+
+### 2. Environment Variables
+
+Set architecture explicitly:
+
+```bash
+set npm_config_arch=x64
+npm run build
+```
+
+### 3. Investigation Areas
+
+- Why `target_arch` isn't being evaluated in binding.gyp conditions
+- Why prebuildify + node-gyp isn't passing architecture defines to MSVC
+- Whether this is a node-gyp bug or a configuration issue
+
+## References
+
+### GitHub Issues
+
+- nodejs/node-gyp#600: "gyp: name 'target_arch' is not defined"
+- nodejs/node-gyp#2813: "Build defaults to x86 on a x64 Windows machine"
+- prebuild/prebuildify#53: Changed from `--target-arch` to `--arch`
+
+### Similar Projects
+
+- node-sqlite3: Uses simpler binding.gyp without architecture defines
+- node-canvas: Uses `WIN32_LEAN_AND_MEAN` but not architecture defines
+- serialport: Focus on exception handling and warning suppression
+
+## Why a Wrapper Header Won't Work
+
+**Important Discovery**: A single `windows_compat.h` wrapper cannot solve this issue because:
+
+1. At compile time, we can't distinguish between x64 and ARM64 (both define `_WIN64`)
+2. The architecture information (`target_arch`) is only available in binding.gyp
+3. Prebuildify doesn't properly pass the conditional defines from binding.gyp to the compiler
+
+This means any compile-time logic would incorrectly set x64 defines on ARM64 builds or vice versa.
+
+## Conclusion
+
+The "No Target Architecture" error is a known limitation of how prebuildify and node-gyp handle Windows builds. The current workaround of adding architecture defines directly to source files is the most reliable solution.
+
+**Why Other Projects Don't Have This Issue**:
+
+- Projects like node-sqlite avoid `<windows.h>` by using cross-platform APIs
+- Most Node.js addons don't directly interface with Windows system APIs
+
+**The Solution**:
+Use `scripts/prebuildify-wrapper.ts` which sets the CL environment variable with architecture-specific defines. This approach:
+
+1. Works reliably with prebuildify
+2. Correctly handles both x64 and ARM64 architectures
+3. Doesn't require modifying source files
+4. Avoids hardcoding architecture assumptions
+
+## Next Steps
+
+1. Consider opening an issue with prebuildify about Windows architecture detection
+2. Test if explicit `--arch x64` flags to prebuildify resolve the issue
+3. Investigate why other native modules don't encounter this problem
+4. Consider if our include order or binding.gyp structure is unusual

package/doc/windows-clang-tidy.md

@@ -0,0 +1,72 @@
+# Windows clang-tidy Support
+
+## Status
+
+clang-tidy has limited support on Windows due to fundamental incompatibilities between clang and MSVC headers. While we've implemented Windows support in our unified `scripts/clang-tidy.ts`, users should be aware of the limitations.
+
+## Current Implementation
+
+1. **Unified Script**: `scripts/clang-tidy.ts` works on all platforms including Windows
+2. **Automatic Detection**: Finds Node.js headers, MSVC includes, and Windows SDK paths
+3. **Configuration**: Windows-specific checks in `src/windows/.clang-tidy`
+4. **Fallback**: Minimal configuration (`.clang-tidy-windows-minimal`) for basic checks
+
+## Known Issues
+
+### Header Compatibility
+
+- clang-tidy cannot fully parse MSVC STL headers, resulting in errors like:
+  - `no member named 'max' in namespace 'std'`
+  - `unknown type name 'namespace'`
+  - `no template named 'pointer_traits'`
+
+### Root Cause
+
+- MSVC headers use Microsoft-specific extensions that clang doesn't fully support
+- Node.js native addon headers add additional complexity
+- Even with clang-cl mode, full compatibility isn't achieved
+
+## Recommendations
+
+### For Windows Developers
+
+1. **Use Visual Studio Code Analysis**: The built-in Code Analysis in Visual Studio provides better Windows-specific checking
+2. **Focus on Warnings**: Despite header errors, clang-tidy still catches many issues:
+   - Uninitialized variables
+   - RAII violations
+   - Member initialization issues
+   - Ownership problems
+
+3. **Run Anyway**: Even with errors, the warnings are valuable:
+   ```bash
+   npm run lint:native
+   ```
+
+### For CI/CD
+
+Consider skipping clang-tidy on Windows in CI to avoid noise:
+
+```bash
+# In CI scripts
+if [ "$OS" != "Windows_NT" ]; then
+  npm run lint:native
+fi
+```
+
+## Future Improvements
+
+- Monitor clang-tidy development for better MSVC support
+- Consider using Visual Studio's built-in clang-tidy integration
+- Investigate using `clangd` as an alternative
+
+## What Still Works
+
+Despite the header issues, clang-tidy on Windows can still detect:
+
+- Uninitialized variables
+- Missing RAII usage
+- Resource leaks in your code (not system headers)
+- Code style issues
+- Many security vulnerabilities
+
+The key is to focus on warnings in your own code files, not system header errors.

package/doc/windows-memory-testing.md

@@ -0,0 +1,108 @@
+# Windows Memory Testing
+
+## Overview
+
+On Windows, debug builds with CRT memory leak detection face a loading issue where Node.js cannot load native modules that use Windows long paths (`\\?\` prefix). This document describes our approach to memory leak detection on Windows.
+
+## The Problem
+
+When building with `node-gyp build --debug`, the debug build cannot be loaded by Node.js due to:
+
+1. **Missing Debug Runtime Dependencies**: Debug builds require debug versions of the Visual C++ runtime libraries (`ucrtbased.dll`, `vcruntime*d.dll`)
+2. **UNC Path Issues**: Node.js module loader has issues with Windows extended-length paths (`\\?\C:\...`)
+
+## Our Solution
+
+We've split Windows security testing into three categories:
+
+### 1. Input Security Tests (`windows-input-security.test.ts`)
+
+Tests for handling malicious inputs:
+
+- Path traversal protection
+- Buffer overflow protection
+- Invalid UTF-8 handling
+- Device name rejection
+- Alternate data stream rejection
+
+These tests run with regular Release builds and don't require debug memory detection.
+
+### 2. Resource Security Tests (`windows-resource-security.test.ts`)
+
+Tests for resource/handle leaks during operations:
+
+- Concurrent operations safety
+- Handle cleanup on timeout
+- Basic memory pattern monitoring
+
+These tests validate functionality with Release builds.
+
+### 3. Memory Leak Detection (`windows-memory-check.test.ts`)
+
+JavaScript-based memory monitoring that works with Release builds:
+
+- Heap memory usage tracking with forced garbage collection
+- Memory growth pattern analysis
+- Handle count monitoring (via `process.report`)
+- Per-operation memory cost calculation
+
+## Alternative Windows Memory Leak Detection Tools
+
+**Note: Traditional Windows memory debugging tools have significant compatibility issues with Node.js native modules.**
+
+### Why These Tools Don't Work Well with Node.js
+
+1. **Dr. Memory**: Known incompatibility with Node.js - fails with "Unable to load client library: ucrtbase.dll" error (GitHub issue #2531)
+2. **Visual Leak Detector (VLD)**: Requires debug builds which have loading issues with Node.js native modules
+3. **Application Verifier**: Cannot properly hook into Node.js's memory management system
+4. **Windows CRT Debug Heap**: Requires debug builds that face the UNC path loading issue
+
+### Our Approach
+
+Instead of these traditional tools, we use:
+
+- JavaScript-based memory monitoring with forced garbage collection
+- Process handle tracking via Node.js's built-in `process.report`
+- Heap usage patterns analysis over multiple iterations
+- This approach provides adequate memory leak detection without the compatibility issues
+
+## Running Memory Tests
+
+```bash
+# Run all memory tests (JavaScript-based)
+npm test -- src/windows-memory-check.test.ts
+
+# Run resource security tests
+npm test -- src/windows-resource-security.test.ts
+
+# Run input security tests
+npm test -- src/windows-input-security.test.ts
+
+# Run full memory check suite (includes debug build attempt)
+npm run check:memory
+```
+
+## Key Metrics Monitored
+
+1. **Heap Memory Growth**: Should be < 100KB per operation (Windows APIs may have higher baseline)
+2. **Total Memory Retention**: Should be < 5MB after GC
+3. **Handle Count**: Should not increase by more than 10
+4. **Concurrent Operation Memory**: Should be < 10MB for 100 operations
+
+## Future Improvements
+
+1. Investigate fixing debug build loading:
+   - Ship debug CRT dependencies
+   - Use manifest embedding for dependency resolution
+   - Investigate short path alternatives to avoid UNC paths
+
+2. Enhanced JavaScript monitoring:
+   - Native memory tracking via N-API
+   - V8 heap snapshots comparison
+   - Event loop lag correlation
+   - Windows performance counters integration
+
+3. Native-level monitoring:
+   - Custom memory allocator wrappers
+   - Direct Windows API handle tracking
+   - Integration with Node.js's built-in diagnostics

package/doc/windows-prebuildify-arm64.md

@@ -0,0 +1,232 @@
+# Windows ARM64 Development and Prebuildify Setup
+
+This document provides guidance for Windows ARM64 native module development, prebuildify configuration, and troubleshooting common issues.
+
+## Overview
+
+Windows ARM64 support for native Node.js modules is a rapidly evolving area. This document captures key learnings from investigating build failures and researching the ecosystem.
+
+## Prebuildify Architecture Support
+
+### How Prebuildify Works
+
+1. **Build Time**: Creates prebuilt binaries for different platforms/architectures
+2. **Package Time**: Stores binaries in `./prebuilds/[platform]-[arch]/`
+3. **Runtime**: Uses `node-gyp-build` to load the appropriate binary
+
+### Platform Naming Conventions
+
+- Windows x64: `win32-x64`
+- Windows ARM64: `win32-arm64`
+- Linux x64: `linux-x64`
+- macOS x64: `darwin-x64`
+- macOS ARM64: `darwin-arm64`
+
+### Scoped Package Handling
+
+For scoped npm packages like `@photostructure/fs-metadata`:
+
+- Package scope uses `+` instead of `/` in filenames
+- Example: `@photostructure+fs-metadata.glibc.node`
+- This is standard prebuildify behavior
+
+## Windows ARM64 Specific Considerations
+
+### Current State of the Ecosystem (2025)
+
+1. **GitHub Actions Support**: Windows ARM64 runners are in public preview
+   - Free for public repositories
+   - Run Windows 11 Desktop image
+   - Native ARM64 compilation (no emulation)
+
+2. **Common Issues in Other Projects**:
+   - **Cypress**: Requires manual addition of "win32-arm64" as valid OS
+   - **LMDB-JS**: Reports "No native build was found for platform=win32 arch=arm64"
+   - **Cloudflare workerd**: Requires manual workarounds in install scripts
+   - Many projects fall back to x64 binaries (run via emulation)
+
+### Architecture Detection Challenges
+
+Windows SDK headers require architecture-specific defines before including `<windows.h>`:
+
+- x64: `/D_M_X64 /D_WIN64 /D_AMD64_`
+- ARM64: `/D_M_ARM64 /D_WIN64`
+
+Our solution: Set `CL` environment variable in build scripts (see `scripts/prebuildify-wrapper.ts`)
+
+## GitHub Actions Configuration
+
+### Best Practices
+
+```yaml
+prebuild-win-arm64:
+  runs-on: windows-11-arm # Native ARM64 runner
+  steps:
+    - uses: actions/checkout@v4
+    - uses: actions/setup-node@v4
+      with:
+        node-version: 20
+        cache: "npm"
+    - run: npm ci --ignore-scripts
+    - run: npm run build:native
+    - uses: actions/upload-artifact@v4
+      with:
+        name: prebuilds-windows-11-arm
+        path: prebuilds/
+```
+
+### Testing Configuration
+
+```yaml
+test-win-arm64:
+  needs: [prebuild-win-arm64]
+  runs-on: windows-11-arm
+  steps:
+    - uses: actions/download-artifact@v4
+      with:
+        path: ./prebuilds
+        merge-multiple: true
+    - run: npm ci
+    - run: npm run tests
+```
+
+## Common Problems and Solutions
+
+### Problem: Jest Worker Process Failures
+
+**Symptoms**:
+
+- "Jest worker encountered 4 child process exceptions, exceeding retry limit"
+- Tests fail only in CI, not locally
+
+**Root Causes**:
+
+1. Module resolution differs in worker threads
+2. `__dirname` context changes in Jest environment
+3. Relative paths may not resolve correctly
+
+**Solutions**:
+
+1. Use multiple fallback paths when loading native modules
+2. Try `process.cwd()` in addition to relative paths
+3. Ensure prebuilds are in expected locations
+
+### Problem: Native Module Not Found
+
+**Symptoms**:
+
+- "No native build was found for platform=win32 arch=arm64"
+- Module loads on x64 but not ARM64
+
+**Root Causes**:
+
+1. Missing prebuilt binary for the platform
+2. Incorrect platform detection
+3. node-gyp-build not finding the prebuild
+
+**Solutions**:
+
+1. Verify prebuild exists in `prebuilds/win32-arm64/`
+2. Check that `process.platform` === "win32" and `process.arch` === "arm64"
+3. Use `node-gyp-build` diagnostic output to debug loading
+
+### Problem: Build Failures on Windows ARM64
+
+**Symptoms**:
+
+- "No Target Architecture" errors
+- Windows SDK header compilation errors
+
+**Root Causes**:
+
+- Missing architecture defines for Windows SDK
+- Prebuildify not passing through defines from binding.gyp
+
+**Solution**:
+Set environment variables before building:
+
+```javascript
+if (process.platform === "win32" && process.arch === "arm64") {
+  process.env.CL = "/D_M_ARM64 /D_WIN64";
+}
+```
+
+## Testing Strategies
+
+### 1. Memory Testing
+
+Traditional Windows memory tools don't work with Node.js:
+
+- Dr. Memory: "Unable to load client library"
+- Debug CRT: Cannot be loaded by Node.js
+- Visual Leak Detector: Requires debug builds
+
+Use JavaScript-based testing instead (see `src/windows-memory-check.test.ts`)
+
+### 2. Worker Thread Testing
+
+- Test native module loading in both main and worker threads
+- Use integration tests, not mocks
+- Verify actual functionality, not just loading
+
+### 3. Cross-Platform Testing
+
+- Test on actual ARM64 hardware when possible
+- Use GitHub Actions for CI testing
+- Be aware of performance differences (ARM64 can be slower)
+
+## Debugging Tips
+
+### 1. Verify Prebuild Location
+
+```bash
+# List prebuilds
+ls -la prebuilds/
+
+# Check specific platform
+ls -la prebuilds/win32-arm64/
+```
+
+### 2. Test Native Module Loading
+
+```javascript
+// test-native-load.js
+const nodeGypBuild = require("node-gyp-build");
+try {
+  const binding = nodeGypBuild(process.cwd());
+  console.log("✓ Native module loaded");
+  console.log("Functions:", Object.keys(binding));
+} catch (error) {
+  console.error("✗ Failed to load:", error.message);
+}
+```
+
+### 3. Check Process Information
+
+```javascript
+console.log("Platform:", process.platform);
+console.log("Architecture:", process.arch);
+console.log("Node version:", process.version);
+```
+
+## Future Considerations
+
+1. **Windows ARM64 Adoption**: Expected to grow with Qualcomm-based laptops
+2. **Tooling Improvements**: Build tools catching up with ARM64 support
+3. **Performance**: Native ARM64 binaries avoid x64 emulation overhead
+4. **Testing**: More ARM64 CI runners becoming available
+
+## References
+
+- [Prebuildify Documentation](https://github.com/prebuild/prebuildify)
+- [Node-gyp-build](https://github.com/prebuild/node-gyp-build)
+- [Windows ARM64 GitHub Actions](https://github.blog/changelog/2025-04-14-windows-arm64-hosted-runners-now-available-in-public-preview/)
+- [Node.js Native Addons](https://nodejs.org/api/addons.html)
+- [Windows on ARM Documentation](https://docs.microsoft.com/en-us/windows/arm/)
+
+## Related Files
+
+- `scripts/prebuildify-wrapper.ts` - Handles architecture-specific build configuration
+- `scripts/install.cjs` - Sets up environment for Windows builds
+- `.github/workflows/build.yml` - CI configuration including ARM64 jobs
+- `CLAUDE.md` - Project-specific build notes

package/jest.config.cjs

@@ -10,10 +10,33 @@ const isESM =
   process.env.TEST_ESM === "1" ||
   process.env.NODE_OPTIONS?.includes("--experimental-vm-modules");
 
+const nodeVersion = parseInt(process.version.slice(1).split(".")[0], 10);
+
+const isNode24ESM = nodeVersion >= 24 && isESM;
+const isWindowsCI = platform === "win32" && process.env.CI;
+
+// Determine maxWorkers based on environment
+let maxWorkers = undefined; // undefined means Jest uses default (number of cores - 1)
+let workerIdleMemoryLimit = undefined; // undefined means Jest uses default
+
+if (isWindowsCI) {
+  console.log("[Jest Config] Windows CI detected, applying workarounds:");
+  maxWorkers = 1;
+  workerIdleMemoryLimit = "1GB";
+} else if (isNode24ESM) {
+  console.log(
+    "[Jest Config] Node 24+ with ESM detected, applying workarounds:",
+  );
+  // Node 24 + ESM detection (avoid "module is already linked")
+  maxWorkers = 1;
+}
+
 /** @type {import('ts-jest').JestConfigWithTsJest} */
 const config = {
   displayName: `@photostructure/fs-metadata (${isESM ? "ESM" : "CJS"})`,
   testEnvironment: "jest-environment-node",
+  ...(maxWorkers != null ? { maxWorkers } : {}),
+  ...(workerIdleMemoryLimit != null ? { workerIdleMemoryLimit } : {}),
   roots: ["<rootDir>/src"],
   coverageProvider: "v8",
   moduleFileExtensions: ["ts", "tsx", "js", "jsx", "json", "node"],
@@ -23,6 +46,7 @@ const config = {
   setupFilesAfterEnv: [
     "jest-extended/all",
     "<rootDir>/src/test-utils/jest-matchers.ts",
+    "<rootDir>/src/test-utils/jest-setup.ts",
   ],
   collectCoverage: !argv.includes("--no-coverage"),
   coverageDirectory: "coverage",