@hybridlabor-api/aos 4.3.0 → 4.4.0

package/mcps/windows-computer-use-mcp/docs/_stale/development/archive/FILE_VALIDATION_GUIDE.md

@@ -1,575 +0,0 @@
-> **Archived fleet import** — From **notepadpp-mcp** / generic fleet dev notes. **Not** pywinauto-mcp source of truth. See [DEVELOPMENT.md](../../DEVELOPMENT.md) and [TESTING.md](../../TESTING.md).
-
-
-# Robust File Validation for MCP Sync
-
-**How to prevent crashes from problematic markdown files**
-
----
-
-## The Problem
-
-MCP servers crash when encountering:
-- ❌ Weird filenames (unicode, special chars, too long)
-- ❌ Zero-size files
-- ❌ Binary files disguised as `.md`
-- ❌ Broken encoding (UTF-8 vs Latin-1)
-- ❌ Malformed frontmatter YAML
-- ❌ Permission errors
-- ❌ Extremely long lines
-- ❌ Mixed line endings
-
-**Result:** Sync stops, users frustrated, no error messages.
-
----
-
-## The Solution
-
-### 1. **Robust File Validator**
-
-```python
-from file_validator import FileValidator
-
-validator = FileValidator(
-    max_file_size=10 * 1024 * 1024,  # 10 MB
-    allow_empty=True,                 # Warn but don't fail
-    strict_frontmatter=False          # Warn but don't fail
-)
-
-result = validator.validate_file(file_path)
-
-if result.is_valid:
-    # Safe to process
-    process_file(result.content)
-else:
-    # Log and skip
-    logger.warning("invalid_file",
-                  path=file_path,
-                  errors=result.errors)
-```
-
----
-
-## Validation Checks
-
-### Filename Validation
-
-✅ **Checks:**
-- Length < 200 characters
-- No dangerous characters (`<>:"|?*`)
-- Not Windows reserved (`CON`, `PRN`, `AUX`, etc.)
-- No control characters
-
-⚠️  **Warnings:**
-- Non-ASCII (unicode) characters
-- Spaces in filename
-- Wrong extension (not `.md` or `.markdown`)
-
-### Size Validation
-
-✅ **Checks:**
-- Not empty (configurable)
-- Not too large (default: 10 MB)
-
-⚠️  **Warnings:**
-- Empty file (0 bytes)
-- Large file (> 1 MB)
-
-### Content Validation
-
-✅ **Checks:**
-- Readable with common encodings
-- Not binary (no null bytes)
-
-⚠️  **Warnings:**
-- Mixed line endings
-- Very long lines (> 10,000 chars)
-
-**Tries multiple encodings:**
-1. UTF-8
-2. UTF-8 with BOM
-3. Latin-1
-4. Windows-1252
-5. ISO-8859-1
-
-### Frontmatter Validation
-
-✅ **Checks (strict mode):**
-- Valid YAML syntax
-- Proper markers (`---` at start and end)
-
-⚠️  **Warnings (lenient mode):**
-- Missing closing marker
-- Malformed YAML
-- Invalid structure
-
----
-
-## Integration with Sync
-
-### Before (CRASHES):
-
-```python
-async def scan_files():
-    for file_path in files:
-        content = file_path.read_text()  # ❌ CRASHES on encoding issues
-        frontmatter = yaml.load(content) # ❌ CRASHES on bad YAML
-        process(content)
-```
-
-### After (ROBUST):
-
-```python
-from file_validator import FileValidator
-
-validator = FileValidator()
-
-async def scan_files():
-    for file_path in files:
-        # Validate first
-        result = validator.validate_file(file_path)
-        
-        if not result.is_valid:
-            logger.warning("skipping_invalid_file",
-                          path=file_path,
-                          errors=result.errors)
-            sync_monitor.metrics.files_skipped += 1
-            continue
-        
-        # Log warnings but continue
-        for warning in result.warnings:
-            logger.info("file_warning",
-                       path=file_path,
-                       warning=warning)
-        
-        # Safe to process
-        process(result.content, result.frontmatter)
-```
-
----
-
-## Test Coverage
-
-### Weird Filenames
-```python
-def test_unicode_filename():
-    # 日本語ファイル.md
-    assert result.is_valid
-    assert "Non-ASCII" in result.warnings
-
-def test_very_long_filename():
-    # 250 character filename
-    assert not result.is_valid
-    assert "too long" in result.errors
-
-def test_reserved_windows_name():
-    # CON.md, PRN.md
-    assert not result.is_valid
-    assert "Reserved" in result.errors
-```
-
-### Size Issues
-```python
-def test_empty_file():
-    assert result.is_valid  # If allow_empty=True
-    assert "Empty file" in result.warnings
-
-def test_huge_file():
-    # 10+ MB file
-    assert not result.is_valid
-    assert "too large" in result.errors
-```
-
-### Encoding Issues
-```python
-def test_utf8_file():
-    assert result.is_valid
-    assert result.encoding == "utf-8"
-
-def test_latin1_file():
-    assert result.is_valid
-    assert result.encoding == "latin-1"
-
-def test_binary_file():
-    assert not result.is_valid
-    assert "Binary" in result.errors
-```
-
-### Frontmatter Issues
-```python
-def test_broken_yaml():
-    # Lenient mode
-    assert result.is_valid
-    assert "YAML" in result.warnings
-    
-    # Strict mode
-    assert not result.is_valid
-    assert "YAML" in result.errors
-
-def test_missing_closing_marker():
-    assert result.is_valid  # Lenient
-    assert "incomplete" in result.warnings
-```
-
----
-
-## Batch Validation
-
-Validate entire directories:
-
-```python
-validator = FileValidator()
-
-# Get all markdown files
-files = list(Path("docs").rglob("*.md"))
-
-# Validate all
-results = validator.validate_batch(files)
-
-# Generate summary
-summary = validator.get_summary(results)
-print(summary)
-```
-
-**Output:**
-```
-# File Validation Summary
-
-**Total Files:** 1896
-**Valid:** 1850 (97.6%)
-**Invalid:** 46 (2.4%)
-
-**Errors:** 52
-**Warnings:** 234
-
-## Invalid Files
-
-### weird€file.md
-- ❌ Dangerous characters in filename: {'€'}
-
-### huge-doc.md
-- ❌ File too large (15728640 > 10485760)
-
-### binary.md
-- ❌ Binary file detected (contains null bytes)
-```
-
----
-
-## Configuration Options
-
-```python
-validator = FileValidator(
-    # Maximum file size
-    max_file_size=10 * 1024 * 1024,  # 10 MB default
-    
-    # Allow empty files?
-    allow_empty=True,                 # Warn but allow
-    
-    # Strict frontmatter?
-    strict_frontmatter=False          # Warn but allow
-)
-```
-
----
-
-## CLI Usage
-
-Validate files from command line:
-
-```bash
-# Single file
-python -m file_validator file.md
-
-# Directory
-python -m file_validator docs/
-
-# Output
-✅ VALID: docs/guide.md
-
-❌ INVALID: docs/broken.md
-  ❌ Binary file detected (contains null bytes)
-  ⚠️  Non-ASCII characters in filename
-```
-
----
-
-## Metrics & Monitoring
-
-Track validation failures:
-
-```python
-class SyncMetrics:
-    files_total: int = 0
-    files_scanned: int = 0
-    files_skipped: int = 0      # NEW
-    files_with_warnings: int = 0 # NEW
-    
-    # Breakdown by error type
-    encoding_errors: int = 0     # NEW
-    frontmatter_errors: int = 0  # NEW
-    filename_errors: int = 0     # NEW
-    size_errors: int = 0         # NEW
-```
-
-Log validation issues:
-
-```python
-if not result.is_valid:
-    # Count by error type
-    for error in result.errors:
-        if "encoding" in error.lower():
-            metrics.encoding_errors += 1
-        elif "frontmatter" in error.lower():
-            metrics.frontmatter_errors += 1
-        elif "filename" in error.lower():
-            metrics.filename_errors += 1
-        elif "size" in error.lower():
-            metrics.size_errors += 1
-    
-    logger.error("file_validation_failed",
-                path=file_path,
-                errors=result.errors,
-                error_types=[
-                    "encoding" if metrics.encoding_errors else None,
-                    # ...
-                ])
-```
-
----
-
-## Best Practices
-
-### DO:
-✅ **Validate before processing**
-```python
-result = validator.validate_file(path)
-if result.is_valid:
-    process(result.content)
-```
-
-✅ **Log warnings but continue**
-```python
-for warning in result.warnings:
-    logger.info("file_warning", warning=warning)
-```
-
-✅ **Use lenient mode for user content**
-```python
-validator = FileValidator(
-    allow_empty=True,
-    strict_frontmatter=False
-)
-```
-
-✅ **Track skipped files**
-```python
-metrics.files_skipped += 1
-logger.warning("skipping_file", reason=result.errors)
-```
-
-### DON'T:
-❌ **Don't crash on invalid files**
-```python
-# BAD
-content = path.read_text()  # Might crash!
-
-# GOOD
-result = validator.validate_file(path)
-if result.is_valid:
-    content = result.content
-```
-
-❌ **Don't ignore warnings**
-```python
-# BAD
-result = validator.validate_file(path)
-process(result.content)  # Ignores warnings!
-
-# GOOD
-if result.warnings:
-    logger.info("file_warnings", warnings=result.warnings)
-process(result.content)
-```
-
-❌ **Don't use strict mode for everything**
-```python
-# BAD (fails on minor issues)
-validator = FileValidator(
-    allow_empty=False,
-    strict_frontmatter=True
-)
-
-# GOOD (lenient but safe)
-validator = FileValidator(
-    allow_empty=True,
-    strict_frontmatter=False
-)
-```
-
----
-
-## Error Recovery
-
-Handle specific errors:
-
-```python
-result = validator.validate_file(path)
-
-if not result.is_valid:
-    # Try to fix common issues
-    
-    # Encoding issue? Try to convert
-    if any("encoding" in e.lower() for e in result.errors):
-        fixed_path = fix_encoding(path)
-        result = validator.validate_file(fixed_path)
-    
-    # Frontmatter issue? Remove it
-    elif any("frontmatter" in e.lower() for e in result.errors):
-        fixed_content = remove_frontmatter(path)
-        process(fixed_content)
-    
-    # Filename issue? Rename
-    elif any("filename" in e.lower() for e in result.errors):
-        new_path = sanitize_filename(path)
-        result = validator.validate_file(new_path)
-    
-    # Still invalid? Skip
-    else:
-        logger.error("unfixable_file", path=path)
-        return
-```
-
----
-
-## Real-World Examples
-
-### Example 1: Unicode Filename
-
-**File:** `日本語ファイル.md`
-
-**Result:**
-```python
-result.is_valid = True
-result.warnings = ["Non-ASCII characters in filename"]
-result.content = "# 日本語コンテンツ"
-result.encoding = "utf-8"
-```
-
-**Action:** Process normally, log warning
-
----
-
-### Example 2: Binary File
-
-**File:** `data.md` (actually binary)
-
-**Result:**
-```python
-result.is_valid = False
-result.errors = ["Binary file detected (contains null bytes)"]
-```
-
-**Action:** Skip file, increment `files_skipped`
-
----
-
-### Example 3: Broken Frontmatter
-
-**File:** `article.md`
-```markdown
----
-title: Test
-author: [broken
----
-# Content
-```
-
-**Result (lenient):**
-```python
-result.is_valid = True
-result.warnings = ["Malformed frontmatter YAML"]
-result.frontmatter = None  # Couldn't parse
-result.content = "---\ntitle: Test\n..."
-```
-
-**Action:** Process content, warn about frontmatter
-
----
-
-### Example 4: Empty File
-
-**File:** `placeholder.md` (0 bytes)
-
-**Result:**
-```python
-result.is_valid = True  # If allow_empty=True
-result.warnings = ["Empty file (0 bytes)"]
-result.size_bytes = 0
-result.content = ""
-```
-
-**Action:** Skip or log, don't crash
-
----
-
-## Performance
-
-**Validation overhead:**
-- Small file (< 10 KB): **< 1 ms**
-- Medium file (100 KB): **< 5 ms**
-- Large file (1 MB): **< 50 ms**
-
-**Batch validation:**
-- 1,000 files: **~5 seconds**
-- 10,000 files: **~50 seconds**
-
-**Recommendation:** Validate during scan, accept overhead for robustness.
-
----
-
-## Testing Your Integration
-
-```python
-# Test with problematic files
-test_files = [
-    "normal.md",                    # Should pass
-    "empty.md",                     # Should warn
-    "日本語.md",                    # Should warn
-    "binary.md",                    # Should fail
-    "broken-frontmatter.md",        # Should warn/fail
-]
-
-validator = FileValidator()
-for file in test_files:
-    result = validator.validate_file(file)
-    print(f"{file}: {'✅' if result.is_valid else '❌'}")
-```
-
----
-
-## Summary
-
-✅ **Prevents crashes** from any file issues  
-✅ **Comprehensive validation** of all aspects  
-✅ **Lenient by default** for user content  
-✅ **Detailed error reporting** for debugging  
-✅ **Easy to integrate** into any sync system  
-✅ **Thoroughly tested** with 30+ test cases  
-
-**Your sync will NEVER crash on bad files again!** 🛡️
-
----
-
-## Related Files
-
-- 📄 [file_validator.py](../../src/notepadpp_mcp/file_validator.py) - Validator implementation
-- 📄 [test_file_validator.py](../../tests/test_file_validator.py) - Comprehensive tests
-- 📄 [sync_health.py](../../src/notepadpp_mcp/sync_health.py) - Sync health monitoring
-
----
-
-*"If it can go wrong with a file, we validate for it!"*
-
-*October 10, 2025*
-