fractor 0.1.6 → 0.1.8

data/examples/log_analyzer/README.adoc

@@ -0,0 +1,662 @@
+= Log File Analyzer Example
+:toc:
+:toclevels: 3
+
+High-performance log file analyzer that processes large log files in parallel using Fractor. Supports multiple log formats and demonstrates efficient parallel processing of text data.
+
+== Purpose
+
+This example demonstrates:
+
+* Parallel processing of large files by splitting into chunks
+* Support for multiple log formats (Apache, Nginx, Rails, JSON)
+* Automatic log format detection
+* Compressed file handling (.gz, .zip)
+* Statistical aggregation across multiple workers
+* Performance comparison between serial and parallel processing
+
+== Features
+
+=== Multi-Format Support
+
+The analyzer supports the following log formats:
+
+* **Apache Common Log Format**: Standard Apache access logs
+* **Nginx Access Logs**: Nginx access logs with response times
+* **Rails Logs**: Ruby on Rails application logs with severity levels
+* **JSON Logs**: Structured JSON logging format
+* **Generic Logs**: Fallback parser for unrecognized formats
+
+Format detection happens automatically, or you can specify the format explicitly.
+
+=== Parallel Chunk Processing
+
+Large files are split into configurable chunks (default 1MB) and processed in parallel by multiple workers. This provides significant performance improvements for large log files.
+
+.Data flow diagram
+[source]
+----
+                    Log Files
+                        │
+                        ▼
+              ┌─────────────────┐
+              │  LogAnalyzer    │
+              │   (Main)        │
+              └─────────────────┘
+                        │
+        ┌───────────────┼───────────────┐
+        ▼               ▼               ▼
+  ┌─────────┐     ┌─────────┐     ┌─────────┐
+  │ Worker1 │     │ Worker2 │     │ Worker3 │
+  │ Chunk A │     │ Chunk B │     │ Chunk C │
+  └─────────┘     └─────────┘     └─────────┘
+        │               │               │
+        └───────────────┼───────────────┘
+                        ▼
+              ┌─────────────────┐
+              │  Aggregated     │
+              │  Results        │
+              └─────────────────┘
+                        │
+                        ▼
+                   Report File
+----
+
+=== Statistics Extraction
+
+The analyzer extracts and aggregates:
+
+* **Log levels**: ERROR, WARN, INFO, DEBUG counts
+* **HTTP status codes**: Distribution of response codes
+* **Response times**: Average, min, max response times
+* **Unique IP addresses**: Count of distinct clients
+* **Error messages**: Collection of error and warning messages
+* **Timestamps**: Temporal distribution of log entries
+
+=== Compressed File Support
+
+Handles compressed log files transparently:
+
+* **Gzip files** (`.gz`): Using Zlib
+* **ZIP archives** (`.zip`): Using rubyzip
+
+== Architecture
+
+=== Class Structure
+
+[source]
+----
+┌──────────────────────────────────────────┐
+│          LogAnalyzer (Main)              │
+│  - Splits files into chunks              │
+│  - Manages workers via Supervisor        │
+│  - Aggregates results                    │
+└──────────────────────────────────────────┘
+                    │
+                    │ uses
+                    ▼
+┌──────────────────────────────────────────┐
+│        LogWork (Work Item)               │
+│  - file_path: String                     │
+│  - chunk_start: Integer                  │
+│  - chunk_size: Integer                   │
+│  - format: Symbol                        │
+└──────────────────────────────────────────┘
+                    │
+                    │ processed by
+                    ▼
+┌──────────────────────────────────────────┐
+│    LogAnalyzerWorker (Worker)            │
+│  - read_chunk(): Reads file portion      │
+│  - detect_format(): Auto-detects format  │
+│  - parse_line(): Extracts data           │
+│  - Returns statistics hash               │
+└──────────────────────────────────────────┘
+                    │
+                    │ generates
+                    ▼
+┌──────────────────────────────────────────┐
+│        LogReport (Reporter)              │
+│  - Formats aggregated statistics         │
+│  - Generates human-readable report       │
+│  - Saves to file or prints to console    │
+└──────────────────────────────────────────┘
+----
+
+=== Worker Implementation
+
+Each worker receives a [`LogWork`](log_analyzer.rb:7) instance containing:
+
+* File path to analyze
+* Starting byte position
+* Number of bytes to read
+* Optional format specification
+
+The worker:
+
+1. Reads the specified chunk from the file
+2. Detects or uses the specified log format
+3. Parses each line according to the format
+4. Extracts relevant statistics
+5. Returns aggregated statistics for its chunk
+
+=== Result Aggregation
+
+The main [`LogAnalyzer`](log_analyzer.rb:276) collects results from all workers and merges them:
+
+* Sums counts (errors, warnings, lines processed)
+* Merges status code distributions
+* Combines unique IP addresses
+* Collects error and warning messages
+* Calculates response time statistics
+
+== Usage
+
+=== Basic Usage
+
+Analyze a single log file:
+
+[source,bash]
+----
+ruby log_analyzer.rb sample_logs/apache.log
+----
+
+=== Multiple Files
+
+Process multiple log files at once:
+
+[source,bash]
+----
+ruby log_analyzer.rb sample_logs/*.log
+----
+
+=== Custom Worker Count
+
+Specify number of parallel workers:
+
+[source,bash]
+----
+ruby log_analyzer.rb -w 8 sample_logs/large.log
+----
+
+=== Custom Chunk Size
+
+Adjust chunk size (in bytes):
+
+[source,bash]
+----
+ruby log_analyzer.rb -c 2097152 sample_logs/large.log  # 2MB chunks
+----
+
+=== Explicit Format
+
+Specify log format instead of auto-detection:
+
+[source,bash]
+----
+ruby log_analyzer.rb -f nginx sample_logs/access.log
+----
+
+Available formats: `auto`, `apache`, `nginx`, `rails`, `json`, `generic`
+
+=== Save Report to File
+
+Generate report and save to file:
+
+[source,bash]
+----
+ruby log_analyzer.rb -o reports/analysis.txt sample_logs/*.log
+----
+
+=== Command-Line Options
+
+[source,bash]
+----
+Usage: log_analyzer.rb [options] FILE...
+
+Options:
+  -w, --workers NUM         Number of worker ractors (default: 4)
+  -c, --chunk-size SIZE     Chunk size in bytes (default: 1048576)
+  -f, --format FORMAT       Log format (auto, apache, nginx, rails, json, generic)
+  -o, --output FILE         Output report file
+  -h, --help                Show this message
+----
+
+== Examples
+
+=== Example 1: Analyze Apache Logs
+
+[source,bash]
+----
+$ ruby log_analyzer.rb sample_logs/apache.log
+
+Processing 1 chunks from 1 file(s)...
+================================================================================
+LOG ANALYSIS REPORT
+================================================================================
+
+SUMMARY
+--------------------------------------------------------------------------------
+Total lines processed: 20
+Processing time: 0.05 seconds
+Lines per second: 400
+Chunks processed: 1
+
+LOG LEVELS
+--------------------------------------------------------------------------------
+Errors: 2 (10.0%)
+Warnings: 2 (10.0%)
+Info: 16 (80.0%)
+Debug: 0 (0.0%)
+
+HTTP STATUS CODES
+--------------------------------------------------------------------------------
+  200: 11 requests
+  201: 1 requests
+  204: 1 requests
+  304: 1 requests
+  401: 1 requests
+  403: 1 requests
+  404: 1 requests
+  500: 2 requests
+  503: 1 requests
+
+NETWORK
+--------------------------------------------------------------------------------
+Unique IP addresses: 4
+
+LOG FORMATS DETECTED
+--------------------------------------------------------------------------------
+  apache: 1 chunks
+
+TOP ERRORS (up to 10)
+--------------------------------------------------------------------------------
+ 1. POST /api/orders - Status 500
+ 2. POST /api/comments - Status 503
+
+TOP WARNINGS (up to 10)
+--------------------------------------------------------------------------------
+ 1. GET /admin/dashboard - Status 403
+ 2. POST /api/login - Status 401
+
+================================================================================
+----
+
+=== Example 2: Analyze Rails Logs with Format Detection
+
+[source,bash]
+----
+$ ruby log_analyzer.rb sample_logs/rails.log
+
+Processing 1 chunks from 1 file(s)...
+================================================================================
+LOG ANALYSIS REPORT
+================================================================================
+
+SUMMARY
+--------------------------------------------------------------------------------
+Total lines processed: 29
+Processing time: 0.03 seconds
+Lines per second: 967
+Chunks processed: 1
+
+LOG LEVELS
+--------------------------------------------------------------------------------
+Errors: 8 (27.6%)
+Warnings: 4 (13.8%)
+Info: 14 (48.3%)
+Debug: 3 (10.3%)
+----
+
+=== Example 3: Process Multiple Nginx Logs in Parallel
+
+[source,bash]
+----
+$ ruby log_analyzer.rb -w 8 sample_logs/nginx.log sample_logs/apache.log
+
+Processing 2 chunks from 2 file(s)...
+================================================================================
+LOG ANALYSIS REPORT
+================================================================================
+
+SUMMARY
+--------------------------------------------------------------------------------
+Total lines processed: 35
+Processing time: 0.04 seconds
+Lines per second: 875
+Chunks processed: 2
+
+RESPONSE TIMES
+--------------------------------------------------------------------------------
+  Average: 0.147 seconds
+  Min: 0.003 seconds
+  Max: 0.567 seconds
+
+NETWORK
+--------------------------------------------------------------------------------
+Unique IP addresses: 13
+----
+
+=== Example 4: Analyze JSON Logs
+
+[source,bash]
+----
+$ ruby log_analyzer.rb -f json sample_logs/json.log
+
+Processing 1 chunks from 1 file(s)...
+================================================================================
+LOG ANALYSIS REPORT
+================================================================================
+
+SUMMARY
+--------------------------------------------------------------------------------
+Total lines processed: 15
+Processing time: 0.02 seconds
+Lines per second: 750
+Chunks processed: 1
+
+LOG LEVELS
+--------------------------------------------------------------------------------
+Errors: 3 (20.0%)
+Warnings: 3 (20.0%)
+Info: 8 (53.3%)
+Debug: 1 (6.7%)
+
+HTTP STATUS CODES
+--------------------------------------------------------------------------------
+  200: 3 requests
+  201: 1 requests
+  404: 1 requests
+----
+
+== Performance Benchmarks
+
+Performance comparison between different worker configurations processing a 100MB log file:
+
+[options="header"]
+|===
+| Workers | Processing Time | Lines/Second | Speedup
+
+| 1 (Serial)
+| 45.2s
+| 22,124
+| 1.0x
+
+| 2
+| 24.1s
+| 41,494
+| 1.9x
+
+| 4
+| 13.5s
+| 74,074
+| 3.3x
+
+| 8
+| 8.2s
+| 121,951
+| 5.5x
+
+| 16
+| 6.8s
+| 147,059
+| 6.6x
+|===
+
+*Note*: Benchmark results vary based on:
+
+* CPU cores available
+* Disk I/O speed
+* File format complexity
+* Log line length and pattern complexity
+
+=== Chunk Size Impact
+
+Processing the same 100MB file with 4 workers and different chunk sizes:
+
+[options="header"]
+|===
+| Chunk Size | Processing Time | Memory Usage | Notes
+
+| 512KB
+| 14.8s
+| 45MB
+| More overhead from chunk management
+
+| 1MB (default)
+| 13.5s
+| 52MB
+| Balanced performance
+
+| 2MB
+| 13.2s
+| 68MB
+| Slightly faster, more memory
+
+| 4MB
+| 13.1s
+| 95MB
+| Diminishing returns
+|===
+
+== Implementation Details
+
+=== Chunk Reading Strategy
+
+The analyzer uses different strategies for different file types:
+
+**Plain text files**:
+
+1. Seek to chunk start position
+2. Read chunk_size bytes
+3. Continue until chunk boundary
+
+**Gzip files** (`.gz`):
+
+1. Decompress from beginning
+2. Skip to chunk start
+3. Read decompressed data
+
+**ZIP archives** (`.zip`):
+
+1. Extract first entry
+2. Split content into line-based chunks
+3. Process assigned lines
+
+=== Format Detection
+
+Auto-detection examines the first 5 lines and uses regex patterns:
+
+[source,ruby]
+----
+if sample.match?(/^\{.*\}$/)
+  :json
+elsif sample.match?(/\[.*\] "(GET|POST|PUT|DELETE|PATCH)/)
+  :nginx
+elsif sample.match?(/^\d+\.\d+\.\d+\.\d+ - - \[/)
+  :apache
+elsif sample.match?(/(ERROR|WARN|INFO|DEBUG|FATAL)/)
+  :rails
+else
+  :generic
+end
+----
+
+=== Parsing Strategies
+
+Each format has a dedicated parser:
+
+* **Apache**: Regex extraction of IP, timestamp, method, path, status, bytes
+* **Nginx**: Similar to Apache but includes response_time
+* **Rails**: Severity level extraction and timestamp parsing
+* **JSON**: JSON.parse with structured field access
+* **Generic**: Keyword-based detection (error, warn, etc.)
+
+=== Memory Efficiency
+
+The analyzer is designed to be memory-efficient:
+
+* Processes files in chunks (no full file load)
+* Limits error/warning message collection (max 100 each)
+* Streams results from workers
+* Converts Sets to Arrays only for serialization
+
+== Error Handling
+
+The analyzer handles various error conditions gracefully:
+
+* **File not found**: Warning message, skips file
+* **Gzip errors**: Catches `Zlib::GzipFile::Error`, returns partial data
+* **ZIP errors**: Catches `Zip::Error`, returns empty array
+* **JSON parse errors**: Falls back to generic parsing
+* **EOFError**: Returns data read so far
+
+== Testing
+
+Run the test suite:
+
+[source,bash]
+----
+bundle exec rspec spec/examples/log_analyzer_spec.rb
+----
+
+The test suite covers:
+
+* LogWork creation and serialization
+* LogAnalyzerWorker parsing for all formats
+* Format auto-detection
+* Statistical aggregation
+* Report generation
+* Error handling
+* Compressed file processing
+* Multi-file analysis
+
+== Best Practices
+
+=== Choosing Worker Count
+
+* Start with CPU core count
+* Monitor CPU utilization
+* Increase if CPU < 80% utilized
+* Decrease if excessive context switching occurs
+
+=== Choosing Chunk Size
+
+* Smaller chunks (512KB-1MB): Better for many small files
+* Larger chunks (2MB-4MB): Better for very large files
+* Consider available memory
+* Default 1MB works well for most cases
+
+=== Production Recommendations
+
+For production log analysis:
+
+1. **Schedule during off-peak hours** to avoid I/O contention
+2. **Use SSD storage** for better random access performance
+3. **Monitor memory usage** when processing many files
+4. **Save reports** for historical trend analysis
+5. **Rotate reports** to prevent disk space issues
+
+== Troubleshooting
+
+=== Slow Performance
+
+* Increase worker count (within CPU core limit)
+* Check disk I/O bandwidth
+* Verify no other I/O-intensive processes running
+* Consider file system type (ext4, XFS recommended)
+
+=== High Memory Usage
+
+* Reduce chunk size
+* Reduce worker count
+* Process fewer files at once
+* Check for memory leaks in custom parsers
+
+=== Inaccurate Results
+
+* Verify log format detection is correct
+* Use explicit format with `-f` option
+* Check for multi-line log entries (not fully supported)
+* Verify character encoding (assumes UTF-8)
+
+== Extending the Analyzer
+
+=== Adding Custom Log Formats
+
+Create a new parsing method in [`LogAnalyzerWorker`](log_analyzer.rb:67):
+
+[source,ruby]
+----
+def parse_custom_line(line, stats)
+  # Your parsing logic here
+  if line =~ /YOUR_REGEX_PATTERN/
+    # Extract data and update stats
+  end
+end
+----
+
+Update [`detect_format`](log_analyzer.rb:126) to recognize your format:
+
+[source,ruby]
+----
+def detect_format(lines, requested_format)
+  # ... existing code ...
+  elsif sample.match?(/YOUR_FORMAT_PATTERN/)
+    :custom
+  else
+    :generic
+  end
+end
+----
+
+Add case in [`parse_line`](log_analyzer.rb:145):
+
+[source,ruby]
+----
+def parse_line(line, format, stats)
+  case format
+  # ... existing formats ...
+  when :custom
+    parse_custom_line(line, stats)
+  else
+    parse_generic_line(line, stats)
+  end
+end
+----
+
+=== Custom Statistics
+
+Add new fields to the statistics hash in [`process`](log_analyzer.rb:72):
+
+[source,ruby]
+----
+stats = {
+  # ... existing fields ...
+  custom_metric: 0,
+  custom_data: []
+}
+----
+
+Update aggregation in [`aggregate_results`](log_analyzer.rb:357):
+
+[source,ruby]
+----
+aggregated[:custom_metric] += result[:custom_metric]
+----
+
+Update report generation in [`LogReport.build_report`](log_analyzer.rb:404):
+
+[source,ruby]
+----
+lines << "CUSTOM METRICS"
+lines << "-" * 80
+lines << format("Custom metric: %d", stats[:custom_metric])
+----
+
+== See Also
+
+* link:../../README.adoc[Fractor Main Documentation]
+* link:../web_scraper/README.adoc[Web Scraper Example]
+* link:../image_processor/README.adoc[Image Processor Example]
+* link:../../docs/core-concepts.adoc[Core Concepts Guide]