parsekit 0.1.0.pre.1 → 0.1.1

data/lib/parsekit/NATIVE_API.md

@@ -0,0 +1,125 @@
+# ParseKit Native API Documentation
+
+This document describes the methods implemented in the Rust native extension for ParseKit::Parser.
+
+## Instance Methods
+
+### `initialize(options = {})`
+Initialize a new Parser instance with optional configuration.
+
+**Parameters:**
+- `options` [Hash] Configuration options
+  - `:encoding` [String] Input encoding (default: UTF-8)
+  - `:strict_mode` [Boolean] Enable strict parsing mode (default: false)
+  - `:max_depth` [Integer] Maximum nesting depth (default: 100)
+  - `:max_size` [Integer] Maximum file size in bytes (default: 100MB)
+
+### `parse(input)`
+Parse an input string (for text content).
+
+**Parameters:**
+- `input` [String] The input to parse
+
+**Returns:**
+- [String] The parsed result
+
+**Raises:**
+- `ArgumentError` If input is empty
+
+### `parse_file(path)`
+Parse a file (supports PDF, Office documents, text files, images with OCR).
+
+**Parameters:**
+- `path` [String] Path to the file to parse
+
+**Returns:**
+- [String] The extracted text content
+
+**Raises:**
+- `IOError` If file cannot be read
+- `RuntimeError` If parsing fails
+
+### `parse_bytes(data)`
+Parse binary data.
+
+**Parameters:**
+- `data` [Array<Integer>] Binary data as byte array
+
+**Returns:**
+- [String] The extracted text content
+
+**Raises:**
+- `ArgumentError` If data is empty
+- `RuntimeError` If parsing fails
+
+### `config`
+Get the current parser configuration.
+
+**Returns:**
+- [Hash] The parser configuration including encoding, strict_mode, max_depth, and max_size
+
+### `supports_file?(path)`
+Check if a file format is supported.
+
+**Parameters:**
+- `path` [String] File path to check
+
+**Returns:**
+- [Boolean] True if the file format is supported
+
+### `strict_mode?`
+Check if strict mode is enabled.
+
+**Returns:**
+- [Boolean] True if strict mode is enabled
+
+## Format-Specific Parsers
+
+These methods are also available but typically called internally via `parse_file` or `parse_bytes`:
+
+### `parse_pdf(data)`
+Parse PDF files using MuPDF (statically linked).
+
+### `parse_docx(data)`
+Parse Microsoft Word documents.
+
+### `parse_pptx(data)`
+Parse Microsoft PowerPoint presentations.
+
+### `parse_xlsx(data)`
+Parse Microsoft Excel spreadsheets.
+
+### `parse_json(data)`
+Parse and pretty-print JSON data.
+
+### `parse_xml(data)`
+Parse XML/HTML files and extract text content.
+
+### `parse_text(data)`
+Parse plain text files.
+
+### `ocr_image(data)`
+Perform OCR on images (PNG, JPEG, TIFF, BMP) using Tesseract.
+
+## Class Methods
+
+### `Parser.supported_formats`
+Get list of supported file formats.
+
+**Returns:**
+- [Array<String>] List of supported file extensions
+
+**Example:**
+```ruby
+ParseKit::Parser.supported_formats
+# => ["txt", "json", "xml", "html", "docx", "xlsx", "xls", "csv", "pdf", "png", "jpg", "jpeg", "tiff", "bmp", ...]
+```
+
+## Implementation Notes
+
+All these methods are implemented in Rust via the native extension. The Ruby layer (`lib/parsekit/parser.rb`) provides additional convenience methods and helpers that wrap these native methods.
+
+The native extension uses:
+- **MuPDF** for PDF parsing (statically linked)
+- **Tesseract** for OCR functionality (bundled)
+- **Various Rust crates** for Office document parsing (docx-rs, calamine, etc.)

data/lib/parsekit/parser.rb

@@ -3,65 +3,24 @@
 module ParseKit
   # Ruby wrapper for the native Parser class
   #
-  # The Ruby layer now handles format detection and routing to specific parsers,
-  # while Rust provides the actual parsing implementations.
+  # This class provides document parsing capabilities through a native Rust extension.
+  # For documentation of native methods, see NATIVE_API.md
+  #
+  # The Ruby layer provides convenience methods and helpers while the Rust
+  # extension handles the actual parsing of PDF, Office documents, images (OCR), etc.
   class Parser
-    # These methods are implemented in the native extension
-    # and are documented here for YARD
-    
-    # Initialize a new Parser instance
-    # @param options [Hash] Configuration options
-    # @option options [String] :encoding Input encoding (default: UTF-8)
-    # def initialize(options = {})
-    #   # Implemented in native extension
-    # end
-    
-    # Parse an input string (for text content)
-    # @param input [String] The input to parse
-    # @return [String] The parsed result
-    # @raise [ArgumentError] If input is empty
-    # def parse(input)
-    #   # Implemented in native extension
-    # end
-    
-    # Parse a file (supports PDF, Office documents, text files)
-    # @param path [String] Path to the file to parse
-    # @return [String] The extracted text content
-    # @raise [IOError] If file cannot be read
-    # @raise [RuntimeError] If parsing fails
-    # def parse_file(path)
-    #   # Implemented in native extension
-    # end
-    
-    # Parse binary data
-    # @param data [Array<Integer>] Binary data as byte array
-    # @return [String] The extracted text content
-    # @raise [ArgumentError] If data is empty
-    # @raise [RuntimeError] If parsing fails
-    # def parse_bytes(data)
-    #   # Implemented in native extension
-    # end
-    
-    # Get the current configuration
-    # @return [Hash] The parser configuration
-    # def config
-    #   # Implemented in native extension
-    # end
-    
-    # Check if a file format is supported
-    # @param path [String] File path to check
-    # @return [Boolean] True if the file format is supported
-    # def supports_file?(path)
-    #   # Implemented in native extension
-    # end
-    
-    # Get list of supported file formats
-    # @return [Array<String>] List of supported file extensions
-    # def self.supported_formats
-    #   # Implemented in native extension
-    # end
-    
-    # Ruby-level helper methods
+    # Native methods implemented in Rust:
+    # - initialize(options = {})
+    # - parse(input)
+    # - parse_file(path)
+    # - parse_bytes(data)
+    # - config
+    # - supports_file?(path)
+    # - strict_mode?
+    # - parse_pdf, parse_docx, parse_xlsx, parse_pptx, parse_json, parse_xml, parse_text, ocr_image
+    # See NATIVE_API.md for detailed documentation
+    
+    # Ruby convenience methods and helpers
     
     # Create a parser with strict mode enabled
     # @param options [Hash] Additional options
@@ -81,6 +40,7 @@ module ParseKit
     end
     
     # Detect format from file path
+    # @deprecated Use the native format detection in parse_file instead
     # @param path [String] File path
     # @return [Symbol, nil] Format symbol or nil if unknown
     def detect_format(path)
@@ -89,6 +49,7 @@ module ParseKit
       
       case ext.downcase
       when 'docx' then :docx
+      when 'pptx' then :pptx
       when 'xlsx', 'xls' then :xlsx
       when 'pdf' then :pdf
       when 'json' then :json
@@ -100,67 +61,134 @@ module ParseKit
     end
     
     # Detect format from binary data
+    # @deprecated Use the native format detection in parse_bytes instead
     # @param data [String, Array<Integer>] Binary data
     # @return [Symbol] Format symbol
     def detect_format_from_bytes(data)
       # Convert to bytes if string
       bytes = data.is_a?(String) ? data.bytes : data
-      return :text if bytes.empty?
-      
-      # Check magic bytes
-      if bytes[0..3] == [0x25, 0x50, 0x44, 0x46]  # %PDF
-        :pdf
-      elsif bytes[0..1] == [0x50, 0x4B]  # PK (ZIP archive)
-        # Could be DOCX or XLSX, default to xlsx for now
-        # In the future, could inspect ZIP contents to determine
-        :xlsx
-      elsif bytes[0..3] == [0xD0, 0xCF, 0x11, 0xE0]  # Old Excel
+      return :text if bytes.empty?  # Return :text for empty data
+      
+      # Check magic bytes for various formats
+      
+      # PDF
+      if bytes.size >= 4 && bytes[0..3] == [0x25, 0x50, 0x44, 0x46]  # %PDF
+        return :pdf
+      end
+      
+      # PNG
+      if bytes.size >= 8 && bytes[0..7] == [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A]
+        return :png
+      end
+      
+      # JPEG
+      if bytes.size >= 3 && bytes[0..2] == [0xFF, 0xD8, 0xFF]
+        return :jpeg
+      end
+      
+      # BMP
+      if bytes.size >= 2 && bytes[0..1] == [0x42, 0x4D]  # BM
+        return :bmp
+      end
+      
+      # TIFF (little-endian or big-endian)
+      if bytes.size >= 4
+        if bytes[0..3] == [0x49, 0x49, 0x2A, 0x00]  # II*\0 (little-endian)
+          return :tiff
+        elsif bytes[0..3] == [0x4D, 0x4D, 0x00, 0x2A]  # MM\0* (big-endian)
+          return :tiff
+        end
+      end
+      
+      # OLE Compound Document (old Excel/Word) - return :xlsx for compatibility
+      if bytes.size >= 4 && bytes[0..3] == [0xD0, 0xCF, 0x11, 0xE0]
+        return :xlsx  # Return :xlsx for compatibility with existing tests
+      end
+      
+      # ZIP archive (could be DOCX, XLSX, PPTX)
+      if bytes.size >= 2 && bytes[0..1] == [0x50, 0x4B]  # PK
+        # Try to determine the specific Office format by checking ZIP contents
+        # For now, we'll need to inspect the ZIP structure
+        return detect_office_format_from_zip(bytes)
+      end
+      
+      # XML
+      if bytes.size >= 5
+        first_chars = bytes[0..4].pack('C*')
+        if first_chars == '<?xml' || first_chars.start_with?('<!')
+          return :xml
+        end
+      end
+      
+      # HTML
+      if bytes.size >= 14
+        first_chars = bytes[0..13].pack('C*').downcase
+        if first_chars.include?('<!doctype') || first_chars.include?('<html')
+          return :xml  # HTML is treated as XML
+        end
+      end
+      
+      # JSON
+      if bytes.size > 0
+        first_char = bytes[0]
+        # Skip whitespace
+        idx = 0
+        while idx < bytes.size && [0x20, 0x09, 0x0A, 0x0D].include?(bytes[idx])
+          idx += 1
+        end
+        
+        if idx < bytes.size
+          first_non_ws = bytes[idx]
+          if first_non_ws == 0x7B || first_non_ws == 0x5B  # { or [
+            return :json
+          end
+        end
+      end
+      
+      # Default to text if not recognized
+      :text
+    end
+    
+    # Detect specific Office format from ZIP data
+    # @param bytes [Array<Integer>] ZIP file bytes
+    # @return [Symbol] :docx, :xlsx, :pptx, or :unknown
+    def detect_office_format_from_zip(bytes)
+      # This is a simplified detection - in practice you'd parse the ZIP
+      # For the test, we'll check for known patterns in the ZIP structure
+      
+      # Convert bytes to string for pattern matching
+      content = bytes[0..2000].pack('C*')  # Check first 2KB
+      
+      # Look for Office-specific directory names in the ZIP
+      if content.include?('word/') || content.include?('word/_rels')
+        :docx
+      elsif content.include?('xl/') || content.include?('xl/_rels')
         :xlsx
-      elsif bytes[0..4] == [0x3C, 0x3F, 0x78, 0x6D, 0x6C]  # <?xml
-        :xml
-      elsif bytes[0..4] == [0x3C, 0x68, 0x74, 0x6D, 0x6C]  # <html
-        :xml
-      elsif bytes[0] == 0x7B || bytes[0] == 0x5B  # { or [
-        :json
+      elsif content.include?('ppt/') || content.include?('ppt/_rels')
+        :pptx
       else
-        :text
+        # Default to xlsx for generic ZIP
+        :xlsx
       end
     end
     
     # Parse file using format-specific parser
-    # This method now detects format and routes to the appropriate parser
+    # This method delegates to parse_file which uses centralized dispatch in Rust
     # @param path [String] File path
     # @return [String] Parsed content
     def parse_file_routed(path)
-      format = detect_format(path)
-      data = File.read(path, mode: 'rb').bytes
-      
-      case format
-      when :docx then parse_docx(data)
-      when :xlsx then parse_xlsx(data) 
-      when :pdf then parse_pdf(data)
-      when :json then parse_json(data)
-      when :xml then parse_xml(data)
-      else parse_text(data)
-      end
+      # Simply delegate to parse_file which already has dispatch logic
+      parse_file(path)
     end
     
     # Parse bytes using format-specific parser
-    # This method detects format and routes to the appropriate parser
+    # This method delegates to parse_bytes which uses centralized dispatch in Rust
     # @param data [String, Array<Integer>] Binary data
     # @return [String] Parsed content
     def parse_bytes_routed(data)
-      format = detect_format_from_bytes(data)
+      # Simply delegate to parse_bytes which already has dispatch logic
       bytes = data.is_a?(String) ? data.bytes : data
-      
-      case format
-      when :docx then parse_docx(bytes)
-      when :xlsx then parse_xlsx(bytes)
-      when :pdf then parse_pdf(bytes)
-      when :json then parse_json(bytes)
-      when :xml then parse_xml(bytes)
-      else parse_text(bytes)
-      end
+      parse_bytes(bytes)
     end
     
     # Parse with a block for processing results
@@ -177,25 +205,49 @@ module ParseKit
     # @param input [String] The input to validate
     # @return [Boolean] True if input is valid
     def valid_input?(input)
-      return false unless input.is_a?(String)
-      return false if input.empty?
-      true
+      input.is_a?(String) && !input.empty?
     end
     
     # Validate file before parsing
     # @param path [String] The file path to validate
     # @return [Boolean] True if file exists and format is supported
     def valid_file?(path)
+      return false if path.nil? || path.empty?
       return false unless File.exist?(path)
+      return false if File.directory?(path)
       supports_file?(path)
     end
     
     # Get file extension
     # @param path [String] File path
-    # @return [String, nil] File extension in lowercase
+    # @return [String, nil] File extension in lowercase without leading dot
     def file_extension(path)
-      ext = File.extname(path)
-      ext.empty? ? nil : ext[1..].downcase
+      return nil if path.nil? || path.empty?
+      
+      # Handle trailing whitespace
+      clean_path = path.strip
+      
+      # Handle trailing slashes (directory indicator)
+      return nil if clean_path.end_with?('/')
+      
+      # Get the extension
+      ext = File.extname(clean_path)
+      
+      # Handle special cases
+      if ext.empty?
+        # Check for hidden files like .gitignore (the whole name after dot is the "extension")
+        basename = File.basename(clean_path)
+        if basename.start_with?('.') && basename.length > 1 && !basename[1..-1].include?('.')
+          return basename[1..-1].downcase
+        end
+        return nil
+      elsif ext == '.'
+        # File ends with a dot but no extension
+        return nil
+      else
+        # Normal extension, remove the dot and downcase
+        ext[1..-1].downcase
+      end
     end
   end
 end

data/lib/parsekit/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module ParseKit
-  VERSION = "0.1.0.pre.1"
+  VERSION = "0.1.1"
 end

data/lib/parsekit.rb

@@ -14,6 +14,22 @@ require_relative "parsekit/parser"
 
 # ParseKit is a Ruby document parsing toolkit with PDF and OCR support
 module ParseKit
+  # Supported file formats and their extensions
+  SUPPORTED_FORMATS = {
+    pdf: ['.pdf'],
+    docx: ['.docx'],
+    xlsx: ['.xlsx'],
+    xls: ['.xls'],
+    pptx: ['.pptx'],
+    png: ['.png'],
+    jpeg: ['.jpg', '.jpeg'],
+    tiff: ['.tiff', '.tif'],
+    bmp: ['.bmp'],
+    json: ['.json'],
+    xml: ['.xml', '.html'],
+    text: ['.txt', '.md', '.csv']
+  }.freeze
+
   class << self
     # The parse_file and parse_bytes methods are defined in the native extension
     # We just need to document them here or add wrapper logic if needed
@@ -50,6 +66,22 @@ module ParseKit
       Parser.new.supports_file?(path)
     end
     
+    # Detect file format from filename/extension
+    # @param filename [String, nil] The filename to check
+    # @return [Symbol] The detected format, or :unknown
+    def detect_format(filename)
+      return :unknown if filename.nil? || filename.empty?
+      
+      ext = File.extname(filename).downcase
+      return :unknown if ext.empty?
+      
+      SUPPORTED_FORMATS.each do |format, extensions|
+        return format if extensions.include?(ext)
+      end
+      
+      :unknown
+    end
+    
     # Get the native library version
     # @return [String] Version of the native library
     def native_version

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: parsekit
 version: !ruby/object:Gem::Version
-  version: 0.1.0.pre.1
+  version: 0.1.1
 platform: ruby
 authors:
 - Chris Petersen
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2025-08-21 00:00:00.000000000 Z
+date: 2025-09-06 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rb_sys
@@ -96,9 +96,11 @@ files:
 - ext/parsekit/Cargo.toml
 - ext/parsekit/extconf.rb
 - ext/parsekit/src/error.rs
+- ext/parsekit/src/format_detector.rs
 - ext/parsekit/src/lib.rs
 - ext/parsekit/src/parser.rs
 - lib/parsekit.rb
+- lib/parsekit/NATIVE_API.md
 - lib/parsekit/error.rb
 - lib/parsekit/parsekit.bundle
 - lib/parsekit/parser.rb