plaintext 0.3.7 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ffcae76367d81bd94ad8e2ad40b0a59a00b0aa0d00ef8790b8e72e96c37c8a5d
-  data.tar.gz: 343182ccc4861b94ca0ac4e1733b60e44e60a840347b1eb877acaf9a4f54f440
+  metadata.gz: 4cff613a25c31b858cbe360c7311e468933d0714e9f80e217a25e3c74e9e9d72
+  data.tar.gz: 2ab92de5c21eac147f36cac18ebb01248f2dfa1aeb44a3c470bc503f528d3635
 SHA512:
-  metadata.gz: 99dfbc41ec270e97ce31332a6eb69ddd346212c8c74df85cc77c218bb257ece73dded27d3db8ee35799d76ba73596f38f7942b0e8d2962c5de9a93a643cbe794
-  data.tar.gz: beb1371a6b51afe487dfb06574471b0eb3ce5755ab7a1c87ae8a4b609f2db8f0753406e775efe418b94d81e3e324346f8bdee43fd454dfa47b34334b454ec2e8
+  metadata.gz: ef16b7430e3344f5213fa217b5e17a0abb79fba36f89e89a2f2a3cfb3a0401a69abf590f36a512ba3e6f6f101892ddfe2a32c5a3c5e26bcff0273ccea0dc797c
+  data.tar.gz: 99a650e3fe37ff7c1f036389806241e9e3f4107c47c13a717a93714d6c8fa998dc4b56510d27e27b7cc9c7ebbae9ea1c45ddef34060a123a1dea3ed9105d0a80

data/CHANGELOG

@@ -4,6 +4,47 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [Unreleased]
+
+## [0.4.0] - 2026-09-21
+
+### Added
+- `Resolver#preserve_whitespace`: when set, the extracted text is returned
+  with the whitespace the handler emitted instead of being collapsed into
+  single spaces. Keeps the line structure for plain text and the formats
+  extracted by an external command (PDF, DOC, XLS, PPT, RTF, images); the
+  zipped XML handlers join their text elements with a space and are
+  unaffected. (#13)
+
+### Changed
+- Replaced `String#mb_chars`, deprecated and to be removed in Rails 8.2, with
+  `unicode_normalize` and `truncate_bytes`. File handlers are now expected to
+  return UTF-8: `mb_chars` used to silently retag whatever it was given, which
+  is what hid the encoding bugs fixed below. Handlers shipped with this gem all
+  do, a custom handler returning binary strings now raises.
+- An extraction command exiting with a non-zero status now raises
+  `Plaintext::CommandFailed`. The exit status used to be ignored, so a
+  misconfigured or failing command silently yielded an empty string for every
+  file of its type.
+
+### Fixed
+- Extracted text no longer has its non-ASCII characters replaced by question
+  marks. Command output was read as ASCII-8BIT no matter what the command
+  actually produced, and only the pdftotext handler was exempt, so umlauts and
+  accented characters from doc, xls, ppt, rtf and image files were destroyed.
+- Cutting the output at `max_plaintext_bytes` no longer corrupts the whole
+  text. A byte limit routinely cuts through a multi byte character; only that
+  character is replaced now instead of every non-ASCII character in the
+  document, and `Resolver#text` no longer raises `ArgumentError: invalid byte
+  sequence in UTF-8` in that case.
+- RTF files no longer lose their accented characters. unrtf cannot be told to
+  write UTF-8 — its `--text` output is always Latin-1 — so its output is
+  converted rather than expected to be UTF-8.
+- Extraction no longer hangs forever when unrtf writes its header but no end
+  marker. Scanning for the marker did not stop at the end of the stream.
+- Corrected the unrtf and tesseract example configurations, which did not match
+  the defaults used by the gem.
+
 ## [0.3.7] - 2025-11-25
 
 ### Added

data/README.md

@@ -44,6 +44,11 @@ Or install it yourself as:
 In a Rails application save `plaintext.yml.example` in `config/plaintext.yml` and overwrite the settings to 
 your needs.
 
+Extraction commands are expected to write UTF-8 to STDOUT. Byte sequences that
+are not valid UTF-8 are replaced by a question mark, so make sure to pass
+whatever switch your command needs to produce UTF-8 — the defaults shipped with
+this gem already do.
+
 Then load that configuration file in an initializer. Add the following lines to `config/initializers/plaintext.rb`:
 
 ```ruby
@@ -109,9 +114,23 @@ catdoc:
 fulltext = Plaintext::Resolver.new(file, content_type).text
 ```
 
+`text` returns `nil` if no handler matches the content type and raises
+`Plaintext::CommandFailed` if an extraction command exits with a non-zero
+status, for example because it was given a switch it does not support.
+
 To limit the number of bytes returned (default is 4MB), set the
 `max_plaintext_bytes` property on the resolver instance before calling `text`.
 
+All whitespace is collapsed into single spaces by default. To keep the line
+structure emitted by the extraction command instead, set `preserve_whitespace`
+on the resolver instance before calling `text`. That keeps the document
+structure for plain text and the formats extracted by an external command
+(PDF, DOC, XLS, PPT, RTF and images), while the handlers for the zipped XML
+formats (OpenDocument and Office Open XML) join their text elements with a
+space. The text is returned as the handler emits it, so it may still contain
+form feeds between PDF pages, `\r\n` line endings and leading or trailing
+whitespace.
+
 ## License
 
 The `plaintext` gem is free software; you can redistribute it and/or modify it under the terms of the GNU General 

data/lib/plaintext/codeset_util.rb

@@ -4,22 +4,17 @@ module Plaintext
   module CodesetUtil
     def self.to_utf8(str, encoding)
       return str if str.nil?
-      str.force_encoding('ASCII-8BIT')
-      if str.empty?
-        str.force_encoding('UTF-8')
-        return str
-      end
-      enc = (encoding.nil? || encoding.size == 0) ? 'UTF-8' : encoding
+
+      enc = encoding.blank? ? 'UTF-8' : encoding
+      str.force_encoding(enc)
       if enc.upcase != 'UTF-8'
-        str.force_encoding(enc)
         str = str.encode('UTF-8', invalid: :replace,
                          undef: :replace, replace: '?')
-      else
-        str.force_encoding('UTF-8')
-        if !str.valid_encoding?
-          str = str.encode('US-ASCII', invalid: :replace,
-                           undef: :replace, replace: '?').encode('UTF-8')
-        end
+      elsif !str.valid_encoding?
+        # only replace the invalid byte sequences, leave the rest of the
+        # string alone. Reading up to a byte limit routinely cuts through a
+        # multi byte character at the very end of the string.
+        str = str.scrub('?')
       end
       str
     end

data/lib/plaintext/file_handler/external_command_handler/pdf_handler.rb

@@ -10,11 +10,5 @@ module Plaintext
       @content_type = 'application/pdf'
       @command = Plaintext::Configuration['pdftotext'] || DEFAULT
     end
-
-    protected
-
-    def utf8_stream?
-      true
-    end
   end
 end

data/lib/plaintext/file_handler/external_command_handler/rtf_handler.rb

@@ -15,10 +15,17 @@ module Plaintext
     UNRTF_HEADER = "###  Translation from RTF performed by UnRTF"
     END_MARKER   = "-----------------\n"
 
+    # unrtf has no switch to make it write UTF-8: its --text output is always
+    # Latin-1, it ignores the locale, and it replaces anything it cannot map
+    # into Latin-1 with a question mark itself.
+    def output_encoding
+      'ISO-8859-1'
+    end
+
     def read(io, max_size = nil)
       if line = io.read(UNRTF_HEADER.length)
         string = if line.starts_with? UNRTF_HEADER
-          io.gets while $_ != END_MARKER
+          io.each_line { |l| break if l == END_MARKER }
           io.read max_size
         else
           if max_size.nil?
@@ -29,7 +36,7 @@ module Plaintext
             line[0,max_size]
           end
         end
-        Plaintext::CodesetUtil.to_utf8 string, "ASCII-8BIT"
+        Plaintext::CodesetUtil.to_utf8 string, output_encoding
       end
     end
   end

data/lib/plaintext/file_handler/external_command_handler.rb

@@ -3,6 +3,10 @@
 require 'pathname'
 
 module Plaintext
+  # Raised when an extraction command exits with a non-zero status. Whatever
+  # the command wrote to STDOUT before failing is discarded.
+  class CommandFailed < StandardError; end
+
   class ExternalCommandHandler < FileHandler
     # TODO: Extract this to a proper module
     # Executes the given command through IO.popen and yields an IO object
@@ -13,15 +17,16 @@ module Plaintext
     require 'fileutils'
 
     FILE_PLACEHOLDER = '__FILE__'.freeze
-    DEFAULT_STREAM_ENCODING = 'ASCII-8BIT'.freeze
 
     def shellout(cmd, options = {}, &block)
       mode = "r+"
-      IO.popen(cmd, mode) do |io|
-        set_stream_encoding(io)
+      result = IO.popen(cmd, mode) do |io|
+        io.binmode
         io.close_write unless options[:write_stdin]
         block.call(io) if block_given?
       end
+      raise CommandFailed, "#{cmd.join(' ')} failed: #{$?}" unless $?.success?
+      result
     end
 
     def text(file, options = {})
@@ -43,32 +48,17 @@ module Plaintext
       new.available?
     end
 
-    protected
-
-    def utf8_stream?
-      false
-    end
-
     private
 
-    def set_stream_encoding(io)
-      return unless io.respond_to?(:set_encoding)
-
-      if utf8_stream?
-        io.set_encoding('UTF-8'.freeze)
-      else
-        io.set_encoding(DEFAULT_STREAM_ENCODING)
-      end
+    # Encoding the command writes its output in. Commands that can be told to
+    # produce UTF-8 are configured to do so (see plaintext.yml.example), the
+    # output of those that cannot is converted by #read.
+    def output_encoding
+      'UTF-8'
     end
 
     def read(io, max_size = nil)
-      piece = io.read(max_size)
-
-      if utf8_stream?
-        piece
-      else
-        Plaintext::CodesetUtil.to_utf8 piece, DEFAULT_STREAM_ENCODING
-      end
+      Plaintext::CodesetUtil.to_utf8 io.read(max_size), output_encoding
     end
   end
 end

data/lib/plaintext/resolver.rb

@@ -6,6 +6,10 @@ module Plaintext
     # maximum length of returned plain text in bytes. Default: 4MB
     attr_accessor :max_plaintext_bytes
 
+    # keep the whitespace emitted by the handler instead of collapsing it
+    # into single spaces. Default: false
+    attr_accessor :preserve_whitespace
+
     class << self
       attr_accessor :cached_file_handlers
 
@@ -29,6 +33,7 @@ module Plaintext
       @file = file
       @content_type = content_type
       @max_plaintext_bytes = 4_194_304 # 4 megabytes
+      @preserve_whitespace = false
     end
 
 
@@ -38,10 +43,12 @@ module Plaintext
       if handler = find_handler and
           text = handler.text(@file, max_size: max_plaintext_bytes)
 
-        text = +text
-        text.gsub!(/\s+/m, ' ')
-        text.strip!
-        text.mb_chars.compose.limit(max_plaintext_bytes).to_s
+        unless preserve_whitespace
+          text = +text
+          text.gsub!(/\s+/m, ' ')
+          text.strip!
+        end
+        text.unicode_normalize(:nfc).truncate_bytes(max_plaintext_bytes, omission: nil)
       end
     end
 

data/lib/plaintext/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Plaintext
-  VERSION = "0.3.7"
+  VERSION = "0.4.0"
 end

data/plaintext.yml.example

@@ -3,8 +3,12 @@
 #
 # Text extraction helper programs.
 #
-# commands should write the resulting plain text to STDOUT. Use __FILE__ as
-# placeholder for the file path. The values below are the defaults.
+# commands should write the resulting plain text to STDOUT, encoded in UTF-8.
+# Use __FILE__ as placeholder for the file path. The values below are the
+# defaults.
+#
+# Byte sequences that are not valid UTF-8 are replaced by a question mark, so
+# make sure to pass whatever switch your command needs to produce UTF-8.
 
 # apt-get install poppler-utils
 # pdftotext:
@@ -15,8 +19,12 @@
 #   - '-'
 
 # apt-get install unrtf
+# unrtf has no switch to write UTF-8, its --text output is always Latin-1 and
+# is converted by the rtf handler. Replacing it with a command that does write
+# UTF-8 will garble accented characters.
 # unrtf:
 #   - /usr/bin/unrtf
+#   - --nopict
 #   - --text
 #   - __FILE__
 
@@ -37,5 +45,6 @@
 # apt-get install tesseract-ocr
 # tesseract:
 #   - /usr/bin/tesseract
-#   - -dutf-8
-#   - __FILE__
+#   - __FILE__
+#   - stdout
+#   - quiet

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: plaintext
 version: !ruby/object:Gem::Version
-  version: 0.3.7
+  version: 0.4.0
 platform: ruby
 authors:
 - Jens Krämer
@@ -10,7 +10,7 @@ authors:
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2025-11-25 00:00:00.000000000 Z
+date: 2026-09-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activesupport