readword 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 2c151bf301945512c356cbaadc83fc03f939c712293015a052be64a62c0c56f1
+  data.tar.gz: 3d9f112fa2fff0582d9040b6e146d9b7e587b6933330773df746e41414ac0b2f
+SHA512:
+  metadata.gz: 876795d221df033bbddf1b407e1cbf6969ac8d39e0956e7b81056b72bae81b90d2df7a887e729494a4ce5e83cce8fbf19db02a5a6e5d69887653d5836f868bf7
+  data.tar.gz: 5ed50d5820cb66dc8b529ea78ce09b1fee80b7783e3977b5239db1bb0d2372798ad8d0483fbbc1ed1963a4d4c96b0dd7369fbadbdbaaee00c6b79d1c18eb4a81

data/CHANGELOG.md

@@ -0,0 +1,15 @@
+# Changelog
+
+## 0.1.1 (2026-10-22)
+
+Same code as 0.1.0. Only the author name in the gem's details changed.
+
+## 0.1.0 (2026-10-22)
+
+First release.
+
+- Reads Word 97-2003 `.doc` and Word 2007+ `.docx` files and prints Markdown: paragraphs, headings with their numbers, nested bullet and numbered lists with the numbers Word shows, and tables as pipe tables.
+- Reads headers, footers, footnotes, endnotes and text boxes. Tracked changes read as accepted, field codes are left out, and Insert Symbol characters come out as the real characters.
+- `readword` command for one or more files or standard input, with `--max-characters`.
+- Refuses password-protected files, Word 6 and 95 files, and files that are not Word documents, each with its own reason.
+- Size limits and loop guards for files it does not trust.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 urbullet
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,124 @@
+# readword
+
+Turn Word documents into Markdown, in plain Ruby. It reads both formats, Word 97-2003 `.doc` and Word 2007+ `.docx`, without LibreOffice, Java or antiword. It was built for feeding Word files to LLMs and coding agents, and it is careful with files it does not trust.
+
+```console
+$ gem install readword
+$ readword structured_report.doc
+# Migration test report
+
+# 1 Scope
+
+Substances tested:
+
+- Bisphenol A
+  - food simulant A
+- Toluene
+
+# 2 Method
+
+## 2.1 Conditions
+
+1. Prepare the sample
+   - a) at 40 C
+   - b) for 10 days
+2. Measure
+
+## 2.2 Results
+
+| Substance | Result (ug/kg) |
+| --- | --- |
+| Benzene | 10 |
+| Steps | 1. Fill 1. Seal |
+
+End of report.
+```
+
+## What it keeps
+
+- Headings as `#` lines, with the numbers Word puts in front of them (`## 2.1 Scope`). Headings come from Word's heading styles, from outline levels, and from styles based on them, in any language.
+- Bullet and numbered lists with their nesting and the numbers Word shows: `1.`, `a)`, `iv.`, `1.2.3`, custom labels, restarted and continued lists.
+- Tables as Markdown pipe tables, one line per row. Merged cells keep their columns, and lists inside a cell stay in that cell.
+- Headers, footers, footnotes, endnotes and text boxes.
+- Tracked changes as if accepted: inserted text stays and deleted text goes.
+- Field results, such as page numbers, dates and merge fields. Field codes are left out.
+- Characters added with Insert Symbol, such as µ, ±, ≤ and °, as the real characters.
+
+It does not keep bold, italics, links, images, charts, comments or embedded objects (an Excel table pasted into the document as an object, for example).
+
+## Use it with a coding agent
+
+Install the gem, then add a line like this to your `CLAUDE.md` or `AGENTS.md`:
+
+```markdown
+To read a Word file (.doc or .docx), run `readword path/to/file` and read the Markdown it prints.
+```
+
+## Command line
+
+```console
+$ readword [--max-characters N] FILE...
+```
+
+- With several files, each one starts with a `==> path <==` line.
+- Use `-` as the file name to read from standard input.
+- If a file cannot be read, the reason goes to standard error and the exit code is 1. The other files are still printed.
+- `--max-characters N` refuses documents longer than N characters, which is useful to stay inside a model's context window.
+
+## Use it from Ruby
+
+```ruby
+require 'readword'
+
+Readword.convert_file('report.doc')                      # => "# Test report\n\n..."
+Readword.convert(File.binread('report.docx'), max_characters: 150_000)
+```
+
+`convert` takes the file's bytes and works out the format from them, so a `.docx` saved with a `.doc` name still reads. Files that cannot be read raise `Readword::Error`. Its `reason` is one of:
+
+| Reason | When |
+| --- | --- |
+| `:unreadable` | Not a Word document (RTF or plain text named `.doc`, for example), or damaged |
+| `:no_text` | The document has no text, only images for example |
+| `:too_long` | Over `max_characters`, or over the reader's own size limits |
+| `:password_protected` | Encrypted with a password |
+| `:old_format` | Saved by Word 6 or Word 95 |
+
+## Safe on untrusted files
+
+- The `.doc` reader is plain Ruby, so a crafted file cannot cause the memory errors that C readers are prone to. The `.docx` reader uses Nokogiri and rubyzip.
+- Size limits on unpacked XML, XML nodes, text and formatting, so a small zip bomb or a crafted file cannot make it use gigabytes of memory.
+- XML parts that declare their own entities are refused, which stops entity bombs.
+- Loop guards in the `.doc` container reader, so a crafted sector chain cannot make it run forever.
+- Fuzzed with 3,000,000 broken copies of real Word files, and 300,000 more after headings and lists were added, with no crash and no hang. The worst crafted files found so far use under 50 MB of memory.
+
+## How it was checked
+
+readword was compared with Apache Tika, LibreOffice and macOS textutil on 650 Word files from the Apache POI and LibreOffice test suites.
+
+| | antiword 0.37 | docx gem 0.13 | readword |
+| --- | --- | --- | --- |
+| Reads `.doc` | yes | no | yes |
+| Reads `.docx` | no | yes | yes |
+| Files missing words that both Tika and textutil found | 84 of 246 `.doc` files | 60 of 207 `.docx` files | 8 `.doc` files, no `.docx` files |
+| Headers and footers | not printed | | read |
+| Tables | long or two-paragraph cells spill into fake rows | paragraphs and tables come back as separate lists, so their order is lost | one line per row, in reading order |
+| Track Changes | deleted text kept | inserted text dropped | deleted text left out, inserted text kept |
+| 10,000 crafted `.doc` files | 164 crashes and 387 memory errors | | no crashes |
+| A 1 MB zip that unpacks to 1 GB | | 3.2 GB of memory used | refused in 0.17 s |
+| Install | system package (no Homebrew formula) | gem | gem |
+
+The words readword misses are field codes, embedded objects, charts and picture names.
+
+List numbers were compared with LibreOffice's on the 45 test files that have numbered lists. All numbers match in 41 of them. Of the other four, two are deliberately corrupted fuzz files, one changed its numbering with Track Changes, and one numbers an empty paragraph that holds a section break, where LibreOffice drops the paragraph.
+
+## Development
+
+```console
+$ bundle install
+$ bundle exec rake    # tests and RuboCop
+```
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).

data/exe/readword

@@ -0,0 +1,44 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require 'optparse'
+require 'readword'
+
+options = {}
+parser = OptionParser.new do |opts|
+  opts.banner = 'Usage: readword [options] FILE...'
+  opts.separator ''
+  opts.separator 'Prints Word documents (.doc and .docx) as Markdown. Use - to read from standard input.'
+  opts.separator ''
+  opts.on('--max-characters N', Integer, 'Fail when a document has more than N characters') do |count|
+    options[:max_characters] = count
+  end
+  opts.on('-v', '--version', 'Print the version') do
+    puts Readword::VERSION
+    exit
+  end
+  opts.on('-h', '--help', 'Print this help') do
+    puts opts
+    exit
+  end
+end
+
+begin
+  paths = parser.parse(ARGV)
+rescue OptionParser::ParseError => e
+  abort("readword: #{e.message}\n\n#{parser}")
+end
+abort(parser.to_s) if paths.empty?
+
+failed = false
+paths.each_with_index do |path, index|
+  bytes = path == '-' ? $stdin.binmode.read : File.binread(path)
+  text = Readword.convert(bytes, **options)
+  puts if index.positive?
+  puts "==> #{path} <==" if paths.size > 1
+  puts text
+rescue Readword::Error, SystemCallError => e
+  warn "readword: #{path}: #{e.message}"
+  failed = true
+end
+exit(1) if failed

data/lib/readword/binary.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+module Readword
+  # Little-endian number readers for the binary Office formats (MS-CFB, MS-DOC).
+  # They return nil past the end of the data, so a crafted offset reads as missing instead of raising.
+  module Binary
+    private
+
+    def u16(bytes, offset)
+      bytes.unpack1('v', offset: offset) if offset + 2 <= bytes.bytesize
+    end
+
+    def u32(bytes, offset)
+      bytes.unpack1('V', offset: offset) if offset + 4 <= bytes.bytesize
+    end
+  end
+end

data/lib/readword/compound_file.rb

@@ -0,0 +1,148 @@
+# frozen_string_literal: true
+
+module Readword
+  # Reads named streams out of a Compound File (MS-CFB), the container every .doc is stored in.
+  class CompoundFile
+    include Binary
+
+    # Compound File Header: the signature, then fields at fixed byte offsets in the first 512 bytes.
+    SIGNATURE = "\xD0\xCF\x11\xE0\xA1\xB1\x1A\xE1".b.freeze
+    HEADER_BYTES = 512
+    SECTOR_SHIFT = 0x1E
+    MINI_SECTOR_SHIFT = 0x20
+    FAT_SECTOR_COUNT = 0x2C
+    FIRST_DIRECTORY_SECTOR = 0x30
+    MINI_STREAM_CUTOFF = 0x38
+    FIRST_MINI_FAT_SECTOR = 0x3C
+    FIRST_DIFAT_SECTOR = 0x44
+    DIFAT_SECTOR_COUNT = 0x48
+    HEADER_DIFAT = 0x4C
+    HEADER_DIFAT_BYTES = 436
+    # Sector numbers above MAXREGSECT are markers such as ENDOFCHAIN, never real sectors.
+    MAX_REGULAR_SECTOR = 0xFFFFFFFA
+    # Directory entries are 128 bytes; NOSTREAM marks an empty link and object type 2 is a stream.
+    NO_ENTRY = 0xFFFFFFFF
+    STREAM = 2
+    ENTRY_BYTES = 128
+
+    def initialize(bytes)
+      @bytes = bytes.b
+      raise Error, UNREADABLE unless @bytes.bytesize >= HEADER_BYTES && @bytes.start_with?(SIGNATURE)
+
+      @sector_size = 1 << u16(@bytes, SECTOR_SHIFT)
+      @mini_sector_size = 1 << u16(@bytes, MINI_SECTOR_SHIFT)
+      raise Error, UNREADABLE unless [512, 4096].include?(@sector_size) && @mini_sector_size == 64
+
+      @fat = fat_sector_ids.flat_map { |sector_id| words(sector(sector_id)) }
+      @directory = chain_data(u32(@bytes, FIRST_DIRECTORY_SECTOR))
+      @root = entry(0) || raise(Error, UNREADABLE)
+    end
+
+    def stream(name)
+      found = root_children.find { |child| child[:type] == STREAM && child[:name].casecmp?(name) }
+      found && read(found)
+    end
+
+    def stream?(name)
+      root_children.any? { |child| child[:type] == STREAM && child[:name].casecmp?(name) }
+    end
+
+    private
+
+    # The header holds the first 109 FAT sector numbers; bigger files continue the list in chained DIFAT sectors.
+    def fat_sector_ids
+      limit = [u32(@bytes, FAT_SECTOR_COUNT), (sector_count / (@sector_size / 4)) + 1].min
+      ids = words(@bytes.byteslice(HEADER_DIFAT, HEADER_DIFAT_BYTES))
+      next_id = u32(@bytes, FIRST_DIFAT_SECTOR)
+      remaining = u32(@bytes, DIFAT_SECTOR_COUNT)
+      seen = Set.new
+      while remaining.positive? && ids.size < limit && next_id <= MAX_REGULAR_SECTOR && seen.add?(next_id)
+        difat = words(sector(next_id))
+        ids.concat(difat[0...-1])
+        next_id = difat.last.to_i
+        remaining -= 1
+      end
+      ids.first(limit).select { |id| id <= MAX_REGULAR_SECTOR }
+    end
+
+    # Directory entries form a tree through their left, right and child links.
+    def root_children
+      @root_children ||= begin
+        pending = [@root[:child]]
+        seen = Set.new
+        children = []
+        until pending.empty?
+          id = pending.pop
+          next if id == NO_ENTRY || !seen.add?(id)
+
+          child = entry(id) or next
+          children << child
+          pending.push(child[:left], child[:right])
+        end
+        children
+      end
+    end
+
+    def entry(index)
+      raw = @directory.byteslice(index * ENTRY_BYTES, ENTRY_BYTES)
+      return unless raw&.bytesize == ENTRY_BYTES
+
+      left, right, child = raw.unpack('@68V3')
+      { name: entry_name(raw), type: raw.getbyte(66), left: left, right: right, child: child,
+        start: raw.unpack1('@116V'), size: raw.unpack1('@120V') }
+    end
+
+    def entry_name(raw)
+      length = raw.unpack1('@64v').clamp(2, 64) - 2
+      raw.byteslice(0, length).force_encoding(Encoding::UTF_16LE).encode(Encoding::UTF_8, invalid: :replace, undef: :replace)
+    end
+
+    # Streams under the mini stream cutoff (4096 bytes) live in 64-byte mini sectors inside the root entry's stream.
+    def read(found)
+      data = found[:size] < u32(@bytes, MINI_STREAM_CUTOFF) ? mini_chain_data(found[:start]) : chain_data(found[:start])
+      data.byteslice(0, found[:size])
+    end
+
+    def chain_data(start)
+      chain(start, @fat).map { |sector_id| sector(sector_id) }.join
+    end
+
+    def mini_chain_data(start)
+      chain(start, mini_fat).map { |sector_id| mini_stream.byteslice(sector_id * @mini_sector_size, @mini_sector_size) }.join
+    end
+
+    def mini_fat
+      @mini_fat ||= chain(u32(@bytes, FIRST_MINI_FAT_SECTOR), @fat).flat_map { |sector_id| words(sector(sector_id)) }
+    end
+
+    def mini_stream
+      @mini_stream ||= chain_data(@root[:start])
+    end
+
+    # Follows a FAT or mini FAT chain; a chain longer than its table can only be a loop in a crafted file.
+    def chain(start, table)
+      ids = []
+      id = start
+      while id <= MAX_REGULAR_SECTOR
+        raise Error, UNREADABLE if id >= table.size || ids.size >= table.size
+
+        ids << id
+        id = table[id]
+      end
+      ids
+    end
+
+    # Sector 0 starts right after the header sector.
+    def sector(id)
+      @bytes.byteslice((id + 1) * @sector_size, @sector_size) || raise(Error, UNREADABLE)
+    end
+
+    def sector_count
+      @bytes.bytesize / @sector_size
+    end
+
+    def words(bytes)
+      bytes.to_s.unpack('V*')
+    end
+  end
+end