standard-procedure-consolidate 0.1.4 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f45c16dc8e6f37f91043673804db8076030a804ab6c08fa1139f4d86cdf54b79
-  data.tar.gz: 7c77802d6b578ac50b9f9c8b5bad9ff097918fab5644eb3c771b2baf98451a36
+  metadata.gz: f61fdd9cdf73d8609574952bca3a57839983f74f100aea6f8b54a4248b526efa
+  data.tar.gz: 0a64110c54d435f10c1465b96dd61e150c2ff56ae887788b5880d7bf489e93d5
 SHA512:
-  metadata.gz: 2ed822b460867d75395fdb9253ef3501619781d3ab31acaca7db4daf699658f8253d4a063336bcc2321bf6dd08f0207e91600172f8ec84e4263af055ee9107a7
-  data.tar.gz: 37a8f85beb0b9c65ba76bf9457edcb9e141d1e87aca1f65c01b5a4e43f45f143c66695d5922eb3ffafca43625f9e3745db8fd2e3c516dd6ffb73172a8c9437ac
+  metadata.gz: ac0a18d02265eb79b2c9ffda09b2a13825e520a08faef45c1a0f3fd29877b11abf46776554a7c7e2e4157e560f0269889a95f3a9dc3dc4f49b66bda51848e37c
+  data.tar.gz: 1fce15f81e519792c058dc731cd298db46dc4163d7c0c6cd753c87c2ec7a672aa4b9e350d526d8a641ebdb13bd3dd98b1a9ebd8bb854cd6d0e2342899103b883

data/CHANGELOG.md

@@ -1,3 +1,8 @@
+## [0.2.0] - 2023-09-13
+
+Thrown away the mail-merge implementation and replaced it with a simple search/replace.  
+Added in the command line utilities for examining and consolidating documents.  
+
 ## [0.1.4] - 2023-09-11
 
 Updated which files get exclusions after crashes in production with some client files

data/README.md

@@ -1,47 +1,80 @@
 # Standard::Procedure::Consolidate
 
-A simple gem for performing mailmerge on Microsoft Word .docx files.
+A simple gem for performing search and replace on Microsoft Word .docx files.
 
-Important: I can't claim the credit for this - I found [this gist](https://gist.github.com/ericmason/7200448) and have just adapted it.
+Important: I can't claim the credit for this - I found [this gist](https://gist.github.com/ericmason/7200448) and have adapted it for my needs.
 
-It's pretty simple, so it probably won't work with complex Word documents, but it does what I need.  YMMV.
+## Search/Replace for field placeholders
 
+If you have a Word .docx file that looks like this (ignoring formatting): 
 
-## Installation
+```
+Dear {{ first_name }},
 
-    $ bundle add standard-procedure-consolidate
+Thank you for your purchase of {{ product }} on the {{ date }}.  We hope that with the proper care and attention it will give you years of happy use.  
+
+Regards
+
+{{ user_name }}
+```
+
+We have marked out the "fields" by using squiggly brackets - in a manner similar to (but simpler than) Liquid or Mustache templates.  In this example, we have fields for first_name, product, date and user_name.  
+
+Consolidate reads your .docx file, locates these fields and then replaces them with the values you have supplied, writing the output to a new file.  
+
+NOTE: These are not traditional Word "mail-merge fields" - these are just fragments of text that are within the Word document.  See the history section for why this does not work with merge-fields.  
 
 ## Usage
 
-To list the merge fields within a document:
+There is a command-line tool or a ruby library that you can include into your code.  
+
+Using the ruby library: 
 
 ```ruby
-Consolidate::Docx::Merge.open "/path/to/docx" do |merge|
-  merge.examine
-end and nil
+Consolidate::Docx::Merge.open "/path/to/file.docx" do |doc|
+  puts doc.field_names 
+
+  doc.data first_name: "Alice", product: "Palm Pilot", date: "23rd January 2002", user_name: "Bob"
+  doc.write_to "/path/to/merge-file.docx"
+end 
 ```
-To perform a merge, replacing merge fields with supplied values:
 
-```ruby
-Consolidate::Docx::Merge.open "/path/to/docx" do |merge|
-  merge.data "Name" => "Alice Aadvark", "Company" => "TinyCo", "Job_Title" => "CEO"
-  merge.write_to "/path/to/output.docx"
-end
+Using the command line: 
+
+```sh
+examine /path/to/file.docx 
+
+consolidate /path/to/file.docx /path/to/merge-file.docx first_name=Alice "product=Palm Pilot" "date=23rd January 2022" "user_name=Bob" 
+```
+
+If you want to see what the routine is doing you can add the `verbose` option.  
+
+```sh
+examine /path/to/file.docx verbose
+
+consolidate /path/to/file.docx /path/to/merge-file.docx first_name=Alice "product=Palm Pilot" "date=23rd January 2022" "user_name=Bob" verbose
 ```
 
-NOTE: The merge fields are case-sensitive - which is why they should be supplied as strings (using the older `{ "key" => "value" }` style ruby hash).
+### History
 
-If your merge isn't working, you can pass `verbose: true` to `open` and it will list the internal .xml documents it finds, the fields it finds within those .xml documents and the substitutions it is trying to perform.
+Originally, this gem was intended to open a Word .docx file, find the mailmerge fields within it and then substitute new values.  
 
-## How it works
+I managed to get a basic version of this working with a variety of different Word files and all seemed good.  Until my client reported that when they went to print the document, the mailmerge fields reappeared!  My best guess is that Word is thinking that print-time is a trigger for merging in a data source (for example, printing out a form letter to 200 customers), so all the substitution work that this gem does is then discarded and Word asks for the merge data again.  The frustrating thing is I can't figure out how Word keeps the references to the merge fields after they've been substituted.  
+
+So instead, this does a simple search and replace - looking for fields within squiggly brackets and substituting them.  
+
+### How it works
 
 This is a bit sketchy and pieced together from the [code I found]((https://gist.github.com/ericmason/7200448)) and various bits of skimming through the published file format.
 
-A .docx file (unlike the previous .doc file), is actually a .zip file that contains a number of .xml files.  The contents of the document are stored in these .xml files, along with configuration information and stuff like fonts and styles.
+A .docx file is actually a .zip file that contains a number of .xml documents.  Some of these are for storing formatting information (fonts, styles and various metadata) and some are for storing the actual document contents.  
+
+Consolidate looks for word/document.xml files, plus any files that match word/header*.xml, word/footer*.xml, word/footnotes*.xml and word/endnotes*.xml.  It parses the XML, looking for text nodes that contain squiggly brackets.  If it finds them, it then checks to see if we have supplied a data value for the matching field-name and replaces the contents of the node.  
 
-When setting up a merge field, Word adds some special tags into the .xml file.  There appear to be two different versions of how it does this - using `w:fldSimple` or `w:instrText` tags.  Consolidate goes through each .xml file within the document (ignoring some which are configuration related) and looks for these two types of tag.
+## Installation
+
+    $ bundle add standard-procedure-consolidate
 
-The `data` method uses the hash of `field: value` data you supply, copies the .xml files and performs a straight text substitution on any matching merge fields.  Then `write_to`
 
 ## Development
 

data/checksums/standard-procedure-consolidate-0.2.0.gem.sha512

@@ -0,0 +1 @@
+872129b729c3531b72fbecc9bea75262ebc771d8e0a9f4d41b30ad87f73ee664aaccf8d602a410c3b500f067cd636f88c74eecb887cb68a345b6ea9d575c2140

data/exe/consolidate

@@ -1,3 +1,34 @@
 #!/usr/bin/env ruby
 
 require "consolidate"
+
+if ARGV[0].nil? || ARGV[0] == ""
+  puts "# Standard::Procedure::Consolidate"
+  puts "## Mailmerge for simple Microsoft Word .docx files."
+  puts ""
+  puts "Create a new file with the mailmerge fields replaced by the data you have supplied"
+  puts "USAGE: consolidate path/to/myfile.docx path/to/mynewfile.docx \"field1=value1\" \"field2=value2\" \"field3=value3\""
+  puts ""
+else
+  input = ARGV[0]
+  output = ARGV[1]
+  data = {}
+  verbose = false
+  2.upto ARGV.length do |index|
+    arg = ARGV[index]
+    next if arg.nil?
+    if arg.strip == "verbose"
+      verbose = true
+    else
+      pieces = arg.split("=")
+      key = pieces.first.strip
+      value = pieces.last.strip
+      data[key] = value
+    end
+  end
+
+  Consolidate::Docx::Merge.open input, verbose: verbose do |doc|
+    doc.data data
+    doc.write_to output
+  end and nil
+end

data/exe/examine

@@ -0,0 +1,18 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "consolidate"
+
+if ARGV[0].nil? || ARGV[0] == ""
+  puts "# Standard::Procedure::Consolidate"
+  puts "## Mailmerge for simple Microsoft Word .docx files."
+  puts ""
+  puts "Examine the mailmerge fields inside a .docx file"
+  puts "USAGE: examine path/to/myfile.docx"
+  puts "Option: examine path/to/myfile.docx verbose"
+  puts ""
+else
+  Consolidate::Docx::Merge.open ARGV[0], verbose: (ARGV[1] == "verbose") do |doc|
+    doc.examine
+  end and nil
+end

data/lib/consolidate/docx/merge.rb

@@ -12,23 +12,47 @@ module Consolidate
       end
 
       def examine
-        puts "Documents: #{extract_document_names}"
-        puts "Merge fields: #{extract_field_names}"
+        documents = document_names.join(", ")
+        fields = field_names.join(", ")
+        puts "Documents: #{documents}"
+        puts "Merge fields: #{fields}"
+      end
+
+      def field_names
+        documents.collect do |name, document|
+          (document / "//w:t").collect do |text_node|
+            next unless (matches = text_node.content.match(/{{\s*(\S+)\s*}}/))
+            field_name = matches[1].strip
+            puts "...field #{field_name} found in #{name}" if verbose
+            field_name
+          end.compact
+        end.flatten
+      end
+
+      def document_names
+        @zip.entries.collect { |entry| entry.name }
       end
 
       def data fields = {}
         fields = fields.transform_keys(&:to_s)
 
+        if verbose
+          puts "...substitutions..."
+          fields.each do |key, value|
+            puts "      #{key} => #{value}"
+          end
+        end
+
         @documents.each do |name, document|
           result = document.dup
-          result = substitute_style_one_with result, fields
-          result = substitute_style_two_with result, fields
+          result = substitute result, fields, name
 
           @output[name] = result.serialize save_with: 0
         end
       end
 
       def write_to path
+        puts "...writing to #{path}" if verbose
         Zip::File.open(path, Zip::File::CREATE) do |out|
           zip.each do |entry|
             out.get_output_stream(entry.name) do |o|
@@ -54,8 +78,8 @@ module Consolidate
         begin
           @zip = Zip::File.open(path)
           @zip.entries.each do |entry|
-            next unless entry.name =~ /word\/(document|header|footer|footnotes|endnotes).?\.xml/
-            puts "Reading #{entry.name}" if verbose
+            next unless entry.name.match?(/word\/(document|header|footer|footnotes|endnotes).?\.xml/)
+            puts "...reading #{entry.name}" if verbose
             xml = @zip.get_input_stream entry
             @documents[entry.name] = Nokogiri::XML(xml) { |x| x.noent }
           end
@@ -65,56 +89,16 @@ module Consolidate
         end
       end
 
-      def extract_document_names
-        @zip.entries.collect { |entry| entry.name }.join(", ")
-      end
-
-      def extract_field_names
-        (extract_style_one + extract_style_two).uniq.join(", ")
-      end
-
-      def extract_style_one
-        documents.collect do |name, document|
-          (document / "//w:fldSimple").collect do |field|
-            value = field.attributes["instr"].value.strip
-            puts "...found #{value} (v1) in #{name}" if verbose
-            value.include?("MERGEFIELD") ? value.gsub("MERGEFIELD", "").strip : nil
-          end.compact
-        end.flatten
-      end
-
-      def extract_style_two
-        documents.collect do |name, document|
-          (document / "//w:instrText").collect do |instr|
-            value = instr.inner_text
-            puts "...found #{value} (v2) in #{name}" if verbose
-            value.include?("MERGEFIELD") ? value.gsub("MERGEFIELD", "").strip : nil
-          end.compact
-        end.flatten
-      end
-
-      def substitute_style_one_with document, fields
-        # Word's first way of doing things
-        (document / "//w:fldSimple").each do |field|
-          if field.attributes["instr"].value =~ /MERGEFIELD (\S+)/
-            text_node = (field / ".//w:t").first
-            next unless text_node
-            puts "...substituting v1 #{field.attributes["instr"]} with #{fields[$1]}" if verbose
-            text_node.inner_html = fields[$1].to_s
-          end
-        end
-        document
-      end
-
-      def substitute_style_two_with document, fields
-        # Word's second way of doing things
-        (document / "//w:instrText").each do |instr|
-          if instr.inner_text =~ /MERGEFIELD (\S+)/
-            text_node = instr.parent.next_sibling.next_sibling.xpath(".//w:t").first
-            text_node ||= instr.parent.next_sibling.next_sibling.next_sibling.xpath(".//w:t").first
-            next unless text_node
-            puts "...substituting v2 #{instr.inner_text} with #{fields[$1]}" if verbose
-            text_node.inner_html = fields[$1].to_s
+      def substitute document, fields, document_name
+        (document / "//w:t").each do |text_node|
+          next unless (matches = text_node.content.match(/{{\s*(\S+)\s*}}/))
+          field_name = matches[1].strip
+          if fields.has_key? field_name
+            field_value = fields[field_name]
+            puts "...substituting #{field_name} with #{field_value} in #{document_name}" if verbose
+            text_node.content = text_node.content.gsub(matches[1], field_value).gsub("{{", "").gsub("}}", "")
+          elsif verbose
+            puts "...found #{field_name} but no replacement value"
           end
         end
         document

data/lib/consolidate/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Consolidate
-  VERSION = "0.1.4"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: standard-procedure-consolidate
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.2.0
 platform: ruby
 authors:
 - Rahoul Baruah
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-09-11 00:00:00.000000000 Z
+date: 2023-09-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rubyzip
@@ -43,6 +43,7 @@ email:
 - rahoulb@standardprocedure.app
 executables:
 - consolidate
+- examine
 extensions: []
 extra_rdoc_files: []
 files:
@@ -60,7 +61,9 @@ files:
 - checksums/standard-procedure-consolidate-0.1.2.gem.sha512
 - checksums/standard-procedure-consolidate-0.1.3.gem.sha512
 - checksums/standard-procedure-consolidate-0.1.4.gem.sha512
+- checksums/standard-procedure-consolidate-0.2.0.gem.sha512
 - exe/consolidate
+- exe/examine
 - lib/consolidate.rb
 - lib/consolidate/docx/merge.rb
 - lib/consolidate/version.rb