neu-mods 0.13.1 → 0.14.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: cc4e71fca2d864b7e1d27cba079bf00f30dd6540747845cb37c18e64d88059e5
-  data.tar.gz: 268e50d8ffa81fe8c97760f524d3af8bb9b412fd925721e967db3281c7ffed98
+  metadata.gz: abcf4a1fa8c50f807e790a3de6999df9e93356275530ebd7aaac32a7b5c93dd0
+  data.tar.gz: b4c3c5b43549e420bfb0ce17b8db3983522f32e5695bf21c7e4a2b2e32fcb2e3
 SHA512:
-  metadata.gz: 19b72bda07149f0e0f7d4797f20fa05866c3296a7612f4f90ac9b0c112d6ef12fef76b61e64f38f98e999102bf479e938a755b62f7dc8594d71a43ad026b5505
-  data.tar.gz: e63764144ec251467950d65a5e58728a7d9941fe85ce53080228453d8baf2e4ecdb2eef300b3af157ced06aad0051db8142b7b29ed3b537b78e95510130dd93d
+  metadata.gz: 1dca7512c1fd31592e5d9bcac19092574924d26da0345eb634e90481938b7e873f3c42c74064b73a276387ebf2ede68f1efdff789451587be0bd8682d508b145
+  data.tar.gz: 1471ba6e147cf9a4ab0f2cc5d19605a3bcfd82e5661ec63c81a9e385c271ebdb5bc6d982f726468dd6a667ca1080dec8dc097e12d41a49896b920ab0b686eb55

data/.version

@@ -1 +1 @@
-0.13.1
+0.14.1

data/CHANGELOG.md

@@ -0,0 +1,89 @@
+# Changelog
+
+Consumers pin this gem, so a change to a projected SHAPE belongs here: Atlas
+derives its access-copy attribute set and its Solr indexer from the projection,
+and Cerberus pre-fills its edit forms from it. A shape change that reaches them
+unannounced is one they discover as a nil value or a missing display row.
+
+Releases before 0.14.0 are not recorded. Their diffs are in git; this file
+starts where the convention does.
+
+## 0.14.1
+
+No projected shape changes. Two fixes change what the write path does, and the
+code is reorganised with the public method list unchanged.
+
+### Fixed
+
+- **The builders find the MODS namespace by URI.** `build_node` looked for the
+  literal prefix `mods`. Under any other prefix, such as `<m:mods xmlns:m=...>`,
+  it built elements in no namespace. The projection could not see them, and
+  `editable_creator_nodes` could not replace them, so each save added another
+  copy. A document whose root declares no MODS namespace now raises
+  `ArgumentError`.
+- **A blank authority attribute counts as absent on the edit path.** A name
+  with `authority=" "` was reported uncontrolled by `authority_of` but kept out
+  of the editable creators. It is now editable, matching the projection.
+
+### Changed
+
+- `Projection` is one mixin per MODS area under `lib/neu/mods/projection/`.
+  `Document`'s public and private methods and `to_h` are unchanged.
+  `Projection.compose_title`, `Projection.fold_type` and
+  `Projection::HEADING_SEPARATOR` still resolve.
+- The node builders move from `Selectors` to a `Builders` mixin. `Document`
+  includes both, so a call through a `Document` is unchanged.
+- Each file requires what it uses, so `require "neu/mods/document"` works
+  without the top-level entry file.
+- The date accessors and their `FIELDS` rows are generated from
+  `Projection::Dates::DATE_FIELDS` and `DATE_KEYS`. The field names are
+  unchanged.
+- The per-area explanation moves from source comments to `docs/`.
+
+## 0.14.0
+
+The projection gains the facts a consumer needs to offer a metadata value as a
+browse link. Additive, but four entry shapes grow keys.
+
+### Added
+
+- **The vocabulary a value was taken from**, as `authority:`,
+  `authority_uri:` and `value_uri:` on `names`, `origin_agents`, `languages`,
+  `genres` and `subject_headings`. Resolved off the element holding the value,
+  then off an enclosing `<subject>`; never off a `<role>`/`<roleTerm>`, which
+  names the relator's vocabulary rather than the name's. The gem tested these
+  attributes to decide editability (`Selectors#editable_creator_name?`,
+  `#keyword_subject?`) and projected none of them.
+- **`subject_headings[:heading]`** — the parts joined with ` -- `. The join
+  moves into the gem because the composed heading is now both the string a
+  display renders and the string a browse index holds; two callers joining
+  independently is how those two drift apart. `parts:` stays, for the Advanced
+  edit form and for a consumer wanting one step.
+- **`subject_headings[:axis]`** — the MODS element the heading's main term came
+  from, which is its first child carrying heading text. A consumer cannot
+  derive it from `parts:`, and it is what says which browse a heading belongs
+  to: `Salt marshes -- Massachusetts` is a topic heading with a place
+  subdivision, not a place. A `<name>` axis splits by `@type` into
+  `personal_name` or `corporate_name`.
+
+### Changed
+
+- **`corporate_name_subjects` now includes a `<subject><name>` with no
+  `@type`.** Such a name already displayed, because `#subject_heading_part`
+  never consulted `@type`, and it reached no axis, because both axis
+  projections required it. Corporate rather than personal: MODS expects
+  `@type="personal"` on a person, and the untyped subject names DRS holds are
+  institutional. `personal_name_subjects` is unchanged.
+
+### Not changed
+
+- **The per-axis subject projections stay plain string arrays.**
+  `topical_subjects` and its siblings carry no authority: `subject_headings`
+  reports it per heading, which is what a display and a browse index both read,
+  and nothing consumes a vocabulary on the flat axes.
+- **The other labeled fields carry no authority either.** `format`, `extent`,
+  `classification` and the rest keep `{ value:, display_label:, href: }`.
+  Nothing gates on their vocabulary, and three more keys on fifteen fields is
+  JSON no consumer reads.
+- **No new `FIELDS` entry.** Every change above is a key on an existing entry,
+  so a consumer deriving its schema from the registry needs no new column.

data/README.md

@@ -1,23 +1,27 @@
 # neu-mods
 
-Northeastern-flavored MODS v3 **projection + selection** for the DRS, shared by
+The MODS v3 reading contract for Northeastern's DRS, shared by
 [Cerberus](https://github.com/NEU-Libraries/cerberus) (front end) and
 [Atlas](https://github.com/NEU-Libraries/atlas) (API backend).
 
-It is a **Nokogiri-native, dependency-light contract over MODS documents** — pure
-functions over a parsed document, nothing else. No Rails, no persistence, no
-HTTP. It answers two questions:
+It is pure functions over a parsed MODS document, with no Rails, no persistence
+and no HTTP. It depends on Nokogiri alone, not on the `sul-dlss/mods` and
+`nom-xml` stack. It answers two questions:
 
-- **"Where is X?"** — `Selectors` return *live Nokogiri nodes*, so they serve both
-  the read path (projection reads their text) and the write path (an editor
-  mutates the returned node in place). The node an editor changes is provably the
-  node the projection reads.
-- **"What does this project to?"** — `Projection` returns *plain data*
-  (hashes/strings/arrays — never opaque typed objects) for indexing/display.
+- **"Where is X?"** `Selectors` return live Nokogiri nodes. They serve the read
+  path and the write path, so the node an editor changes is the node the
+  projection reads.
+- **"What does this project to?"** `Projection` returns plain data: hashes,
+  strings and arrays, for indexing and display.
 
-It depends on **Nokogiri alone** — deliberately *not* the `sul-dlss/mods` +
-`nom-xml` stack (which is sunsetting alongside Stanford's move to Cocina). See
-the design note in the DRS gap-reports for the full rationale.
+## Installation
+
+```ruby
+# Gemfile
+gem "neu-mods"
+```
+
+The gem needs Ruby 3.0 or later.
 
 ## Usage
 
@@ -26,201 +30,66 @@ require "neu-mods"
 
 doc = NEU::MODS::Document.parse(xml_string)
 
-# Projection (plain data)
+# Projection: plain data
+doc.to_h           # => every field in NEU::MODS::FIELDS
 doc.plain_title    # => "What's New. How We Respond to Disaster. Episode 1"
 doc.title_parts    # => { non_sort:, subtitle:, title:, part_name:, part_number: }
-                   #    byte-faithful -- the edit forms pre-fill from these
-doc.abstract       # => normalized, paragraph-joined String
-doc.languages      # => [{ term: "English", object_part: nil, script: nil }, ...]
-                   #    a code-only <languageTerm>eng</> is read through the
-                   #    ISO 639 registry. @objectPart rides along because
-                   #    objectPart="subtitles" says the SUBTITLES are Spanish,
-                   #    not the resource
-doc.topical_subjects # => ["Civil society", ...]   (every <topic>, for the access copy)
-doc.keywords       # => [...]   (only the editable attribute-free keyword subjects)
-doc.date_created_parts
-                   # => { value:, precision:, end_value:, end_precision:,
-                   #      qualifier:, key_date:, text: }   everything the record
-                   #    declared about one date. w3cdtf YYYY, YYYY-MM and
-                   #    YYYY-MM-DD all parse, and the precision says which
-                   #    shape it gave, so display cannot invent a month or a
-                   #    day. The points are read by @point, not by document
-                   #    order, and the end carries its OWN precision.
-                   #    A keyDate="yes" node chooses the value, ahead of
-                   #    @point and document order; one date per type is the
-                   #    rule, so an unflagged repeat is discarded.
-                   #    A value that is not a w3cdtf date projects NO date
-                   #    and keeps its literal in :text -- "19uu", "ca. 1920"
-                   #    and "undated" are statements a cataloguer made, and
-                   #    guessing a date for them is worse than either losing
-                   #    them or showing them as written.
-                   #    Same for the other six originInfo dates --
-                   #    date_issued, copyright_date, date_captured,
-                   #    date_valid, date_other and date_modified. Each part is
-                   #    also a reader of its own, e.g.
-                   #    doc.date_created_qualifier.
-doc.place_of_publication
-                   # => [{ value: "Boston", display_label:, href: }, ...]
-                   #    the type="text" placeTerm wins, and a bare marccountry
-                   #    code drops rather than reaching a places facet as a
-                   #    place name. A bare code under any other authority
-                   #    still projects
-
-# A name entry also carries @usage (fixed="primary" in the schema, so a record
-# that sets it has said which name leads) and :alternative_names, the MODS 3.7
-# alternativeName composed with the ENCLOSING name's @type.
-
-doc.origin_agents  # => [{ name:, roles:, affiliation:, display_label:, href:,
-                   #      event_type: }, ...]
-                   #    originInfo/agent, new in MODS 3.8: who performed the
-                   #    event the block records
-
-# An originInfo child also carries its block's @eventType, and a place carries
-# the NAMES of the date elements beside it (:date_elements) -- "Creation place"
-# and "Publication place" are the same element under a different date, and the
-# place says nothing about the event itself. Each of the seven dates gains
-# <date>_display_label and <date>_event_type from the same block.
-
-# Every DISPLAYED projection carries the @displayLabel and xlink:href of the
-# element its header comes from, as { value:, display_label:, href: } -- or as
-# two extra keys where the entry already had a shape of its own. MODS puts the
-# pair on originInfo and physicalDescription rather than on the publisher,
-# place, extent or digitalOrigin inside them, so those children read it off
-# their parent. The two attribute sets overlap rather than match (26 elements
-# take @displayLabel, 14 take xlink:href); an element the schema gives neither
-# projects nil. The four fields that JOIN several elements into one string --
-# abstract and the three accessCondition fields -- take companion scalars
-# instead (doc.abstract_display_label, doc.abstract_href).
-doc.host_collections
-                   # => [{ title:, volume:, issue:, start_page:, end_page:,
-                   #      date:, text:, details: [...], extents: [...] }, ...]
-                   #    this work's position in its host. The entry survives on
-                   #    its part alone, so a host with no titleInfo is kept
-doc.identifiers    # => [{ type: "isbn", value: "...", invalid: false,
-                   #      display_label:, href: }, ...]
-                   #    @invalid means cancelled or superseded, so it travels
-doc.table_of_contents
-                   # => [{ value: "Ch 1\nCh 2", ... }]
-                   #    line breaks kept: in a contents list
-                   #    the break is the structure, not stray formatting
-doc.notes          # => [{ type: "funding", value: "...", display_label:, href: }, ...]
-doc.related_items  # => [{ type: "otherFormat", title: "...",
-                   #      display_label:, href: }, ...]
-                   #    every relatedItem that is not a series or a host
-doc.location       # => [{ physical_location:, shelf_location:, url:,
-                   #      display_label:, href: }, ...]
-doc.map_data       # => [{ scale:, projection:, coordinates: }, ...]
-doc.title_subjects # => ["The Great Gatsby"]   composed like the main title
-doc.hierarchical_geographic_subjects
-                   # => [{ country:, state:, city:, ... }, ...]   eleven levels,
-                   #    structured for the reason map_data is
-doc.record_info    # => { content_source:, origin:, description_standard:,
-                   #      creation_date:, change_date:, language_of_cataloging: }
-                   #    describes the CATALOGUING, not the resource
-doc.to_h           # => full projection, keyed to Atlas's Metadata::MODS attributes
-
-# The field registry -- the single declaration of what this gem projects.
-# name => :one or :many. to_h is derived from it, and a consumer builds its own
-# schema from it rather than re-listing the field set by hand. Cardinality
-# follows what MODS marks repeatable, so a field can never silently truncate.
+doc.names          # => [{ name: "Cohen, Daniel J.(Daniel Jared), 1968-", roles: ["Creator"], ... }]
+doc.subject_headings
+                   # => [{ parts: [...], heading: "Salt marshes -- Massachusetts",
+                   #      axis: "topic", authority: "lcsh", ... }]
+doc.date_issued    # => a DateTime or nil, beside date_issued_precision, _end, ...
+
+# The field registry: field name => :one or :many
 NEU::MODS::FIELDS  # => { main_title: :one, names: :many, ... }
 
-# Pure title composition (no document needed) — for callers that already hold
-# the parts (e.g. Atlas's access-copy model) and must not re-parse XML on read.
-NEU::MODS.compose_title(non_sort: "", title: "What's New",
-                        part_name: "How We Respond to Disaster", part_number: "Episode 1")
-# => "What's New. How We Respond to Disaster. Episode 1"   (== doc.plain_title)
-# The part NUMBER precedes the part NAME: "Part 2. The Marshes" is the
-# cataloguing convention, and titleInfo is an unordered choice in the schema.
+# Title composition over parts a caller already holds, with no XML
+NEU::MODS.compose_title(title: "What's New", part_name: "How We Respond to Disaster",
+                        part_number: "Episode 1")
+# => "What's New. How We Respond to Disaster. Episode 1"
 
-# Selectors (live nodes — for editing)
+# Selectors: live nodes, for editing
 node = doc.primary_title_info.at_xpath("mods:title", NEU::MODS::NAMESPACE)
 node.content = "New Title" unless NEU::MODS.whitespace_equivalent?(node.text, "New Title")
-doc.to_xml
 
-# Editable creators (for an "advanced metadata" form): structured read,
-# node selection (for replace-on-save), and structure-aware build.
-doc.editable_personal_creators   # => [{ given:, family: }]  (plain, Creator role)
-doc.editable_corporate_creators  # => [{ name: }]
-doc.preserved_names              # => [{ name:, roles: }]  (authority-bearing / non-Creator — read-only)
-doc.editable_creator_nodes("personal")            # => live <name> nodes to replace
-doc.build_personal_name(given: "Jenny", family: "Smith")      # => a plain personal <name> node
-doc.build_corporate_name(name: "Northeastern University")     # => a plain corporate <name> node
+# Builders: new nodes in the document's MODS namespace
+doc.doc.root.add_child(doc.build_corporate_name(name: "Northeastern University"))
+doc.to_xml
 ```
 
-The "editable creator" set is plain names — **no `@authority`/`@authorityURI`/
-`@valueURI`** — with a **Creator** role; everything else (authority-controlled or
-other-role names) is `preserved_names`, shown read-only. This mirrors the
-keyword-subject curated-vs-editable split. `build_*_name`'s `role:` defaults to
-`"Creator"` but is parameterised, so a later role-selectable form is non-breaking.
-
-## Two normalizers, two jobs
-
-- `NEU::MODS.whitespace_equivalent?` / `.canonical_ws` — the **no-op guard**: did an
-  edit change anything, or only insignificant whitespace? (Used to avoid minting
-  an unchanged OCFL MODS version.)
-- `NEU::MODS.normalize_paragraphs` / `.normalize` — clean **curator freetext** for
-  the JSON/Solr access copy (dash/smart-punctuation transliteration, control
-  stripping, paragraph handling). The XML preservation copy is never touched.
-
-Titles and prose share the one freetext vocabulary: `to_h[:main_title]` is
-normalized like `abstract`, so an invisible format mark, a Windows-1252 control
-or an exotic space cannot reach Solr or a display template.
-
-**The boundary matters.** Normalization belongs on projections that only feed
-display and the index. `title_parts` is deliberately *not* normalized, because
-Cerberus pre-fills its Metadata and Advanced forms from it and `MODSMerge` writes
-the posted value back into the MODS XML — cleaning there would rewrite the
-curator's own characters in the preservation copy on the next save. Cerberus
-makes the same call for prose: its editable source is the bare `<abstract>` node,
-not `doc.abstract`. Add a normalized *sibling* rather than normalizing a
-projection an edit form reads.
-
-## Behavior fidelity & known caveats
-
-The projection is **behavior-preserving** with Atlas's prior `mods`-gem-based
-extraction, pinned by `spec/conformance_spec.rb` against `work-mods.xml`. Two
-intentional notes:
-
-- **Name display** reproduces the `mods` gem's `display_value_w_date` *including
-  its quirks* (e.g. multiple `given` nameParts concatenate with no separator),
-  to preserve existing Solr/display output. Cleanups are a deliberate future
-  contract change, not a silent one.
-- **Languages are translated; roles are not.** Both read the `type="text"` term
-  first. A code-only `languageTerm` is then translated through the vendored ISO
-  639 registry (`lib/neu/mods/data/iso639-2.txt`, from the Library of Congress),
-  so `eng` projects `English`. That happens here rather than in a consumer's
-  display layer because otherwise Solr indexes `eng` while the page shows
-  `English`, and the language facet reads in codes.
-  A code-only `roleTerm` stays raw. A MARC relator is a display *label*, and the
-  label vocabulary belongs to the consumer — Cerberus's edit form and Atlas's
-  display word the same role differently. An unrecognised language code also
-  stays raw, since the record still said something.
-- **`description` is not projected.** MODS does define `name/description`, but
-  that annotates a *name*, not the resource, so it is not the field Atlas once
-  called `description`. The two candidates for that one — an `abstract` variant
-  and `physicalDescription/note` — describe different things. Projecting a guess
-  would put wrong data in the field rather than leave an empty one, so it waits
-  on a decision.
-- **A date carries more than a value.** Each of `dateCreated`, `dateIssued` and
-  `copyrightDate` projects a value, its precision, an end value with its own
-  precision, the `@qualifier` and the `@keyDate` flag. The gem does not *pick*
-  the key date, because "which date to sort on" and "which date to display" are
-  not necessarily the same answer, and choosing is the consumer's job.
-
-## Source convention
-
-Every character-class regex in `TextNormalizer` is built **programmatically from
-codepoints**, so the source stays pure ASCII (no literal smart-quotes/dashes, no
-raw control bytes). A spec enforces this. Keep it that way.
+What each field holds, and why, is documented per MODS area in
+[`docs/`](docs/README.md). Start with [`docs/fields.md`](docs/fields.md).
+
+## Behavior fidelity and known caveats
+
+The projection preserves the output of Atlas's earlier `mods`-gem-based
+extraction. `spec/conformance_spec.rb` pins it against `work-mods.xml`, so a
+change to what the gem projects is a deliberate contract change.
+
+- **Name display reproduces the `mods` gem's `display_value_w_date`, quirks
+  included.** For example, two `given` name parts join with no separator. This
+  preserves Atlas's Solr and display output. A cleanup would be a contract
+  change. See [`docs/names.md`](docs/names.md).
+- **Languages are translated; roles are not.** A code-only `languageTerm` is
+  read through the vendored ISO 639 registry, so `eng` projects "English", and
+  Solr and the page agree. A code-only `roleTerm` stays raw, because a MARC
+  relator is a display label and each consumer words it differently. An
+  unrecognised language code also stays raw.
+- **`description` is not projected.** MODS `name/description` annotates a name,
+  not the resource. The two candidates for a resource description, an
+  `abstract` variant and `physicalDescription/note`, describe different things.
+  A guess would put wrong data in the field, so it waits on a decision.
+- **A date carries more than a value.** Each of the seven `originInfo` date
+  elements projects its value, precision, range end, qualifier, key-date flag,
+  literal text and block header. The gem does not choose which date to sort on
+  or display; that is the consumer's call. See [`docs/dates.md`](docs/dates.md).
 
 ## Development
 
-```bash
+```sh
 bundle install
-bundle exec rspec
-bundle exec rubocop
+bundle exec rake    # specs, then rubocop
 ```
 
-Versioned via the `.version` file (read by `lib/neu/mods/version.rb`); released
-with `bundler/gem_tasks` (`rake release`), mirroring `atlas_rb`.
+The version lives in `.version`, which `lib/neu/mods/version.rb` reads. Release
+with `bundler/gem_tasks` (`rake release`).

data/lib/neu/mods/builders.rb

@@ -0,0 +1,63 @@
+# frozen_string_literal: true
+
+require "nokogiri"
+
+require_relative "namespaces"
+
+module NEU
+  module MODS
+    # Node CREATION for the write path. Cerberus's MODSMerge builds every
+    # element it adds through these, so their names and signatures are part of
+    # the contract. See docs/editing.md.
+    module Builders
+      # A MODS element reusing the root's namespace declaration, matched by URI:
+      # built outside MODS, an element is invisible to every XPath here.
+      def build_node(name, text = nil)
+        node = Nokogiri::XML::Node.new(name, doc)
+        node.namespace = mods_namespace_definition
+        node.content = text unless text.nil?
+        node
+      end
+
+      # A plain personal creator: given and family parts and a text roleTerm.
+      def build_personal_name(given:, family:, role: "Creator")
+        name = build_node("name")
+        name["type"] = "personal"
+        name.add_child(name_part(given, "given")) unless given.to_s.strip.empty?
+        name.add_child(name_part(family, "family")) unless family.to_s.strip.empty?
+        name.add_child(role_node(role))
+        name
+      end
+
+      # A plain corporate creator: one name part and a text roleTerm.
+      def build_corporate_name(name:, role: "Creator")
+        node = build_node("name")
+        node["type"] = "corporate"
+        node.add_child(name_part(name)) unless name.to_s.strip.empty?
+        node.add_child(role_node(role))
+        node
+      end
+
+      private
+
+      def mods_namespace_definition
+        doc.root.namespace_definitions.find { |d| d.href == NAMESPACE["mods"] } ||
+          raise(ArgumentError, "document declares no MODS namespace on its root element")
+      end
+
+      def name_part(text, type = nil)
+        np = build_node("namePart", text.to_s.strip)
+        np["type"] = type if type
+        np
+      end
+
+      def role_node(role)
+        role_el = build_node("role")
+        term = build_node("roleTerm", role)
+        term["type"] = "text"
+        role_el.add_child(term)
+        role_el
+      end
+    end
+  end
+end

data/lib/neu/mods/canonicalize.rb

@@ -2,29 +2,22 @@
 
 module NEU
   module MODS
-    # Lightweight whitespace canonicalization used by the *no-op guard* -- does an
-    # edit actually change anything, or only insignificant whitespace? (Cerberus's
-    # MODSMerge uses this to avoid minting an unchanged OCFL MODS version.) This is
-    # deliberately distinct from TextNormalizer below: this one only folds
-    # whitespace; TextNormalizer cleans curator freetext for the access copy.
+    # Whitespace canonicalization for the no-op guard: did an edit change
+    # anything, or only insignificant whitespace? Distinct from TextNormalizer,
+    # which cleans text for the access copy. See docs/text-normalization.md.
     module Canonicalize
       module_function
 
       NBSP = [0xA0].pack("U") # U+00A0 non-breaking space, built from codepoint
 
-      # \s doesn't match U+00A0 (NBSP) in Ruby's default mode, so fold NBSP to a
-      # plain space first, then collapse any whitespace run to one space + strip.
+      # Fold NBSP first: Ruby's \s does not match U+00A0.
       def canonical_ws(str)
         str.to_s.tr(NBSP, " ").gsub(/\s+/, " ").strip
       end
 
-      # canonical_ws per line, keeping the line breaks. For a field where a
-      # newline is structure rather than formatting -- tableOfContents, where
-      # the break separates one entry from the next. Blank lines drop, so a
-      # double-spaced list does not project empty entries.
+      # canonical_ws per line, keeping the breaks and dropping blank lines.
       def canonical_lines(str)
-        str.to_s.tr(NBSP, " ").split("\n").map { |line| line.gsub(/\s+/, " ").strip }
-           .reject(&:empty?).join("\n")
+        str.to_s.split("\n").map { |line| canonical_ws(line) }.reject(&:empty?).join("\n")
       end
 
       # Treat values differing only by insignificant whitespace (NBSP vs space,
@@ -33,143 +26,5 @@ module NEU
         canonical_ws(current) == canonical_ws(incoming)
       end
     end
-
-    # Normalises curator-authored freetext on the way into the JSON access copy
-    # (and Solr); the XML preservation copy stays untouched. Ported from Atlas's
-    # TextNormalizer (which carries DRS v1 prior art) so the gem reproduces Atlas's
-    # projection byte-for-byte.
-    #
-    # IMPORTANT: every character-class regex is built *programmatically* from
-    # codepoint lists via `format('\\u%04X', cp)`, so this source file stays pure
-    # ASCII -- no literal smart-quotes, dashes, or (critically) raw control bytes
-    # land on disk. Keep it that way.
-    #
-    # Pipeline: force UTF-8 + scrub invalid bytes; NFC; map Unicode dashes to '-'
-    # (swung-dash to '~'); transliterate the General Punctuation block (smart
-    # quotes, ellipsis, etc.) to ASCII; map the separator controls to a newline
-    # and drop what is invisible (the soft hyphen, the rest of C0/C1, keeping
-    # tab/newline); collapse
-    # horizontal-whitespace runs to one space; for paragraph fields, collapse
-    # 2+ newlines to exactly two; strip.
-    #
-    #   .normalize(str)            -- single-line fields (newlines -> spaces)
-    #   .normalize_paragraphs(str) -- fields that may carry paragraph breaks
-    #                                 (abstract, accessCondition)
-    module TextNormalizer
-      module_function
-
-      # Build a character-class Regexp from an array of integer codepoints, as
-      # \uXXXX escapes (keeps this source ASCII).
-      def self.char_class(codepoints, prefix: "")
-        Regexp.new("[#{prefix}#{codepoints.map { |cp| format('\\u%04X', cp) }.join}]")
-      end
-
-      # NOTE: U+2053 (swung dash) is intentionally excluded from dashes -- it is
-      # named "dash" but conventionally maps to ASCII '~', not '-' (V1 prior art).
-      DASH_CODEPOINTS = [
-        0x002D, 0x058A, 0x05BE, 0x1400, 0x1806,
-        0x2010, 0x2011, 0x2012, 0x2013, 0x2014, 0x2015,
-        0x2043, 0x207B, 0x208B, 0x2212,
-        0x2E17, 0x2E1A, 0x2E3A, 0x2E3B, 0x2E40,
-        0x301C, 0x3030, 0x30A0, 0xFE31, 0xFE32, 0xFE58,
-        0xFE63, 0xFF0D
-      ].freeze
-      DASH_RE = char_class(DASH_CODEPOINTS).freeze
-
-      SWUNG_DASH_RE = Regexp.new(format('\\u%04X', 0x2053)).freeze
-
-      # U+00AD is a hint about where a word may break, not a dash: it renders as
-      # nothing, and Solr discards it, so "co<00AD>operation" already matches a
-      # search for "cooperation". Mapping it to an ASCII hyphen instead would
-      # index "co" and "operation" as two tokens and lose the word, so it is
-      # dropped and kept out of DASH_CODEPOINTS.
-      SOFT_HYPHEN_RE = Regexp.new(format('\\u%04X', 0x00AD)).freeze
-
-      # U+000B (vertical tab) and U+000C (form feed) separate words rather than
-      # meaning nothing: Word writes a manual line break as U+000B and a page
-      # break as U+000C. They map to a newline, because deleting one runs the
-      # words either side of it together -- normalize_paragraphs then reads that
-      # newline as the soft wrap the line break was, and normalize turns it into
-      # a space.
-      SEPARATOR_CONTROL_CODEPOINTS = [0x000B, 0x000C].freeze
-      SEPARATOR_CONTROL_RE = char_class(SEPARATOR_CONTROL_CODEPOINTS).freeze
-
-      # C0 (U+0000..U+0008, U+000D..U+001F) and C1 (U+007F..U+009F) -- what is
-      # left once the separators above are accounted for, and none of it carries
-      # meaning in curator text. U+0009 (tab) and U+000A (newline) are preserved.
-      # U+000D is not: dropping it reduces a CRLF line ending to the single
-      # newline it stands for.
-      CONTROL_CODEPOINTS = ((0x0000..0x0008).to_a + (0x000D..0x001F).to_a + (0x007F..0x009F).to_a).freeze
-      CONTROL_RE = char_class(CONTROL_CODEPOINTS).freeze
-
-      HORIZONTAL_WS_CODEPOINTS = [
-        0x0009, 0x00A0, 0x1680,
-        0x2000, 0x2001, 0x2002, 0x2003, 0x2004, 0x2005, 0x2006,
-        0x2007, 0x2008, 0x2009, 0x200A, 0x202F, 0x205F, 0x3000
-      ].freeze
-      # Leading literal space included in the class (the " " prefix); `+` so a run
-      # of horizontal whitespace collapses to a single space.
-      HORIZONTAL_WS_RE = Regexp.new("#{char_class(HORIZONTAL_WS_CODEPOINTS, prefix: " ").source}+").freeze
-
-      PARAGRAPH_RUN_RE = /\n{2,}/
-
-      # General Punctuation block (U+2000..U+206F). Codepoints not listed pass
-      # through unchanged. Empty-string values deliberately drop invisible/bidi/
-      # format marks so they cannot leak into the access copy.
-      GENERAL_PUNCTUATION = {
-        0x2000 => " ", 0x2001 => " ", 0x2002 => " ", 0x2003 => " ",
-        0x2004 => " ", 0x2005 => " ", 0x2006 => " ", 0x2007 => " ",
-        0x2008 => " ", 0x2009 => " ", 0x200A => " ",
-        0x200B => "",  0x200C => "",  0x200D => "",
-        0x200E => "",  0x200F => "",
-        0x2018 => "'", 0x2019 => "'", 0x201A => ",", 0x201B => "'",
-        0x201C => '"', 0x201D => '"', 0x201E => '"', 0x201F => '"',
-        0x2020 => "+", 0x2021 => "+",
-        0x2022 => "*", 0x2023 => "*", 0x2024 => ".", 0x2025 => "..",
-        0x2026 => "...",
-        0x2028 => "\n", 0x2029 => "\n\n",
-        0x202A => "",  0x202B => "", 0x202C => "", 0x202D => "",
-        0x202E => "",  0x202F => " ",
-        0x2030 => "%", 0x2032 => "'", 0x2033 => '"', 0x2035 => "'",
-        0x2036 => '"',
-        0x2039 => "<", 0x203A => ">", 0x203C => "!!", 0x203D => "?",
-        0x2044 => "/", 0x2052 => "%",
-        0x205F => " ", 0x2060 => "", 0x2061 => "", 0x2062 => "",
-        0x2063 => "",  0x2064 => "",
-        0x206A => "",  0x206B => "", 0x206C => "", 0x206D => "",
-        0x206E => "",  0x206F => ""
-      }.transform_keys { |cp| [cp].pack("U") }.freeze
-      GENERAL_PUNCTUATION_RE = Regexp.new("[#{format('\\u%04X-\\u%04X', 0x2000, 0x206F)}]").freeze
-
-      def normalize(str)
-        return "" if str.nil?
-
-        s = base_normalize(str.to_s)
-        s = s.tr("\n", " ")
-        s.gsub(HORIZONTAL_WS_RE, " ").strip
-      end
-
-      def normalize_paragraphs(str)
-        return "" if str.nil?
-
-        s = base_normalize(str.to_s)
-        s = s.gsub(HORIZONTAL_WS_RE, " ")
-        s = s.gsub(/ *\n */, "\n")
-        s.split(PARAGRAPH_RUN_RE).map { |p| p.tr("\n", " ").strip }
-                                 .reject(&:empty?).join("\n\n")
-      end
-
-      def base_normalize(str)
-        s = str.dup.force_encoding("UTF-8")
-        s = s.scrub("")
-        s = s.unicode_normalize(:nfc)
-        s = s.gsub(DASH_RE, "-")
-        s = s.gsub(SWUNG_DASH_RE, "~")
-        s = s.gsub(GENERAL_PUNCTUATION_RE) { |c| GENERAL_PUNCTUATION.fetch(c, c) }
-        s = s.gsub(SEPARATOR_CONTROL_RE, "\n")
-        s = s.gsub(SOFT_HYPHEN_RE, "")
-        s.gsub(CONTROL_RE, "")
-      end
-    end
   end
 end