fair_champion_harvester 0.1.12 → 0.1.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6fa689e4da4938f714f1b3635db3b5a2065c14dcb8836cc10237528af84a045d
-  data.tar.gz: e2e45e0a23b5c31eaab37bbdaef962ce90bc4e92e8fc6996decd7c4d34c6a506
+  metadata.gz: 98197689724c7efe3d1d725d74b96d841ef1e9b4da0ab6068a721109192cd901
+  data.tar.gz: eab9a398f94b2770d60aa9bde1a81ca035bc21ee69bc22d2b0be0fc46a148e5e
 SHA512:
-  metadata.gz: e1664661b38f7d460a72cab769bf32039a6026119d1f4156d6e049f680feb17e1884e39e194f1b02f582869e888c68b53c390c9d374f4fb8d67e3fc4c26bab1a
-  data.tar.gz: 7d2a6cdbd9b1f81de505f7064f8092db09b0d31a28c087435ebc3c874f2950ce57d42c31426a4e177f8541cafacd9bb274e4dd1686cc33247bfcc083cb2993a3
+  metadata.gz: ac9ca34832e5045adc5761f4579d69eec86e7ee87e54aeeb6ccb6ae87a3d04b31d3c03bcbbcf730d4274ce2066fbdd4b5e7e28afc5528544d8a84078bf4d1d61
+  data.tar.gz: 3d1f6868db870bcd874f3797a93160f5e330b7614c7de0aaa514d3d914ee8fc17a5f6cc2c99ecba0386c030419af446052faf6c7428b2969dd9ce06a1ce95507

data/CHANGELOG.md

@@ -1,13 +1,29 @@
+# Changelog
+
 ## [Unreleased]
 
+## [0.1.13] - 2026-06-30
+
+### Fixed
+
+- JSON-LD documents with a remote `@context` (e.g. `"@context": "http://schema.org"`) were not having their context resolved, so properties like `license` that carry `"@type": "@id"` in the schema.org context were being parsed as string literals instead of IRI resources. This caused FAIR license assessment tests to warn "Found the Schema license predicate, but it does not have a Resource as its value." Fixed by routing JSON-LD through `JSON::LD::API.expand` (processing mode 1.1) before conversion to RDF, which applies `@type: @id` coercions. `json/ld/preloaded` (already required) supplies schema.org and other common contexts without network overhead.
+- Array-wrapped JSON-LD (a top-level JSON array, as returned by extruct and some APIs) no longer raises `TypeError: no implicit conversion of String into Integer` during the context debug log.
+- Added graceful fallback: if JSON-LD context expansion raises `JSON::LD::JsonLdError`, the document is re-parsed without expansion and a warning is recorded rather than failing silently.
+
+### Added
+
+- RSpec unit tests for `parse_rdf` JSON-LD path: license IRI coercion, successful expansion comment, array-wrapped document handling.
+
 ## [0.1.12] - 2026-05-27
 
 ### Fixed
+
 - 0.1.11 regression: `FAIRChampionHarvester::Uri` was missing from the gem because `uri_resolver.rb` was not tracked by git and the gemspec uses `git ls-files` to determine packaged files; merged the class back into `lib/uri.rb` so a single tracked file handles both stdlib forwarding and the class definition
 
 ## [0.1.11] - 2026-05-26
 
 ### Fixed
+
 - `lib/uri.rb` was shadowing Ruby's stdlib `require "uri"` because `lib/` is on the load path; moved `FAIRChampionHarvester::Uri` class to `lib/uri_resolver.rb` and turned `lib/uri.rb` into a stdlib-forwarding shim — this was the root cause of all HTTP fetch failures (`uninitialized constant URI`)
 - `parse_link_http_headers`: handle multiple separate `Link:` headers (Array input) in addition to comma-separated single-string headers, using `Array(links).flat_map { |l| l.split(",") }`
 - `parse_link_http_headers`: `rel` regex `\w+` → `[\w-]+` so hyphenated rel types like `cite-as` are captured correctly rather than silently truncated
@@ -18,13 +34,13 @@
 - `Core.fetch`: backtrace now always logged on `StandardError` (removed `if ENV["DEBUG"]` guard)
 
 ### Added
+
 - RSpec test suite with 14 unit tests for `parse_link_http_headers` and 6 live integration tests against `https://fairsharing.org/1547`; live tests clear stale `/tmp/*_error` cache files before each run
 
 ## [0.1.10] - 2026-05-26
 
 - variable url was not defined, but caught by begin block so no complaints
 
-
 ## [0.1.0] - 2026-03-27
 
 - Initial release

data/lib/fair_champion_harvester/handoff.md

@@ -0,0 +1,313 @@
+# JSON-LD Context Expansion Fix — Handoff for Claude Code Integration
+
+**Date:** 2026-06-30  
+**Issue:** JSON-LD license properties being parsed as blank nodes instead of IRI resources  
+**Root Cause:** Remote `@context` resolution not enabled in RDF reader instantiation  
+**Impact:** FAIR assessment tests failing because `schema.org` license coercion (`@type: @id`) not applied
+
+---
+
+## Problem Summary
+
+### What's Happening
+When the FAIR Champion Harvester encounters JSON-LD with `"@context": "http://schema.org"` and `"license": "https://creativecommons.org/licenses/by/4.0/legalcode"`:
+
+1. **Expected behavior:** JSON-LD processor fetches schema.org context, sees `license` has `"@type": "@id"`, coerces the URL string to an IRI resource
+2. **Actual behavior:** Parser treats license as a bare string literal, or normalizes it into a blank node structure during serialization
+
+### Why It Fails
+The current `parse_rdf` method in `FAIRChampionHarvester` uses:
+```ruby
+reader = formattype.reader.new(body)
+meta.merge_rdf(reader.to_a)
+```
+
+This instantiates `JSON::LD::Reader` with **no processing options**, so:
+- Remote `@context` URLs are **not fetched** (security/performance default)
+- JSON-LD expansion rules are **not applied**
+- `@type: @id` coercions don't happen
+- Bare string IRIs stay as literals
+
+### Real-World Impact
+ESRF dataset metadata has:
+```json
+{
+  "@context": "http://schema.org",
+  "license": "https://creativecommons.org/licenses/by/4.0/legalcode"
+}
+```
+
+Without context expansion → license becomes a literal string or blank node → FAIR license test fails with warning:
+```
+WARN: Found the Schema license predicate, but it does not have a Resource as its value.
+```
+
+---
+
+## Solution: Enable JSON-LD Expansion with Remote Context Resolution
+
+### Key Code Decisions
+
+**Decision 1: Format-specific handling**
+- JSON-LD gets special treatment: use `JSON::LD::API.expand()` + `toRdf()` pipeline
+- Other RDF formats (Turtle, N-Triples, RDF/XML) use existing reader pattern
+- Rationale: Only JSON-LD needs context resolution; other formats are already RDF
+
+**Decision 2: Processing options**
+```ruby
+processingOptions: {
+  processingMode: 'json-ld-1.1',           # CRITICAL: 1.0 mode has weaker @type: @id semantics
+  documentLoader: JSON::LD::DocumentLoader.new  # Explicitly enable remote context fetching
+}
+```
+- `processingMode: 'json-ld-1.1'` ensures `@type: @id` is properly applied (1.0 had edge cases)
+- `documentLoader.new` explicitly enables fetching of remote `@context` URIs (may be disabled by default)
+
+**Decision 3: Graceful fallback**
+- If remote context resolution fails (network, malformed URI), catch `JSON::LD::ProcessingError`
+- Attempt parsing without expansion as fallback
+- Log warning so operator knows context wasn't resolved
+- Rationale: Metadata is still parseable as plain RDF even without context; better to get something than fail completely
+
+**Decision 4: Cache interaction**
+- Continue using existing `FAIRChampionHarvester::Cache` for non-JSON-LD formats
+- For JSON-LD, cache the **raw body**, not the reader (reader is stateful, hard to serialize)
+- After expansion, the RDF graph is cached via normal merge process
+- Rationale: Keeps cache simple; re-expansion is cheap
+
+---
+
+## Implementation Strategy
+
+### File to Modify
+`lib/fair_champion_harvester/evaluator.rb` (or wherever `parse_rdf` method lives)
+
+### Change Scope
+- **Lines affected:** Reader instantiation block (approximately where `formattype.reader.new(body)` appears)
+- **Methods touched:** `parse_rdf` static method
+- **New requires:** `require 'json/ld'` (add at top of method or class)
+- **No breaking changes:** Falls back to existing behavior if JSON-LD expansion fails
+
+### Integration Checklist
+- [ ] Locate `parse_rdf` method definition
+- [ ] Add `require 'json/ld'` at method start or class level
+- [ ] Replace simple reader pattern with format-aware conditional
+- [ ] Test against ESRF metadata (original + expanded forms)
+- [ ] Verify license triple is now IRI resource, not literal
+- [ ] Run FAIR license assessment test — should pass or show nuanced result
+- [ ] Check cache hit/miss behavior for JSON-LD documents
+- [ ] Monitor for JSON::LD::ProcessingError in production logs
+
+---
+
+## Code Structure: What Gets Added
+
+### Before (Current)
+```ruby
+reader = formattype.reader.new(body)
+if reader.size == 0
+  # handle empty graph
+end
+reader = formattype.reader.new(body)
+FAIRChampionHarvester::Cache.writeRDFCache(reader, body)
+reader = formattype.reader.new(body)
+meta.merge_rdf(reader.to_a)
+```
+
+### After (Proposed)
+```ruby
+if formattype.to_s.include?('JSON::LD')
+  # JSON-LD: expand with remote context resolution
+  json_body = JSON.parse(body)
+  
+  expanded = JSON::LD::API.expand(json_body, processingOptions: {
+    processingMode: 'json-ld-1.1',
+    documentLoader: JSON::LD::DocumentLoader.new
+  })
+  
+  graph = RDF::Graph.new
+  JSON::LD::API.toRdf(expanded) do |statement|
+    graph << statement
+  end
+  
+  meta.merge_rdf(graph.to_a)
+  FAIRChampionHarvester::Cache.writeRDFCache(graph, body)  # cache expanded RDF
+else
+  # Turtle, N-Triples, RDF/XML, etc: use standard reader
+  reader = formattype.reader.new(body)
+  
+  if reader.size == 0
+    meta.comments << "WARN: Though linked data was found, it failed to parse...\n"
+    return meta
+  end
+  
+  reader = formattype.reader.new(body)
+  FAIRChampionHarvester::Cache.writeRDFCache(reader, body)
+  reader = formattype.reader.new(body)
+  meta.merge_rdf(reader.to_a)
+end
+```
+
+### Error Handling
+Wrap in `rescue JSON::LD::ProcessingError` to catch context resolution failures:
+```ruby
+rescue JSON::LD::ProcessingError => e
+  meta.comments << "WARN: JSON-LD context resolution failed: #{e.message}\n"
+  # Try fallback: parse without expansion
+  begin
+    reader = formattype.reader.new(body)
+    meta.merge_rdf(reader.to_a)
+    meta.comments << "WARN: Parsed JSON-LD without remote context resolution. Results may be incomplete.\n"
+  rescue => fallback_error
+    meta.comments << "CRITICAL: JSON-LD parsing failed even without context expansion: #{fallback_error.message}\n"
+  end
+```
+
+---
+
+## Validation: How to Test the Fix
+
+### Unit Test: ESRF Metadata
+```ruby
+# Test with the original ESRF JSON-LD
+json_body = JSON.parse(File.read('spec/fixtures/esrf_dataset.jsonld'))
+
+expanded = JSON::LD::API.expand(json_body, processingOptions: {
+  processingMode: 'json-ld-1.1',
+  documentLoader: JSON::LD::DocumentLoader.new
+})
+
+graph = RDF::Graph.new
+JSON::LD::API.toRdf(expanded) { |stmt| graph << stmt }
+
+# Check license triple
+license_triples = graph.query([nil, RDF::URI('http://schema.org/license'), nil])
+license_triples.each do |stmt|
+  expect(stmt.object.resource?).to be true
+  expect(stmt.object.anonymous?).to be false
+  expect(stmt.object.to_s).to eq 'https://creativecommons.org/licenses/by/4.0/legalcode'
+end
+```
+
+### Integration Test: FAIR Assessment
+Run your existing FAIR license test against the fixed harvester:
+```ruby
+result = FAIRChampionHarvester.evaluate('https://doi.org/10.15151/esrf-es-2210534378')
+
+# Should now pass (or at least not complain about literal value)
+expect(result.output.score).to eq 'pass'
+expect(result.output.comments).to include('SUCCESS: Found the Schema license predicate with a Resource')
+```
+
+### Diagnostic Output
+Add debug logging to trace expansion:
+```ruby
+warn "JSON-LD Expansion: Input has #{json_body.keys.length} root keys"
+warn "JSON-LD Expansion: Context is #{json_body['@context']}"
+expanded = JSON::LD::API.expand(json_body, processingOptions: {...})
+warn "JSON-LD Expansion: Output has #{expanded.length} objects"
+```
+
+---
+
+## Performance Considerations
+
+### Remote Context Fetching
+- **Latency:** First fetch of `http://schema.org` context takes ~200-500ms
+- **Caching:** Ruby's `json-ld` gem caches fetched contexts in memory — subsequent documents are fast
+- **Network:** Single HTTP GET per unique `@context` URL per process lifetime
+- **Recommendation:** OK for production; consider warm-up if processing high volume at startup
+
+### JSON-LD Expansion
+- **CPU:** Expansion is O(n) in document size; typically <50ms for datasets
+- **Memory:** Expanded form is usually 1.5-2x original size
+- **Recommendation:** Acceptable; RDF graph construction is already expensive
+
+### Cache Strategy
+- Cache the **expanded RDF graph**, not the raw JSON-LD
+- This way, subsequent accesses skip expansion
+- For documents with same `@context`, expansion is cached by `json-ld` gem anyway
+
+---
+
+## Known Limitations & Edge Cases
+
+### 1. Custom or Malformed Contexts
+If a document's `@context` URL returns 404 or invalid JSON, expansion will fail.
+- **Mitigation:** Fallback to parsing without expansion (implemented in error handler)
+- **Log:** Warning message indicates context couldn't be resolved
+
+### 2. Blank Nodes vs. Named Resources
+Schema.org allows some properties to be blank nodes. Example: a license as a `CreativeWork` object.
+```json
+{
+  "license": {
+    "@type": "CreativeWork",
+    "name": "CC-BY",
+    "url": "https://creativecommons.org/licenses/by/4.0/"
+  }
+}
+```
+- **Current test logic:** Rejects blank nodes (`!object.anonymous?`)
+- **Better approach:** Accept blank nodes IF they have properties; only fail on bare string literals
+- **Future refinement:** Update license test to differentiate:
+  - Named resource IRI (best)
+  - Anonymous resource with properties (good)
+  - String literal (warn)
+
+### 3. Context Conflicts
+If document has both `"@context": "http://schema.org"` and inline context definitions, inline wins.
+- **Expected behavior:** This is JSON-LD spec-compliant
+- **No action needed:** Rare in practice
+
+---
+
+## Debugging Tips for Claude Code Session
+
+### If License Still Appears as Literal
+1. Check if `JSON::LD::API.expand()` is actually being called
+   - Add `warn "Expanding JSON-LD with schema.org context"`
+2. Verify documentLoader is enabled:
+   - Try adding `documentLoader: JSON::LD::DocumentLoader.new` explicitly
+3. Check if remote context is actually fetching:
+   - Try: `context = JSON.parse(Net::HTTP.get(URI('https://schema.org/docs/jsonldcontext.json')))`
+   - If that times out, network issue; if it parses, `json-ld` gem should too
+
+### If Blank Nodes Still Appear
+- Check if license is nested structure: `{ "@type": "CreativeWork", ... }`
+- If so, that's valid RDF; update FAIR test to accept anonymous resources with properties
+- If it's a bare blank node with no properties, something is still wrong with expansion
+
+### If Context Resolution Fails
+- Check gem version: `bundle show json-ld`
+- Ensure `json-ld >= 3.2.0` (older versions had weaker context support)
+- Try manually: `JSON::LD::API.expand(json_body, processingOptions: { documentLoader: JSON::LD::DocumentLoader.new })`
+- Check network: Can you reach `http://schema.org` from the runtime environment?
+
+---
+
+## Commit Message Template
+
+```
+Fix: Enable JSON-LD context expansion for remote @context resolution
+
+- Add processingMode: 'json-ld-1.1' to apply @type: @id coercions
+- Enable documentLoader for remote context fetching (schema.org, etc.)
+- Separate JSON-LD handling from other RDF formats
+- Add graceful fallback if context resolution fails
+- Resolves: License properties now correctly expanded as IRI resources
+- Tested against: ESRF dataset metadata with schema.org context
+
+Fixes #ISSUE_NUMBER (if applicable)
+```
+
+---
+
+## References
+
+- **JSON-LD 1.1 Spec:** https://www.w3.org/TR/json-ld11/
+- **schema.org context:** https://schema.org/docs/jsonldcontext.json
+- **ruby json-ld gem:** https://github.com/ruby-rdf/json-ld
+- **schema.org license property:** https://schema.org/license
+- **FAIR license assessment:** Your FAIR test queries for `http://schema.org/license` with `@type: @id`
+

data/lib/fair_champion_harvester/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module FairChampionHarvester
-  VERSION = "0.1.12"
+  VERSION = "0.1.13"
 end

data/lib/harvester.rb

@@ -148,38 +148,80 @@ module FAIRChampionHarvester
         return meta
       end
       meta.comments << "INFO: The response message body component appears to contain #{formattype}.\n"
-      reader = ""
-      begin
-        reader = formattype.reader.new(body)
-      rescue StandardError
-        meta.comments << "WARN: Though linked data was found, it failed to parse.  This likely indicates some syntax error in the data.  As a result, no metadata will be extracted from this message.\n"
-        return meta
-      end
-
-      begin
-        # $stderr.puts "Reader Class #{reader.class}\n\n #{reader.inspect}"
-        if reader.size == 0
+      if formattype.to_s.include?('JSON::LD')
+        # JSON-LD: expand with remote context resolution to correctly apply @type: @id coercions
+        # (e.g. schema.org "license" coerced to IRI resource instead of string literal)
+        # json/ld/preloaded provides schema.org and other common contexts without network calls
+        begin
+          json_body = JSON.parse(body)
+          context_hint = json_body.is_a?(Array) ? json_body.first&.[]('@context') : json_body['@context']
+          warn "JSON-LD Expansion: Context is #{context_hint}"
+
+          expanded = JSON::LD::API.expand(json_body, processingMode: 'json-ld-1.1')
+          warn "JSON-LD Expansion: Output has #{expanded.length} objects"
+
+          graph = RDF::Graph.new
+          JSON::LD::API.toRdf(expanded) { |statement| graph << statement }
+
+          if graph.size == 0
+            meta.comments << "WARN: Though linked data was found, it failed to parse.  This likely indicates some syntax error in the data.  As a result, no metadata will be extracted from this message.\n"
+            return meta
+          end
+
+          meta.comments << "INFO: JSON-LD context resolved and expanded successfully.\n"
+          FAIRChampionHarvester::Cache.writeRDFCache(graph, body)
+          meta.merge_rdf(graph.to_a)
+
+        rescue JSON::LD::JsonLdError => e
+          meta.comments << "WARN: JSON-LD context resolution failed: #{e.message}. Attempting parse without expansion.\n"
+          warn "JSON-LD JsonLdError: #{e.message}"
+          begin
+            reader = formattype.reader.new(body)
+            meta.merge_rdf(reader.to_a)
+            meta.comments << "WARN: Parsed JSON-LD without context expansion. Results may be incomplete.\n"
+          rescue => fallback_error
+            meta.comments << "CRITICAL: JSON-LD parsing failed even without context expansion: #{fallback_error.message}\n"
+            return meta
+          end
+        rescue StandardError => e
           meta.comments << "WARN: Though linked data was found, it failed to parse.  This likely indicates some syntax error in the data.  As a result, no metadata will be extracted from this message.\n"
+          warn "JSON-LD StandardError: #{e.message}"
           return meta
         end
-        #       reader.rewind!
-        # for some reason, the rewind method isn't working here...??
-        reader = formattype.reader.new(body) # have to re-read it here, but now its safe because we have already caught errors
-        warn "WRITING TO CACHE"
-        FAIRChampionHarvester::Cache.writeRDFCache(reader, body) # write to the special RDF graph cache
-        warn "WRITING DONE"
-        reader = formattype.reader.new(body)
-        warn "RE-READING DONE"
-        meta.merge_rdf(reader.to_a)
-        warn "MERGE DONE"
-      rescue RDF::ReaderError => e
-        meta.comments << "CRITICAL: The Linked Data was malformed and caused the parser to crash with error message: #{e.message} ||  (sample of what was parsed:  #{body[0..300].delete("\n")})\n"
-        warn "CRITICAL: The Linked Data was malformed and caused the parser to crash with error message: #{e.message} ||  (sample of what was parsed:  #{body[0..300].delete("\n")})\n"
-        nil
-      rescue Exception => e
-        meta.comments << "CRITICAL: An unknown error occurred while parsing the (apparent) Linked Data (sample of what was parsed:  #{body[0..300].delete("\n")}).  Moving on...\n"
-        warn "\n\nCRITICAL: #{e.inspect} An unknown error occurred while parsing the (apparent) Linked Data (full body:  #{body}).  Moving on...\n\n"
-        nil
+      else
+        reader = ""
+        begin
+          reader = formattype.reader.new(body)
+        rescue StandardError
+          meta.comments << "WARN: Though linked data was found, it failed to parse.  This likely indicates some syntax error in the data.  As a result, no metadata will be extracted from this message.\n"
+          return meta
+        end
+
+        begin
+          # $stderr.puts "Reader Class #{reader.class}\n\n #{reader.inspect}"
+          if reader.size == 0
+            meta.comments << "WARN: Though linked data was found, it failed to parse.  This likely indicates some syntax error in the data.  As a result, no metadata will be extracted from this message.\n"
+            return meta
+          end
+          #       reader.rewind!
+          # for some reason, the rewind method isn't working here...??
+          reader = formattype.reader.new(body) # have to re-read it here, but now its safe because we have already caught errors
+          warn "WRITING TO CACHE"
+          FAIRChampionHarvester::Cache.writeRDFCache(reader, body) # write to the special RDF graph cache
+          warn "WRITING DONE"
+          reader = formattype.reader.new(body)
+          warn "RE-READING DONE"
+          meta.merge_rdf(reader.to_a)
+          warn "MERGE DONE"
+        rescue RDF::ReaderError => e
+          meta.comments << "CRITICAL: The Linked Data was malformed and caused the parser to crash with error message: #{e.message} ||  (sample of what was parsed:  #{body[0..300].delete("\n")})\n"
+          warn "CRITICAL: The Linked Data was malformed and caused the parser to crash with error message: #{e.message} ||  (sample of what was parsed:  #{body[0..300].delete("\n")})\n"
+          nil
+        rescue Exception => e
+          meta.comments << "CRITICAL: An unknown error occurred while parsing the (apparent) Linked Data (sample of what was parsed:  #{body[0..300].delete("\n")}).  Moving on...\n"
+          warn "\n\nCRITICAL: #{e.inspect} An unknown error occurred while parsing the (apparent) Linked Data (full body:  #{body}).  Moving on...\n\n"
+          nil
+        end
       end
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: fair_champion_harvester
 version: !ruby/object:Gem::Version
-  version: 0.1.12
+  version: 0.1.13
 platform: ruby
 authors:
 - markwilkinson
@@ -301,6 +301,7 @@ files:
 - lib/doi.rb
 - lib/extruct.rb
 - lib/fair_champion_harvester.rb
+- lib/fair_champion_harvester/handoff.md
 - lib/fair_champion_harvester/version.rb
 - lib/fairsharing.rb
 - lib/handle.rb