solis 0.114.0 → 0.116.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0c79cee8453b41aeab243fc45e69132a9192f9bc3481b447e68db95895d8e1e2
-  data.tar.gz: 1afd13dbd1e5e95505f5363766c829730e88d56e3e89471b8f770044f4e8b6e8
+  metadata.gz: affbb3891268851909a6a35684c11a36f6e67d617dea1e977015124ca0bd8118
+  data.tar.gz: 920ad82dc52d0bdefe3bd1e08b631e6907d6f13952f5573fc3bb8f392b03521d
 SHA512:
-  metadata.gz: 1e8fc91acdcf2f17f029907895be2ea35b3e0813aca875dc6f46f3787907ec7a02b66c8c46d012eb4e7f65e1920035e9eb57d576751c474ac6078c73963ee3aa
-  data.tar.gz: 1ba7210919daa84eed4be4994515834c0561f5d22f1d03ddb18f78cffb499be1f0c400946f10f5f8792437098f6cd204cb8ad0741bbf93aa9f55f54156d4c022
+  metadata.gz: 9b42751289ba605e41d7e02c85a6a7df99a41a80237ca282a4502d3c4a48198ba8057a35b884a51914b828657a88b1d6c3899d3dd703f10c30a0b9ff779dd364
+  data.tar.gz: 8d627312e95e719ab5ddabbd8bdbd0f964aa0a98ed6372df07f18462ecf69a9bdd4ce1e53f21b29392ad450a5874ebfb6bc5b510bc2adbb5fe634a1edf325cc0

data/README.md

@@ -4,7 +4,7 @@ Solis means 'the sun' in Latin or just Silos spelled backwards.
 A simple idea to use SHACL to describe models and API on top of a data store.
 
 # Status
-Although I'm using it in production I must shamefully acknowledge that it actually needs refactoring. Solis is currently still grown organically driven by my needs and understanding of problem domain(RDF, triples, ...).
+Although I'm using it in production I must shamefully acknowledge that it actually needs refactoring. Solis is currently still grown organically driven by my needs and understanding of the problem domain(RDF, triples, ...).
 When my current project is live I'll start to refactor the code and release it as 1.0.
 
 # From Google Sheets to RDF, Shacl, ... to API
@@ -422,6 +422,88 @@ Solis::ConfigFile.path = './'
 solis = Solis::Graph.new(Solis::Shape::Reader::File.read(Solis::ConfigFile[:solis][:shacl]), Solis::ConfigFile[:solis][:env])
 ```
 
+## Performance Optimizations
+
+Entity create/update operations have been optimized to reduce SPARQL round-trips from ~25+ down to ~5-7 for a typical update with embedded entities.
+
+### What changed
+
+| Optimization | Impact |
+|---|---|
+| **Cached `up?` health check** (30s TTL) | Eliminates ~10 redundant ASK queries per operation |
+| **Removed UUID collision check** | Saves 2 round-trips per new entity (UUID v4 collision is ~1 in 2^122) |
+| **Eliminated redundant re-fetches in `update()`** | Saves 1-3 SELECT queries by returning in-memory data |
+| **`known_entities` cache in graph building** | Avoids re-fetching entities from store during `build_ttl_objekt` recursion |
+| **Batched existence checks** (`batch_exists?`) | Replaces N individual ASK queries with 1 SELECT for embedded entities |
+| **Batched orphan reference checks** (`batch_referenced?`) | Replaces N individual ASK queries with 1 SELECT for orphan detection |
+| **SPARQL client reuse** | `save()` passes its client to `update()` when delegating |
+
+## PATCH vs PUT Updates
+
+The `update()` method supports both PUT (full replace) and PATCH (partial merge) semantics via the `patch:` keyword argument.
+
+```ruby
+# PUT (default) — replaces embedded arrays entirely, orphans removed entities
+entity.update({ 'id' => '1', 'students' => [{ 'id' => 's1' }] })
+
+# PATCH — merges into existing arrays, no orphan deletion
+entity.update({ 'id' => '1', 'students' => [{ 'id' => 's3' }] }, true, true, nil, patch: true)
+```
+
+### PUT mode (default, `patch: false`)
+- Embedded entity arrays are **fully replaced** by the provided data
+- Entities removed from the array are detected as orphans and may be deleted
+- Omitted attributes keep their original values
+
+### PATCH mode (`patch: true`)
+- Embedded entity arrays are **merged**: new IDs are added, existing IDs are updated, unmentioned IDs are kept
+- **No orphan detection or deletion** — removing an entity requires an explicit PUT or destroy
+- Omitted embedded attributes are left completely untouched
+
+## Orphan Protection (Embedded Entity Lifecycle)
+
+When updating an entity with PUT semantics and removing a reference to an embedded entity, Solis decides whether to delete the orphaned entity using this decision chain:
+
+| # | Check | Result |
+|---|-------|--------|
+| 1 | Entity in `embedded_readonly`? | **Never delete** (code tables, lookup values) |
+| 2 | Top-level entity + NOT in `embedded_delete`? | **Unlink only** (safe default) |
+| 3 | Top-level entity + in `embedded_delete`? | **Delete** (opt-in cascade) |
+| 4 | Still referenced by other entities? | **Never delete** (safety net) |
+
+A **top-level entity** is one that has its own `sh:NodeShape` + `sh:targetClass` in the SHACL schema — it is independently addressable with its own CRUD endpoint.
+
+### Configuration
+
+```yaml
+:solis:
+  # Code tables / lookup values: never modified, never deleted
+  :embedded_readonly:
+    - :Codetable
+
+  # Child/join entities that should cascade-delete when orphaned
+  :embedded_delete:
+    - :About
+    - :Contributor
+    - :Comment
+    - :Link
+    - :ImageObject
+    - :Identifier
+    - :Provider
+```
+
+### Example
+
+Given an Item with 5 About entities (join between Item and Tag+Role):
+
+```ruby
+# PUT: reduce from 5 Abouts to 1 — the other 4 are deleted (About is in embedded_delete)
+item.update({ 'id' => '1', 'about' => [{ 'id' => 'a1' }] })
+
+# But if Item references a Tag directly, removing it just unlinks — Tag is top-level, not in embedded_delete
+```
+
+Entities like `Tag`, `Agens`, `Person`, `Organization`, `Owner`, `Book` are protected automatically because they are top-level entities not listed in `embedded_delete`.
 
 TODO:
  - extract sparql layer into its own gem

data/lib/solis/graph.rb

@@ -369,6 +369,10 @@ module Solis
 
       @sparql_client = Solis::Store::Sparql::Client.new(@sparql_endpoint)
       result = @sparql_client.query("with <#{graph_name}> delete {?s ?p ?o} where{?s ?p ?o}")
+
+      # Clear the query cache since all data has been flushed
+      Solis::Query.shared_query_cache.clear if defined?(Solis::Query)
+
       LOGGER.info(result)
       true
     end

data/lib/solis/model.rb

@@ -1,6 +1,7 @@
 require 'securerandom'
 require 'iso8601'
 require 'hashdiff'
+require 'set'
 require_relative 'query'
 
 module Solis
@@ -139,6 +140,9 @@ values ?s {<#{self.graph_id}>}
         end
       end
 
+      # Invalidate cached queries for this entity type
+      Solis::Query.invalidate_cache_for(self.class.name)
+
       result
     end
 
@@ -150,44 +154,71 @@ values ?s {<#{self.graph_id}>}
 
       if self.exists?(sparql)
         data = properties_to_hash(self)
-        result = update(data)
+        result = update(data, validate_dependencies, top_level, sparql)
       else
         data = properties_to_hash(self)
         attributes = data.include?('attributes') ? data['attributes'] : data
         readonly_list = (Solis::Options.instance.get[:embedded_readonly] || []).map(&:to_s)
 
+        # Collect all embedded entities first for batched existence check
+        all_embedded = []
         attributes.each_pair do |key, value|
           unless self.class.metadata[:attributes][key][:node].nil?
             value = [value] unless value.is_a?(Array)
             value.each do |sub_value|
               embedded = self.class.graph.shape_as_model(self.class.metadata[:attributes][key][:datatype].to_s).new(sub_value)
+              all_embedded << embedded
+            end
+          end
+        end
 
-              if readonly_entity?(embedded, readonly_list)
-                # Readonly entities (code tables) should never be modified
-                # Only verify they exist, do not create or update them
-                unless embedded.exists?(sparql)
-                  Solis::LOGGER.warn("#{embedded.class.name} (id: #{embedded.id}) is readonly but does not exist in database. Skipping.")
-                end
-              else
-                # Non-readonly entities can be created or updated
-                if embedded.exists?(sparql)
-                  embedded_data = properties_to_hash(embedded)
-                  embedded.update(embedded_data, validate_dependencies, false)
-                else
-                  embedded.save(validate_dependencies, false)
-                end
-              end
+        # Batch check existence of all embedded entities in one query
+        existing_ids = self.class.batch_exists?(sparql, all_embedded)
+
+        # Separate embedded entities into creates and updates
+        to_create_embedded = []
+        to_update_embedded = []
+
+        all_embedded.each do |embedded|
+          entity_exists = existing_ids.include?(embedded.graph_id)
+
+          if readonly_entity?(embedded, readonly_list)
+            unless entity_exists
+              Solis::LOGGER.warn("#{embedded.class.name} (id: #{embedded.id}) is readonly but does not exist in database. Skipping.")
             end
+          elsif entity_exists
+            to_update_embedded << embedded
+          else
+            to_create_embedded << embedded
           end
         end
 
-        graph = as_graph(self, validate_dependencies)
+        # Batch insert all new embedded entities in one SPARQL INSERT
+        unless to_create_embedded.empty?
+          embedded_graph = RDF::Graph.new
+          embedded_graph.name = RDF::URI(self.class.graph_name)
+          known = {}
+          to_create_embedded.each { |e| collect_known_entities(e, known) }
+          to_create_embedded.each { |e| build_ttl_objekt(embedded_graph, e, [], validate_dependencies, known, skip_store_fetch: true) }
+          sparql.insert_data(embedded_graph, graph: embedded_graph.name)
+        end
+
+        # Updates still processed individually (each needs its own DELETE/INSERT)
+        to_update_embedded.each do |embedded|
+          embedded_data = properties_to_hash(embedded)
+          embedded.update(embedded_data, validate_dependencies, false)
+        end
+
+        graph = as_graph(self, validate_dependencies, {}, skip_store_fetch: true)
 
         Solis::LOGGER.info SPARQL::Client::Update::InsertData.new(graph, graph: graph.name).to_s if ConfigFile[:debug]
 
         result = sparql.insert_data(graph, graph: graph.name)
       end
 
+      # Invalidate cached queries for this entity type
+      Solis::Query.invalidate_cache_for(self.class.name)
+
       after_create_proc&.call(self)
       self
     rescue StandardError => e
@@ -195,79 +226,132 @@ values ?s {<#{self.graph_id}>}
       raise e
     end
 
-    def update(data, validate_dependencies = true, top_level = true)
+    # Update an entity.
+    #
+    # @param data [Hash] the attributes to update. Must include 'id'.
+    # @param validate_dependencies [Boolean] whether to validate dependencies (default: true)
+    # @param top_level [Boolean] whether this is a top-level call (default: true)
+    # @param sparql_client [SPARQL::Client, nil] optional reusable SPARQL client
+    # @param patch [Boolean] when true, uses PATCH semantics:
+    #   - Only provided attributes are changed; omitted attributes are untouched
+    #   - Embedded entity arrays are merged (new IDs added, existing IDs updated,
+    #     missing IDs are kept as-is — not orphaned)
+    #   - No orphan detection or deletion
+    #   When false (default), uses PUT semantics:
+    #   - Embedded entity arrays are fully replaced
+    #   - Entities removed from the array are orphaned and deleted if unreferenced
+    # @param prefetched_original [Solis::Model, nil] optional pre-fetched original entity to avoid re-querying
+    def update(data, validate_dependencies = true, top_level = true, sparql_client = nil, patch: false, prefetched_original: nil)
       raise Solis::Error::GeneralError, "I need a SPARQL endpoint" if self.class.sparql_endpoint.nil?
 
       attributes = data.include?('attributes') ? data['attributes'] : data
       raise "id is mandatory when updating" unless attributes.keys.include?('id')
 
       id = attributes.delete('id')
-      sparql = SPARQL::Client.new(self.class.sparql_endpoint)
+      sparql = sparql_client || SPARQL::Client.new(self.class.sparql_endpoint)
 
-      original_klass = self.query.filter({ language: self.class.language, filters: { id: [id] } }).find_all.map { |m| m }&.first
+      original_klass = prefetched_original || self.query.filter({ language: self.class.language, filters: { id: [id] } }).find_all.map { |m| m }&.first
       raise Solis::Error::NotFoundError if original_klass.nil?
       updated_klass = original_klass.deep_dup
 
       # Cache readonly entities list once
       readonly_list = (Solis::Options.instance.get[:embedded_readonly] || []).map(&:to_s)
 
-      # Track entities to potentially delete
+      # Track entities to potentially delete (only used in PUT mode)
       entities_to_check_for_deletion = {}
 
+      # First pass: collect all embedded entities for batched existence check
+      embedded_by_key = {}
       attributes.each_pair do |key, value|
         unless original_klass.class.metadata[:attributes][key][:node].nil?
           value = [value] unless value.is_a?(Array)
+          embedded_by_key[key] = value.map do |sub_value|
+            self.class.graph.shape_as_model(original_klass.class.metadata[:attributes][key][:datatype].to_s).new(sub_value)
+          end
+        end
+      end
 
-          # Get original embedded entities for this attribute
-          original_embedded = original_klass.instance_variable_get("@#{key}")
-          original_embedded = [original_embedded] unless original_embedded.nil? || original_embedded.is_a?(Array)
-          original_embedded ||= []
+      all_embedded = embedded_by_key.values.flatten
+      existing_ids = self.class.batch_exists?(sparql, all_embedded)
 
-          # Track original IDs
-          original_ids = original_embedded.map { |e| solis_model?(e) ? e.id : nil }.compact
+      # Build lookup of original embedded entities by ID for pre-fetched updates
+      original_embedded_lookup = {}
+      embedded_by_key.each_key do |key|
+        orig = original_klass.instance_variable_get("@#{key}")
+        next if orig.nil?
+        Array(orig).each do |e|
+          original_embedded_lookup[e.id] = e if solis_model?(e) && e.id
+        end
+      end
 
-          # Build new array of embedded entities
-          new_embedded_values = []
-          new_ids = []
+      # Second pass: process embedded entities using batched results
+      embedded_by_key.each do |key, embedded_list|
+        value = attributes[key]
+        value = [value] unless value.is_a?(Array)
 
-          value.each do |sub_value|
-            embedded = self.class.graph.shape_as_model(original_klass.class.metadata[:attributes][key][:datatype].to_s).new(sub_value)
-            new_ids << embedded.id if embedded.id
+        # Get original embedded entities for this attribute
+        original_embedded = original_klass.instance_variable_get("@#{key}")
+        original_embedded = [original_embedded] unless original_embedded.nil? || original_embedded.is_a?(Array)
+        original_embedded ||= []
 
-            if readonly_entity?(embedded, readonly_list)
-              # Readonly entities (code tables) should never be modified
-              # Only verify they exist, do not create or update them
-              if embedded.exists?(sparql)
-                new_embedded_values << embedded
-              else
-                Solis::LOGGER.warn("#{embedded.class.name} (id: #{embedded.id}) is readonly but does not exist in database. Skipping.")
-              end
+        # Track original IDs
+        original_ids = original_embedded.map { |e| solis_model?(e) ? e.id : nil }.compact
+
+        # Build new array of embedded entities
+        new_embedded_values = []
+        new_ids = []
+
+        embedded_list.each do |embedded|
+          new_ids << embedded.id if embedded.id
+          entity_exists = existing_ids.include?(embedded.graph_id)
+
+          if readonly_entity?(embedded, readonly_list)
+            if entity_exists
+              new_embedded_values << embedded
             else
-              # Non-readonly entities can be created or updated
-              if embedded.exists?(sparql)
-                embedded_data = properties_to_hash(embedded)
-                embedded.update(embedded_data, validate_dependencies, false)
-                new_embedded_values << embedded
-              else
-                embedded_value = embedded.save(validate_dependencies, false)
-                new_embedded_values << embedded_value
-              end
+              Solis::LOGGER.warn("#{embedded.class.name} (id: #{embedded.id}) is readonly but does not exist in database. Skipping.")
+            end
+          else
+            if entity_exists
+              embedded_data = properties_to_hash(embedded)
+              # Pass pre-fetched original to avoid N+1 query in embedded update
+              prefetched = original_embedded_lookup[embedded.id]
+              embedded.update(embedded_data, validate_dependencies, false, nil, prefetched_original: prefetched)
+              new_embedded_values << embedded
+            else
+              embedded_value = embedded.save(validate_dependencies, false)
+              new_embedded_values << embedded_value
             end
           end
+        end
+
+        if patch
+          # PATCH mode: merge new embedded entities into the original array.
+          # Keep original entities that were not mentioned in the update data.
+          unmentioned = original_embedded.select do |e|
+            solis_model?(e) && !new_ids.include?(e.id)
+          end
+          merged_values = unmentioned + new_embedded_values
+        else
+          # PUT mode: replace the entire array; detect orphans for deletion
+          merged_values = new_embedded_values
 
-          # Identify orphaned entities (in original but not in new)
-          # Note: Readonly entities will be filtered out in delete_orphaned_entities
           orphaned_ids = original_ids - new_ids
           unless orphaned_ids.empty?
             orphaned_entities = original_embedded.select { |e| solis_model?(e) && orphaned_ids.include?(e.id) }
             entities_to_check_for_deletion[key] = orphaned_entities
           end
-
-          # Replace entire array with new values
-          maxcount = original_klass.class.metadata[:attributes][key][:maxcount]
-          value = maxcount && maxcount == 1 ? new_embedded_values.first : new_embedded_values
         end
 
+        maxcount = original_klass.class.metadata[:attributes][key][:maxcount]
+        embedded_value = maxcount && maxcount == 1 ? merged_values.first : merged_values
+        updated_klass.instance_variable_set("@#{key}", embedded_value)
+      end
+
+      # Process non-embedded attributes
+      attributes.each_pair do |key, value|
+        next unless original_klass.class.metadata[:attributes][key][:node].nil?
+
         updated_klass.instance_variable_set("@#{key}", value)
       end
 
@@ -278,9 +362,12 @@ values ?s {<#{self.graph_id}>}
 
       if Hashdiff.best_diff(properties_original_klass, properties_updated_klass).empty?
         Solis::LOGGER.info("#{original_klass.class.name} unchanged, skipping")
-        data = self.query.filter({ filters: { id: [id] } }).find_all.map { |m| m }&.first
+        data = original_klass
       else
-        delete_graph = as_graph(original_klass, false)
+        # Pre-populate known entities separately to avoid cross-contamination
+        # between delete and insert graphs (same ID, different attribute values)
+        delete_known = collect_known_entities(original_klass)
+        delete_graph = as_graph(original_klass, false, delete_known)
         where_graph = RDF::Graph.new(graph_name: RDF::URI("#{self.class.graph_name}#{tableized_class_name(self)}/#{id}"), data: RDF::Repository.new)
 
         if id.is_a?(Array)
@@ -291,23 +378,31 @@ values ?s {<#{self.graph_id}>}
           where_graph << [RDF::URI("#{self.class.graph_name}#{tableized_class_name(self)}/#{id}"), :p, :o]
         end
 
-        insert_graph = as_graph(updated_klass, true)
+        insert_known = collect_known_entities(updated_klass)
+        insert_graph = as_graph(updated_klass, true, insert_known)
 
         delete_insert_query = SPARQL::Client::Update::DeleteInsert.new(delete_graph, insert_graph, where_graph, graph: insert_graph.name).to_s
         delete_insert_query.gsub!('_:p', '?p')
 
-        data = sparql.query(delete_insert_query)
+        sparql.query(delete_insert_query)
+
+        # Invalidate cache before verification to avoid stale reads
+        Solis::Query.invalidate_cache_for(self.class.name)
 
+        # Verify the update succeeded by re-fetching; fallback to insert if needed
         data = self.query.filter({ filters: { id: [id] } }).find_all.map { |m| m }&.first
         if data.nil?
           sparql.insert_data(insert_graph, graph: insert_graph.name)
-          data = self.query.filter({ filters: { id: [id] } }).find_all.map { |m| m }&.first
+          data = updated_klass
         end
 
-        # Delete orphaned entities after successful update
-        delete_orphaned_entities(entities_to_check_for_deletion, sparql)
+        # Delete orphaned entities after successful update (PUT mode only)
+        delete_orphaned_entities(entities_to_check_for_deletion, sparql) unless patch
       end
 
+      # Invalidate cached queries for this entity type
+      Solis::Query.invalidate_cache_for(self.class.name)
+
       after_update_proc&.call(updated_klass, data)
 
       data
@@ -333,18 +428,83 @@ values ?s {<#{self.graph_id}>}
       sparql.query("ASK WHERE { <#{self.graph_id}> ?p ?o }")
     end
 
+    # Save multiple entities in a single SPARQL INSERT operation.
+    # Entities that already exist are updated individually.
+    # @param entities [Array<Solis::Model>] entities to save
+    # @param validate_dependencies [Boolean] whether to validate dependencies
+    # @param batch_size [Integer] max entities per INSERT (default 100)
+    # @return [Array<Solis::Model>] the saved entities
+    def self.batch_save(entities, validate_dependencies: true, batch_size: 100)
+      raise "I need a SPARQL endpoint" if sparql_endpoint.nil?
+      return [] if entities.empty?
+
+      sparql = SPARQL::Client.new(sparql_endpoint)
+
+      # Batch check existence of all entities
+      existing_ids = batch_exists?(sparql, entities)
+
+      to_create = []
+      to_update = []
+
+      entities.each do |entity|
+        if existing_ids.include?(entity.graph_id)
+          to_update << entity
+        else
+          to_create << entity
+        end
+      end
+
+      # Batch insert: combine new entity graphs into single INSERT DATA operations
+      unless to_create.empty?
+        to_create.each_slice(batch_size) do |batch|
+          combined_graph = RDF::Graph.new
+          combined_graph.name = RDF::URI(graph_name)
+
+          # Pre-collect known entities from all entities being created
+          known = {}
+          batch.each { |e| e.send(:collect_known_entities, e, known) }
+
+          batch.each do |entity|
+            entity.before_create_proc&.call(entity)
+            entity.send(:build_ttl_objekt, combined_graph, entity, [], validate_dependencies, known, skip_store_fetch: true)
+          end
+
+          sparql.insert_data(combined_graph, graph: combined_graph.name)
+
+          batch.each { |entity| entity.after_create_proc&.call(entity) }
+        end
+
+        # Invalidate cache once for the entity type
+        Solis::Query.invalidate_cache_for(name)
+      end
+
+      # Updates still processed individually (DELETE/INSERT requires per-entity WHERE)
+      to_update.each do |entity|
+        data = entity.send(:properties_to_hash, entity)
+        entity.update(data, validate_dependencies, true, sparql)
+      end
+
+      entities
+    end
+
+    # Check existence of multiple entities in a single SPARQL query
+    # Returns a Set of graph_ids that exist
+    def self.batch_exists?(sparql, entities)
+      return Set.new if entities.empty?
+      return Set.new([entities.first.graph_id]) if entities.size == 1 && entities.first.exists?(sparql)
+      return Set.new if entities.size == 1
+
+      values = entities.map { |e| "<#{e.graph_id}>" }.join(' ')
+      query = "SELECT DISTINCT ?s WHERE { VALUES ?s { #{values} } . ?s ?p ?o }"
+      results = sparql.query(query)
+      Set.new(results.map { |r| r[:s].to_s })
+    end
+
     def self.make_id_for(model)
-      raise "I need a SPARQL endpoint" if self.sparql_endpoint.nil?
-      sparql = Solis::Store::Sparql::Client.new(self.sparql_endpoint)
       id = model.instance_variable_get("@id")
       if id.nil? || (id.is_a?(String) && id&.empty?)
-        id_retries = 0
-
-        while id.nil? || sparql.query("ASK WHERE { ?s <#{self.graph_name}id>  \"#{id}\" }")
-          id = SecureRandom.uuid
-          id_retries += 1
-        end
-        LOGGER.info("ID(#{id}) generated for #{self.name} in #{id_retries} retries") if ConfigFile[:debug]
+        id = SecureRandom.uuid
+        LOGGER.info("ID(#{id}) generated for #{self.name}") if ConfigFile[:debug]
         model.instance_variable_set("@id", id)
       elsif id.to_s =~ /^https?:\/\//
         id = id.to_s.split('/').last
@@ -414,20 +574,22 @@ values ?s {<#{self.graph_id}>}
     end
 
     def self.model(level = 0)
-      m = { type: self.name.tableize, attributes: {} }
+      m = { type: self.name.tableize, attributes: [] }
       self.metadata[:attributes].each do |attribute, attribute_metadata|
-        is_array = ((attribute_metadata[:maxcount].nil? || (attribute_metadata[:maxcount].to_i > 1)) && !attribute_metadata[:datatype].eql?(:lang_string))
-        attribute_name = is_array  ? "#{attribute}[]" : attribute
-        attribute_name = attribute_metadata[:mincount].to_i > 0 ? "#{attribute_name}*" : attribute_name
         if attribute_metadata.key?(:class) && !attribute_metadata[:class].nil? && attribute_metadata[:class].value =~ /#{self.graph_name}/ && level == 0
           cm = self.graph.shape_as_model(self.metadata[:attributes][attribute][:datatype].to_s).model(level + 1)
-          m[:attributes][attribute_name.to_sym] = cm[:attributes]
-        else
-          m[:attributes][attribute_name.to_sym] = { description: attribute_metadata[:comment]&.value,
-                                               mandatory: (attribute_metadata[:mincount].to_i > 0),
-                                               data_type: attribute_metadata[:datatype] }
-          m[:attributes][attribute_name.to_sym][:order] = attribute_metadata[:order]&.value.to_i if attribute_metadata.key?(:order) && !attribute_metadata[:order].nil?
         end
+
+        attribute_data = { name: attribute,
+                           data_type: attribute_metadata[:datatype],
+                           mandatory: (attribute_metadata[:mincount].to_i > 0),
+                           description: attribute_metadata[:comment]&.value
+        }
+        attribute_data[:order] = attribute_metadata[:order]&.value.to_i if attribute_metadata.key?(:order) && !attribute_metadata[:order].nil?
+        attribute_data[:category] = attribute_metadata[:category]&.value if attribute_metadata.key?(:category) && !attribute_metadata[:category].nil?
+        attribute_data[:attributes] = cm[:attributes] if cm && cm[:attributes]
+
+        m[:attributes] << attribute_data
       end
 
       m
@@ -486,6 +648,21 @@ values ?s {<#{self.graph_id}>}
 
     private
 
+    # Walk the entity tree and collect all in-memory entities by UUID.
+    # Prevents redundant store fetches during recursive graph building.
+    def collect_known_entities(entity, collected = {})
+      uuid = entity.instance_variable_get("@id")
+      return collected if uuid.nil? || collected.key?(uuid)
+      collected[uuid] = entity
+      entity.class.metadata[:attributes].each do |attr, meta|
+        next if meta[:node_kind].nil?
+        val = entity.instance_variable_get("@#{attr}")
+        next if val.nil?
+        Array(val).each { |v| collect_known_entities(v, collected) if solis_model?(v) }
+      end
+      collected
+    end
+
     # Helper method to check if an object is a Solis model
     def solis_model?(obj)
       obj.class.ancestors.include?(Solis::Model)
@@ -497,6 +674,13 @@ values ?s {<#{self.graph_id}>}
       (entity.class.ancestors.map(&:to_s) & readonly_list).any?
     end
 
+    # Helper method to check if an entity is a top-level entity (has its own shape definition).
+    # Top-level entities are independently addressable and should not be cascade-deleted
+    # when unlinked from a parent, unless explicitly opted in via embedded_delete config.
+    def top_level_entity?(entity)
+      self.class.graph.shape?(entity.class.name)
+    end
+
     # Helper method to get tableized class name
     def tableized_class_name(obj)
       obj.class.name.tableize
@@ -514,50 +698,85 @@ values ?s {<#{self.graph_id}>}
       RDF::URI("#{self.class.graph_name}#{class_name.tableize}/#{id}")
     end
 
-    # Delete orphaned entities that are no longer referenced
+    # Delete orphaned entities that are no longer referenced.
+    #
+    # Decision logic (in order of precedence):
+    # 1. embedded_readonly → never delete (code tables)
+    # 2. Top-level entity (has own shape) + NOT in embedded_delete → unlink only, don't delete
+    # 3. Top-level entity + listed in embedded_delete → delete (opt-in override)
+    # 4. Still referenced by other entities (batch_referenced?) → never delete (safety net)
     def delete_orphaned_entities(entities_to_check, sparql)
       return if entities_to_check.nil? || entities_to_check.empty?
 
       readonly_list = (Solis::Options.instance.get[:embedded_readonly] || []).map(&:to_s)
+      delete_list = (Solis::Options.instance.get[:embedded_delete] || []).map(&:to_s)
 
-      entities_to_check.each do |key, orphaned_entities|
+      # Collect all deletable orphans
+      deletable_orphans = []
+      entities_to_check.each do |_key, orphaned_entities|
         next if orphaned_entities.nil?
-
         orphaned_entities.each do |orphaned_entity|
           next unless solis_model?(orphaned_entity)
 
-          # Skip if it's a readonly entity (like code tables)
+          # 1. Never delete readonly entities (code tables)
           if readonly_entity?(orphaned_entity, readonly_list)
             Solis::LOGGER.info("#{orphaned_entity.class.name} (id: #{orphaned_entity.id}) is in embedded_readonly list. Skipping deletion.")
             next
           end
 
-          # Check if the entity is still referenced elsewhere
-          if orphaned_entity.is_referenced?(sparql)
-            Solis::LOGGER.info("#{orphaned_entity.class.name} (id: #{orphaned_entity.id}) is still referenced elsewhere. Skipping deletion.")
-            next
+          # 2. Top-level entities are never auto-deleted unless opted in via embedded_delete
+          if top_level_entity?(orphaned_entity)
+            explicitly_deletable = (orphaned_entity.class.ancestors.map(&:to_s) & delete_list).any?
+            unless explicitly_deletable
+              Solis::LOGGER.info("#{orphaned_entity.class.name} (id: #{orphaned_entity.id}) is a top-level entity. Skipping deletion (unlink only).")
+              next
+            end
           end
 
-          # Safe to delete the orphan
-          begin
-            Solis::LOGGER.info("Deleting orphaned entity: #{orphaned_entity.class.name} (id: #{orphaned_entity.id})")
-            orphaned_entity.destroy
-          rescue StandardError => e
-            Solis::LOGGER.error("Failed to delete orphaned entity #{orphaned_entity.class.name} (id: #{orphaned_entity.id}): #{e.message}")
-          end
+          deletable_orphans << orphaned_entity
+        end
+      end
+
+      return if deletable_orphans.empty?
+
+      # 4. Safety net: batch check which orphans are still referenced elsewhere
+      referenced_ids = batch_referenced?(sparql, deletable_orphans)
+
+      deletable_orphans.each do |orphaned_entity|
+        if referenced_ids.include?(orphaned_entity.graph_id)
+          Solis::LOGGER.info("#{orphaned_entity.class.name} (id: #{orphaned_entity.id}) is still referenced elsewhere. Skipping deletion.")
+          next
+        end
+
+        begin
+          Solis::LOGGER.info("Deleting orphaned entity: #{orphaned_entity.class.name} (id: #{orphaned_entity.id})")
+          orphaned_entity.destroy
+        rescue StandardError => e
+          Solis::LOGGER.error("Failed to delete orphaned entity #{orphaned_entity.class.name} (id: #{orphaned_entity.id}): #{e.message}")
         end
       end
     end
 
-    def as_graph(klass = self, resolve_all = true)
+    # Batch check which entities are still referenced by other entities
+    # Returns a Set of graph_ids that are referenced
+    def batch_referenced?(sparql, entities)
+      return Set.new if entities.empty?
+
+      values = entities.map { |e| "<#{e.graph_id}>" }.join(' ')
+      query = "SELECT DISTINCT ?o WHERE { VALUES ?o { #{values} } . ?s ?p ?o . FILTER(!CONTAINS(STR(?s), 'audit') && !CONTAINS(STR(?p), 'audit')) }"
+      results = sparql.query(query)
+      Set.new(results.map { |r| r[:o].to_s })
+    end
+
+    def as_graph(klass = self, resolve_all = true, known_entities = {}, skip_store_fetch: false)
       graph = RDF::Graph.new
       graph.name = RDF::URI(self.class.graph_name)
-      id = build_ttl_objekt(graph, klass, [], resolve_all)
+      id = build_ttl_objekt(graph, klass, [], resolve_all, known_entities, skip_store_fetch: skip_store_fetch)
 
       graph
     end
 
-    def build_ttl_objekt(graph, klass, hierarchy = [], resolve_all = true)
+    def build_ttl_objekt(graph, klass, hierarchy = [], resolve_all = true, known_entities = {}, skip_store_fetch: false)
       hierarchy.push("#{klass.class.name}(#{klass.instance_variables.include?(:@id) ? klass.instance_variable_get("@id") : ''})")
 
       graph_name = self.class.graph_name
@@ -568,8 +787,11 @@ values ?s {<#{self.graph_id}>}
 
       graph << [id, RDF::RDFV.type, klass_metadata[:target_class]]
 
-      # load existing object and overwrite
-      original_klass = klass.query.filter({ filters: { id: [uuid] } }).find_all { |f| f.id == uuid }.first || nil
+      # Use cached entity if available, otherwise query the store (unless skip_store_fetch)
+      original_klass = known_entities[uuid]
+      if original_klass.nil? && !skip_store_fetch
+        original_klass = klass.query.filter({ filters: { id: [uuid] } }).find_all { |f| f.id == uuid }.first || nil
+      end
 
       if original_klass.nil?
         original_klass = klass
@@ -583,8 +805,11 @@ values ?s {<#{self.graph_id}>}
         end
       end
 
+      # Cache entity for potential reuse in recursive calls
+      known_entities[uuid] = original_klass
+
       begin
-        make_graph(graph, hierarchy, id, original_klass, klass_metadata, resolve_all)
+        make_graph(graph, hierarchy, id, original_klass, klass_metadata, resolve_all, known_entities, skip_store_fetch: skip_store_fetch)
       rescue => e
         Solis::LOGGER.error(e.message)
         raise e
@@ -594,14 +819,19 @@ values ?s {<#{self.graph_id}>}
       id
     end
 
-    def make_graph(graph, hierarchy, id, klass, klass_metadata, resolve_all)
+    def make_graph(graph, hierarchy, id, klass, klass_metadata, resolve_all, known_entities = {}, skip_store_fetch: false)
       klass_metadata[:attributes].each do |attribute, metadata|
         data = klass.instance_variable_get("@#{attribute}")
 
         if data.nil? && metadata.key?(:mincount) && (metadata[:mincount].nil? || metadata[:mincount] > 0) && graph.query(RDF::Query.new({ attribute.to_sym => { RDF.type => metadata[:node] } })).size == 0
           if data.nil?
             uuid = id.value.split('/').last
-            original_klass = klass.query.filter({ filters: { id: [uuid] } }).find_all { |f| f.id == uuid }.first || nil
+            # Use cached entity if available (skip store fetch for new entities)
+            original_klass = known_entities[uuid]
+            if original_klass.nil? && !skip_store_fetch
+              original_klass = klass.query.filter({ filters: { id: [uuid] } }).find_all { |f| f.id == uuid }.first || nil
+              known_entities[uuid] = original_klass if original_klass
+            end
             unless original_klass.nil?
               klass = original_klass
               data = klass.instance_variable_get("@#{attribute}")
@@ -642,10 +872,10 @@ values ?s {<#{self.graph_id}>}
             if self.class.graph.shape_as_model(d.class.name).metadata[:attributes].select { |_, v| v[:node_kind].is_a?(RDF::URI) }.size > 0 &&
               hierarchy.select { |s| s =~ /^#{d.class.name}/ }.size == 0
               internal_resolve = false
-              d = build_ttl_objekt(graph, d, hierarchy, internal_resolve)
+              d = build_ttl_objekt(graph, d, hierarchy, internal_resolve, known_entities)
             elsif self.class.graph.shape_as_model(d.class.name) && hierarchy.select { |s| s =~ /^#{d.class.name}/ }.size == 0
               internal_resolve = false
-              d = build_ttl_objekt(graph, d, hierarchy, internal_resolve)
+              d = build_ttl_objekt(graph, d, hierarchy, internal_resolve, known_entities)
             else
               d = "#{klass.class.graph_name}#{d.class.name.tableize}/#{d.id}"
             end
@@ -714,96 +944,6 @@ values ?s {<#{self.graph_id}>}
       raise e
     end
 
-    def build_ttl_objekt_old(graph, klass, hierarchy = [], resolve_all = true)
-      hierarchy.push("#{klass.class.name}(#{klass.instance_variables.include?(:@id) ? klass.instance_variable_get("@id") : ''})")
-      sparql_endpoint = self.class.sparql_endpoint
-      if klass.instance_variables.include?(:@id) && hierarchy.length > 1
-        unless sparql_endpoint.nil?
-          existing_klass = klass.query.filter({ filters: { id: [klass.instance_variable_get("@id")] } }).find_all { |f| f.id == klass.instance_variable_get("@id") }
-          if !existing_klass.nil? && !existing_klass.empty? && existing_klass.first.is_a?(klass.class)
-            klass = existing_klass.first
-          end
-        end
-      end
-
-      uuid = klass.instance_variable_get("@id") || SecureRandom.uuid
-      id = RDF::URI("#{self.class.graph_name}#{klass.class.name.tableize}/#{uuid}")
-      graph << [id, RDF::RDFV.type, klass.class.metadata[:target_class]]
-
-      klass.class.metadata[:attributes].each do |attribute, metadata|
-        data = klass.instance_variable_get("@#{attribute}")
-        if data.nil? && metadata[:datatype_rdf].eql?('http://www.w3.org/2001/XMLSchema#boolean')
-          data = false
-        end
-
-        if metadata[:datatype_rdf].eql?("http://www.w3.org/1999/02/22-rdf-syntax-ns#JSON")
-          data = data.to_json
-        end
-
-        if data.nil? && metadata[:mincount] > 0
-          raise Solis::Error::InvalidAttributeError, "#{hierarchy.join('.')}.#{attribute} min=#{metadata[:mincount]} and max=#{metadata[:maxcount]}"
-        end
-
-        next if data.nil? || ([Hash, Array, String].include?(data.class) && data&.empty?)
-
-        data = [data] unless data.is_a?(Array)
-        model = nil
-        model = klass.class.graph.shape_as_model(klass.class.metadata[:attributes][attribute][:datatype].to_s) unless klass.class.metadata[:attributes][attribute][:node_kind].nil?
-
-        data.each do |d|
-          original_d = d
-          if model
-            target_node = model.metadata[:target_node].value.split('/').last.gsub(/Shape$/, '')
-            if model.ancestors[0..model.ancestors.find_index(Solis::Model) - 1].map { |m| m.name }.include?(target_node)
-              parent_model = model.graph.shape_as_model(target_node)
-            end
-          end
-
-          if model && d.is_a?(Hash)
-            model_instance = model.descendants.map { |m| m&.new(d) rescue nil }.compact.first || nil
-            model_instance = model.new(d) if model_instance.nil?
-
-            if resolve_all
-              d = build_ttl_objekt(graph, model_instance, hierarchy, false)
-            else
-              real_model = model_instance.query.filter({ filters: { id: model_instance.id } }).find_all { |f| f.id == model_instance.id }&.first
-              d = RDF::URI("#{self.class.graph_name}#{real_model ? real_model.class.name.tableize : model_instance.class.name.tableize}/#{model_instance.id}")
-            end
-          elsif model && d.is_a?(model)
-            if resolve_all
-              if parent_model
-                model_instance = parent_model.new({ id: d.id })
-                d = build_ttl_objekt(graph, model_instance, hierarchy, false)
-              else
-                d = build_ttl_objekt(graph, d, hierarchy, false)
-              end
-            else
-              real_model = model.new.query.filter({ filters: { id: d.id } }).find_all { |f| f.id == d.id }&.first
-              d = RDF::URI("#{self.class.graph_name}#{real_model ? real_model.class.name.tableize : model.name.tableize}/#{d.id}")
-            end
-          else
-            datatype = RDF::Vocabulary.find_term(metadata[:datatype_rdf] || metadata[:node])
-            if datatype && datatype.datatype?
-              d = if metadata[:datatype_rdf].eql?('http://www.w3.org/1999/02/22-rdf-syntax-ns#langString')
-                    RDF::Literal.new(d, language: self.class.language)
-                  else
-                    if metadata[:datatype_rdf].eql?('http://www.w3.org/2001/XMLSchema#anyURI')
-                      RDF::Literal.new(d.to_s, datatype: RDF::XSD.anyURI)
-                    else
-                      RDF::Literal.new(d, datatype: datatype)
-                    end
-                  end
-              d = (d.object.value rescue d.object) unless d.valid?
-            end
-          end
-
-          graph << [id, RDF::URI("#{self.class.graph_name}#{attribute}"), d]
-        end
-      end
-      hierarchy.pop
-      id
-    end
-
     def properties_to_hash(model)
       n = {}
       model.class.metadata[:attributes].each_key do |m|

data/lib/solis/query.rb

@@ -58,6 +58,30 @@ module Solis
       Solis::Options.instance.get.key?(:graphs) ? Solis::Options.instance.get[:graphs].select{|s| s['type'].eql?(:main)}&.first['name'] : ''
     end
 
+    # Shared class-level query cache to ensure consistent reads/writes/invalidations
+    def self.shared_query_cache
+      cache_dir = File.absolute_path(Solis::Options.instance.get[:cache])
+      @shared_query_cache ||= Moneta.new(:HashFile, dir: cache_dir)
+    end
+
+    # Reset the shared cache (useful when config changes, e.g., in tests)
+    def self.reset_shared_query_cache!
+      @shared_query_cache = nil
+    end
+
+    # Invalidate all cached query results for a given model type.
+    def self.invalidate_cache_for(model_class_name, cache_dir = nil)
+      cache = shared_query_cache
+      tag_key = "TAG:#{model_class_name}"
+      if cache.key?(tag_key)
+        cache[tag_key].each { |key| cache.delete(key) }
+        cache.delete(tag_key)
+        Solis::LOGGER.info("CACHE: invalidated entries for #{model_class_name}") if ConfigFile[:debug]
+      end
+    rescue StandardError => e
+      Solis::LOGGER.warn("CACHE: invalidation failed for #{model_class_name}: #{e.message}")
+    end
+
     def initialize(model)
       @construct_cache = File.absolute_path(Solis::Options.instance.get[:cache])
       @model = model
@@ -73,7 +97,7 @@ module Solis
       @sort = 'ORDER BY ?s'
       @sort_select = ''
       @language = Graphiti.context[:object]&.language || Solis::Options.instance.get[:language] || 'en'
-      @query_cache = Moneta.new(:HashFile, dir: @construct_cache)
+      @query_cache = self.class.shared_query_cache
     end
 
     def each(&block)
@@ -135,6 +159,16 @@ module Solis
 
     private
 
+    # Track a cache key under its model type tag for targeted invalidation
+    def track_cache_key(query_key)
+      tag_key = "TAG:#{@model.model_class_name}"
+      existing_keys = @query_cache.key?(tag_key) ? @query_cache[tag_key] : []
+      unless existing_keys.include?(query_key)
+        existing_keys << query_key
+        @query_cache[tag_key] = existing_keys
+      end
+    end
+
     def model_construct?
       construct_name = @model.model_class_name.tableize.singularize rescue @model.class.name.tableize.singularize
       File.exist?("#{ConfigFile.path}/constructs/#{construct_name}.sparql")
@@ -208,6 +242,9 @@ order by ?s
         Solis::LOGGER.info("CACHE: to #{query_key}") if ConfigFile[:debug]
       end
 
+      # Always ensure the cache key is tracked under its model type tag
+      track_cache_key(query_key)
+
       result
     rescue StandardError => e
       Solis::LOGGER.error(e.message)

data/lib/solis/store/sparql/client.rb

@@ -27,12 +27,18 @@ module Solis
         end
 
         def up?
-          result = nil
+          now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+          if @up_checked_at && (now - @up_checked_at) < 30
+            return @up_result
+          end
+          @up_result = nil
           @pool.with do |c|
-            result = c.query("ASK WHERE { ?s ?p ?o }")
+            @up_result = c.query("ASK WHERE { ?s ?p ?o }")
           end
-          result
+          @up_checked_at = now
+          @up_result
         rescue HTTP::Error => e
+          @up_checked_at = nil
           return false
         end
 

data/lib/solis/version.rb

@@ -1,3 +1,3 @@
 module Solis
-  VERSION = "0.114.0"
+  VERSION = "0.116.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: solis
 version: !ruby/object:Gem::Version
-  version: 0.114.0
+  version: 0.116.0
 platform: ruby
 authors:
 - Mehmet Celik