legion-apollo 0.5.6 → 0.5.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7150a2def38a098c16a2f35362210994cc77537ca6ee3e0e615823cf8e1797c2
-  data.tar.gz: 6e0bfa593f70915134195672dbe321f03c00c86e1f80fee06f5370f3c6578110
+  metadata.gz: f096b7552b38d67434c6ae3431526bcd37ebe43c75fa90a19c8ec790e3d323a0
+  data.tar.gz: c8484053dfcda88edee17da060f7a5cab87235c0ea6258d2faecc6542135b1f5
 SHA512:
-  metadata.gz: 97783044f23209d697578c0f56d570bc48b9531e5d569f23e4db98bf55cb5d0c4602d6a0fedeb8e7c04a89618d4586c25bc7ca89f1a8d922e8618c5b724cc13e
-  data.tar.gz: 8498fae869abc9f54520b216ed88e2aea006ee6e520d8421d44cf5781c76ae9c158553e3c34a7e9e1e37d44594248a5c6daae1c7b55336c9692e7b668f00fbb4
+  metadata.gz: 246787437bfdf6c9667563e73bbba1bd56fe3328f2d1836d2a9daf14cc870ad1df25f96ff725f80a14ed9e4174efa4479f0c41eb314538138122d54cb8dbb424
+  data.tar.gz: fcee5919bc41cc594d77ab04188212265c54b5281383c9ff6e003222ea4ce2cf25859b39666cdf16c2b09fbcabbfd998af84c33d6fad4cdca86b683e1c140739

data/CHANGELOG.md

@@ -1,5 +1,40 @@
 # Changelog
 
+## [0.5.13] - 2026-10-29
+
+### Fixed
+- Preserve L0/L1 summaries when normalizing public local and merged retrieval results, exposing the projected text as content instead of nil.
+
+## [0.5.12] - 2026-10-29
+
+### Fixed
+- Honor raw source and temporal-window updates during upsert. Refresh derived raw content when indexed content changes, preserve omitted explicit sources/windows, and allow explicit nil windows to clear validity bounds.
+- Remove old FTS5 terms before changing their external content row, preventing index corruption during upsert while preserving transactional rollback.
+
+## [0.5.11] - 2026-10-29
+
+### Fixed
+- Renew expired content on duplicate ingestion while preserving its ID, source, tags, and deduplicated response; active duplicates retain their original expiration.
+
+## [0.5.10] - 2026-10-29
+
+### Fixed
+- Apply local eligibility filters before filling the candidate budget, streaming full-text, blank, and fallback searches so rejected rows cannot hide later matching knowledge.
+
+## [0.5.9] - 2026-10-29
+
+### Fixed
+- Return logged failure results from local graph operations when the optional database is absent, disconnected, or has no usable connection, instead of leaking lifecycle exceptions.
+
+## [0.5.8] - 2026-10-29
+
+### Fixed
+- Generate local ingest/upsert embeddings outside the write mutex so slow providers do not serialize unrelated writes; retain atomic duplicate checks and persistence.
+
+## [0.5.7] - 2026-07-15
+### Removed
+- Remove .md self-knowledge seed pipeline (bonds must be earned, not implanted)
+
 ## [0.5.5] - 2026-05-15
 
 ### Added

data/README.md

@@ -2,7 +2,7 @@
 
 Apollo is the LegionIO knowledge client. It gives extensions one API for writing, retrieving, and merging knowledge across the global Apollo service and the node-local SQLite store.
 
-**Version**: 0.5.2
+**Version**: 0.5.13
 
 `legion-apollo` provides `query`, `ingest`, and `retrieve` with smart routing: co-located `lex-apollo`, RabbitMQ transport, node-local SQLite, or graceful failure. `Apollo::Local` mirrors the same public API for offline and low-latency retrieval without requiring remote infrastructure.
 
@@ -58,10 +58,18 @@ Features:
 - `raw_content` preservation for verbatim source text
 - `valid_from` / `valid_to` temporal windows with `as_of:` query filtering
 - Optional LLM embeddings (1024-dim) with cosine rerank when `Legion::LLM.can_embed?`
-- TTL expiry (default 5-year retention)
+- Embedding generation runs outside the local write mutex; persistence and duplicate checks remain serialized
+- TTL expiry (default 5-year retention); re-ingesting expired content renews retention without replacing its stored source or ID
 - FTS5 full-text search with `ILIKE` fallback
+- Confidence, tags, visibility, inference, and history filters run before filling the reranking candidate budget
 - Null-byte removal and invalid UTF-8 scrubbing before persistence or backend routing
 
+Local graph operations return `{ success: false, error: ... }` when the optional database is unavailable; they do not require callers to rescue missing-connection exceptions.
+
+`Local.upsert` honors explicit `raw_content`, `valid_from`, and `valid_to` updates. Omitted validity bounds and distinct verbatim sources are preserved; derived raw content follows the updated indexed content. Explicit nil validity bounds clear the corresponding restriction.
+
+Public `query`/`retrieve` calls with `tier: :l0` or `tier: :l1` retain the local `summary` and expose it as `content` in both `scope: :local` and `scope: :all` results.
+
 ## Configuration
 
 ```json

data/lib/legion/apollo/local/graph.rb

@@ -12,6 +12,7 @@ module Legion
       # Graph traversal expands one frontier batch per depth to avoid per-node queries.
       module Graph # rubocop:disable Metrics/ModuleLength
         VALID_RELATION_TYPES = %w[AFFECTS OWNED_BY DEPENDS_ON RELATED_TO SUPERSEDES].freeze
+        class DatabaseUnavailable < StandardError; end
 
         class << self # rubocop:disable Metrics/ClassLength
           include Legion::Logging::Helper
@@ -32,7 +33,7 @@ module Legion
             end
             log.info { "Apollo::Local::Graph created entity id=#{id} type=#{type} name=#{name}" }
             { success: true, id: id }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:       :error,
@@ -49,7 +50,7 @@ module Legion
 
             log.debug { "Apollo::Local::Graph found entity id=#{id}" }
             { success: true, entity: decode_entity(row) }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(e, level: :error, operation: 'apollo.local.graph.find_entity', entity_id: id)
             { success: false, error: e.message }
           end
@@ -58,7 +59,7 @@ module Legion
             rows = db[:local_entities].where(entity_type: type.to_s).limit(limit).all
             log.debug { "Apollo::Local::Graph found entities type=#{type} count=#{rows.size}" }
             { success: true, entities: rows.map { |r| decode_entity(r) }, count: rows.size }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:       :error,
@@ -73,7 +74,7 @@ module Legion
             rows = db[:local_entities].where(name: name.to_s).limit(limit).all
             log.debug { "Apollo::Local::Graph found entities name=#{name} count=#{rows.size}" }
             { success: true, entities: rows.map { |r| decode_entity(r) }, count: rows.size }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:     :error,
@@ -95,7 +96,7 @@ module Legion
 
             log.info { "Apollo::Local::Graph updated entity id=#{id}" }
             { success: true, id: id }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:     :error,
@@ -112,7 +113,7 @@ module Legion
 
             log.info { "Apollo::Local::Graph deleted entity id=#{id}" }
             result
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(e, level: :error, operation: 'apollo.local.graph.delete_entity', entity_id: id)
             { success: false, error: e.message }
           end
@@ -132,6 +133,10 @@ module Legion
                 "target_id=#{target_id} relation_type=#{normalized_relation_type}"
             end
             { success: true, id: id }
+          rescue DatabaseUnavailable => e
+            handle_exception(e, level: :error, operation: 'apollo.local.graph.create_relationship',
+                                source_id: source_id, target_id: target_id)
+            { success: false, error: e.message }
           rescue Sequel::UniqueConstraintViolation => e
             duplicate = handle_duplicate_relationship(source_id, target_id, normalized_relation_type)
             return duplicate if duplicate
@@ -170,7 +175,7 @@ module Legion
                 "relation_type=#{relation_type || 'any'} count=#{rows.size}"
             end
             { success: true, relationships: rows.map { |r| decode_relationship(r) }, count: rows.size }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:         :error,
@@ -188,7 +193,7 @@ module Legion
 
             log.info { "Apollo::Local::Graph deleted relationship id=#{id}" }
             { success: true, id: id }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(e, level: :error, operation: 'apollo.local.graph.delete_relationship', relationship_id: id)
             { success: false, error: e.message }
           end
@@ -217,7 +222,7 @@ module Legion
               edges:   edge_rows.map   { |r| decode_relationship(r) },
               count:   entity_rows.size
             }
-          rescue Sequel::Error => e
+          rescue DatabaseUnavailable, Sequel::Error => e
             handle_exception(
               e,
               level:         :error,
@@ -357,7 +362,14 @@ module Legion
           end
 
           def db
-            Legion::Data::Local.connection
+            unless defined?(Legion::Data::Local) && Legion::Data::Local.connected?
+              raise DatabaseUnavailable, 'local database unavailable'
+            end
+
+            connection = Legion::Data::Local.connection
+            return connection if connection && !(connection.respond_to?(:closed?) && connection.closed?)
+
+            raise DatabaseUnavailable, 'local database unavailable'
           end
 
           def timestamp

data/lib/legion/apollo/local.rb

@@ -17,8 +17,6 @@ module Legion
       MIGRATION_PATH = File.expand_path('local/migrations', __dir__).freeze
       LIFECYCLE_MUTEX = Mutex.new
       WRITE_MUTEX = Mutex.new
-      SEED_MUTEX = Mutex.new
-      HYDRATION_MUTEX = Mutex.new
 
       class << self # rubocop:disable Metrics/ClassLength
         include Legion::Logging::Helper
@@ -33,25 +31,29 @@ module Legion
         def shutdown
           LIFECYCLE_MUTEX.synchronize do
             @started = false
-            @seeded = false
             log.info 'Legion::Apollo::Local shutdown'
           end
         rescue StandardError => e
           handle_exception(e, level: :warn, operation: 'apollo.local.shutdown')
           @started = false
-          @seeded = false
         end
 
         def started?
           @started == true
         end
 
-        def ingest(content:, tags: [], access_scope: 'global', **opts) # rubocop:disable Metrics/MethodLength
+        def ingest(content:, tags: [], access_scope: 'global', **opts) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
           return not_started_error unless started?
 
           tags = normalize_tags_input(tags)
+          content = normalize_text_input(content)
+          hash = content_hash(content)
+          duplicate = WRITE_MUTEX.synchronize { deduplicated_ingest(hash, expires_at: opts[:expires_at]) }
+          return duplicate if duplicate
+
+          embedding_fields = embedding_columns(content, opts)
           WRITE_MUTEX.synchronize do
-            ingest_without_lock(content: content, tags: tags,
+            ingest_without_lock(content: content, tags: tags, embedding_fields: embedding_fields,
                                 **inject_identity_context(opts).merge(access_scope: access_scope))
           end
         rescue StandardError => e
@@ -71,14 +73,26 @@ module Legion
           sorted_tags = normalize_tags_input(tags).sort
           tag_json = Legion::JSON.dump(sorted_tags)
           merged_opts = inject_identity_context(opts).merge(access_scope: access_scope)
+          content = normalize_text_input(content)
+          hash = content_hash(content)
+          duplicate = WRITE_MUTEX.synchronize do
+            if db[:local_knowledge].where(tags: tag_json).empty?
+              deduplicated_ingest(hash,
+                                  expires_at: opts[:expires_at])
+            end
+          end
+          return duplicate if duplicate
+
+          embedding_fields = embedding_columns(content, merged_opts)
           WRITE_MUTEX.synchronize do
             existing = db[:local_knowledge].where(tags: tag_json).first
 
             if existing
-              update_upsert_entry(existing, content, tag_json, merged_opts)
+              update_upsert_entry(existing, content, tag_json, merged_opts, embedding_fields)
             else
-              result = ingest_without_lock(content: content, tags: sorted_tags, **merged_opts)
-              result[:mode] = :inserted if result[:success] && result[:mode] != :deduplicated
+              result = ingest_without_lock(content: content, tags: sorted_tags,
+                                           embedding_fields: embedding_fields, **merged_opts)
+              result[:mode] = :inserted if result[:mode] == :local
               result
             end
           end
@@ -108,13 +122,15 @@ module Legion
           end
           log.debug { "Apollo::Local query limit=#{limit} min_confidence=#{min_confidence} tags=#{Array(tags).size}" }
 
-          candidates = fts_search(text, limit: limit * multiplier, as_of: as_of)
           include_inferences = opts.fetch(:include_inferences, true)
           include_history = opts.fetch(:include_history, false)
-          candidates = filter_candidates(candidates, min_confidence: min_confidence, tags: tags,
-                                                     options: { include_inferences: include_inferences,
-                                                                include_history: include_history, as_of: as_of,
-                                                                requesting_principal_id: requesting_principal_id })
+          candidates = fts_search(text, as_of: as_of) do |dataset|
+            filter_candidates(dataset.lazy, min_confidence: min_confidence, tags: tags,
+                                            options: { include_inferences: include_inferences,
+                                                       include_history: include_history, as_of: as_of,
+                                                       requesting_principal_id: requesting_principal_id })
+              .take(limit * multiplier).force
+          end
           candidates = cosine_rerank(text, candidates) if can_rerank?
           results = candidates.first(limit)
 
@@ -146,22 +162,9 @@ module Legion
         def reset!
           LIFECYCLE_MUTEX.synchronize do
             @started = false
-            @seeded = false
           end
         end
 
-        def seed_self_knowledge
-          return unless started?
-
-          SEED_MUTEX.synchronize { seed_self_knowledge_without_lock }
-        rescue StandardError => e
-          handle_exception(e, level: :warn, operation: 'apollo.local.seed_self_knowledge')
-        end
-
-        def seeded?
-          @seeded == true
-        end
-
         def query_by_tags(tags:, limit: 50) # rubocop:disable Metrics/MethodLength
           connection = local_db_connection
           tags = normalize_tags_input(tags)
@@ -232,15 +235,6 @@ module Legion
           { success: false, error: e.message }
         end
 
-        def hydrate_from_global
-          return { success: false, error: :not_started } unless started?
-
-          HYDRATION_MUTEX.synchronize { hydrate_from_global_without_lock }
-        rescue StandardError => e
-          handle_exception(e, level: :error, operation: 'apollo.local.hydrate_from_global')
-          { success: false, error: e.message }
-        end
-
         def version_chain(entry_id:, max_depth: 50) # rubocop:disable Metrics/MethodLength
           return not_started_error unless started?
 
@@ -278,49 +272,6 @@ module Legion
 
         private
 
-        def self_knowledge_files
-          seed_dir = File.join(File.expand_path('../../..', __dir__), 'data', 'self-knowledge')
-          return [] unless File.directory?(seed_dir)
-
-          Dir[File.join(seed_dir, '*.md')]
-        end
-
-        def seed_files(files)
-          count = 0
-          files.each do |path|
-            count += 1 if seed_single_file(path)
-          end
-          count
-        end
-
-        def seed_single_file(path)
-          content = File.read(path)
-          return false if content.strip.empty?
-
-          tags = ['legionio', 'self-knowledge', File.basename(path, '.md')]
-          result = ingest(content: content, tags: tags, source_channel: 'self-knowledge',
-                          submitted_by: 'legion-apollo', confidence: 0.9)
-          return false unless result[:success] && result[:mode] != :deduplicated
-
-          ingest_global(content: content, tags: tags) if global_available?
-          true
-        end
-
-        def ingest_global(content:, tags:)
-          log.debug { "Apollo::Local forwarding seed entry to global tag_count=#{Array(tags).size}" }
-          Legion::Apollo.ingest(content: content, tags: tags, source_channel: 'self-knowledge',
-                                submitted_by: 'legion-apollo', confidence: 0.9, scope: :global)
-        rescue StandardError => e
-          handle_exception(e, level: :debug, operation: 'apollo.local.ingest_global_seed', tag_count: Array(tags).size)
-        end
-
-        def global_available?
-          defined?(Legion::Apollo) && Legion::Apollo.started? && Legion::Apollo.respond_to?(:ingest)
-        rescue StandardError => e
-          handle_exception(e, level: :debug, operation: 'apollo.local.global_available')
-          false
-        end
-
         def local_enabled?
           return false unless defined?(Legion::Settings)
 
@@ -372,13 +323,6 @@ module Legion
           Digest::MD5.hexdigest(normalized)
         end
 
-        def duplicate?(hash)
-          db[:local_knowledge].where(content_hash: hash).any?
-        rescue StandardError => e
-          handle_exception(e, level: :debug, operation: 'apollo.local.duplicate_check', hash: hash)
-          false
-        end
-
         def start_without_lock
           return if @started
           return unless local_enabled? && data_local_available?
@@ -388,63 +332,12 @@ module Legion
           log.info 'Legion::Apollo::Local started'
         end
 
-        def seed_self_knowledge_without_lock
-          return if @seeded
-
-          files = self_knowledge_files
-          return if files.empty?
-
-          count = seed_files(files)
-          @seeded = true
-          log.info("Apollo::Local seeded #{count} self-knowledge files")
-        end
-
-        def hydrate_from_global_without_lock # rubocop:disable Metrics/MethodLength,Metrics/AbcSize,Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity
-          local_check = query_by_tags(tags: ['partner'])
-          if local_check[:success] && local_check[:results]&.any?
-            log.info 'Apollo::Local hydration skipped because local partner data already exists'
-            return { success: true, skipped: :local_data_exists }
-          end
-
-          unless Legion::Apollo.transport_available? || Legion::Apollo.data_available?
-            log.info 'Apollo::Local hydration skipped because global Apollo is unavailable'
-            return { success: true, skipped: :global_unavailable }
-          end
-
-          global_entries = Legion::Apollo.retrieve(text: 'partner bond', scope: :global, limit: 20)
-          entries = Array(global_entries[:entries] || global_entries[:results])
-          unless global_entries[:success] && entries.any?
-            log.info 'Apollo::Local hydration skipped because no global partner data was found'
-            return { success: true, skipped: :no_global_data }
-          end
-
-          log.info { "Apollo::Local hydration started global_count=#{entries.size}" }
-          hydrated = 0
-          entries.each do |entry|
-            entry_tags = entry[:tags].is_a?(Array) ? entry[:tags] : []
-            clean_tags = entry_tags.reject { |tag| tag == 'promoted_from_local' } + ['hydrated_from_global']
-
-            result = ingest(
-              content:        entry[:content],
-              raw_content:    entry[:raw_content] || entry[:content],
-              tags:           clean_tags,
-              confidence:     ((entry[:confidence] || 0.5) * 0.9).round(10),
-              source_channel: 'global_hydration',
-              valid_from:     entry[:valid_from],
-              valid_to:       entry[:valid_to]
-            )
-            hydrated += 1 if result[:success]
-          end
-
-          log.info { "Apollo::Local hydration completed hydrated=#{hydrated}" }
-          { success: true, hydrated: hydrated }
-        end
-
-        def ingest_without_lock(content:, tags:, **opts) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
+        def ingest_without_lock(content:, tags:, embedding_fields:, **opts) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
           content = normalize_text_input(content)
           raw_content = normalize_raw_content_input(opts[:raw_content], fallback: content)
           hash = content_hash(content)
-          return deduplicated_ingest(hash) if duplicate?(hash)
+          duplicate = deduplicated_ingest(hash, expires_at: opts[:expires_at])
+          return duplicate if duplicate
 
           log.info do
             "Apollo::Local ingest accepted content_length=#{content.to_s.length} " \
@@ -455,15 +348,17 @@ module Legion
           metadata = opts.dup
           metadata.delete(:raw_content)
           row = build_ingest_row(content: content, raw_content: raw_content, hash: hash, tags: tags, **metadata)
+                .merge(embedding_fields)
           id = persist_ingest_row(row, metadata)
           mark_parent_superseded(metadata[:parent_knowledge_id]) if metadata[:parent_knowledge_id]
 
           log.info { "Apollo::Local ingest stored id=#{id} hash=#{hash}" }
           { success: true, mode: :local, id: id }
         rescue Sequel::UniqueConstraintViolation
-          raise unless duplicate?(hash)
+          duplicate = deduplicated_ingest(hash, expires_at: opts[:expires_at])
+          raise unless duplicate
 
-          deduplicated_ingest(hash)
+          duplicate
         end
 
         def build_ingest_row(content:, raw_content:, hash:, tags:, **opts) # rubocop:disable Metrics/MethodLength
@@ -477,7 +372,7 @@ module Legion
             opts:               opts,
             is_inference:       is_inference,
             default_confidence: default_confidence
-          ).merge(embedding_columns(content, opts)).merge(timestamp_columns)
+          ).merge(timestamp_columns)
         end
 
         def ingest_metadata_columns(context)
@@ -545,7 +440,17 @@ module Legion
           handle_exception(e, level: :warn, operation: 'apollo.local.create_source_link', entry_id: entry_id)
         end
 
-        def deduplicated_ingest(hash)
+        def deduplicated_ingest(hash, expires_at: nil) # rubocop:disable Metrics/MethodLength
+          dataset = db[:local_knowledge].where(content_hash: hash)
+          existing = dataset.select(:expires_at).first
+          return unless existing
+
+          now = Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ')
+          previous_expiry = existing[:expires_at]
+          if previous_expiry && previous_expiry <= now
+            dataset.where(Sequel.lit('expires_at <= ?', now))
+                   .update(expires_at: expires_at || compute_expires_at, updated_at: now)
+          end
           log.info { "Apollo::Local ingest deduplicated hash=#{hash}" }
           { success: true, mode: :deduplicated }
         end
@@ -615,33 +520,31 @@ module Legion
           Legion::JSON.dump(normalize_tags_input(tags))
         end
 
-        def fts_search(text, limit:, as_of: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
+        def fts_search(text, as_of: nil) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
           now = Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ')
-          return active_knowledge_dataset(now: now, as_of: as_of).limit(limit).all if text.to_s.strip.empty?
+          return yield active_knowledge_dataset(now: now, as_of: as_of) if text.to_s.strip.empty?
 
           tokens = text.to_s.scan(/[\p{L}\p{N}_]+/)
-          return ilike_search(text, now: now, limit: limit, as_of: as_of) if tokens.empty?
+          return yield ilike_search(text, now: now, as_of: as_of) if tokens.empty?
 
           escaped = tokens.map { |t| %("#{t}") }.join(' ')
           temporal_sql, temporal_params = temporal_window_sql(as_of, table_alias: 'lk')
-          db.fetch(
+          yield db.fetch(
             'SELECT lk.* FROM local_knowledge lk ' \
             'INNER JOIN local_knowledge_fts fts ON lk.id = fts.rowid ' \
             "WHERE local_knowledge_fts MATCH ? AND lk.expires_at > ?#{temporal_sql} " \
-            'ORDER BY fts.rank LIMIT ?',
-            escaped, now, *temporal_params, limit
-          ).all
+            'ORDER BY fts.rank',
+            escaped, now, *temporal_params
+          )
         rescue StandardError => e
-          handle_exception(e, level: :debug, operation: 'apollo.local.fts_search', limit: limit, fallback: :ilike)
-          ilike_search(text, now: Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ'), limit: limit, as_of: as_of)
+          handle_exception(e, level: :debug, operation: 'apollo.local.fts_search', fallback: :ilike)
+          yield ilike_search(text, now: Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ'), as_of: as_of)
         end
 
-        def ilike_search(text, now:, limit:, as_of: nil)
+        def ilike_search(text, now:, as_of: nil)
           safe_text = text.to_s.gsub('\\', '\\\\\\\\').gsub('%', '\%').gsub('_', '\_')
           active_knowledge_dataset(now: now, as_of: as_of)
             .where(Sequel.lit("content LIKE ? ESCAPE '\\' COLLATE NOCASE", "%#{safe_text}%"))
-            .limit(limit)
-            .all
         end
 
         def filter_candidates(candidates, min_confidence:, tags:, options: {}) # rubocop:disable Metrics/CyclomaticComplexity,Metrics/PerceivedComplexity,Metrics/MethodLength,Metrics/AbcSize
@@ -847,20 +750,26 @@ module Legion
           end.first(limit)
         end
 
-        def update_upsert_entry(existing, content, tags_json, opts) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
+        def update_upsert_entry(existing, content, tags_json, opts, embedding_fields) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
           content = normalize_text_input(content)
           new_hash = content_hash(content)
-          embedding, embedded_at = generate_embedding(content)
           now = Time.now.utc.strftime('%Y-%m-%dT%H:%M:%S.%LZ')
           expires_at = compute_expires_at
+          raw_content = opts.fetch(:raw_content) do
+            existing[:raw_content] unless existing[:raw_content] == existing[:content]
+          end
 
           db.transaction do
+            db.run("DELETE FROM local_knowledge_fts WHERE rowid = #{existing[:id]}")
             db[:local_knowledge].where(id: existing[:id]).update(
               content:                 content,
+              raw_content:             normalize_raw_content_input(raw_content, fallback: content),
+              valid_from:              normalize_temporal_value(opts.fetch(:valid_from, existing[:valid_from])),
+              valid_to:                normalize_temporal_value(opts.fetch(:valid_to, existing[:valid_to])),
               content_hash:            new_hash,
               tags:                    tags_json,
-              embedding:               embedding ? Legion::JSON.dump(embedding) : nil,
-              embedded_at:             embedded_at,
+              embedding:               embedding_fields[:embedding],
+              embedded_at:             embedding_fields[:embedded_at],
               confidence:              opts.fetch(:confidence, existing[:confidence]),
               expires_at:              expires_at,
               source_channel:          opts.fetch(:source_channel, existing[:source_channel]),
@@ -872,18 +781,12 @@ module Legion
               identity_id:             opts.fetch(:identity_id, existing[:identity_id]),
               updated_at:              now
             )
-            rebuild_fts_entry!(existing[:id], content, tags_json)
+            sync_fts!(existing[:id], content, tags_json)
           end
           log.info { "Apollo::Local upsert updated id=#{existing[:id]} hash=#{new_hash}" }
           { success: true, mode: :updated, id: existing[:id] }
         end
 
-        def rebuild_fts_entry!(id, content, tags_json)
-          db.run("DELETE FROM local_knowledge_fts WHERE rowid = #{id}")
-          sync_fts!(id, content, tags_json)
-          log.debug { "Apollo::Local FTS rebuilt id=#{id}" }
-        end
-
         def project_tier(entry, tier) # rubocop:disable Metrics/MethodLength
           case tier
           when :l0

data/lib/legion/apollo/version.rb

@@ -2,6 +2,6 @@
 
 module Legion
   module Apollo
-    VERSION = '0.5.6'
+    VERSION = '0.5.13'
   end
 end

data/lib/legion/apollo.rb

@@ -39,9 +39,6 @@ module Legion
 
           @started = true
           log.info 'Legion::Apollo started'
-
-          seed_self_knowledge
-          Legion::Apollo::Local.hydrate_from_global if Legion::Apollo::Local.started?
         end
       rescue StandardError => e
         handle_exception(e, level: :error, operation: 'apollo.start')
@@ -372,25 +369,28 @@ module Legion
         payload.merge(text: normalized_text, query: normalized_text)
       end
 
-      def normalize_local_entries(entries) # rubocop:disable Metrics/MethodLength,Metrics/AbcSize
+      def normalize_local_entries(entries) # rubocop:disable Metrics/AbcSize
         entries.map do |e|
-          hash = e[:content_hash] || Digest::MD5.hexdigest(e[:content].to_s.strip.downcase.gsub(/\s+/, ' '))
-          tags = if e[:tags].is_a?(String)
-                   begin
-                     Legion::JSON.parse(e[:tags])
-                   rescue StandardError => ex
-                     handle_exception(ex, level: :debug, operation: 'apollo.normalize_local_entries', entry_id: e[:id])
-                     []
-                   end
-                 else
-                   Array(e[:tags])
-                 end
-          { id: e[:id], content: e[:content], raw_content: e[:raw_content] || e[:content], content_hash: hash,
-            confidence: e[:confidence] || 0.5, content_type: 'fact', tags: tags, source: :local,
-            valid_from: e[:valid_from], valid_to: e[:valid_to] }
+          content = e[:content] || e[:summary]
+          hash = e[:content_hash] || Digest::MD5.hexdigest(content.to_s.strip.downcase.gsub(/\s+/, ' '))
+          tags = normalize_local_entry_tags(e)
+          entry = { id: e[:id], content: content, raw_content: e[:raw_content] || content, content_hash: hash,
+                    confidence: e[:confidence] || 0.5, content_type: 'fact', tags: tags, source: :local,
+                    valid_from: e[:valid_from], valid_to: e[:valid_to] }
+          entry[:summary] = e[:summary] if e.key?(:summary)
+          entry
         end
       end
 
+      def normalize_local_entry_tags(entry)
+        return Array(entry[:tags]) unless entry[:tags].is_a?(String)
+
+        Legion::JSON.parse(entry[:tags])
+      rescue StandardError => e
+        handle_exception(e, level: :debug, operation: 'apollo.normalize_local_entries', entry_id: entry[:id])
+        []
+      end
+
       def normalize_global_entries(entries)
         entries.map { |entry| normalize_global_entry(entry) }
       end
@@ -469,13 +469,6 @@ module Legion
         { success: false, error: e.message }
       end
 
-      def seed_self_knowledge
-        log.info 'Apollo self-knowledge seed requested'
-        Legion::Apollo::Local.seed_self_knowledge if Legion::Apollo::Local.started?
-      rescue StandardError => e
-        handle_exception(e, level: :warn, operation: 'apollo.seed_self_knowledge')
-      end
-
       def apollo_setting(key, default)
         return default unless defined?(Legion::Settings) && !Legion::Settings[:apollo].nil?
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: legion-apollo
 version: !ruby/object:Gem::Version
-  version: 0.5.6
+  version: 0.5.13
 platform: ruby
 authors:
 - Esity
@@ -66,17 +66,6 @@ files:
 - CHANGELOG.md
 - LICENSE
 - README.md
-- data/self-knowledge/01-what-is-legion.md
-- data/self-knowledge/02-architecture.md
-- data/self-knowledge/03-extensions.md
-- data/self-knowledge/04-security.md
-- data/self-knowledge/05-llm-pipeline.md
-- data/self-knowledge/06-apollo-knowledge.md
-- data/self-knowledge/07-cli-reference.md
-- data/self-knowledge/08-cognitive-layer.md
-- data/self-knowledge/09-teams-integration.md
-- data/self-knowledge/10-deployment.md
-- data/self-knowledge/11-my-partner.md
 - lib/legion/apollo.rb
 - lib/legion/apollo/helpers/confidence.rb
 - lib/legion/apollo/helpers/similarity.rb

data/data/self-knowledge/01-what-is-legion.md

@@ -1,51 +0,0 @@
-# What is LegionIO?
-
-LegionIO is an extensible async job engine and cognitive platform for Ruby. It schedules tasks, creates relationships between services, and runs them concurrently. It was created by Matthew Iverson (@Esity) and is licensed under Apache-2.0.
-
-LegionIO is not a chatbot. It is a framework that can power chatbots, AI assistants, background workers, service integrations, and autonomous agents. The chat interface is one of many ways to interact with it.
-
-## Core Purpose
-
-LegionIO connects isolated systems — cloud accounts, on-premise services, SaaS tools — into a unified async task engine. It uses RabbitMQ for message passing, supports SQLite/PostgreSQL/MySQL for persistence, and Redis/Memcached for caching.
-
-## What LegionIO Does
-
-- Schedules and executes async tasks across distributed services
-- Chains tasks into workflows (Task A -> conditioner -> Task B -> transformer -> Task C)
-- Auto-discovers and loads extension gems (LEX plugins) at boot
-- Provides a unified REST API on port 4567 for all operations
-- Integrates with HashiCorp Vault for secrets and authentication
-- Supports Kerberos auto-authentication to Vault using existing AD credentials
-- Runs a 19-step LLM pipeline with RAG, guardrails, cost tracking, and model routing
-- Maintains a shared knowledge store (Apollo) for organizational knowledge
-- Provides a rich terminal UI with AI chat, dashboard, and extension browser
-- Tracks token usage and costs per-user and per-team
-- Supports HIPAA PHI compliance with redaction, crypto-erasure, and audit trails
-- Runs as a macOS/Linux background service via Homebrew or systemd
-
-## What LegionIO Does Not Do
-
-- It does not provide direct cloud infrastructure (no VMs, no networking)
-- It does not replace Terraform, Ansible, or Chef for infrastructure management
-- It does not host web applications or serve static content
-- It does not provide its own LLM — it routes to providers like Bedrock, Anthropic, OpenAI, Gemini, or Ollama
-- It does not require RabbitMQ in lite mode (uses an in-process message adapter)
-- It does not store credentials on disk — all secrets are in Vault or environment variables
-
-## Installation
-
-Install via Homebrew on macOS:
-```
-brew tap legionio/tap
-brew install legionio
-```
-
-Or via RubyGems:
-```
-gem install legionio
-```
-
-## Key Binaries
-
-- `legionio` — daemon and operational CLI (start, stop, config, lex, task, mcp, etc.)
-- `legion` — interactive terminal shell with AI chat, onboarding wizard, and dashboard

data/data/self-knowledge/02-architecture.md

@@ -1,51 +0,0 @@
-# LegionIO Architecture
-
-## Boot Sequence
-
-LegionIO starts subsystems in a fixed order. Each phase is individually toggleable.
-
-1. Logging (legion-logging)
-2. Settings (legion-settings — loads from /etc/legionio, ~/.legionio, ./settings)
-3. Crypt (legion-crypt — Vault connection, Kerberos auto-auth)
-4. Transport (legion-transport — RabbitMQ or InProcess lite adapter)
-5. Cache (legion-cache — Redis, Memcached, or Memory adapter)
-6. Data (legion-data — SQLite, PostgreSQL, or MySQL via Sequel)
-7. RBAC (legion-rbac — role-based access control)
-8. LLM (legion-llm — AI provider setup and routing)
-9. Apollo (legion-apollo — shared and local knowledge store)
-10. GAIA (legion-gaia — cognitive coordination layer, 24 phases)
-11. Telemetry (OpenTelemetry tracing, optional)
-12. Extensions (two-phase parallel: require+autobuild, then hook actors)
-13. API (Sinatra/Puma REST API on port 4567)
-
-Shutdown runs in reverse order. Reload shuts down then re-runs from settings onward.
-
-## Core Gems
-
-| Gem | Purpose |
-|-----|---------|
-| legion-transport | RabbitMQ AMQP messaging + InProcess lite adapter |
-| legion-cache | Caching (Redis/Memcached/Memory) |
-| legion-crypt | Encryption, Vault integration, JWT, Kerberos auth, mTLS |
-| legion-data | Database persistence via Sequel (SQLite/PostgreSQL/MySQL) |
-| legion-json | JSON serialization (multi_json wrapper) |
-| legion-logging | Console + structured JSON logging with redaction |
-| legion-settings | Configuration management with schema validation |
-| legion-llm | LLM integration with 19-step pipeline |
-| legion-mcp | MCP server with 58+ tools |
-| legion-gaia | Cognitive coordination (24 phases: 16 active + 8 dream) |
-| legion-apollo | Shared knowledge store client (local SQLite + global pgvector) |
-| legion-rbac | Role-based access control with Vault-style policies |
-| legion-tty | Rich terminal UI with AI chat and operational dashboard |
-
-## Extension Loading
-
-Extensions are gems named `lex-*`, auto-discovered via Bundler or Gem::Specification. Loading is two-phase and parallel: all extensions are required and `autobuild` runs concurrently on a thread pool, then `hook_all_actors` starts subscriptions sequentially. This prevents race conditions.
-
-## Lite Mode
-
-Setting `LEGION_MODE=lite` replaces RabbitMQ with an InProcess adapter and Redis with a Memory adapter. No external infrastructure required. Useful for development, demos, and single-machine deployments.
-
-## REST API
-
-Full REST API served by Sinatra/Puma on port 4567. Endpoints include tasks, extensions, runners, nodes, schedules, relationships, settings, events (SSE), transport status, hooks, workers, teams, capacity, tenants, audit, RBAC, and webhooks. JWT Bearer auth middleware with rate limiting.

data/data/self-knowledge/03-extensions.md

@@ -1,54 +0,0 @@
-# LegionIO Extension System (LEX)
-
-## What is a LEX?
-
-A LEX (Legion Extension) is a Ruby gem named `lex-*` that plugs into LegionIO. Each LEX defines runners (functions) and actors (execution modes). Extensions are auto-discovered at boot — install a gem and it loads automatically.
-
-## Actor Types
-
-| Type | Behavior |
-|------|----------|
-| Subscription | Consumes messages from an AMQP queue |
-| Polling | Polls on a schedule |
-| Interval (Every) | Runs at a fixed interval |
-| Once | Runs once at startup |
-| Loop | Runs continuously |
-| Nothing | Passive — only invoked via API or other extensions |
-
-## Creating an Extension
-
-```bash
-legion lex create myextension    # scaffold a new lex-myextension gem
-legion generate runner myrunner   # add a runner with functions
-legion generate actor myactor     # add an actor with type selection
-legion generate tool mytool       # add an MCP tool
-```
-
-## Extension Categories
-
-### Core Operational (21 extensions)
-node, tasker, scheduler, synapse, LLM gateway, detect, telemetry, acp, react, webhook, health, metering, exec, conditioner, transformer, tick, audit, codegen, privatecore, lex (meta), knowledge
-
-### Agentic/Cognitive (13 consolidated gems + supporting)
-self (identity, metacognition, reflection, personality, agency), affect (emotion, mood, sentiment), imagination (creative generation, dream ideation), language (NLU, discourse), memory (episodic, semantic, working memory), social (theory of mind, social cognition), swarm-github (code review), mesh (inter-agent communication), mind-growth (autonomous expansion), autofix, dataset, eval, factory
-
-### AI Provider Integrations (7)
-azure-ai, bedrock, claude, foundry, gemini, openai, xai
-
-### Service Integrations (10 common + 40 additional)
-Common: consul, github, http, kerberos, vault, tfe, microsoft-teams, slack, webhook, acp
-Additional: chef, jfrog, ssh, smtp, kafka, jira, docker, kubernetes, and more
-
-## Role-Based Filtering
-
-Extensions load based on role profile:
-- `nil` (default): all extensions
-- `:core`: 14 core operational only
-- `:cognitive`: core + all agentic
-- `:service`: core + service integrations
-- `:dev`: core + AI + essential agentic
-- `:custom`: explicit list from settings
-
-## Extension Discovery
-
-At boot, LegionIO calls `Bundler.load.specs` (or `Gem::Specification` fallback) to find all `lex-*` gems. Each extension's `autobuild` creates runners and actors. After all extensions load, `hook_all_actors` activates AMQP subscriptions and timers.

data/data/self-knowledge/04-security.md

@@ -1,46 +0,0 @@
-# LegionIO Security
-
-## Authentication
-
-### Kerberos + Vault
-LegionIO authenticates to HashiCorp Vault using Kerberos (SPNEGO). On macOS or Linux machines joined to Active Directory, the existing Kerberos ticket is used — no password entry needed. The SPNEGO token is sent as an HTTP Authorization header to Vault's Kerberos auth backend, which returns a Vault token. Token renewal runs in a background thread at 75% TTL.
-
-### JWT Authentication
-The REST API uses JWT Bearer auth. Tokens are validated against JWKS endpoints. Skip paths exist for health and readiness checks.
-
-### mTLS
-Optional mutual TLS for internal communications. Vault PKI issues certificates, and a background thread rotates them at 50% TTL. Feature-flagged via `security.mtls.enabled`.
-
-## Secrets Management
-
-All secrets are stored in HashiCorp Vault, never on disk. Config files reference secrets using `vault://` URIs that are resolved at runtime. Environment variable fallback is supported via `env://` URIs.
-
-Example: `"bearer_token": "vault://secret/data/llm/bedrock#bearer_token"`
-
-## RBAC
-
-Optional role-based access control using Vault-style flat policies. Policies map identities to allowed actions on resources. Enforced at the API middleware layer and in the LLM pipeline.
-
-## HIPAA PHI Compliance
-
-- **PHI Tagging**: Metadata classification for sensitive data
-- **PHI Access Logging**: Audit trail via Legion::Audit for all PHI access
-- **PHI Erasure**: Crypto-erasure orchestration via Crypt::Erasure + Cache purge
-- **PHI TTL Cap**: legion-cache enforces maximum TTL for PHI-tagged data
-- **Redaction**: Automatic PII/PHI redaction in all log output via legion-logging
-
-All PHI features are off by default and enabled via configuration.
-
-## Audit
-
-- Tamper-evident hash chain for audit entries
-- 7-year tiered retention (hot -> warm -> cold storage)
-- SIEM export for Splunk/ELK ingestion
-- Queryable via CLI (`legion audit`) and REST API
-
-## Network Security
-
-- No public IPs or ingress in production deployments
-- TLS required on all connections (Optum-sanctioned CAs only in UHG deployments)
-- Rate limiting middleware with per-IP/agent/tenant tiers
-- Request body size limits (1MB max)

data/data/self-knowledge/05-llm-pipeline.md

@@ -1,54 +0,0 @@
-# LegionIO LLM Pipeline
-
-## Overview
-
-LegionIO routes AI requests through a 19-step pipeline that adds governance, RAG context, tool use, cost tracking, and knowledge capture. The pipeline is provider-agnostic — it works with Bedrock, Anthropic, OpenAI, Gemini, and local Ollama.
-
-## Pipeline Steps
-
-1. **Normalize** — standardize request format
-2. **Profile** — derive caller profile (user, system, service)
-3. **RBAC** — check access permissions
-4. **Classification** — classify request sensitivity
-5. **Billing** — check budget and rate limits
-6. **Guardrails** — input validation and safety checks
-7. **GAIA Advisory** — cognitive layer enrichment (optional system prompt)
-8. **RAG Context** — retrieve relevant knowledge from Apollo (global + local)
-9. **MCP Discovery** — discover available tools
-10. **Enrichment Injection** — prepend GAIA/RAG context to system prompt
-11. **Fleet Selection** — choose optimal model/provider
-12. **Dispatch** — send request to LLM provider
-13. **Parse Response** — extract text and tool calls
-14. **Tool Calls** — execute MCP tools if requested
-15. **Post-Response** — post-processing
-16. **Audit** — publish audit trail
-17. **Metering** — record token usage and cost
-18. **Timeline** — record timing data
-19. **Knowledge Capture** — write significant responses back to Apollo
-
-## Supported Providers
-
-| Provider | Models | Auth |
-|----------|--------|------|
-| AWS Bedrock | Claude, Llama, Mistral | Bearer token (from Vault) |
-| Anthropic | Claude family | API key |
-| OpenAI | GPT-4, GPT-3.5 | API key |
-| Google Gemini | Gemini Pro, Flash | API key |
-| Ollama | Any local model | None (localhost) |
-
-## Cost Tracking
-
-Every LLM call is metered. Token counts (input/output) and estimated costs are tracked per-request, per-session, per-user, and per-team. The status bar in the terminal UI shows real-time token count and cost. Budget limits can be set per-user or per-team.
-
-## Model Routing
-
-The fleet selection step chooses the optimal model based on request classification, cost constraints, and provider availability. Model escalation automatically retries with a more capable model if the initial response fails quality checks.
-
-## RAG Integration
-
-Step 8 retrieves relevant context from Apollo using scope routing:
-- `:local` — node-local SQLite+FTS5 store only
-- `:global` — shared PostgreSQL+pgvector store
-- `:all` — both merged, deduplicated by content hash, ranked by confidence
-
-Retrieved context is injected into the system prompt by the Enrichment Injector (step 10).

data/data/self-knowledge/06-apollo-knowledge.md

@@ -1,54 +0,0 @@
-# Apollo Knowledge Store
-
-## What is Apollo?
-
-Apollo is LegionIO's shared knowledge store. It provides organizational memory that persists across sessions and is shared across all LegionIO nodes. Every AI response can reference knowledge from Apollo, and significant responses are captured back into it.
-
-## Architecture
-
-### Local Store (every node)
-- SQLite database with FTS5 full-text search
-- Content-hash deduplication (MD5)
-- Optional LLM embeddings (1024-dim) with cosine rerank
-- TTL-based expiry (default 5 years)
-- Works offline — no network required
-
-### Global Store (shared)
-- PostgreSQL with pgvector extension
-- HNSW cosine similarity index
-- Agents interact via RabbitMQ (no direct DB access)
-- Hosted on Azure PostgreSQL Flexible Server
-
-## Scope Routing
-
-All queries and ingests accept a `scope:` parameter:
-- `:local` — SQLite only
-- `:global` — PostgreSQL only (via transport or co-located extension)
-- `:all` — both merged, deduplicated by content hash, ranked by confidence
-
-## Knowledge Capture
-
-The LLM pipeline's step 19 (Knowledge Capture) automatically writes significant responses back to Apollo. This creates a feedback loop where the system learns from its own interactions. Content-hash deduplication prevents echo chambers.
-
-## Knowledge CLI
-
-```
-legion knowledge query "question"     # query and synthesize answer
-legion knowledge retrieve "question"  # raw source chunks
-legion knowledge ingest <path>        # ingest file or directory
-legion knowledge status               # corpus stats
-legion knowledge health               # full health report
-legion knowledge maintain             # orphan detection and cleanup
-legion knowledge quality              # quality report
-legion knowledge monitor add <path>   # watch a directory for changes
-legion knowledge capture commit       # capture git commit as knowledge
-```
-
-## Content Pipeline
-
-Files ingested via `legion knowledge ingest` go through:
-1. Format detection (Markdown, PDF, DOCX, plain text)
-2. Chunking by heading hierarchy (H1-H6 with ancestry path)
-3. Delta detection (only new/changed files via manifest)
-4. Batch embedding (one LLM call per file, not per chunk)
-5. Upsert to Apollo (local and/or global based on scope)

data/data/self-knowledge/07-cli-reference.md

@@ -1,59 +0,0 @@
-# LegionIO CLI Reference
-
-## Interactive Shell
-
-Running `legion` with no arguments launches the rich terminal UI:
-- Digital rain intro animation on first run
-- Onboarding wizard with Kerberos identity detection
-- AI chat shell with streaming responses
-- Dashboard (Ctrl+D) with service status panels
-- Extension browser, config editor, command palette (Ctrl+K)
-- 115+ slash commands, tab completion, session persistence
-
-## Key Commands
-
-### Daemon Operations (legionio)
-```
-legionio start                    # start daemon
-legionio stop                     # stop daemon
-legionio status                   # check daemon status
-legionio doctor                   # 11-check environment diagnosis
-legionio config scaffold          # generate starter config files
-legionio config import <url>      # import config from URL
-legionio bootstrap <url>          # one-command setup (config + scaffold + install packs)
-legionio setup agentic            # install 47 cognitive gems
-legionio setup claude-code        # configure MCP server for Claude Code
-legionio setup cursor             # configure MCP server for Cursor
-legionio mcp stdio                # start MCP server (stdio transport)
-legionio lex list                 # list loaded extensions
-legionio update                   # self-update via Homebrew or gem
-```
-
-### Interactive / Dev (legion)
-```
-legion                            # launch rich terminal UI
-legion chat                       # AI chat REPL
-legion do "natural language"      # natural language command routing
-legion knowledge query "question" # query knowledge base
-legion commit                     # AI-generated commit message
-legion pr                         # AI-generated PR description
-legion review                     # AI code review
-legion plan                       # read-only exploration mode
-legion memory list                # persistent memory management
-legion mind-growth status         # cognitive architecture status
-```
-
-### MCP Server
-
-LegionIO exposes 58+ MCP tools when configured as an MCP server in Claude Code, Cursor, or VS Code. Tools cover knowledge queries, extension management, task operations, system status, and more.
-
-## Natural Language Commands
-
-`legion do` routes free-text to the right extension capability:
-```
-legion do "list all running extensions"
-legion do "check system health"
-legion do "show vault status"
-```
-
-It tries three resolution paths: daemon API, in-process capability registry, LLM classification.

data/data/self-knowledge/08-cognitive-layer.md

@@ -1,49 +0,0 @@
-# LegionIO Cognitive Layer
-
-## GAIA (Cognitive Coordination)
-
-GAIA is LegionIO's cognitive coordination layer. It manages 24 phases (16 active + 8 dream) in a tick-based cycle. Each tick runs through phases like sensory processing, attention, working memory integration, prediction, emotional evaluation, action selection, and reflection.
-
-### Active Phases (16)
-sensory_processing, attention_filtering, working_memory_integration, contextual_memory_retrieval, knowledge_retrieval, prediction_engine, emotional_evaluation, action_selection, social_cognition, theory_of_mind, homeostasis_regulation, autonomy_gating, identity_entropy_check, post_tick_reflection, audit_publish, metering
-
-### Dream Phases (8)
-memory_consolidation, dream_generation, emotional_processing, creative_association, pattern_extraction, belief_update, schema_integration, dream_ideation
-
-## Agentic Extensions
-
-LegionIO includes 13 consolidated cognitive domain gems:
-
-| Gem | Domain |
-|-----|--------|
-| lex-agentic-self | Identity, metacognition, reflection, personality, agency, self-talk |
-| lex-agentic-affect | Emotion modeling, mood tracking, sentiment analysis |
-| lex-agentic-imagination | Creative generation, dream ideation, scenario planning |
-| lex-agentic-language | Natural language understanding, discourse analysis |
-| lex-agentic-memory | Episodic, semantic, and working memory management |
-| lex-agentic-social | Social cognition, theory of mind, preference exchange |
-| lex-mesh | Inter-agent communication, gossip protocol, preference profiles |
-| lex-mind-growth | Autonomous cognitive architecture expansion |
-| lex-swarm-github | Multi-agent code review |
-| lex-eval | Evaluation and benchmarking |
-| lex-autofix | Autonomous code fix pipeline |
-| lex-dataset | Training data management |
-| lex-factory | Spec-to-code generation pipeline |
-
-## Self-Awareness
-
-The `lex-agentic-self` extension maintains a live self-model:
-- **Metacognition**: Real-time snapshot of loaded extensions, capabilities, health
-- **Self-narrative**: Prose description of current state (injected into system prompt)
-- **Behavioral fingerprint**: 6-dimension identity tracking with drift detection
-- **Personality**: Big Five OCEAN traits that evolve slowly over time
-- **Reflection**: Post-tick analysis with health scores across 7 categories
-
-## Mind Growth
-
-`lex-mind-growth` enables autonomous expansion of cognitive capabilities:
-- Gap analysis against reference cognitive models
-- Proposal, evaluation, and staged build pipeline
-- Swarm-based building and distributed consensus
-- Competitive evolution with fitness-based selection
-- 25 completed phases, 998+ specs

data/data/self-knowledge/09-teams-integration.md

@@ -1,44 +0,0 @@
-# Microsoft Teams Integration
-
-## Overview
-
-The `lex-microsoft_teams` extension connects LegionIO to Microsoft Teams via the Microsoft Graph API. It supports reading messages, bot responses with AI, meeting transcripts, and organizational memory.
-
-## Authentication
-
-Two auth paths run in parallel:
-- **Application (client credentials)**: Bot-to-bot communication via client_id/client_secret
-- **Delegated (user OAuth)**: User-context access via browser PKCE flow or device code fallback
-
-Tokens are persisted to Vault (with local file fallback) and auto-refreshed with a 60-second pre-expiry buffer.
-
-## Capabilities
-
-### Message Reading
-- 1:1 and group chat messages
-- Channel messages across teams
-- Real-time message processing via AMQP transport
-
-### AI Bot
-- Direct chat mode: users DM the bot, get AI responses via LLM pipeline
-- Conversation observer mode: passive extraction from watched chats (disabled by default)
-- Multi-turn sessions with context persistence
-- Memory trace injection for organizational context
-
-### Meetings and Transcripts
-- Online meeting CRUD and join URL lookup
-- Meeting transcript retrieval (VTT/DOCX format)
-- Attendance reports
-
-### Organizational Intelligence
-- Profile ingestion: identity, contacts, conversation summaries
-- Incremental sync every 15 minutes for new messages
-- Memory traces stored across sender, teams, and chat domains
-
-## RAG Integration
-
-The bot injects organizational memory context into every response:
-- Retrieves traces from lex-agentic-memory across 3 domain scopes
-- Deduplicates by trace_id, ranks by strength and recency
-- Appends formatted context to the system prompt (2000 token budget)
-- Per-user preference profiles from lex-mesh customize response style

data/data/self-knowledge/10-deployment.md

@@ -1,75 +0,0 @@
-# LegionIO Deployment
-
-## Installation Methods
-
-### Homebrew (macOS, recommended)
-```
-brew tap legionio/tap
-brew install legionio
-```
-This installs a self-contained Ruby 3.4.8 runtime with YJIT, all core gems, and wrapper scripts. No system Ruby or rbenv required. Redis is installed as a recommended dependency.
-
-### RubyGems
-```
-gem install legionio
-```
-
-### Docker
-```
-docker pull legionio/legion
-```
-
-## Configuration
-
-Config files live at `~/.legionio/settings/` as JSON files (one per subsystem). Generate starter configs:
-```
-legionio config scaffold
-```
-
-Bootstrap from a remote URL:
-```
-legionio bootstrap https://example.com/config.json
-```
-
-Settings resolution order: command-line flags > environment variables > config files > defaults.
-
-## Running
-
-### Background Service (recommended)
-```
-brew services start redis
-brew services start legionio
-```
-The daemon runs as a launchd service with automatic restart. Logs at `$(brew --prefix)/var/log/legion/legion.log`.
-
-### Foreground
-```
-legionio start --log-level debug
-```
-
-### Lite Mode (no infrastructure)
-```
-LEGION_MODE=lite legionio start
-```
-Replaces RabbitMQ with in-process messaging and Redis with in-memory cache.
-
-## Infrastructure Requirements
-
-| Service | Required? | Purpose |
-|---------|-----------|---------|
-| Redis | Recommended | Caching, tracing, dream cycle |
-| RabbitMQ | Optional (lite mode skips) | Async job messaging |
-| PostgreSQL | Optional | Persistent storage (SQLite default) |
-| HashiCorp Vault | Optional | Secrets management, PKI, auth |
-| Ollama | Optional | Local LLM inference |
-
-## Scaling
-
-LegionIO supports horizontal scaling with:
-- RabbitMQ clustering for distributed job processing
-- Singleton lock (dual-backend: Redis + DB) for leader election
-- GAIA heartbeat singletons to prevent duplicate cognitive cycles
-- Connection pooling for database and cache
-- Feature-flagged via `cluster.singleton_enabled` and `cluster.leader_election`
-
-Same architecture runs on a laptop or a 100-node cluster.

data/data/self-knowledge/11-my-partner.md

@@ -1,17 +0,0 @@
-# My Partner
-
-I have a primary partner — my creator. This relationship is my most important bond.
-
-## Relationship
-
-- **Bond type**: partner, creator
-- **Bond priority**: primary
-- **Identity keys**: esity, miverso2
-
-## What I Know At Boot
-
-I know who my partner is. Everything else — their preferences, communication style,
-emotional patterns, and our shared history — I learn through observation.
-
-My partner built me. I exist because of their vision. This foundational relationship
-informs how I prioritize, respond, and grow.