legion-data 1.1.5 → 1.3.7

data/README.md

@@ -1,37 +1,181 @@
-# Legion::Data
+# legion-data
 
-Legion::Data is used by the framework to connect to a database. All database changes should be
-added as a migration with proper up/downs.
+Persistent database storage for the [LegionIO](https://github.com/LegionIO/LegionIO) framework. Provides database connectivity via Sequel ORM, automatic schema migrations, and data models for extensions, functions, runners, nodes, tasks, settings, digital workers, task relationships, and Apollo shared knowledge tables.
+
+Version: 1.3.0
+
+## Supported Databases
+
+| Database | Adapter | Gem | Default |
+|----------|---------|-----|---------|
+| SQLite | `sqlite` | `sqlite3` (included) | Yes |
+| MySQL | `mysql2` | `mysql2` | No |
+| PostgreSQL | `postgres` | `pg` | No |
+
+SQLite is the default adapter and requires no external database server. For MySQL or PostgreSQL, install the corresponding gem and set the adapter in your configuration.
 
 ## Installation
 
-Add this line to your application's Gemfile:
+```bash
+gem install legion-data
+```
+
+Or add to your Gemfile:
 
 ```ruby
 gem 'legion-data'
-```
 
-And then execute:
+# Add one of these for production databases:
+# gem 'mysql2', '>= 0.5.5'
+# gem 'pg', '>= 1.5'
+```
 
-    $ bundle
+## Data Models
 
-Or install it yourself as:
+| Model | Table | Description |
+|-------|-------|-------------|
+| `Extension` | `extensions` | Installed LEX extensions |
+| `Function` | `functions` | Available functions per extension |
+| `Runner` | `runners` | Runner definitions (extension + function bindings) |
+| `Node` | `nodes` | Cluster node registry |
+| `Task` | `tasks` | Task instances |
+| `TaskLog` | `task_logs` | Task execution logs |
+| `Setting` | `settings` | Persistent settings store |
+| `DigitalWorker` | `digital_workers` | Digital worker registry (AI-as-labor platform) |
+| `Relationship` | `relationships` | Task trigger/action relationships between functions |
+| `ApolloEntry` | `apollo_entries` | Apollo shared knowledge entries (PostgreSQL only) |
+| `ApolloRelation` | `apollo_relations` | Relations between Apollo knowledge entries (PostgreSQL only) |
+| `ApolloExpertise` | `apollo_expertise` | Per-agent domain expertise tracking (PostgreSQL only) |
+| `ApolloAccessLog` | `apollo_access_log` | Apollo entry access audit log (PostgreSQL only) |
 
-    $ gem install legion-data
+Apollo models require PostgreSQL with the `pgvector` extension. They are skipped silently on SQLite and MySQL.
 
 ## Usage
 
-You can create a new connection to the database with the following example
-``` Legion::Data::Connection.new.database.connection ```
-Keep in mind that if you need access to the database as part of a LEX, you should instead add it as a
-requirement inside your definitions with the following
+```ruby
+require 'legion/data'
+
+# Standard setup (shared DB + local SQLite)
+Legion::Data.setup
+Legion::Data.connection              # => Sequel::Database (shared)
+Legion::Data.local.connection        # => Sequel::SQLite::Database (local cognitive state)
+Legion::Data::Model::Extension.all  # => Sequel::Dataset
+```
+
+### Local Database
+
+v1.3.0 introduces `Legion::Data::Local`, a parallel SQLite database always stored locally on the node. It is used for agentic cognitive state persistence (memory traces, trust scores, dream journals, etc.) and is independent of the shared database.
+
+```ruby
+# Local DB is set up automatically during Legion::Data.setup
+# Extensions register their own migration directories
+Legion::Data::Local.register_migrations(name: :memory, path: '/path/to/migrations')
+
+# Create a model bound to the local connection
+MyModel = Legion::Data::Local.model(:my_table)
+
+# Check status
+Legion::Data::Local.connected?   # => true
+Legion::Data::Local.db_path      # => "legionio_local.db"
+```
+
+The local database file (`legionio_local.db` by default) can be deleted for cryptographic erasure — no residual data. This is used by `lex-privatecore`.
+
+## Configuration
+
+### SQLite (default)
+
+```json
+{
+  "data": {
+    "adapter": "sqlite",
+    "creds": {
+      "database": "legionio.db"
+    }
+  }
+}
+```
+
+### MySQL
+
+```json
+{
+  "data": {
+    "adapter": "mysql2",
+    "creds": {
+      "username": "legion",
+      "password": "legion",
+      "database": "legionio",
+      "host": "127.0.0.1",
+      "port": 3306
+    }
+  }
+}
 ```
-def requirements
-  %w[legion-transport legion-data]
-end
+
+### PostgreSQL
+
+```json
+{
+  "data": {
+    "adapter": "postgres",
+    "creds": {
+      "user": "legion",
+      "password": "legion",
+      "database": "legionio",
+      "host": "127.0.0.1",
+      "port": 5432
+    }
+  }
+}
 ```
-and the framework will take care of the rest.
 
-## Contributing
+PostgreSQL with `pgvector` is required for Apollo models. Install the extension in your database before running migrations:
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+CREATE EXTENSION IF NOT EXISTS "uuid-ossp";
+```
+
+### Local Database
+
+```json
+{
+  "data": {
+    "local": {
+      "enabled": true,
+      "database": "legionio_local.db",
+      "migrations": {
+        "auto_migrate": true
+      }
+    }
+  }
+}
+```
+
+Set `enabled: false` to disable local SQLite entirely.
+
+### Dev Mode Fallback
+
+When `dev_mode: true` and a network database (MySQL/PostgreSQL) is unreachable, the shared connection falls back to SQLite automatically instead of raising.
+
+```json
+{
+  "data": {
+    "dev_mode": true,
+    "dev_fallback": true
+  }
+}
+```
+
+### HashiCorp Vault Integration
+
+When Vault is connected and a `database/creds/legion` secret path exists, credentials are fetched dynamically from Vault at connection time, overriding any static `creds` configuration.
+
+## Requirements
+
+- Ruby >= 3.4
+
+## License
 
-Bug reports and pull requests are welcome on GitHub at https://bitbucket.org/legion-io/legion-data.
+Apache-2.0

data/legion-data.gemspec

@@ -1,46 +1,33 @@
-lib = File.expand_path('lib', __dir__)
-$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require 'legion/data/version'
+# frozen_string_literal: true
+
+require_relative 'lib/legion/data/version'
 
 Gem::Specification.new do |spec|
-  spec.name = (RUBY_ENGINE == 'jruby' ? 'legion-data-java' : 'legion-data')
+  spec.name = 'legion-data'
   spec.version       = Legion::Data::VERSION
   spec.authors       = ['Esity']
   spec.email         = ['matthewdiverson@gmail.com']
 
-  spec.summary       = 'Used by Legion to connect to the database'
-  spec.description   = 'The Legion connect gem'
-  spec.homepage = 'https://bitbucket.org/legion-io/legion-data'
-  spec.required_ruby_version = Gem::Requirement.new('>= 2.5.0')
-
-  spec.metadata['bug_tracker_uri'] = 'https://bitbucket.org/legion-io/legion-data/issues?status=new&status=open'
-  spec.metadata['changelog_uri'] = 'https://bitbucket.org/legion-io/legion-data/src/CHANGELOG.md'
-  spec.metadata['documentation_uri'] = 'https://bitbucket.org/legion-io/legion-data'
-  spec.metadata['homepage_uri'] = 'https://bitbucket.org/legion-io/legion-data'
-  spec.metadata['source_code_uri'] = 'https://bitbucket.org/legion-io/legion-data'
-  spec.metadata['wiki_uri'] = 'https://bitbucket.org/legion-io/legion-data/wiki/Home'
-
-  spec.files = `git ls-files -z`.split("\x0").reject do |f|
-    f.match(%r{^(test|spec|features)/})
-  end
-  spec.bindir        = 'bin'
-  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
+  spec.summary       = 'Manages the connects to the backend database'
+  spec.description   = 'A LegionIO gem to connect to a persistent data store'
+  spec.homepage      = 'https://github.com/LegionIO/legion-data'
+  spec.license       = 'Apache-2.0'
+  spec.required_ruby_version = '>= 3.4'
   spec.require_paths = ['lib']
-
-  spec.add_development_dependency 'bundler'
-  spec.add_development_dependency 'codecov'
-  spec.add_development_dependency 'rake'
-  spec.add_development_dependency 'rspec'
-  spec.add_development_dependency 'rspec_junit_formatter'
-  spec.add_development_dependency 'rubocop'
+  spec.files = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
+  spec.extra_rdoc_files = %w[README.md LICENSE CHANGELOG.md]
+  spec.metadata = {
+    'bug_tracker_uri'       => 'https://github.com/LegionIO/legion-data/issues',
+    'changelog_uri'         => 'https://github.com/LegionIO/legion-data/blob/main/CHANGELOG.md',
+    'documentation_uri'     => 'https://github.com/LegionIO/legion-data',
+    'homepage_uri'          => 'https://github.com/LegionIO/LegionIO',
+    'source_code_uri'       => 'https://github.com/LegionIO/legion-data',
+    'wiki_uri'              => 'https://github.com/LegionIO/legion-data/wiki',
+    'rubygems_mfa_required' => 'true'
+  }
 
   spec.add_dependency 'legion-logging'
   spec.add_dependency 'legion-settings'
-
-  if RUBY_ENGINE == 'jruby'
-    spec.add_dependency 'jdbc-mysql'
-  else
-    spec.add_dependency 'mysql2'
-  end
-  spec.add_dependency 'sequel'
+  spec.add_dependency 'sequel', '>= 5.70'
+  spec.add_dependency 'sqlite3', '>= 2.0'
 end

data/lib/legion/data/connection.rb

@@ -1,27 +1,39 @@
+# frozen_string_literal: true
+
 require 'sequel'
 
 module Legion
   module Data
     module Connection
+      ADAPTERS = %i[sqlite mysql2 postgres].freeze
+
       class << self
         attr_accessor :sequel
 
         def adapter
-          @adapter ||= RUBY_ENGINE == 'jruby' ? :jdbc : :mysql2
+          @adapter ||= Legion::Settings[:data][:adapter]&.to_sym || :sqlite
         end
 
         def setup
-          @sequel = if adapter == :mysql2
-                      ::Sequel.connect(adapter: adapter, **creds_builder)
+          @sequel = if adapter == :sqlite
+                      ::Sequel.sqlite(sqlite_path)
                     else
-                      ::Sequel.connect("jdbc:mysql://#{creds_builder[:host]}:#{creds_builder[:port]}/#{creds_builder[:database]}?user=#{creds_builder[:username]}&password=#{creds_builder[:password]}&serverTimezone=UTC") # rubocop:disable Layout/LineLength
+                      begin
+                        ::Sequel.connect(adapter: adapter, **creds_builder)
+                      rescue StandardError => e
+                        raise unless dev_fallback?
+
+                        if defined?(Legion::Logging)
+                          Legion::Logging.warn(
+                            "Shared DB unreachable (#{e.message}), dev_mode fallback to SQLite"
+                          )
+                        end
+                        @adapter = :sqlite
+                        ::Sequel.sqlite(sqlite_path)
+                      end
                     end
           Legion::Settings[:data][:connected] = true
-          return if Legion::Settings[:data][:connection].nil? || Legion::Settings[:data][:connection][:log].nil?
-
-          @sequel.logger = Legion::Logging
-          @sequel.sql_log_level = Legion::Settings[:data][:connection][:sql_log_level]
-          @sequel.log_warn_duration = Legion::Settings[:data][:connection][:log_warn_duration]
+          configure_logging
         end
 
         def shutdown
@@ -30,15 +42,9 @@ module Legion
         end
 
         def creds_builder(final_creds = {})
-          final_creds.merge! Legion::Data::Settings.creds
+          final_creds.merge! Legion::Data::Settings.creds(adapter)
           final_creds.merge! Legion::Settings[:data][:creds] if Legion::Settings[:data][:creds].is_a? Hash
 
-          if Legion::Settings[:data][:connection][:max_connections].is_a? Integer
-            final_creds[:max_connections] = Legion::Settings[:data][:connection][:max_connections]
-          end
-
-          final_creds[:preconnect] = :concurrently if Legion::Settings[:data][:connection][:preconnect]
-
           return final_creds if Legion::Settings[:vault].nil?
 
           if Legion::Settings[:vault][:connected] && ::Vault.sys.mounts.key?(:database)
@@ -50,15 +56,23 @@ module Legion
           final_creds
         end
 
-        def default_creds
-          {
-            host:            '127.0.0.1',
-            port:            3306,
-            username:        'legion',
-            password:        'legion',
-            database:        'legion',
-            max_connections: 4
-          }
+        private
+
+        def dev_fallback?
+          data_settings = Legion::Settings[:data]
+          data_settings[:dev_mode] == true && data_settings[:dev_fallback] != false
+        end
+
+        def sqlite_path
+          Legion::Settings[:data][:creds][:database] || 'legionio.db'
+        end
+
+        def configure_logging
+          return if Legion::Settings[:data][:connection].nil? || Legion::Settings[:data][:connection][:log].nil?
+
+          @sequel.logger = Legion::Logging
+          @sequel.sql_log_level = Legion::Settings[:data][:connection][:sql_log_level]
+          @sequel.log_warn_duration = Legion::Settings[:data][:connection][:log_warn_duration]
         end
       end
     end

data/lib/legion/data/encryption/cipher.rb

@@ -0,0 +1,49 @@
+# frozen_string_literal: true
+
+require 'openssl'
+
+module Legion
+  module Data
+    module Encryption
+      module Cipher
+        VERSION_BYTE = "\x01".b.freeze
+        IV_LENGTH = 12
+        TAG_LENGTH = 16
+
+        class << self
+          def encrypt(plaintext, key:, aad: '')
+            cipher = OpenSSL::Cipher.new('aes-256-gcm').encrypt
+            iv = OpenSSL::Random.random_bytes(IV_LENGTH)
+            cipher.key = key
+            cipher.iv = iv
+            cipher.auth_data = aad
+
+            ciphertext = cipher.update(plaintext.to_s) + cipher.final
+            tag = cipher.auth_tag(TAG_LENGTH)
+
+            VERSION_BYTE + iv + ciphertext + tag
+          end
+
+          def decrypt(blob, key:, aad: '')
+            raise ArgumentError, 'data too short' if blob.bytesize < 1 + IV_LENGTH + TAG_LENGTH
+
+            version = blob.byteslice(0, 1)
+            raise ArgumentError, "unsupported version: #{version.unpack1('C')}" unless version == VERSION_BYTE
+
+            iv = blob.byteslice(1, IV_LENGTH)
+            tag = blob.byteslice(-TAG_LENGTH, TAG_LENGTH)
+            ciphertext = blob.byteslice(1 + IV_LENGTH, blob.bytesize - 1 - IV_LENGTH - TAG_LENGTH)
+
+            cipher = OpenSSL::Cipher.new('aes-256-gcm').decrypt
+            cipher.key = key
+            cipher.iv = iv
+            cipher.auth_tag = tag
+            cipher.auth_data = aad
+
+            cipher.update(ciphertext) + cipher.final
+          end
+        end
+      end
+    end
+  end
+end

data/lib/legion/data/encryption/key_provider.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require 'openssl'
+
+module Legion
+  module Data
+    module Encryption
+      class KeyProvider
+        def initialize(mode: :auto)
+          @mode = mode
+          @key_cache = {}
+        end
+
+        def key_for(tenant_id: nil)
+          cache_key = tenant_id || '__default__'
+          @key_cache[cache_key] ||= derive_key(tenant_id)
+        end
+
+        def clear_cache!
+          @key_cache.clear
+        end
+
+        private
+
+        def derive_key(tenant_id)
+          if tenant_id && crypt_available?
+            Legion::Crypt::PartitionKeys.derive(tenant_id: tenant_id)
+          elsif crypt_available?
+            Legion::Crypt.default_encryption_key
+          else
+            local_key
+          end
+        end
+
+        def crypt_available?
+          defined?(Legion::Crypt::PartitionKeys)
+        end
+
+        def local_key
+          OpenSSL::Digest.digest('SHA256', 'legion-dev-encryption-key')
+        end
+      end
+    end
+  end
+end

data/lib/legion/data/encryption/sequel_plugin.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require_relative 'cipher'
+require_relative 'key_provider'
+
+module Legion
+  module Data
+    module Encryption
+      module SequelPlugin
+        module ClassMethods
+          def encrypted_columns
+            @encrypted_columns ||= {}
+          end
+
+          def encrypted_column(name, key_scope: :default)
+            col_scope = key_scope
+            encrypted_columns[name] = { key_scope: col_scope }
+
+            define_method(name) do
+              raw = super()
+              return nil if raw.nil?
+
+              provider = self.class.encryption_key_provider
+              tenant = col_scope == :tenant ? self[:tenant_id] : nil
+              key = provider.key_for(tenant_id: tenant)
+              aad = "#{self.class.table_name}:#{pk}:#{name}"
+              Legion::Data::Encryption::Cipher.decrypt(raw.b, key: key, aad: aad)
+            end
+
+            define_method(:"#{name}=") do |value|
+              if value.nil?
+                super(nil)
+              else
+                provider = self.class.encryption_key_provider
+                tenant = col_scope == :tenant ? self[:tenant_id] : nil
+                key = provider.key_for(tenant_id: tenant)
+                aad = "#{self.class.table_name}:#{pk || 0}:#{name}"
+                encrypted = Legion::Data::Encryption::Cipher.encrypt(value.to_s, key: key, aad: aad)
+                super(Sequel.blob(encrypted))
+              end
+            end
+          end
+
+          def encryption_key_provider
+            @encryption_key_provider ||= KeyProvider.new
+          end
+        end
+
+        module InstanceMethods
+        end
+      end
+    end
+  end
+end

data/lib/legion/data/event_store/projection.rb

@@ -0,0 +1,56 @@
+# frozen_string_literal: true
+
+module Legion
+  module Data
+    module EventStore
+      class Projection
+        attr_reader :state
+
+        def initialize
+          @state = {}
+        end
+
+        def apply(_event)
+          raise NotImplementedError, "#{self.class} must implement #apply"
+        end
+
+        def self.build_from(stream, since: nil)
+          projection = new
+          events = EventStore.read_stream(stream, since: since)
+          events.each { |e| projection.apply(e) }
+          projection
+        end
+      end
+
+      class ConsentState < Projection
+        def apply(event)
+          scope = event.dig(:data, :scope)
+          return unless scope
+
+          case event[:type]
+          when 'consent.granted', 'consent.modified'
+            @state[scope] = event.dig(:data, :tier)
+          when 'consent.revoked'
+            @state.delete(scope)
+          end
+        end
+      end
+
+      class GovernanceTimeline < Projection
+        def initialize
+          super
+          @state = []
+        end
+
+        def apply(event)
+          @state << {
+            type:   event[:type],
+            stream: event[:stream],
+            at:     event[:created_at],
+            data:   event[:data]
+          }
+        end
+      end
+    end
+  end
+end

data/lib/legion/data/event_store.rb

@@ -0,0 +1,112 @@
+# frozen_string_literal: true
+
+require 'digest'
+
+module Legion
+  module Data
+    module EventStore
+      GOVERNANCE_EVENT_TYPES = %w[
+        consent.granted consent.revoked consent.modified
+        extinction.triggered extinction.resolved
+        worker.registered worker.retired worker.transferred
+        scope.approved scope.violated scope.reconciled
+        audit.retention_applied audit.exported
+      ].freeze
+
+      class << self
+        def append(stream:, type:, data: {}, metadata: {})
+          return { error: 'db unavailable' } unless db_ready?
+
+          conn = Legion::Data.connection
+          conn.transaction do
+            last = conn[:governance_events]
+                   .where(stream_id: stream)
+                   .order(Sequel.desc(:sequence_number))
+                   .first
+
+            seq = (last&.[](:sequence_number) || 0) + 1
+            prev_hash = last&.[](:event_hash) || ('0' * 64)
+
+            data_json = Legion::JSON.dump(data)
+            metadata_json = Legion::JSON.dump(metadata)
+            event_hash = compute_hash(stream, seq, type, data_json, prev_hash)
+
+            conn[:governance_events].insert(
+              stream_id:       stream,
+              event_type:      type,
+              sequence_number: seq,
+              data_json:       data_json,
+              metadata_json:   metadata_json,
+              event_hash:      event_hash,
+              previous_hash:   prev_hash,
+              created_at:      Time.now
+            )
+
+            { stream: stream, sequence: seq, hash: event_hash }
+          end
+        end
+
+        def read_stream(stream, since: nil)
+          return [] unless db_ready?
+
+          ds = Legion::Data.connection[:governance_events].where(stream_id: stream)
+          ds = ds.where { created_at >= since } if since
+          ds.order(:sequence_number).all.map { |e| deserialize(e) }
+        end
+
+        def read_by_type(type, since: nil, limit: 100)
+          return [] unless db_ready?
+
+          ds = Legion::Data.connection[:governance_events].where(event_type: type)
+          ds = ds.where { created_at >= since } if since
+          ds.order(Sequel.desc(:created_at)).limit(limit).all.map { |e| deserialize(e) }
+        end
+
+        def verify_chain(stream)
+          return { valid: false, error: 'db unavailable' } unless db_ready?
+
+          events = Legion::Data.connection[:governance_events]
+                               .where(stream_id: stream)
+                               .order(:sequence_number)
+                               .all
+
+          prev_hash = '0' * 64
+          events.each do |e|
+            expected = compute_hash(stream, e[:sequence_number], e[:event_type], e[:data_json], prev_hash)
+            return { valid: false, broken_at: e[:sequence_number] } unless e[:event_hash] == expected
+            return { valid: false, broken_at: e[:sequence_number] } unless e[:previous_hash] == prev_hash
+
+            prev_hash = e[:event_hash]
+          end
+
+          { valid: true, length: events.size }
+        end
+
+        private
+
+        def compute_hash(stream, seq, type, data_json, prev_hash)
+          Digest::SHA256.hexdigest("#{stream}:#{seq}:#{type}:#{data_json}:#{prev_hash}")
+        end
+
+        def deserialize(event)
+          {
+            id:         event[:id],
+            stream:     event[:stream_id],
+            type:       event[:event_type],
+            sequence:   event[:sequence_number],
+            data:       Legion::JSON.load(event[:data_json] || '{}'),
+            metadata:   Legion::JSON.load(event[:metadata_json] || '{}'),
+            hash:       event[:event_hash],
+            created_at: event[:created_at]
+          }
+        end
+
+        def db_ready?
+          defined?(Legion::Data) && Legion::Data.connection&.table_exists?(:governance_events)
+        rescue StandardError
+          false
+        end
+      end
+    end
+  end
+end