s3arch 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1d5bd8c8d47da22b8feb308ff8fbc96205d554ae915ef8e4e2831d5ba534f786
+  data.tar.gz: 4741157e9c522074a08c3478dc4cfddd16445590cfdb107100121083ff34a29a
+SHA512:
+  metadata.gz: 54b5a57df4c767d23eac10d7984ea5ec796f60cfe9047b24619cb3d67600af2b67e71e2f0639704a55939a057f7351f7778a0587b9db6b80fa768eb2398c9e18
+  data.tar.gz: c4a2ff82ac4985b24376ccacab235e9a56132e1ee10b77202eff9defc62d0b1765ab9da31d89c695b42c3560984d1098df476a6b73b6a4561ad6c1cbefaa4749

data/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## [0.0.1] - 2025-06-08
+
+### Added
+
+- Initial release
+- `S3arch::Indexer` — builds per-owner SQLite FTS5 databases from DynamoDB records
+- `S3arch::Searcher` — queries per-owner indexes from Lambda `/tmp` with LRU caching
+- `S3arch::Handler` — pre-built Lambda handler methods for indexer and search
+- `S3arch::Configuration` — declarative configuration with `from_env!` convenience
+- DynamoDB version tracking for cache invalidation
+- EventBridge Pipe architecture (DynamoDB Streams → SQS → Indexer Lambda)
+- Terraform module for infrastructure provisioning

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Stowzilla
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,152 @@
+# S3arch
+
+Full-text search for DynamoDB on AWS Lambda using SQLite FTS5. Per-owner indexes stored on S3, queried from Lambda `/tmp` with LRU eviction and DynamoDB version tracking.
+
+## Why?
+
+DynamoDB doesn't have native full-text search. OpenSearch/Elasticsearch is expensive and complex for many use cases. S3arch gives you fast, typo-tolerant search with zero infrastructure beyond what you already have (Lambda, S3, DynamoDB).
+
+**How it works:**
+
+```
+DynamoDB (source) → Stream → EventBridge Pipe → SQS → Indexer Lambda
+                                                         ↓
+                                                    SQLite FTS5 DB
+                                                         ↓
+                                                    S3 (per-owner)
+                                                         ↓
+                                    Search Lambda ← /tmp cache (LRU)
+```
+
+Each owner (user, tenant, account) gets their own SQLite database. The indexer rebuilds it on every change. The searcher downloads it to `/tmp` on first query, then serves subsequent queries from the warm cache until the version changes.
+
+## Installation
+
+```ruby
+gem 's3arch'
+```
+
+Requires the `sqlite3` native extension available at runtime. On Lambda, use a layer that provides it (e.g., [stowzilla-sqlite3-ruby](https://github.com/stowzilla/stowzilla-sqlite3-ruby)).
+
+## Usage
+
+### Configuration
+
+```ruby
+require 's3arch'
+
+S3arch.configure do |c|
+  c.from_env!                                    # Reads S3ARCH_* env vars
+  c.source_index = 'UserIndex'                   # GSI for owner lookup
+  c.owner_key = 'userId'                         # Partition key for owner
+  c.searchable_fields = %w[name description tags] # FTS5 indexed fields
+  c.metadata_fields = %w[status created_at]       # Stored for filtering (not searched)
+  c.record_filter = ->(item) { item['status'] == 'active' }
+  c.logger = Logger.new($stdout)
+end
+```
+
+### Lambda Handlers
+
+**Indexer** (triggered by SQS from DynamoDB Streams):
+
+```ruby
+def handler(event:, context:)
+  S3arch::Handler.indexer(event)
+end
+```
+
+**Searcher** (invoked directly or via API Gateway):
+
+```ruby
+def handler(event:, context:)
+  result = S3arch::Handler.search(event)
+  # => { record_ids: ["id1", "id2", ...], search_mode: "fts5" }
+end
+```
+
+### Search Event Format
+
+```json
+{
+  "query": "blue chair",
+  "owner_ids": ["user-123", "user-456"],
+  "filters": { "status": "active" }
+}
+```
+
+### Manual Indexing
+
+```ruby
+indexer = S3arch::Indexer.new
+indexer.rebuild("user-123")
+```
+
+## Environment Variables
+
+| Variable | Used By | Description |
+|----------|---------|-------------|
+| `S3ARCH_SOURCE_TABLE` | Indexer | DynamoDB source table name |
+| `S3ARCH_SOURCE_INDEX` | Indexer | GSI name for owner lookup (default: `UserIndex`) |
+| `S3ARCH_INDEX_BUCKET` | Both | S3 bucket for SQLite index files |
+| `S3ARCH_VERSION_TABLE` | Both | DynamoDB version tracking table |
+
+## Infrastructure
+
+A Terraform module is provided at [`terraform/`](./terraform/) that provisions:
+
+- S3 bucket for index storage (with 90-day lifecycle)
+- DynamoDB version tracking table (PAY_PER_REQUEST)
+- SQS queue + DLQ for the indexer
+- EventBridge Pipe (DynamoDB Stream → SQS)
+
+```hcl
+module "s3arch" {
+  source                  = "github.com/stowzilla/s3arch//terraform"
+  app_name                = "myapp"
+  environment             = "production"
+  source_table_name       = aws_dynamodb_table.items.name
+  source_table_arn        = aws_dynamodb_table.items.arn
+  source_table_stream_arn = aws_dynamodb_table.items.stream_arn
+}
+```
+
+Outputs include `indexer_env_vars`, `searcher_env_vars`, `indexer_permissions`, and `searcher_permissions` for easy Lambda configuration.
+
+## Configuration Options
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `source_table` | — | DynamoDB table with source records |
+| `source_index` | — | GSI name for querying by owner |
+| `owner_key` | `"user_id"` | Partition key field for owner lookup |
+| `index_bucket` | — | S3 bucket for SQLite files |
+| `version_table` | — | DynamoDB table for version tracking |
+| `searchable_fields` | `["name", "description"]` | Fields indexed in FTS5 |
+| `metadata_fields` | `["status", "created_at"]` | Fields stored for filtering |
+| `record_filter` | `->(_) { true }` | Proc to filter records during indexing |
+| `owner_extractor` | (extracts from DynamoDB stream image) | Proc to extract owner_id from stream event |
+| `version_ttl` | `30` (seconds) | How long to cache version checks |
+| `max_results` | `50` | Maximum search results returned |
+| `max_cached_dbs` | `20` | Max databases cached in `/tmp` |
+
+## How Search Works
+
+1. **Query arrives** with `owner_ids` and a search string
+2. For each owner, check DynamoDB version table (cached for `version_ttl` seconds)
+3. If version changed (or first request), download SQLite DB from S3 to `/tmp`
+4. Run FTS5 `MATCH` query with prefix matching (`term*`)
+5. Apply metadata filters, sort by rank, return record IDs
+6. LRU eviction when `/tmp` fills up
+
+## Requirements
+
+- Ruby >= 3.2
+- AWS Lambda with `/tmp` storage (recommend 2048MB ephemeral)
+- SQLite3 native extension (via Lambda layer)
+- DynamoDB table with streams enabled
+- S3 bucket for index storage
+
+## License
+
+MIT

data/lib/s3arch/configuration.rb

@@ -0,0 +1,42 @@
+# frozen_string_literal: true
+
+module S3arch
+  class Configuration
+    attr_accessor :source_table, :source_index, :owner_key, :index_bucket, :version_table,
+                  :searchable_fields, :metadata_fields, :record_filter, :owner_extractor,
+                  :logger, :version_ttl, :max_results, :max_cached_dbs, :ephemeral_storage_mb
+
+    def initialize
+      @owner_key = 'user_id'
+      @searchable_fields = %w[name description]
+      @metadata_fields = %w[status created_at]
+      @record_filter = ->(_record) { true }
+      @owner_extractor = ->(stream_record) {
+        image = stream_record.dig('dynamodb', 'NewImage') || stream_record.dig('dynamodb', 'OldImage') || {}
+        image.dig(owner_key, 'S')
+      }
+      @version_ttl = 30
+      @max_results = 50
+      @max_cached_dbs = 20
+      @ephemeral_storage_mb = 2048
+      @logger = nil
+    end
+
+    def from_env!
+      @source_table = ENV['S3ARCH_SOURCE_TABLE']
+      @source_index = ENV['S3ARCH_SOURCE_INDEX'] || 'UserIndex'
+      @index_bucket = ENV['S3ARCH_INDEX_BUCKET']
+      @version_table = ENV['S3ARCH_VERSION_TABLE']
+      self
+    end
+
+    def validate!
+      missing = []
+      missing << 'source_table' unless source_table
+      missing << 'index_bucket' unless index_bucket
+      missing << 'version_table' unless version_table
+      missing << 'source_index' unless source_index
+      raise Error, "S3arch configuration missing: #{missing.join(', ')}" if missing.any?
+    end
+  end
+end

data/lib/s3arch/handler.rb

@@ -0,0 +1,24 @@
+# frozen_string_literal: true
+
+module S3arch
+  module Handler
+    class << self
+      def indexer(event)
+        S3arch::Indexer.new.process_event(event)
+      end
+
+      def search(event)
+        query = event['query']
+        owner_ids = event['owner_ids'] || []
+        filters = event['filters'] || {}
+
+        return { record_ids: [], search_mode: 'fts5' } if query.nil? || owner_ids.empty?
+
+        searcher = S3arch::Searcher.new
+        result = searcher.search(query: query, owner_ids: owner_ids, filters: filters)
+
+        result || { record_ids: nil, search_mode: nil }
+      end
+    end
+  end
+end

data/lib/s3arch/indexer.rb

@@ -0,0 +1,123 @@
+# frozen_string_literal: true
+
+require 'aws-sdk-dynamodb'
+require 'aws-sdk-s3'
+require 'sqlite3'
+require 'json'
+
+module S3arch
+  class Indexer
+    def initialize(config: S3arch.configuration)
+      config.validate!
+      @config = config
+      @dynamodb = Aws::DynamoDB::Client.new
+      @s3 = Aws::S3::Client.new
+    end
+
+    def rebuild(owner_id)
+      records = fetch_records(owner_id)
+      db_path = "/tmp/s3arch_#{owner_id}.sqlite3"
+
+      build_database(db_path, records)
+      upload(owner_id, db_path)
+      increment_version(owner_id, records.size)
+
+      log(:info, 'Index rebuilt', owner_id: owner_id, record_count: records.size)
+    ensure
+      File.delete(db_path) if db_path && File.exist?(db_path)
+    end
+
+    def process_event(event)
+      records = event['Records'] || []
+      owner_ids = records.filter_map { |r|
+        body = JSON.parse(r['body'])
+        @config.owner_extractor.call(body)
+      }.uniq
+
+      log(:info, 'Rebuilding indexes', owner_ids: owner_ids, record_count: records.size)
+      owner_ids.each { |id| rebuild(id) }
+      { statusCode: 200, body: JSON.generate(rebuilt: owner_ids.size) }
+    end
+
+    private
+
+    def fetch_records(owner_id)
+      records = []
+      params = { table_name: @config.source_table, index_name: @config.source_index,
+                 key_condition_expression: "#{@config.owner_key} = :owner",
+                 expression_attribute_values: { ':owner' => owner_id } }
+
+      loop do
+        result = @dynamodb.query(params)
+        result.items.each { |item| records << item if @config.record_filter.call(item) }
+        break unless result.last_evaluated_key
+        params[:exclusive_start_key] = result.last_evaluated_key
+      end
+
+      records
+    end
+
+    def build_database(db_path, records)
+      File.delete(db_path) if File.exist?(db_path)
+      db = SQLite3::Database.new(db_path)
+
+      fts_cols = @config.searchable_fields.join(', ')
+      meta_cols = (['rowid INTEGER PRIMARY KEY', 'record_id TEXT NOT NULL'] +
+                   @config.metadata_fields.map { |f| "#{f} TEXT" }).join(', ')
+
+      db.execute_batch(<<~SQL)
+        CREATE VIRTUAL TABLE records_fts USING fts5(#{fts_cols}, content='');
+        CREATE TABLE records_meta (#{meta_cols});
+      SQL
+
+      db.transaction do
+        records.each_with_index do |record, idx|
+          rowid = idx + 1
+          fts_values = @config.searchable_fields.map { |f| normalize_field(record[f]) }
+          db.execute("INSERT INTO records_fts(rowid, #{fts_cols}) VALUES (#{(['?'] * (fts_values.size + 1)).join(', ')})",
+                     [rowid] + fts_values)
+
+          meta_values = [rowid, normalize_field(record['id'])] + @config.metadata_fields.map { |f| normalize_field(record[f]) }
+          placeholders = (['?'] * meta_values.size).join(', ')
+          db.execute("INSERT INTO records_meta(rowid, record_id, #{@config.metadata_fields.join(', ')}) VALUES (#{placeholders})",
+                     meta_values)
+        end
+      end
+
+      db.close
+    end
+
+    def normalize_field(value)
+      case value
+      when Hash
+        if value.key?('S') then value['S'].to_s
+        elsif value.key?('L') then value['L'].map { |v| v['S'] || v.to_s }.join(' ')
+        elsif value.key?('N') then value['N'].to_s
+        elsif value.key?('SS') then value['SS'].join(' ')
+        else value.values.first.to_s
+        end
+      when Array then value.map { |v| v.is_a?(Hash) ? (v['S'] || v.to_s) : v.to_s }.join(' ')
+      when nil then ''
+      else value.to_s
+      end
+    end
+
+    def upload(owner_id, db_path)
+      @s3.put_object(bucket: @config.index_bucket, key: "#{owner_id}/index.sqlite3", body: File.open(db_path, 'rb'))
+    end
+
+    def increment_version(owner_id, record_count)
+      @dynamodb.update_item(
+        table_name: @config.version_table,
+        key: { @config.owner_key => owner_id },
+        update_expression: 'SET version = if_not_exists(version, :zero) + :one, updated_at = :now, record_count = :count',
+        expression_attribute_values: { ':zero' => 0, ':one' => 1, ':now' => Time.now.iso8601, ':count' => record_count }
+      )
+    end
+
+    def log(level, message, **data)
+      return unless @config.logger
+      @config.logger.send(level, message, **data)
+    end
+  end
+end

data/lib/s3arch/searcher.rb

@@ -0,0 +1,137 @@
+# frozen_string_literal: true
+
+require 'aws-sdk-dynamodb'
+require 'aws-sdk-s3'
+require 'sqlite3'
+
+module S3arch
+  class Searcher
+    @version_cache = {}
+    @db_cache = {}
+
+    class << self
+      attr_accessor :version_cache, :db_cache
+
+      def reset!
+        @version_cache = {}
+        @db_cache = {}
+      end
+    end
+
+    def initialize(config: S3arch.configuration)
+      config.validate!
+      @config = config
+      @dynamodb = Aws::DynamoDB::Client.new
+      @s3 = Aws::S3::Client.new
+    end
+
+    def search(query:, owner_ids:, filters: {})
+      return nil if owner_ids.empty? || query.nil? || query.strip.empty?
+
+      log(:info, 'Search started', query: query, owner_ids: owner_ids, filters: filters)
+
+      results = []
+      owner_ids.each do |owner_id|
+        db = ensure_database(owner_id)
+        unless db
+          log(:info, 'No database available', owner_id: owner_id)
+          next
+        end
+        hits = query_fts(db, query, filters: filters)
+        log(:info, 'Owner search complete', owner_id: owner_id, hits: hits.size)
+        results.concat(hits)
+      end
+
+      log(:info, 'Search complete', total_results: results.size)
+      return nil if results.empty?
+
+      results.sort_by! { |r| r[:rank] }
+      { record_ids: results.first(@config.max_results).map { |r| r[:record_id] }, search_mode: 'fts5' }
+    end
+
+    private
+
+    def ensure_database(owner_id)
+      current_version = fetch_version(owner_id)
+      return nil unless current_version
+
+      cached = self.class.db_cache[owner_id]
+      if cached && cached[:version] == current_version
+        cached[:last_used] = Time.now
+        return cached[:db]
+      end
+
+      log(:info, 'Downloading database from S3', owner_id: owner_id, version: current_version)
+      db = download_database(owner_id)
+      return nil unless db
+
+      evict_lru if self.class.db_cache.size >= @config.max_cached_dbs
+      self.class.db_cache[owner_id] = { db: db, version: current_version, last_used: Time.now }
+      db
+    end
+
+    def fetch_version(owner_id)
+      cached = self.class.version_cache[owner_id]
+      if cached && (Time.now - cached[:checked_at]) < @config.version_ttl
+        return cached[:version]
+      end
+
+      result = @dynamodb.get_item(table_name: @config.version_table,
+                                  key: { @config.owner_key => owner_id },
+                                  projection_expression: 'version')
+      version = result.item&.dig('version')
+      self.class.version_cache[owner_id] = { version: version, checked_at: Time.now } if version
+      version
+    end
+
+    def download_database(owner_id)
+      db_path = "/tmp/s3arch_#{owner_id}.sqlite3"
+      @s3.get_object(bucket: @config.index_bucket, key: "#{owner_id}/index.sqlite3",
+                     response_target: db_path)
+      db = SQLite3::Database.new(db_path)
+      db.results_as_hash = true
+      db
+    rescue Aws::S3::Errors::NoSuchKey
+      nil
+    end
+
+    def query_fts(db, query, filters: {})
+      terms = query.strip.split(/\s+/).map { |t| "#{t.gsub('"', '')}*" }
+      match_expr = terms.join(' AND ')
+      return [] if match_expr.empty?
+
+      sql = <<~SQL
+        SELECT m.record_id, m.*, rank
+        FROM records_fts f
+        JOIN records_meta m ON m.rowid = f.rowid
+        WHERE records_fts MATCH ?
+        ORDER BY rank
+        LIMIT #{@config.max_results}
+      SQL
+
+      rows = db.execute(sql, [match_expr])
+      rows.filter_map do |row|
+        next if filters.any? { |field, value| row[field.to_s] != value }
+        { record_id: row['record_id'], rank: row['rank'] }
+      end
+    rescue SQLite3::Exception => e
+      log(:error, 'FTS5 query failed', error: e.message)
+      []
+    end
+
+    def evict_lru
+      oldest = self.class.db_cache.min_by { |_, v| v[:last_used] }
+      return unless oldest
+
+      owner_id, entry = oldest
+      entry[:db]&.close rescue nil
+      File.delete("/tmp/s3arch_#{owner_id}.sqlite3") rescue nil
+      self.class.db_cache.delete(owner_id)
+    end
+
+    def log(level, message, **data)
+      return unless @config.logger
+      @config.logger.send(level, message, **data)
+    end
+  end
+end

data/lib/s3arch/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module S3arch
+  VERSION = '0.0.1'
+end

data/lib/s3arch.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+require_relative 's3arch/version'
+require_relative 's3arch/configuration'
+require_relative 's3arch/indexer'
+require_relative 's3arch/searcher'
+require_relative 's3arch/handler'
+
+module S3arch
+  class Error < StandardError; end
+
+  class << self
+    def configuration
+      @configuration ||= Configuration.new
+    end
+
+    def configure
+      yield(configuration)
+    end
+
+    def reset_configuration!
+      @configuration = Configuration.new
+    end
+  end
+end

metadata

@@ -0,0 +1,96 @@
+--- !ruby/object:Gem::Specification
+name: s3arch
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Adam Dalton
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-dynamodb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: aws-sdk-s3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+description: Per-owner SQLite FTS5 indexes stored on S3, queried from Lambda /tmp.
+  DynamoDB source records are indexed via streams, with version tracking and LRU caching.
+email:
+- adam@stowzilla.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/s3arch.rb
+- lib/s3arch/configuration.rb
+- lib/s3arch/handler.rb
+- lib/s3arch/indexer.rb
+- lib/s3arch/searcher.rb
+- lib/s3arch/version.rb
+homepage: https://github.com/stowzilla/s3arch
+licenses:
+- MIT
+metadata:
+  rubygems_mfa_required: 'true'
+  homepage_uri: https://github.com/stowzilla/s3arch
+  source_code_uri: https://github.com/stowzilla/s3arch
+  changelog_uri: https://github.com/stowzilla/s3arch/blob/main/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.11
+specification_version: 4
+summary: SQLite FTS5 full-text search for DynamoDB on AWS Lambda
+test_files: []