portolan 0.1.0.pre.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9ddd564427e8a47bf2013520a1f2adb0de2970481c6c83f55f20ea4cd751c818
+  data.tar.gz: ffe366aa7c3f03197e814f68648d1463b54c57f155af0ffa0782495e58390c85
+SHA512:
+  metadata.gz: 1d696d9332f633dec4a9eb3231d538288c6aa17e387277cb4f75234b0d8f69f8a397e2dddd4019e3f0adb9e92616f5a4bcb919c3609d4cc24972f6176d022014
+  data.tar.gz: 96e1499eb19e2f94e7d16811553fa5fa29dc3d686c318a63c318b2c14015970846569794aa5e1d14b443a9952c361392b0167d4a0790b8d88985252a918a3a1d

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 David Busse
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,202 @@
+# Portolan
+
+Let LLMs and reporting tools query your Rails data safely: only the datasets,
+dimensions, and measures you declare.
+
+You declare each dataset in Ruby: the rows it covers, the dimensions it can be
+grouped and filtered by, the measures it can count or sum, and worked examples.
+A dashboard, a scheduled report, or an LLM sends a structured request naming a
+dataset and what to see from it. Portolan checks the request against your
+declaration, builds the SQL itself, and returns only what the declaration
+allows. Nothing outside your code ever writes SQL or reaches a column you
+didn't declare.
+
+Giving an LLM database access usually means trusting every query it writes.
+Portolan turns that around: it can only ask questions you have already made
+answerable.
+
+Portolan is in early development. Today it answers with aggregates: counts,
+sums, and averages, grouped by the dimensions you declared. It requires
+PostgreSQL.
+
+## Installation
+
+```ruby
+gem 'portolan', '~> 0.1.0.pre'
+```
+
+Portolan needs Ruby 3.3 or later, ActiveRecord 7.1 or later, and PostgreSQL.
+
+## Declaring a dataset
+
+A dataset is a class. It says which rows it covers, how they may be grouped and
+filtered, and what may be counted or summed.
+
+```ruby
+class Voyages < Portolan::Dataset
+  description 'One row per voyage.'
+
+  model Voyage
+  relation { |fleet| Voyage.where(fleet:) }
+
+  dimension :harbor, 'the harbor the voyage set out from'
+  dimension :crew, 'how many sailors were aboard', type: :integer
+  dimension :departed_at, 'when the ship set sail', type: :datetime
+  dimension :flag, 'the flag the ship sailed under', joins: :ship, expression: 'ships.flag'
+  dimension :fate, 'how the voyage ended', buckets: {
+    arrived: { where: -> { Voyage.where.not(arrived_at: nil) }, description: 'reached its destination' },
+    lost: -> { Voyage.where(lost: true) },
+  }
+
+  measure :voyages, 'number of voyages'
+  measure :ships, 'number of ships that sailed', count_distinct: :ship_id
+  measure :leagues, 'distance sailed, in leagues', sum: :leagues
+
+  example 'How many voyages set out each month in 1325, and how did they end?',
+    { measure: 'voyages', group_by: %w[departed_at fate], period: 'month',
+      filters: [{ dimension: 'departed_at', op: 'between', values: %w[1325-01-01 1325-12-31] }] }
+end
+```
+
+Register the datasets one audience may query:
+
+```ruby
+FLEET_DATASETS = Portolan::Registry.new(voyages: Voyages)
+```
+
+A registry checks every dataset when it is built, without touching the
+database, so most mistakes in a declaration fail at boot, not in front of a
+caller: a join through an association that doesn't exist, is a `has_many`, or
+is polymorphic, and an example that isn't a valid request. Columns need the
+database, so they are checked on the first query; add a spec that calls
+`FLEET_DATASETS.check!` to catch a misspelled column before you deploy. Without
+a `relation` block, a dataset covers every row of its model.
+
+## Asking a question
+
+```ruby
+result = FLEET_DATASETS.aggregate(
+  { dataset: 'voyages', measure: 'voyages', group_by: %w[fate], filters: [{ dimension: 'crew', op: 'gt', value: 20 }] },
+  context: current_user.fleet
+)
+
+result.success? # => true
+result.rows     # => [{ "fate" => "arrived", "voyages" => 12 }, { "fate" => "lost", "voyages" => 2 }]
+result.to_h     # => the request, rows, notes, and definitions, ready for JSON
+```
+
+A request may come from Ruby with symbols, from JSON, or from Rails params. A
+request with mistakes runs no SQL and returns every error at once, each saying
+what is allowed:
+
+```ruby
+result.errors
+# => ["unknown measure `ships_lost`; allowed: `voyages`, `ships`, `leagues`",
+#     "filter 1: `crew`: `twenty` is not an integer"]
+```
+
+To let an LLM ask questions, give it three tools: one that returns
+`registry.list`, one that returns `registry.describe(name)`, and one that
+returns `registry.aggregate(arguments, context:).to_h`. The descriptions,
+examples, and errors are written for a model to read and correct itself from.
+
+## What keeps it safe
+
+**The context comes from your authentication, never from the request.** The
+relation block receives whatever you pass as `context:`, such as the signed-in
+user or account, and uses it to scope rows. A request cannot name a context:
+a `context` key is rejected as unknown. Never pass anything from the request as
+the context.
+
+**One row per something.** Each dataset has a grain: one row per voyage, per
+order, per spool. Dimensions reach up from that row through `belongs_to`
+associations, never down through `has_many`, because joining a `has_many`
+repeats rows and inflates counts and sums; Portolan refuses such a join. To ask
+across a `has_many`, either declare a dataset at the finer grain, or use a
+bucket whose condition is a subquery:
+
+```ruby
+dimension :cargo, 'what the voyage carried', buckets: {
+  spices: -> { Voyage.where(id: Hold.where(kind: 'spice').select(:voyage_id)) },
+}
+```
+
+**Every query runs read-only, briefly, and in UTC.** Portolan runs each query in
+its own transaction with writes blocked, a 5 second timeout on each SQL
+statement, and the UTC time zone, then rolls it back. When Portolan opens the
+transaction itself, PostgreSQL enforces read-only too; inside a transaction of
+your own, Rails blocks writes. It only changes transaction-local settings, so
+nothing leaks into your app's queries, including behind a pooler such as
+PgBouncer in transaction mode. A query that times out, or a value the database
+cannot use, comes back as an error.
+
+**Values are always bound parameters.** Filter values never become SQL. A
+filter on a column behaves like a Rails `where` condition on it, including
+rounding a decimal to the column's scale. The only SQL in a query is what you
+wrote in `expression:` and in bucket conditions.
+
+## Reference
+
+**Dataset declarations:** `description` and `model` are required. `relation`
+is optional and receives the context; without it, a dataset covers
+`model.all`. A dataset declares at least one `measure`, and any number of
+`dimension`s and `example`s.
+
+**Dimension types**
+
+| Type | Operators | Periods |
+|---|---|---|
+| `:string` (default) | `eq`, `in` | |
+| `:integer` | `eq`, `in`, `gt`, `lt`, `between` | |
+| `:decimal` | `eq`, `in`, `gt`, `lt`, `between` | |
+| `:boolean` | `eq` | |
+| `:date` | `gt`, `lt`, `between` | `day`, `week`, `month`, `quarter`, `year` |
+| `:datetime` | `gt`, `lt`, `between` | `hour`, `day`, `week`, `month`, `quarter`, `year` |
+
+**Dimension options:** `type:`, `values:` (a fixed list for a string
+dimension), `joins:` (a `belongs_to` association path, joined with a left
+join, which needs an `expression:` naming the joined column), `expression:`
+(SQL to read instead of a column), and `buckets:` (named
+conditions, each a lambda returning a relation, with an optional description).
+
+**Measures:** a plain `measure` counts rows; `count_distinct:`, `sum:`, and
+`average:` take a column on the dataset's table.
+
+**What `aggregate` accepts:** `dataset` and `measure` are required.
+`group_by` takes up to three dimensions, at most one of them a date or
+datetime, which then needs a `period`. `filters` is a list of
+`{ dimension:, op:, value: }` (or `values:` for `in` and `between`); dates are
+`YYYY-MM-DD`, and a day given for a datetime covers the whole day in UTC.
+`limit` is 1 to 50 rows, and defaults to 50.
+
+**Periods** start on Monday for weeks, are in UTC, and are labelled by their
+first day (`2026-11-01`), or by the hour for `hour` (`2026-11-03T14:00Z`).
+Periods with no rows are left out. A missing value comes back as `null` (`nil`
+in Ruby).
+
+## Current limitations
+
+- PostgreSQL only.
+- A row can fall into more than one bucket if their conditions overlap.
+- Buckets need a model with a single-column primary key.
+- When a query joins the same table twice, Rails aliases the second join; an
+  `expression:` must name the table as Rails aliases it.
+- Dimensions cannot join through a `has_many` or a polymorphic association; see
+  the grain rule above.
+- The timeout applies to each SQL statement, not to the whole request. Grouping
+  by buckets runs one statement per bucket combination.
+- Your `relation` block runs before Portolan's guard is in place, so it should
+  build a relation, not run queries itself.
+- The timeout (5 seconds), row limit (50), group limit (10,000), and number of
+  bucket combinations (50) are fixed for now.
+
+## Why the name?
+
+A portolan is a medieval sailing chart of known coasts, harbors, and safe
+routes. Portolan does the same for your data: you chart the datasets worth
+visiting, and anything that queries them can go anywhere on the chart and
+nowhere off it.
+
+## License
+
+MIT. See [LICENSE.txt](LICENSE.txt).

data/lib/portolan/dataset.rb

@@ -0,0 +1,131 @@
+# frozen_string_literal: true
+
+module Portolan
+  class Dataset
+    Example = Data.define(:question, :query)
+
+    class << self
+      def description(text = nil)
+        text.nil? ? @description : @description = text
+      end
+
+      def model(model = nil)
+        model.nil? ? @model : @model = model
+      end
+
+      def relation(&block)
+        @relation = block
+      end
+
+      def relation_for(context)
+        check_columns!
+        relation = @relation ? @relation.call(context) : model.all
+        return relation if relation.is_a?(ActiveRecord::Relation) && relation.klass <= model
+
+        invalid!("relation must return a relation of `#{model.name}`")
+      end
+
+      def dimension(name, description, **options)
+        add(dimension_list, Dimension.new(name, description, **options), 'dimension')
+      end
+
+      def measure(name, description, **options)
+        add(measure_list, Measure.new(name, description, **options), 'measure')
+      end
+
+      def example(question, query)
+        @examples = nil
+        example_list << Example.new(question:, query: query.freeze)
+      end
+
+      def dimensions = @dimensions ||= dimension_list.dup.freeze
+      def measures = @measures ||= measure_list.dup.freeze
+      def examples = @examples ||= example_list.dup.freeze
+
+      def describe
+        {
+          description:,
+          dimensions: dimension_list.values.map(&:describe),
+          measures: measure_list.values.map(&:describe),
+          examples: example_list.map(&:to_h),
+        }
+      end
+
+      def check!
+        invalid!('needs a description') unless description.is_a?(String) && description.present?
+        invalid!('needs a model, an ActiveRecord class') unless model.is_a?(Class) && model < ActiveRecord::Base
+        invalid!('needs at least one measure') if measure_list.empty?
+        dimension_list.each_value { |dimension| check_join!(dimension, model, dimension.joins) if dimension.joins }
+      end
+
+      def check_columns!
+        return if @columns_checked
+
+        measure_list.each_value { |measure| check_measure_column!(measure) }
+        dimension_list.each_value { |dimension| check_dimension_column!(dimension) }
+        @columns_checked = true
+      end
+
+      private
+
+      def dimension_list = @dimension_list ||= {}
+      def measure_list = @measure_list ||= {}
+      def example_list = @example_list ||= []
+
+      def add(list, item, kind)
+        invalid!("#{kind} `#{item.name}` is declared twice") if list.key?(item.name)
+        invalid!("`#{item.name}` names both a dimension and a measure") if declared_names.include?(item.name)
+        @dimensions = @measures = nil
+        list[item.name] = item
+      end
+
+      def declared_names = dimension_list.keys + measure_list.keys
+
+      def check_measure_column!(measure)
+        return if measure.column.nil? || model.has_attribute?(measure.column)
+
+        invalid!("measure `#{measure.name}`: `#{measure.column}` is not a column of `#{model.name}`")
+      end
+
+      def check_dimension_column!(dimension)
+        return check_bucket_key!(dimension) if dimension.buckets?
+        return if dimension.expression || model.has_attribute?(dimension.name)
+
+        invalid!("dimension `#{dimension.name}`: `#{dimension.name}` is not a column of `#{model.name}`; " \
+                 'add an `expression` to read it another way')
+      end
+
+      def check_join!(dimension, from, path)
+        case path
+        when ::Hash then path.each { |name, rest| check_join!(dimension, check_join!(dimension, from, name), rest) }
+        when ::Array then path.each { |name| check_join!(dimension, from, name) }
+        else joined_model(dimension, from, path)
+        end
+      end
+
+      def joined_model(dimension, from, name)
+        reflection = from.reflect_on_association(name)
+        invalid!("dimension `#{dimension.name}`: `#{from.name}` has no association `#{name}`") unless reflection
+        if reflection.polymorphic?
+          invalid!("dimension `#{dimension.name}`: `#{name}` is polymorphic, and Rails cannot join it")
+        end
+        if reflection.collection?
+          invalid!("dimension `#{dimension.name}`: `#{name}` is a has_many, and joining it would count rows " \
+                   'more than once; see the grain rule')
+        end
+        reflection.klass
+      end
+
+      def check_bucket_key!(dimension)
+        return if model.primary_key.is_a?(::String)
+
+        invalid!("dimension `#{dimension.name}`: buckets need a model with a single-column primary key; " \
+                 "`#{model.table_name}` has #{Portolan.quoted(Array(model.primary_key))}")
+      end
+
+      def invalid!(problem)
+        raise ArgumentError, "dataset `#{name || 'anonymous'}`: #{problem}"
+      end
+    end
+  end
+end

data/lib/portolan/dimension/bucket.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module Portolan
+  class Dimension
+    class Bucket
+      OPTIONS = %i[where description].freeze
+
+      attr_reader :name, :where, :description
+
+      def initialize(name, options)
+        @name = name.to_s
+        options = { where: options } unless options.is_a?(Hash)
+        check!(options)
+        @where = options[:where]
+        @description = options[:description].presence
+        freeze
+      end
+
+      def apply(scope)
+        primary_key = scope.klass.primary_key
+        scope.where(primary_key => where.call.select(primary_key))
+      end
+
+      private
+
+      def check!(options)
+        unknown = options.keys - OPTIONS
+        invalid!("unknown option #{Portolan.quoted(unknown.map(&:inspect))}") if unknown.any?
+        invalid!('needs a `where` that responds to `call`') unless options[:where].respond_to?(:call)
+      end
+
+      def invalid!(problem)
+        raise ArgumentError, "bucket `#{name}`: #{problem}"
+      end
+    end
+  end
+end

data/lib/portolan/dimension.rb

@@ -0,0 +1,105 @@
+# frozen_string_literal: true
+
+module Portolan
+  class Dimension
+    OPTIONS = %i[type values joins expression buckets].freeze
+
+    attr_reader :name, :description, :type, :joins, :expression, :buckets
+
+    def initialize(name, description, **options)
+      @name = name.to_s
+      check_options!(options.keys, description)
+      @description = description
+      @joins = options[:joins]
+      @expression = options[:expression]
+      @buckets = build_buckets(options[:buckets])
+      @type = build_type(options)
+      invalid!('`joins` needs an `expression` naming the joined column') if joins && expression.nil?
+      freeze
+    end
+
+    def values = type.values
+    def operators = type.operators
+    def periods = type.periods
+    def time? = periods.any?
+    def buckets? = !buckets.nil?
+
+    def describe
+      {
+        name:,
+        type: type.name,
+        description:,
+        operators:,
+        values:,
+        periods: periods.presence,
+        buckets: bucket_descriptions.presence,
+      }.compact
+    end
+
+    def join(scope) = joins ? scope.left_joins(joins) : scope
+
+    def column(scope)
+      return scope.arel_table[name] unless expression
+
+      Arel::Nodes::Grouping.new(Arel.sql(expression))
+    end
+
+    def definition
+      { description:, buckets: bucket_descriptions.presence }.compact
+    end
+
+    private
+
+    def build_type(options)
+      type_name = options.fetch(:type, :string)
+      unless Type::NAMES.include?(type_name)
+        invalid!("unknown type `#{type_name.inspect}` (allowed: #{Type::NAMES.map(&:inspect).join(', ')})")
+      end
+      return bucket_type(type_name, options) if buckets?
+      return Type.for(type_name) if options[:values].nil?
+
+      invalid!('values must be a non-empty array') unless options[:values].is_a?(Array) && options[:values].any?
+      invalid!('values are only for `:string`') unless type_name == :string
+      Type::Enum.new(options[:values])
+    end
+
+    def bucket_type(type_name, options)
+      invalid!('buckets are always `:string`') unless type_name == :string
+      invalid!('buckets cannot have `values`, `joins`, or `expression`') if options[:values] || joins || expression
+      Type::Enum.new(buckets.keys)
+    end
+
+    def build_buckets(bucket_options)
+      return if bucket_options.nil?
+
+      invalid!('buckets must be a hash of names to conditions') unless bucket_options.is_a?(Hash)
+      invalid!('needs at least one bucket') if bucket_options.empty?
+      duplicate = bucket_options.keys.map(&:to_s).tally.find { |_, count| count > 1 }&.first
+      invalid!("bucket `#{duplicate}` is declared twice") if duplicate
+
+      bucket_options.to_h { |bucket_name, options| [bucket_name.to_s, build_bucket(bucket_name, options)] }.freeze
+    end
+
+    def build_bucket(bucket_name, options)
+      Bucket.new(bucket_name, options)
+    rescue ArgumentError => e
+      invalid!(e.message)
+    end
+
+    def bucket_descriptions
+      buckets.to_h.values.select(&:description).to_h { |bucket| [bucket.name, bucket.description] }
+    end
+
+    def check_options!(option_names, description)
+      raise ArgumentError, 'a dimension needs a name' if name.blank?
+
+      unknown = option_names - OPTIONS
+      invalid!("unknown option #{Portolan.quoted(unknown.map(&:inspect))}") if unknown.any?
+      invalid!('needs a description') unless description.is_a?(String) && description.present?
+    end
+
+    def invalid!(problem)
+      raise ArgumentError, "dimension `#{name}`: #{problem}"
+    end
+  end
+end

data/lib/portolan/measure.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module Portolan
+  class Measure
+    AGGREGATES = %i[count_distinct sum average].freeze
+
+    attr_reader :name, :description, :aggregate, :column
+
+    def initialize(name, description, **options)
+      @name = name.to_s
+      check!(options, description)
+      @description = description
+      @aggregate, @column = options.first || [:count, nil]
+      @column = @column&.to_s
+      freeze
+    end
+
+    def call(scope)
+      case aggregate
+      when :count then scope.count
+      when :count_distinct then scope.distinct.count(column.to_sym)
+      when :sum then scope.sum(column.to_sym)
+      when :average then scope.average(column.to_sym)
+      end
+    end
+
+    def describe = { name:, description: }
+    def label(value) = value.is_a?(::BigDecimal) ? value.to_f : value
+
+    private
+
+    def check!(options, description)
+      raise ArgumentError, 'a measure needs a name' if name.blank?
+
+      unknown = options.keys - AGGREGATES
+      invalid!("unknown option #{Portolan.quoted(unknown.map(&:inspect))}") if unknown.any?
+      invalid!('needs a description') unless description.is_a?(String) && description.present?
+      invalid!("takes one of #{Portolan.quoted(AGGREGATES)}, not several") if options.size > 1
+
+      options.each do |kind, column|
+        next if (column.is_a?(Symbol) || column.is_a?(String)) && column.present?
+
+        invalid!("`#{kind}` needs a column name")
+      end
+    end
+
+    def invalid!(problem)
+      raise ArgumentError, "measure `#{name}`: #{problem}"
+    end
+  end
+end