typisch 0.1.5

data/README.md

@@ -0,0 +1,55 @@
+# Typisch
+
+A schema language / type system / validation framework, for semi-structured data and for data in dynamic languages.
+
+The initial implementation is in Ruby, but the aim is that it be capable of describing data equally well in a range of dynamic languages
+and semi-structured data serialization languages -- Ruby, Python, Javascript, YAML, JSON, ...
+
+*N.B.* still a work in progress at this stage, but enough works that it may be interesting to poke around!
+
+## What distinguishes it?
+
+It aims to be more than just an ad-hoc data validation library; rather, a proper type system with formal foundations.
+
+This means that it's able to do more than just validate data; it can settle questions phrased in terms of the
+schemas themselves -- like "is this schema a subtype of this other schema?", or "compute the intersection of
+these two schemas" -- in a well-defined and sound way.
+
+This is nice, since if you're going to bother specifying a schema for your data at all (as opposed to just
+writing some validation code), you're probably doing so because you want to be able to reason statically
+about the structure of the data. Having solid foundations makes that easier and more pleasant.
+
+## As a type system, it features
+
+- Record types with structural subtyping
+- Nominal subtyping based on a hierarchy of type tags (which can be based on the subclassing graph of the host language)
+- Tagged union types (arbitrary untagged unions may be computed too, but some type information may be lost where the tags overlap)
+- Equi-recursive types, eg "Person {parent: Person}", which can be used to type-check cyclic object graphs if required
+- Parameterised polymorphic types for Sequences, Tuples and other collection types
+- A numeric tower with subtyping for the primitive numeric types
+- Refinement types like "Integer greater than 0", "String of at most 10 characters", "Float from the following set of allowed values" etc
+- Decidable subtyping for all the above
+- Ability to compute unions and intersections of types
+
+If that sounds surprisingly powerful, bear in mind there's one very common type system feature which it *lacks*: function types, or typing
+of code. Typisch only cares about typing data, which makes its life significantly easier.
+
+Usually type systems for data are called 'schema languages', their types 'schemas', and type-checking 'validation'.
+Forgive me if I use these terms somewhat interchangeably here.
+
+## Semi-structured data and subtyping
+
+One way to characterise semi-structured data would be: data whose datatype admits structural subtyping.
+
+Structural subtyping allows extra fields beyond those specifically required, to be present on an object without cause for concern
+-- as may frequently be the case with "duck-typed" data in dynamic languages, and data serialised in extensible schemas in formats
+like JSON.
+
+Sometimes you only care to validate, to serialize or to process a *subset* of a large structured object graph.
+Structural typing provides a rather nice way to describe these subsets, as *supertypes* of the more complete datatype.
+
+So, a good notion of subtyping seems useful for a type system designed to cope well with semi-structured data.
+
+## Why Typisch?
+
+Well, it combines Type and Schema. It's also german for "typical", as in "typical, another bloody schema language".

data/lib/typisch/boolean.rb

@@ -0,0 +1,13 @@
+module Typisch
+  class Type::Boolean < Type::Constructor::Singleton
+    def self.tag
+      "Boolean"
+    end
+
+    def shallow_check_type(instance)
+      instance == true || instance == false
+    end
+
+    Registry.register_global_type(:boolean, top_type)
+  end
+end

data/lib/typisch/constructor.rb

@@ -0,0 +1,112 @@
+module Typisch
+  # All types except Top, Bottom and Unions (which may necessarily involve more than one constructor type) have a
+  # tag associated with them which is used at runtime to distinguish its instances somewhat from instances
+  # of other types.
+  #
+  # This is an abstract superclass; each subclass of Type::Constructor is assumed to implement its own, distinct
+  # lattice of types. For simple atomic types like Bool, there will only be one tag, "Bool", in that lattice.
+  #
+  # While the type lattices of different subclass of Type::Constructor are non-overlapping, within a subclass
+  # (such as Type::Object or Type::Numeric) there may be a non-trivial type lattice, eg for Numeric,
+  # Int < Float, and for Object, the type lattice is based on a nominal tag inheritance hierarchy in
+  # the host language together with structural subtyping rules for object properties.
+  #
+  # A list of 'reserved tags' is maintained globally, and any Type::Constructor subtype which allows custom
+  # user-specified tags to be used should ensure that they don't match any reserved tags.
+  class Type::Constructor < Type
+    CONSTRUCTOR_TYPE_SUBCLASSES = []
+
+    class << self
+      def inherited(subclass)
+        # we add any non-abstract subclasses to this list, which is used to
+        # construct the Any type. For now Constructor::Singleton is the only
+        # other abstract Type::Constructor subclass:
+        unless subclass == Type::Constructor::Singleton
+          CONSTRUCTOR_TYPE_SUBCLASSES << subclass
+        end
+        super
+      end
+
+      # This should be the top in the type lattice for this class of taged types.
+      # Its tag should be the top_tag above.
+      # You are passed the overall_top, ie the top type of the overall type lattice,
+      # to use; this is needed by parameterised types which want to parameterise their
+      # top type by the overall top, eg Top = Foo | Bar | Sequence[Top] | ...
+      def top_type(overall_top)
+        raise NotImplementedError
+      end
+
+      # This gets called by the subtyper on a Type::Constructor subclass, with two instances of
+      # that subclass.
+      # It should return true or false; if it needs to check some subgoals,
+      # say on child types of the ones passed in, it should use the supplied
+      # 'recursively_check_subtype' block rather than calling itself recursively
+      # directly. This hides away the details of the corecursive subtyping algorithm
+      # for you.
+      def check_subtype(x, y, &recursively_check_subtype)
+        raise NotImplementedError
+      end
+    end
+
+    # the distinct type lattice within which this type lives.
+    # the type system as a whole can be seen as a set of non-overlapping type lattices, together
+    # with tagged unions drawn from them.
+    #
+    # the interface for a type lattice is just that it responds to 'check_subtype'; by default
+    # the class of a type implements this interface
+    def type_lattice
+      self.class
+    end
+
+    def check_type(instance)
+      shallow_check_type(instance)
+    end
+
+    # the tag of this particular type
+    def tag
+      raise NotImplementedError
+    end
+
+    # these are here so as to implement a common interface with Type::Union
+    def alternative_types
+      [self]
+    end
+
+    # A class of constructor type of which there is only one type, and
+    # hence only one tag.
+    #
+    # Will have no supertype besides Any, and no subtype besides
+    # Nothing.
+    #
+    # (abstract superclass; see Boolean or Null for example subclasses).
+    class Singleton < Type::Constructor
+      class << self
+        private :new
+
+        def tag
+          raise NotImplementedError
+        end
+
+        def top_type(*)
+          @top_type ||= new
+        end
+
+        def check_subtype(x, y)
+          true
+        end
+      end
+
+      def subexpression_types
+        []
+      end
+
+      def tag
+        self.class.tag
+      end
+
+      def to_s(*)
+        @name.inspect
+      end
+    end
+  end
+end

data/lib/typisch/datetime.rb

@@ -0,0 +1,41 @@
+require 'date'
+require 'time'
+
+module Typisch
+  class Type::Date < Type::Constructor::Singleton
+    def self.tag; "Date"; end
+    Registry.register_global_type(:date, top_type)
+
+    # You could add your own to the list, if you have some alternative ruby implementation
+    # with a Date-like interface which you want to typecheck.
+    VALID_IMPLEMENTATION_CLASSES = [::Date]
+
+    def shallow_check_type(instance)
+      case instance when *VALID_IMPLEMENTATION_CLASSES then true else false end
+    end
+  end
+
+  class Type::Time < Type::Constructor::Singleton
+    def self.tag; "Time"; end
+    Registry.register_global_type(:time, top_type)
+    Registry.register_global_type(:datetime, top_type)
+
+    # You could add your own to the list, if you have some alternative ruby implementation
+    # with a Time-like interface which you want to typecheck here.
+    #
+    # Maybe allow DateTime too under ruby? its interface is slightly different though.
+    # Typisch types are aiming to not be overly coupled to the structure of implementation
+    # classes in Ruby though - eg when serializing to JSON we don't really care about
+    # ruby's DateTime vs Time quirks. So, not sure whether to add DateTime here and pretend
+    # its interface is similar enough to that of Time, or just pretend it doesn't exist.
+    #
+    # Or maybe we could just define this as 'anything which respond_to?(:to_time)' or similar.
+    # Although annoyingly DateTime and Time don't even have to_time / to_datetime methods to
+    # convert between them. Poor stdlib design :(
+    VALID_IMPLEMENTATION_CLASSES = [::Time]
+
+    def shallow_check_type(instance)
+      case instance when *VALID_IMPLEMENTATION_CLASSES then true else false end
+    end
+  end
+end

data/lib/typisch/dsl.rb

@@ -0,0 +1,242 @@
+module Typisch
+
+  # Apologies this is a bit messy. Should probably do a bit of a tidy-up
+  # once the dust has settled around the DSL syntax.
+  #
+  # It's a layer ontop of the core type model though - could be worse, it
+  # could be horribly intertwined with the model itself :)
+
+  module DSL
+    def registry
+      raise NotImplementedError
+    end
+
+    def register(name, *type_args, &type_block_arg)
+      result = registry[name] = type(*type_args, &type_block_arg)
+      if @pending_annotations
+        result.annotations.merge!(@pending_annotations)
+        @pending_annotations = nil
+      end
+      result
+    end
+
+    def register_type_for_class(klass, *object_type_args, &object_type_block_arg)
+      name = :"#{klass}"
+      type = register(name, :object, klass, *object_type_args, &object_type_block_arg)
+      registry.register_type_for_class(klass, type)
+      type
+    end
+
+    def register_version_type_for_class(klass, version, *object_type_args, &object_type_block_arg)
+      name = :"#{klass}__#{version}"
+      type = register(name, :object, klass, *object_type_args, &object_type_block_arg)
+      registry.register_version_type_for_class(klass, version, type)
+      type
+    end
+
+    # annotations apply to the next register'd type.
+    #
+    # annotate "Some description", :some_other => 'annotations'
+    # annotate :description => "Some description", :some_other => 'annotations'
+    def annotate(description_or_options, options=nil)
+      if description_or_options.is_a?(::String)
+        options ||= {}; options[:description] = description_or_options
+      else
+        options = description_or_options
+      end
+      @pending_annotations ||= {}
+      @pending_annotations.merge!(options)
+    end
+
+    def type(arg, *more_args, &block_arg)
+      case arg
+      when Type
+        arg
+      when ::Symbol
+        if more_args.empty? && !block_arg
+          registry[arg]
+        elsif more_args.last.is_a?(::Hash) && !block_arg && (version = more_args.last[:version])
+          registry[:"#{arg}__#{version}"]
+        else
+          send(arg, *more_args, &block_arg)
+        end
+      when ::Module
+        type(:"#{arg}", *more_args, &block_arg)
+      else
+        raise ArgumentError, "expected Type or type name or class, but was given #{arg.class}"
+      end
+    end
+
+    def string(refinements=nil)
+      refinements ? Type::String.new(refinements) : registry[:string]
+    end
+
+    def sequence(*args)
+      seq_options = {}
+      if (opts = args.last and opts.is_a?(::Hash))
+        seq_options[:slice] = opts.delete(:slice) if opts.has_key?(:slice)
+        seq_options[:total_length] = opts.delete(:total_length) if opts.has_key?(:total_length)
+        args.pop if opts.empty?
+      end
+      Type::Sequence.new(type(*args), seq_options)
+    end
+
+    def tuple(*types)
+      Type::Tuple.new(*types.map {|t| type(t)})
+    end
+
+    def object(*args, &block)
+      klass, properties = _normalize_object_args(::Object, *args)
+      _object(klass, properties, &block)
+    end
+
+    def _normalize_object_args(default_class, klass_or_properties=nil, properties=nil)
+      case klass_or_properties
+      when ::Hash     then [default_class, klass_or_properties]
+      when ::NilClass then [default_class, {}]
+      when ::Module   then [klass_or_properties, properties || {}]
+      end
+    end
+
+    # back-end for object, which takes args in a normalized format
+    def _object(klass, properties, derive_from=nil, &block)
+      if block
+        object_context = if derive_from
+          DerivedObjectContext.new(self, derive_from)
+        else
+          ObjectContext.new(self)
+        end
+        object_context.instance_eval(&block)
+        properties.merge!(object_context.properties)
+      end
+      properties.keys.each do |k|
+        type_args, type_block_arg = properties[k]
+        properties[k] = type(*type_args, &type_block_arg)
+      end
+      type = Type::Object.new(klass.to_s, properties)
+      if block && (prop_annot = object_context.property_annotations)
+        type.annotations[:properties] = prop_annot
+      end
+      type
+    end
+
+    def union(*types)
+      Type::Union.new(*types.map {|t| type(t)})
+    end
+
+    def nullable(t)
+      union(type(t), :null)
+    end
+
+    def derived_from(original_type, *args, &block_arg)
+      if args.empty? && !block_arg
+        original_type
+      elsif args.last.is_a?(::Hash) && (version = args.last[:version]) && original_type.name
+        # slightly messy; we rely on the convention that 'versions' of named types are registered
+        # as :"#{name}__#{version}", this allows you to specify or override the version when
+        # deriving from a type, even if it is still just a named placeholder, by manipulating the
+        # name.
+        original_name = original_type.name.to_s.sub(/__.*$/, '')
+        :"#{original_name}__#{version}"
+      else
+        original_type = type(original_type).target
+        case original_type
+        when Type::Object
+          klass, properties = _normalize_object_args(original_type.class_or_module, *args)
+          _object(klass, properties, original_type, &block_arg)
+        when Type::Sequence
+          slice_overrides = (args.last.is_a?(::Hash) && args.last[:slice]) ? args.pop : {}
+          Type::Sequence.new(derived_from(original_type.type, *args, &block_arg), slice_overrides)
+        when Type::Union
+          non_null = original_type.excluding_null
+          raise "DSL doesn't support deriving from union types (except simple unions with null)" if Type::Union === non_null
+          nullable(derived_from(non_null, *args, &block_arg))
+        else
+          raise "DSL doesn't support deriving from #{original_type.class} types" if args.length > 0 || block_arg
+          original_type
+        end
+      end
+    end
+
+    class ObjectContext
+      attr_reader :properties, :property_annotations
+
+      def initialize(parent_context)
+        @parent_context = parent_context
+        @properties = {}
+      end
+
+      # property annotations apply to the next declared property.
+      #
+      # annotate "Some description", :some_other => 'annotations'
+      # annotate :description => "Some description", :some_other => 'annotations'
+      def annotate(description_or_options, options=nil)
+        if description_or_options.is_a?(::String)
+          options ||= {}; options[:description] = description_or_options
+        else
+          options = description_or_options
+        end
+        @pending_annotations ||= {}
+        @pending_annotations.merge!(options)
+      end
+
+      def property(name, *type_args, &type_block_arg)
+        raise Error, "property #{name.inspect} declared twice" if @properties[name]
+        @properties[name] = [type_args, type_block_arg]
+        if @pending_annotations
+          @property_annotations ||= {}
+          @property_annotations[name] = @pending_annotations
+          @pending_annotations = nil
+        end
+      end
+
+      def method_missing(name, *args, &block)
+        @parent_context.respond_to?(name) ? @parent_context.send(name, *args, &block) : super
+      end
+    end
+
+    class DerivedObjectContext < ObjectContext
+      def initialize(c, original_object_type)
+        super(c)
+        @original_object_type = original_object_type
+      end
+
+      def derive_properties(*names)
+        names.each {|n| derive_property(n)}
+      end
+
+      # use a property from the original object type being derived from
+      def derive_property(name, *derive_args, &derive_block)
+        type = @original_object_type[name] or raise "use_property: no such property #{name.inspect} on the original type"
+        derived_type = derived_from(type, *derive_args, &derive_block)
+        property name, derived_type
+      end
+
+      # derives all properties from the original type which haven't already been derived.
+      # this is done for you by Typisch::Typed::ClassMethods#register_subtype.
+      def derive_all_properties
+        @original_object_type.property_names_to_types.each do |name, type|
+          property(name, type) unless @properties[name]
+        end
+      end
+
+      def derive_all_properties_except(*props)
+        @original_object_type.property_names_to_types.each do |name, type|
+          next if props.include?(name)
+          property(name, type) unless @properties[name]
+        end
+      end
+    end
+  end
+
+  class DSLContext
+    include DSL
+
+    attr_reader :registry
+
+    def initialize(registry)
+      @registry = registry
+    end
+  end
+
+end

data/lib/typisch/errors.rb

@@ -0,0 +1,11 @@
+module Typisch
+  class Error < StandardError; end
+  class TypeDeclarationError < Error; end
+  class NameResolutionError < TypeDeclarationError
+    def initialize(type_name)
+      super("Problem resolving named placeholder type: cannot find type with name #{type_name} in registry")
+    end
+  end
+  class SerializationError < Error; end
+  class CyclicSerialization < SerializationError; end
+end

data/lib/typisch/meta.rb

@@ -0,0 +1,57 @@
+module Typisch
+  META_TYPES = Registry.new
+  META_TYPES.register do
+
+    register(:"Typisch::Type", :union,
+      :"Typisch::Type::String",
+      :"Typisch::Type::Numeric",
+      :"Typisch::Type::Boolean",
+      :"Typisch::Type::Null",
+      :"Typisch::Type::Date",
+      :"Typisch::Type::Time",
+      :"Typisch::Type::Object",
+      :"Typisch::Type::Sequence",
+      :"Typisch::Type::Tuple",
+      :"Typisch::Type::Union"
+    )
+
+    register_type_for_class(Type::Boolean)
+    register_type_for_class(Type::Null)
+    register_type_for_class(Type::Date)
+    register_type_for_class(Type::Time)
+
+    register_type_for_class(Type::Numeric) do
+      property :tag, :string
+    end
+
+    register_type_for_class(Type::String) do
+      property :values,     nullable(sequence(:string))
+      property :max_length, nullable(:integer)
+    end
+
+    register_type_for_class(::Range) do
+      property :begin, :integer
+      property :end,   :integer
+    end
+
+    register_type_for_class(Type::Sequence) do
+      property :type,         :"Typisch::Type"
+      property :slice,        nullable(:Range)
+      property :total_length, nullable(:boolean)
+    end
+
+    register_type_for_class(Type::Tuple) do
+      property :types, sequence(:"Typisch::Type")
+    end
+
+    register_type_for_class(Type::Object) do
+      property :property_names_to_types, sequence(tuple(:string, :"Typisch::Type"))
+      property :tag, :string
+    end
+
+    register_type_for_class(Type::Union) do
+      property :alternative_types, sequence(:"Typisch::Type")
+    end
+  end
+
+end

data/lib/typisch/named_placeholder.rb

@@ -0,0 +1,67 @@
+module Typisch
+  # This is a proxy wrapper for a type, which we can use as a placeholder for a named
+  # type which hasn't yet been declared. Helps when it comes to cyclic references etc.
+  #
+  # (You can view this as a free variable, where the scope of all free variables is
+  #  implicitly closed over at the top level by the 'registry'. We don't keep variables
+  #  lying around as symbolic things in a syntax tree though, we're just using them as
+  #  temporary placeholders on the way to rewriting it as a syntax *graph*).
+  #
+  # Once a register_types block has finished, the registry ensures that all references
+  # in the type graph to NamedPlaceholders are replaced with references to their targets.
+  class Type::NamedPlaceholder < Type
+    def initialize(name, registry)
+      @registry = registry
+      @name = name
+    end
+
+    def target
+      return @target if @target
+      @target = @registry[@name]
+      case @target when NilClass, Type::NamedPlaceholder
+        raise NameResolutionError.new(@name.inspect)
+      end
+    end
+
+    attr_writer :target
+
+    def target=(target)
+      @target = target.target
+    end
+    private :target=
+
+    # this is slightly naughty - we actually pretend to be of the class
+    # of our target object.
+    #
+    # note that TargetClass === placeholder will still return false.
+
+    def class
+      target.class
+    end
+
+    def is_a?(klass)
+      target.is_a?(klass)
+    end
+    alias :kind_of? :is_a?
+
+    def instance_of?(klass)
+      target.instance_of?(klass)
+    end
+
+    def to_s(*)
+      @name.inspect
+    end
+
+    # let us proxy these through
+    undef :alternative_types, :check_type, :shallow_check_type, :subexpression_types
+    undef :excluding_null, :annotations, :canonicalize!
+
+    def method_missing(name, *args, &block)
+      target.respond_to?(name) ? target.send(name, *args, &block) : super
+    end
+
+    def respond_to?(name, include_private=false)
+      super || target.respond_to?(name, include_private)
+    end
+  end
+end

data/lib/typisch/null.rb

@@ -0,0 +1,17 @@
+module Typisch
+  class Type::Null < Type::Constructor::Singleton
+    def self.tag
+      "Null"
+    end
+
+    def shallow_check_type(instance)
+      instance.nil?
+    end
+
+    def excluding_null
+      Type::Nothing::INSTANCE
+    end
+
+    Registry.register_global_type(:null, top_type)
+  end
+end