qwe 0.0.0

data/README.md

@@ -0,0 +1,134 @@
+# Qwe - pure ruby framework for stuff
+
+The stuff is mostly complex ruby objects living in RAM for a long time, accessed over DRb. Qwe provides database-like interface for managing that with a few extras, while attempting to impose as little bounds
+as possible on the way you write code.
+
+[Full Documentation](DOCS.md)
+
+## Features
+
+### Zero-config startup
+
+```
+bundle exec qwe
+```
+And you are ready to go.
+[Puma plugin](DOCS.md#Puma-plugin) is also available.
+
+### Object persistence
+
+```ruby
+id = Qwe::DB.create(Hash)
+```
+The hash is now accessible from any ruby process with `Qwe::DB[]`,
+ considering you know that id.
+
+```ruby
+Qwe::DB[id][:qwe] = 123
+```
+
+After a period of inactivity, or manually if configured,
+hash will get dumped on disk with `Marshal` and RAM freed.
+Once you call `Qwe::DB[id]` again, object gets un-marshalled and lives in RAM the same way.
+
+### Attributes API
+
+```ruby
+class Person
+  include Qwe::Mixins::Thing
+
+  attribute :age, min: 0, init: 18, convert: :to_i
+end
+```
+is equivalent to
+```ruby
+class Person
+  def initialize
+    @age = 18
+  end
+
+  attr_reader :age
+
+  def age=(val)
+    val = val.to_i
+    val = 0 if val < 0
+    @age = val
+  end
+end
+```
+
+Despite the fact that `attribute` method accepts many arguments and can be extended,
+performance of both variants is the same.
+
+### Transactions
+
+Every attribute change can be tracked, so you can scroll back and forth your object state
+throughout it's lifecycle. Changes are stored as ruby code in plaintext file.
+
+requirements.rb
+```ruby
+class Person
+  include Qwe::Mixins::Root
+
+  attribute :age, init: 0, convert: :to_i
+end
+```
+Server startup
+```
+qwe -r requirements.rb
+```
+ruby console
+```ruby
+id = Qwe::DB.create(:Person)
+Qwe::DB[id].age = 1
+Qwe::DB[id].age += 2
+Qwe::DB[id].age += 3
+puts Qwe::DB[id].record.commits
+```
+Outputs:
+```ruby
+self.instance_variable_set(:@age, 1)
+self.instance_variable_set(:@age, 3)
+self.instance_variable_set(:@age, 6)
+```
+
+While commit "logs" are not handcrafted ruby code, they are still readable.
+Most standard types are supported and subject to extension.
+
+# Requirements
+
+ruby 3+ is required, but since long-running workers greatly benefit from YJIT compiler, and compiler is greatly improved in subsequent releases, it is better to use latest ruby whenever possible.
+
+# Notes
+
+## Is it production ready?
+
+It is extracted from production and backported there, but your case may be different. Feedback is welcome.
+
+## Does it scale?
+
+It depends, but mostly yes.
+
+On MRI one object lives in one thread. Qwe can work with any amount of worker threads,
+but it's for increasing amount of simultaneously served objects.
+Other ruby implementations are not tested.
+
+The primary limitation and at the same time strength is the fact that "root" objects
+(the ones created with Qwe::DB.create) are fully loaded into RAM upon access.
+It makes working with them faster after a short startup delay, but does not allow to effectively perform SQL-like
+requests across multiple root objects.
+
+If root object are independent from each other, system scales easily
+by running independent Qwe servers.
+
+## Note on security
+
+`Qwe::DB[]` provides access to object from another thread with **DRb**, ruby gem from standard library.
+DRb does not provide any access control, and opens separate port for each object it serves.
+It is safe to run Qwe in docker container or other isolated network, and unsafe to run on virtual or physical device with all ports open.
+
+See [DRb docs](https://docs.ruby-lang.org/en/3.3/DRb.html#module-DRb-label-Security) for a more detailed explanation.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,22 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "etc"
+
+task :server do
+  pid = spawn("#{__dir__}/exe/qwe", "-p", "3229", "-d", Etc.systmpdir + "/qwe_db", "-r", "#{__dir__}/test/requirements.rb", "-s", "30", "--gc", "5", "-w", "#{__dir__}/lib")
+  Process.wait(pid)
+end
+
+Rake::TestTask.new(:test) do |t|
+  t.libs << "test"
+  t.libs << "lib"
+  t.test_files = FileList["test/**/test_*.rb"]
+end
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[test rubocop]

data/exe/qwe

@@ -0,0 +1,118 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+$LOAD_PATH.unshift File.expand_path("../lib", __dir__)
+
+require "qwe"
+require "optparse"
+
+args = {}
+opt_parser = OptionParser.new do |parser|
+  parser.banner = "Usage: qwe [options] - start server\n"
+
+  parser.on("-dDIR", "--dir=DIR", "Directory with persistent data") do |a|
+    args[:dir] = a
+  end
+
+  parser.on("-pPORT", "--port=PORT", "Server port") do |a|
+    args[:port] = a
+  end
+
+  parser.on("-wDIR", "--watch=DIR", "Restart server on changes in that directory. Can be passed multiple times") do |a|
+    args[:watch] ||= []
+    args[:watch].push a
+  end
+
+  parser.on("-rFILE", "--require=FILE", "--require_file=FILE", "This file will be required in worker threads") do |a|
+    args[:require_file] = File.expand_path(a)
+  end
+
+  parser.on("-iID", "--interactive=ID", "Launch interactive console bound to record #ID") do |a|
+    args[:interactive] = a.to_i
+  end
+
+  parser.on("-cCLASS", "--create=CLASS", "Connect to running server, create new record of class CLASS, and print its id") do |a|
+    args[:create] = a
+  end
+
+  parser.on("-tTHREADS", "--threads=THREADS", "Worker threads count. Defaults to the number of CPU cores") do |a|
+    args[:threads] = a.to_i
+  end
+
+  parser.on("--gc=SECONDS", "--gc_interval=SECONDS", "Call garbage collector (GC.start) every x seconds. Off by default") do |a|
+    args[:gc_interval] = a.to_i
+  end
+
+  parser.on("-sSECONDS", "--detach=SECONDS", "--detach_timeout=SECONDS", "Detach record after x seconds since creation or loading. 0 - disable, default 300.") do |a|
+    args[:detach_timeout] = a.to_i
+  end
+
+  parser.on("--no-jit", "--no_jit", "Disable JIT. It's on by default") do |a|
+    args[:no_jit] = true
+  end
+
+  parser.on("-h", "--help", "Print this help") do
+    puts parser
+    exit
+  end
+end
+opt_parser.parse!
+
+def spawn_server(args)
+  spawn(__FILE__, *((args.to_a.filter { |a| a[0] != :watch }).map { |a| "--#{a[0]}=#{a[1]}" }))
+end
+
+if args[:watch]
+  begin
+    require "listen"
+  rescue LoadError
+    puts "Can't load 'listen' gem"
+    exit 1
+  end
+  pid = spawn_server(args)
+  Process.detach(pid)
+
+  puts "Watching changes in #{args[:watch]}..."
+  Listen.to(*args[:watch]) do
+    Process.kill("INT", pid)
+    pid = spawn_server(args)
+    Process.detach(pid)
+    puts "\n\n"
+  end.start
+
+  trap :INT do
+    Process.kill("INT", pid)
+    Process.exit
+  end
+  sleep
+elsif args[:interactive]
+  require "irb"
+
+  Qwe::DB.connect("druby://localhost:#{args[:port]}") if args[:port]
+
+  if args[:interactive] > 0
+    obj = Qwe::DB[args[:interactive]]
+    puts "\e[0;32mQwe interactive prompt\e[0m, record \e[1;32m##{args[:interactive]}\e[0m, class \e[1;36m#{obj.record.obj_eval("self.class.to_s")}\e[0m"
+  else
+    obj = Qwe::DB.server.pick_worker
+    puts "\e[0;32mQwe interactive prompt\e[0m, worker \e[1;36m#{obj.uri}\e[0m, require #{obj.requirements}"
+
+    require obj.requirements
+  end
+
+  (obj.instance_exec { binding }).instance_exec do
+    irb(show_code: false)
+  rescue
+    # Older irb versions always print source code around call point
+    IRB.setup(source_location[0], argv: [])
+    workspace = IRB::WorkSpace.new(self)
+    binding_irb = IRB::Irb.new(workspace)
+    binding_irb.context.irb_path = File.expand_path(source_location[0])
+    binding_irb.run(IRB.conf)
+  end
+elsif args[:create]
+  Qwe::DB.connect("druby://localhost:#{args[:port]}") if args[:port]
+  puts Qwe::DB.create(args[:create].to_sym)
+else
+  Qwe::DB.serve(**args)
+end

data/lib/guerilla.rb

@@ -0,0 +1,83 @@
+# frozen_string_literal: true
+
+class BasicObject
+  def self.to_rb
+    name
+  end
+end
+
+class String
+  def to_rb
+    "'" + gsub("'", "\\\\'") + "'"
+  end
+end
+
+class Symbol
+  def to_rb
+    ":\"#{self}\""
+  end
+end
+
+class Hash
+  def to_rb
+    "{" + (to_a.map { |v| "#{v[0].to_rb} => #{v[1].to_rb}" }).join(", ") + "}"
+  end
+end
+
+class Array
+  def to_rb
+    "[" + (map { |v| v.to_rb }).join(", ") + "]"
+  end
+end
+
+class Time
+  def to_rb
+    "Time.at(#{to_r}r)"
+  end
+end
+
+class Float
+  def to_rb
+    if nan?
+      "Float::NAN"
+    elsif infinite?
+      "Float::INFINITY"
+    else
+      to_s
+    end
+  end
+end
+
+class Integer
+  alias_method :to_rb, :to_s
+end
+
+class Rational
+  def to_rb
+    to_s + "r"
+  end
+end
+
+class Vector
+  def to_rb
+    "Vector#{to_a.to_rb}"
+  end
+end
+
+class TrueClass
+  def to_rb
+    "true"
+  end
+end
+
+class FalseClass
+  def to_rb
+    "false"
+  end
+end
+
+class NilClass
+  def to_rb
+    "nil"
+  end
+end

data/lib/puma/plugin/qwe.rb

@@ -0,0 +1,80 @@
+require "puma/plugin"
+
+Puma::Plugin.create do
+  attr_reader :puma_pid, :qwe_pid, :log_writer
+
+  def start(launcher)
+    @log_writer = launcher.log_writer
+    @puma_pid = $$
+
+    in_background do
+      monitor_qwe
+    end
+
+    launcher.events.on_booted do
+      @qwe_pid = fork do
+        Thread.new { monitor_puma }
+        conf = Rails.root.join("config", "qwe.yml")
+        if File.exist?(conf)
+          log "Loading config #{conf}"
+          Qwe::DB.serve(**YAML.parse_file(conf).to_ruby(symbolize_names: true))
+        else
+          Qwe::DB.serve
+        end
+      end
+    end
+
+    launcher.events.on_stopped { stop_qwe }
+    launcher.events.on_restart { stop_qwe }
+  end
+
+  private
+
+  def stop_qwe
+    Process.waitpid(qwe_pid, Process::WNOHANG)
+    log "Stopping Qwe..."
+    Process.kill(:INT, qwe_pid) if qwe_pid
+    Process.wait(qwe_pid)
+  rescue Errno::ECHILD, Errno::ESRCH
+  end
+
+  def monitor_puma
+    monitor(:puma_dead?, "Detected Puma has gone away, stopping Qwe...")
+  end
+
+  def monitor_qwe
+    monitor(:qwe_dead?, "Detected Qwe has gone away, stopping Puma...")
+  end
+
+  def monitor(process_dead, message)
+    loop do
+      if send(process_dead)
+        log message
+        Process.kill(:INT, $$)
+        break
+      end
+      sleep 2
+    end
+  end
+
+  def qwe_dead?
+    if qwe_started?
+      Process.waitpid(qwe_pid, Process::WNOHANG)
+    end
+    false
+  rescue Errno::ECHILD, Errno::ESRCH
+    true
+  end
+
+  def qwe_started?
+    qwe_pid.present?
+  end
+
+  def puma_dead?
+    Process.ppid != puma_pid
+  end
+
+  def log(...)
+    log_writer.log(...)
+  end
+end

data/lib/qwe/attribute.rb

@@ -0,0 +1,193 @@
+# frozen_string_literal: true
+
+module Qwe
+  class Attribute
+    @@possible_params = {}
+
+    [:name, :reader, :writer].each do |k|
+      @@possible_params[k] = true
+      attr_accessor k
+    end
+
+    attr_accessor :klass, :params
+
+    def initialize(klass, name, *booleans, writer: nil, reader: nil, **params)
+      @klass = klass
+      @name = name
+      @params = params
+      raise "Parameter 'name' is reserved" if @params[:name]
+      @params[:name] = name
+      booleans.each { |b| @params[b] = true }
+
+      @params.each_key do |p|
+        raise "Unknown parameter #{p}" unless @@possible_params.include?(p)
+      end
+
+      @reader = if reader
+        if reader.is_a?(Proc)
+          reader
+        else
+          raise "Reader parameter should be a proc, #{reader.class} given"
+        end
+      else
+        Qwe::Function.new(klass, name)
+      end
+
+      @writer = Qwe::Function.new(klass, :"#{name}=")
+      if writer
+        if writer.is_a?(Proc)
+          klass.define_method(:"#{name}_writer", &writer)
+          @writer.stage("value = #{name}_writer(value)")
+        else
+          raise "writer parameter should be a Proc, #{writer.class} given"
+        end
+      end
+    end
+
+    @@extensions = []
+    def self.add(p = nil, &block)
+      if p
+        @@possible_params[p] = true
+        if block
+          @@extensions.push(proc { instance_exec(&block) if self[p] })
+        end
+      elsif block
+        @@extensions.push(block)
+      else
+        raise ArgumentError.new("Attribute.add - param or block required")
+      end
+    end
+
+    def compile
+      writer.arg(:value)
+
+      @@extensions.each do |block|
+        instance_exec(&block)
+      end
+
+      if reader.is_a?(Function)
+        reader.stage "@#{name}"
+        reader.compile
+      else
+        klass.define_method(name, &reader)
+      end
+
+      writer.stage("@#{name} = value")
+      writer.stage("root&.record&.commit(\"\#{to_rb}.instance_variable_set(:@#{name}, \#{value.to_rb})\\n\")")
+      writer.stage("value")
+      writer.compile
+    end
+
+    def method_missing(symbol, *args)
+      if @@possible_params.has_key?(symbol)
+        @params[symbol]
+      else
+        super
+      end
+    end
+
+    def respond_to_missing?(symbol, *args)
+      @@possible_params[symbol] || super
+    end
+
+    def [](key)
+      @params[key]
+    end
+
+    def inspect
+      "Attribute #{klass}.#{params}"
+    end
+
+    def init_or_default_stage(p)
+      if self[p].is_a?(Class)
+        "#{self[p]}.new"
+      else
+        klass.class_variable_set(:"@@#{name}_#{p}", self[p])
+        if self[p].is_a?(Proc)
+          "instance_exec(&@@#{name}_#{p})"
+        else
+          "@@#{name}_#{p}.clone"
+        end
+      end
+    end
+
+    add(:init) do
+      klass.init.stage "self.#{name} = " + init_or_default_stage(:init)
+    end
+
+    add(:default) do
+      reader.stage "self.#{name} = " + init_or_default_stage(:default) + " unless @#{name}"
+    end
+
+    add(:convert) do
+      writer.stage "value = value.#{convert}"
+    end
+
+    add(:min) do
+      klass.class_variable_set(:"@@#{name}_min", convert && min.send(convert) || min)
+      writer.stage "value = @@#{name}_min.clone if value < @@#{name}_min"
+    end
+
+    add(:max) do
+      klass.class_variable_set(:"@@#{name}_max", convert && max.send(convert) || max)
+      writer.stage "value = @@#{name}_max.clone if value > @@#{name}_max"
+    end
+
+    add(:array) do
+      writer.stage "value = Qwe::Attribute::ArrayProxy.from_array(self, :#{name}, value)"
+      klass.init.stage("self.#{name} = []") unless init
+    end
+
+    add(:enum) do
+      raise "#{klass}.#{name} - enum #{enum} doesn't respond to include?" unless enum.respond_to?(:include?)
+      klass.class_variable_set(:"@@#{name}_enum", enum)
+      writer.stage "raise \"Enum - unknown value \#{self}.#{name} = \#{value}\" unless @@#{name}_enum.include?(value)"
+    end
+
+    class ArrayProxy < Array
+      attr_accessor :__qwe_thing
+      attr_accessor :__qwe_name
+
+      include DRb::DRbUndumped
+
+      undef _dump
+
+      def __qwe_commit(m, *a, **k)
+        __qwe_thing&.root&.record&.commit("#{__qwe_thing.to_rb}.#{__qwe_name}.method(:#{m}).super_method.call(*#{a.to_rb}, **#{k.to_rb})\n")
+      end
+
+      TRANSACTION_METHODS = [:<<, :[]=, :clear, :compact!, :concat, :delete_at, :flatten!, :insert, :pop, :push, :replace, :reverse!, :rotate!, :shift, :shuffle!, :slice!, :unshift]
+      REPLACE_METHODS = [:delete, :delete_if, :fill, :filter!, :select!, :keep_if, :map!, :reject!, :sort!, :sort_by!, :uniq!]
+
+      TRANSACTION_METHODS.each do |m|
+        define_method(m) do |*a, **k|
+          __qwe_commit(m, *a, **k)
+          super(*a, **k)
+        end
+      end
+
+      REPLACE_METHODS.each do |m|
+        define_method(m) do |*a, **k, &block|
+          r = super(*a, **k, &block)
+          if block
+            __qwe_thing&.root&.record&.commit "#{__qwe_thing.to_rb}.instance_variable_set(:@#{__qwe_name}, #{to_rb})\n"
+          else
+            __qwe_commit(m, *a, **k)
+          end
+          r
+        end
+      end
+
+      def self.from_array(thing, name, arr = [])
+        a = new(arr)
+        a.__qwe_thing = thing
+        a.__qwe_name = name
+        a
+      end
+
+      def to_rb
+        "Qwe::Attribute::ArrayProxy.from_array(#{__qwe_thing.to_rb}, :#{__qwe_name}, #{super})"
+      end
+    end
+  end
+end

data/lib/qwe/db/commits_file.rb

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+require "zstds"
+require "fileutils"
+
+module Qwe::DB
+  class CommitsFile
+    include DRb::DRbUndumped
+
+    attr_reader :path, :archive_path, :file
+
+    def initialize(dir)
+      @path = File.join(dir, "commits")
+      @archive_path = File.join(dir, "commits.zst")
+      unless File.exist?(@archive_path)
+        @file = File.open(@path, "a+")
+      end
+    end
+
+    def write(str)
+      unless file
+        log "Write #{str} to archived commits file #{path}, unpacking"
+        ZSTDS::File.decompress archive_path, path
+        FileUtils.rm_f(archive_path)
+        @file = File.open(@path, "a+")
+      end
+      file.write(str)
+    rescue => e
+      log "Couldn't write #{str} to #{path}: #{e.full_message}"
+      0
+    end
+
+    def read(line = nil)
+      if file
+        file.flush
+        file.rewind
+        if line
+          lines = file.readlines
+          lines[0, line].join("\n")
+        else
+          file.read
+        end
+      else
+        cs = ZSTDS::Stream::Reader.open(archive_path) do |r|
+          r.set_encoding("UTF-8")
+          r.read
+        end
+        if line
+          cs.lines[0, line].join("")
+        else
+          cs
+        end
+      end
+    end
+
+    def archive!
+      file&.close
+      ZSTDS::File.compress path, archive_path
+      FileUtils.rm_f(path)
+      @file = nil
+    end
+
+    def flush
+      file&.flush
+    end
+
+    def close
+      file&.close
+    end
+  end
+end