dgd-tools 0.1.7 → 0.1.12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2b3639fa674b53796ffe654f0e7e0deb9506868eece4550b99b4f28b98940e97
-  data.tar.gz: b0f115747b8f790ba079a9789d18e5bbe024de6e9b1c6cc1c476d21066a1b48a
+  metadata.gz: ed7c35c57fad646f3083312c2c4a0d3b2ed1dcc9fa0d7f95f4b2c89d49ddadba
+  data.tar.gz: 8124dbf072f51dfd84ba1bde82fe60e3521d20b7eb53ef8dc413b8d31f78a57b
 SHA512:
-  metadata.gz: c739260975e503e32a44bfeaf4775d6693e606f35a247f6d432008d0b9e813ac7b29208d1db641dee46be7aec8bbb86e92c1f8fe7e9c7b1c2aadb2ce93e4fb8e
-  data.tar.gz: 50cc3537969a52ca379857a8936b900212e18567bdc10c1cb1152cffb7901f88db53b75a04126b8848e0edb3151d74a4514cc3fe02d7c1ae1f0ece0cbbe7d6f3
+  metadata.gz: 0db04c0632d1ee81659be61f9d5341602f1112a148577b32f89fe9d327ec607910d979e2843cd3b6466269fdb7bb11ddd1c5eda29a111f019e6e753d5619c7bb
+  data.tar.gz: 0b888a3f260a47cf77d3024555cdb38322b646ce016511a5edc7331ae6b0e43852c6f698e2fdf0a022071a763e31ebdfbb22e30d9cdc7a673036b8d549d6ea28

data/Gemfile.lock

@@ -1,9 +1,10 @@
 PATH
   remote: .
   specs:
-    dgd-tools (0.1.7)
+    dgd-tools (0.1.11)
       nokogiri (~> 1.10.5)
       optimist (~> 3.0.1)
+      treetop
 
 GEM
   remote: https://rubygems.org/
@@ -13,7 +14,10 @@ GEM
     nokogiri (1.10.10)
       mini_portile2 (~> 2.4.0)
     optimist (3.0.1)
+    polyglot (0.3.5)
     rake (12.3.3)
+    treetop (1.6.11)
+      polyglot (~> 0.3)
 
 PLATFORMS
   ruby

data/README.md

@@ -20,7 +20,7 @@ You can find example "goods" (library) files under the "goods" subdirectory of t
 
 ## WOE Objects and skotos-xml-diff
 
-SkotOS-based games use an XML format for in-game objects called WOE, which is [documented in SkotOS-Doc](https://ChatTheatre.github.io/SkotOS-Doc). The skotos-xml-diff utility will diff between WOE objects or directories of WOE objects.
+SkotOS-based games use an XML format for in-game objects ("/base/obj/thing"), which is [documented in SkotOS-Doc](https://ChatTheatre.github.io/SkotOS-Doc). The skotos-xml-diff utility will diff between Things or directories of Things.
 
 See SkotOS-Doc for more detail about how this can be used with a SkotOS game.
 
@@ -30,15 +30,21 @@ Run "skotos-xml-diff --help" for a list of options. You can tell it to ignore wh
 
 You would normally install DGDTools directly: `gem install dgd-tools`.
 
-It's possible to add it to a Ruby's application's Gemfile. Ordinarily you wouldn't.
+If you have a Ruby Gemfile or gems.rb that uses it, you can "bundle install" as normal.
 
 ## Usage
 
-If you have a DGD application that uses DGD Manifest, run `dgd-manifest install` to download its dependencies and create a fully-assembled DGD directory for it. You can also `dgd-manifest test` to make sure its dependencies are downloaded and satisfied without building an application directory.
+If you have a DGD application that uses DGD Manifest, run `dgd-manifest install` to download its dependencies and create a clean, fully-assembled DGD directory for it. You can also `dgd-manifest test` to make sure its dependencies are downloaded and satisfied without building an application directory. After the initial install, "dgd-manifest update" will avoid deleting any files you may have added.
+
+NOTE: "dgd-manifest install" will delete any extra files you may have created in the DGD root. "dgd-manifest update" will not. Neither of these is always the right answer.
 
 That fully-assembled DGD directory is named ".root". To run your dgd server, type "dgd-manifest server".
 
-If you update files in your root and want to update files under the generated root directory, use "dgd-manifest update".
+## Creating a New DGD Manifest App
+
+You can type "dgd-manifest new my_app_name" to create a new application using the appropriate DGD manifest structure. This is an easy way to set up an appropriate .gitignore file and similar.
+
+You can, of course, do the same without the command if you like.
 
 ## Using DGD Manifest with your DGD Application
 
@@ -94,6 +100,12 @@ Here's what those look like:
 }
 ```
 
+Fields in the Goods file are the same as fields in unbundled_goods.
+
+## Implementation Limits
+
+Right now it's not possible to use the same repo multiple times with different branches. So for instance, you can't use one branch of ChatTheatre/SkotOS for your main library while using a different branch of it for some other library. For this reason, it's recommended that you extract smaller libraries into their own repositories rather than keeping multiple libraries in the same repo.
+
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests.

data/dgd-tools.gemspec

@@ -26,4 +26,5 @@ Gem::Specification.new do |spec|
 
   spec.add_runtime_dependency "nokogiri", "~>1.10.5"
   spec.add_runtime_dependency "optimist", "~>3.0.1"
+  spec.add_runtime_dependency "treetop"
 end

data/exe/dgd-manifest

@@ -1,70 +1,82 @@
 #!/usr/bin/env ruby
 
+require "optimist"
 require "dgd-tools/manifest"
 
-if ARGV.size == 0
-    ARGV.push "install"
-end
+SUB_COMMANDS = %w(new test install update server)
 
-if ARGV == ["--version"]
-    puts "dgd-tools version #{DGD::VERSION}"
-    exit
-end
+OPTS = Optimist::options do
+    version "DGD-tools version #{DGD::VERSION}"
+    banner <<BANNER
+Use dgd.manifest to assemble your DGD application.
 
-if ARGV.size == 1 && ["-h", "--help"].include?(ARGV[0])
-    puts <<HELP_INFO
-dgd-manifest commands:
+Available subcommands:
+  new [project_name]: create a new DGD-manifest project
+  test: make sure the dgd.manifest file is well-formed and usable
+  install: compile the DGD application to a config file and a root directory
+  update: copy files into generated root directory but do *not* clear 'extra' files (e.g. user data)
+  server: run DGD with the generated root and configuration
 
-new [project_name]: create a new DGD-manifest project
-test: make sure the dgd.manifest file is well-formed and usable
-install: compile the DGD application to a config file and a root directory
-HELP_INFO
-    exit
+Available options:
+BANNER
+    opt :verbose, "Print verbose output where available"
+    stop_on SUB_COMMANDS
 end
 
-case ARGV[0]
-when "new"
-    unless ARGV.size == 2
-        puts "Usage: dgd-manifest new [project name]"
-        raise "Must supply exactly one argument to dgd-manifest new!"
-    end
-    appdir = DGD::Manifest::AppDirectory.new(File.expand_path ARGV[1])
-    appdir.name = ARGV[1]
-    appdir.create!
-when "test"
-    unless File.exist?("dgd.manifest")
-        raise "I don't see a dgd.manifest file in this directory!"
-    end
-    puts "Running dgd.manifest installer..."
-    repo = DGD::Manifest::Repo.new
-    repo.manifest_file("dgd.manifest")
-    repo.precheck(".")
-    puts "Verified Manifest packages: this looks likely correct."
-when "install"
-    unless File.exist?("dgd.manifest")
-        raise "I don't see a dgd.manifest file in this directory!"
-    end
-    puts "Running DGD Manifest installer..."
-    repo = DGD::Manifest::Repo.new
-    repo.manifest_file("dgd.manifest")
-    current_dir = File.expand_path(".")
-    repo.precheck(current_dir)
-    repo.assemble_app(current_dir)
-    puts "Assembled DGD application into #{current_dir}"
-when "update"
-    unless File.exist?("dgd.manifest")
-        raise "I don't see a dgd.manifest file in this directory!"
+ARGV.push("install") if ARGV.size == 0
+cmd = ARGV.shift
+cmd_opts = case cmd
+    when "test"
+        #Optimist::options do
+        #  opt :harsh, "check as exactly as possible"
+        #end
+
+        unless File.exist?("dgd.manifest")
+            raise "I don't see a dgd.manifest file in this directory!"
+        end
+        puts "Running dgd.manifest installer..."
+        repo = DGD::Manifest::Repo.new
+        repo.manifest_file("dgd.manifest")
+        repo.precheck(".", verbose: OPTS[:verbose])
+        puts "Verified Manifest packages: this looks likely correct."
+
+    when "install"
+        unless File.exist?("dgd.manifest")
+            raise "I don't see a dgd.manifest file in this directory!"
+        end
+        puts "Running DGD Manifest installer..."
+        repo = DGD::Manifest::Repo.new
+        repo.manifest_file("dgd.manifest")
+        current_dir = File.expand_path(".")
+        repo.precheck(current_dir, verbose: OPTS[:verbose])
+        repo.assemble_app(current_dir, verbose: OPTS[:verbose])
+        puts "Assembled DGD application into #{current_dir}"
+
+    when "update"
+        unless File.exist?("dgd.manifest")
+            raise "I don't see a dgd.manifest file in this directory!"
+        end
+        puts "Running DGD Manifest installer..."
+        repo = DGD::Manifest::Repo.new
+        repo.manifest_file("dgd.manifest")
+        current_dir = File.expand_path(".")
+        repo.precheck(current_dir, verbose: OPTS[:verbose])
+        repo.update_app(current_dir, verbose: OPTS[:verbose])
+        puts "Updated DGD application in #{current_dir}"
+
+    when "server"
+        puts "Starting DGD server..."
+        DGD::Manifest.system_call("~/.dgd-tools/dgd/bin/dgd dgd.config")
+
+    when "new"
+        unless ARGV.size == 1
+            puts "Usage: dgd-manifest new [project name]"
+            Optimist::die "Must supply exactly one argument to dgd-manifest new!"
+        end
+        appdir = DGD::Manifest::AppDirectory.new(File.expand_path ARGV[0])
+        appdir.name = ARGV[0]
+        appdir.create!
+
+    else
+        Optimist::die "Unknown subcommand: #{cmd.inspect}"
     end
-    puts "Running DGD Manifest installer..."
-    repo = DGD::Manifest::Repo.new
-    repo.manifest_file("dgd.manifest")
-    current_dir = File.expand_path(".")
-    repo.precheck(current_dir)
-    repo.update_app(current_dir)
-    puts "Updated DGD application in #{current_dir}"
-when "server"
-    puts "Starting DGD server..."
-    DGD::Manifest.system_call("~/.dgd-tools/dgd/bin/dgd dgd.config")
-else
-    raise "Unrecognised #{$0} command: #{ARGV[0].inspect}!"
-end

data/exe/skotos-xml-diff

@@ -14,6 +14,7 @@ Usage:
     skotos-xml-diff [options] <file_or_dir_1> <file_or_dir_2>
 where [options] are:
 BANNER
+    opt :version, "Print DGD version and exit"
     opt :merry_only, "Only diff Merry scripts"
     opt :ignore_whitespace, "Ignore whitespace in final diff"
     opt :ignore_types, "Comma-separated list of XML node types to ignore (e.g. Combat:Base,Base:Exit)", type: :string

data/lib/dgd-tools/dgd-doc.rb

@@ -0,0 +1,56 @@
+require "dgd-tools/version"
+
+require "treetop"
+
+require_relative "dgd_grammar_support"
+Treetop.load "#{__dir__}/dgd_grammar.tt"
+
+# Preprocessor incantation: cpp -C -I. -D__DGD__ name_of_file.c
+
+module DGD; end
+
+module DGD::Doc
+
+  class SourceFile
+    attr_reader :parser
+
+    def initialize(path, dgd_root:)
+      unless File.exist?(path)
+        raise "No such source file for DGD::Doc::Sourcefile: #{path.inspect}"
+      end
+
+      @parser = DGDGrammarParser.new
+      @path = path
+      @dgd_root = dgd_root
+
+      parse_contents
+    end
+
+    private
+
+    def parse_contents
+      preproc_output = `cpp -C -I#{@dgd_root}/include -D__DGD__ #{@path}`
+      preproc_output.gsub!(/\n+/, "\n")  # The preprocessor often winds up leaving a lot of newlines from various sources
+      preproc_output.gsub!(/^# \d+ ".*"\w*\d*$/, "")
+      data = @parser.parse(preproc_output)
+      if data
+        clean_tree(data)
+        #puts data.inspect
+      else
+        puts @parser.failure_reason
+        puts @parser.failure_line
+        puts @parser.failure_column
+
+        #puts "Preproc output was:#{preproc_output}\n\n\n"
+        raise "Parse error!"
+      end
+    end
+
+    def clean_tree(root_node)
+      return if(root_node.elements.nil?)
+      root_node.elements.each {|node| self.clean_tree(node) }
+      root_node.elements.delete_if {|node| node.class.name == "Treetop::Runtime::SyntaxNode" && node.text_value.size < 20 }
+    end
+
+  end
+end

data/lib/dgd-tools/dgd_grammar.tt

@@ -0,0 +1,373 @@
+grammar DGDGrammar
+  rule source_file
+    ws (inherit ws)* (top_level_decl ws)*
+  end
+
+  rule inherit
+    ("private" WS)? "inherit" WS (ident WS)? ('object' WS)? string_exp ws ';' <Inherit>
+  end
+
+  rule top_level_decl
+    data_decl / func_decl
+  end
+
+  rule ident
+    [a-zA-Z_] [a-zA-Z_0-9]* <Identifier>
+  end
+
+  rule c_comment
+    '/*'
+    (
+      !'*/'
+      (. / "\n")
+    )*
+    '*/'
+  end
+
+  rule preproc_hash_line
+    "\n"? "# " decimal " "+ simple_string (" "+ decimal)? "\n"
+  end
+
+  rule simple_string
+    '"' (
+      !["\n\\]
+      .
+      )*
+    '"' <StringLiteral>
+  end
+
+  rule complex_string
+    '"' ( ((!["\n\\] ) .) / . )+ '"' <StringLiteral>
+  end
+
+  rule simple_char
+    "'" ((!['\n\\]) .)+ "'" <CharLiteral>
+  end
+
+  rule complex_char
+    "'" (((!['\n\\]) .) / ("\\" .))+ "'" <CharLiteral>
+  end
+
+  rule decimal
+    [1-9] [0-9]* <DecimalLiteral>
+  end
+
+  rule octal
+    '0' [0-7]* <OctalLiteral>
+  end
+
+  rule hexadecimal
+    '0' [xX] [a-fA-F0-9]+ <HexadecimalLiteral>
+  end
+
+  rule float
+    ([0-9]+ "." [0-9]+ ([eE] [-+]? [0-9]+)?) <FloatLiteral> /
+    ("." [0-9]+ ([eE] [-+]? [0-9]+)?) <FloatLiteral> /
+    ([0-9]+ [eE] [-+]? [0-9]+) <FloatLiteral>
+  end
+
+  rule single_WS
+    [ \t\v\f\r\n] / c_comment / preproc_hash_line
+  end
+
+  rule ws
+    single_WS*
+  end
+
+  rule WS
+    single_WS+
+  end
+
+  rule string
+    simple_string /
+    complex_string
+  end
+
+  rule string_exp
+    string /
+    '(' ws composite_string ws ')'
+  end
+
+  rule composite_string
+    string_exp / string_exp '+' composite_string
+  end
+
+  rule klass
+    (class_spec WS)* class_spec
+  end
+
+  rule class_spec
+    'private' / 'static' / 'atomic' / 'nomask' / 'varargs'
+  end
+
+  rule type_spec
+    'int' / 'float' / 'string' / 'object' / 'mapping' / 'mixed' / 'void'
+  end
+
+  rule stars
+    ('*' ws)*
+  end
+
+  rule formals
+    # If you put these in a different order with '' and 'void' first, parses start failing. That seems bad.
+    formal_list (ws '...')? / '' / 'void'
+  end
+
+  rule formal_list
+    formal (ws ',' ws formal)*
+  end
+
+  rule formal
+    class_type data_declarator /
+    ident
+  end
+
+  rule data_decl
+    class_type WS declarators ws ';' <DataDecl>
+  end
+
+  rule func_decl
+    class_type WS function_declarator ws compound_statement <FuncDecl> /
+    klass WS function_name ws '(' ws formals ws ')' ws compound_statement <FuncDecl>
+  end
+
+  rule class_type
+    (class_spec WS)* type_spec /
+    (class_spec WS)* 'object' list_exp
+  end
+
+  rule data_declarator
+    stars ws ident
+  end
+
+  rule function_name
+    ident /
+    'operator' ws '+' /
+    'operator' ws '-' /
+    'operator' ws '*' /
+    'operator' ws '/' /
+    'operator' ws '%' /
+    'operator' ws '&' /
+    'operator' ws '^' /
+    'operator' ws '|' /
+    'operator' ws '<' /
+    'operator' ws '>' /
+    'operator' ws '>=' /
+    'operator' ws '<=' /
+    'operator' ws '<<' /
+    'operator' ws '>>' /
+    'operator' ws '~' /
+    'operator' ws '++' /
+    'operator' ws '--' /
+    'operator' ws '[' ws ']' /
+    'operator' ws '[' ws ']' ws '=' /
+    'operator' ws '[' ws '..' ws ']'
+  end
+
+  rule function_declarator
+    stars ws function_name ws '(' ws formals ws ')'
+  end
+
+  rule declarator
+    function_declarator / data_declarator
+  end
+
+  rule declarators
+    declarator (ws ',' ws declarator)*
+  end
+
+  rule locals
+    (ws data_decl ws)*
+  end
+
+  rule statements
+    (ws statement ws)*
+  end
+
+  rule statement
+    'if' ws '(' ws list_exp ws ')' ws statement (WS 'else' statement)? /
+    'do' WS statement WS 'while' ws '(' ws list_exp ws ')' ws ';' /
+    'while' ws '(' ws list_exp ws ')' ws statement /
+    'for' ws '(' ws list_exp? ws ';' ws list_exp? ws ';' ws list_exp? ws ')' ws statement /
+    'rlimits' ws '(' ws list_exp ws ';' ws list_exp ws ')' ws compound_statement /
+    'catch' ws compound_statement ws ':' statement /
+    'catch' ws compound_statement /
+    'switch' ws '(' ws list_exp ws ')' ws compound_statement /
+    'case' WS exp ws ':' ws statement /
+    'case' WS exp ws '..' ws exp ws ':' statement /
+    'default' ws ':' ws statement /
+    'goto' WS ident ws ';' /
+    'break' ws ';' /
+    'continue' ws ';' /
+    'return' WS list_exp WS ';' /
+    'return' ws ';' /
+    ident ws ':' ws statement /  # label statement
+    list_exp /
+    compound_statement /
+    ';'
+  end
+
+  rule compound_statement
+    '{' ws locals ws statements ws '}'
+  end
+
+  rule function_call
+    ('::' ws function_name) /
+    (ident ws '::' ws function_name) /
+    function_name
+  end
+
+  rule list_exp
+    exp ws ',' ws list_exp /
+    exp
+  end
+
+  rule exp
+    cond_exp ws '=' ws exp /
+    cond_exp ws '+=' ws exp /
+    cond_exp ws '-=' ws exp /
+    cond_exp ws '*=' ws exp /
+    cond_exp ws '/=' ws exp /
+    cond_exp ws '%=' ws exp /
+    cond_exp ws '<<=' ws exp /
+    cond_exp ws '>>=' ws exp /
+    cond_exp ws '&=' ws exp /
+    cond_exp ws '^=' ws exp /
+    cond_exp ws '|=' ws exp /
+    cond_exp
+  end
+
+  rule cond_exp
+    or_exp ws '?' ws list_exp ws ':' ws cond_exp / or_exp
+  end
+
+  rule or_exp
+    and_exp ws '||' ws or_exp / and_exp
+  end
+
+  rule and_exp
+    bitor_exp ws '&&' ws and_exp / bitor_exp
+  end
+
+  rule bitor_exp
+    bitxor_exp ws '|' ws bitor_exp / bitxor_exp
+  end
+
+  rule bitxor_exp
+    bitand_exp ws '^' ws bitxor_exp / bitand_exp
+  end
+
+  rule bitand_exp
+    equ_exp ws '&' bitand_exp / equ_exp
+  end
+
+  rule equ_exp
+    rel_exp ws '==' ws equ_exp /
+    rel_exp ws '!=' ws equ_exp /
+    rel_exp
+  end
+
+  rule rel_exp
+    shift_exp ws '<' ws rel_exp /
+    shift_exp ws '>' ws rel_exp /
+    shift_exp ws '<=' ws rel_exp /
+    shift_exp ws '>=' ws rel_exp /
+    shift_exp
+  end
+
+  rule shift_exp
+    add_exp ws '<<' ws shift_exp /
+    add_exp ws '>>' ws shift_exp /
+    add_exp
+  end
+
+  rule add_exp
+    mult_exp ws '+' ws add_exp /
+    mult_exp ws '-' ws add_exp /
+    mult_exp
+  end
+
+  rule mult_exp
+    cast_exp ws '*' ws mult_exp /
+    cast_exp ws '/' ws mult_exp /
+    cast_exp ws '%' ws mult_exp /
+    cast_exp
+  end
+
+  rule cast_exp
+    '(' ws class_type ws stars ws ')' ws cast_exp /
+    prefix_exp
+  end
+
+  rule prefix_exp
+    '++' ws cast_exp /
+    '--' ws cast_exp /
+    '+' ws cast_exp /
+    '-' ws cast_exp /
+    '!' ws cast_exp /
+    '~' ws cast_exp /
+    postfix_exp
+  end
+
+  rule postfix_exp
+    exp2 (ws ('++' / '--'))?
+  end
+
+  rule exp2
+    exp1 ws '->' ws ident ws '(' ws opt_arg_list ws ')' /
+    exp1 ws '<-' ws string_exp /
+    exp1 ((ws '[' ws list_exp ws ']') / (ws '[' ws (list_exp ws)? '..' ws (list_exp ws)? ']'))*
+  end
+
+  rule exp1
+    # This works (for now) with function_call up front, but not when it was lower down...
+    # There's some kind of bad stuff going on with precedence that's causing parse failures.
+    # That's weird, since I assume it should try all combinations/precedences before giving up...
+    function_call ws '(' ws opt_arg_list ws ')' /
+    'catch' ws '(' list_exp ws ')' /
+    'new' (WS 'object')? ws string_exp (ws '(' ws opt_arg_list ws ')')? /
+    'nil' /
+    '(' ws '{' ws opt_arg_list_comma ws '}' ws ')' /
+    '(' ws '[' ws opt_assoc_list_comma ws ']' ws ')' /
+    '::' ws ident /
+    decimal /
+    octal /
+    hexadecimal /
+    simple_char /
+    complex_char /
+    float /
+    string /
+    ident /
+    '(' ws list_exp ws ')'
+  end
+
+  rule arg_list
+    exp ws ',' ws arg_list /
+    exp
+  end
+
+  rule opt_arg_list
+    arg_list ws '...' /
+    arg_list?
+  end
+
+  rule opt_arg_list_comma
+    arg_list ws ',' /
+    arg_list?
+  end
+
+  rule assoc_list
+    assoc_pair ws ',' ws assoc_list /
+    assoc_pair
+  end
+
+  rule assoc_pair
+    exp ws ':' ws exp
+  end
+
+  rule opt_assoc_list_comma
+    assoc_list ws ',' /
+    assoc_list?
+  end
+
+end

data/lib/dgd-tools/dgd_grammar_support.rb

@@ -0,0 +1,36 @@
+#class StringLiteral < Treetop::Runtime::SyntaxNode
+#  def some_name
+#    # This is a pretty crude way of handling character escaping, etc.
+#    return eval self.text_value
+#  end
+#end
+
+class DataDecl < Treetop::Runtime::SyntaxNode
+end
+
+class FuncDecl < Treetop::Runtime::SyntaxNode
+end
+
+class Inherit < Treetop::Runtime::SyntaxNode
+end
+
+class Identifier < Treetop::Runtime::SyntaxNode
+end
+
+class DecimalLiteral < Treetop::Runtime::SyntaxNode
+end
+
+class OctalLiteral < Treetop::Runtime::SyntaxNode
+end
+
+class HexadecimalLiteral < Treetop::Runtime::SyntaxNode
+end
+
+class FloatLiteral < Treetop::Runtime::SyntaxNode
+end
+
+class StringLiteral < Treetop::Runtime::SyntaxNode
+end
+
+class CharLiteral < Treetop::Runtime::SyntaxNode
+end

data/lib/dgd-tools/manifest.rb

@@ -22,7 +22,7 @@ module DGD::Manifest
         puts "Running command: #{cmd.inspect}..."
         system(cmd, out: $stdout, err: :out)
         unless $?.success?
-            raise "Error running command: #{cmd.inspect}!"
+            raise "Error running command in #{Dir.pwd}: #{cmd.inspect}!"
         end
     end
 
@@ -64,19 +64,20 @@ module DGD::Manifest
             raise "Already have a dgd.manifest file!" unless @no_manifest_file
 
             @no_manifest_file = false
-            @manifest_file ||= AppFile.new(self, path)
+            @manifest_file ||= AppFile.new(self, path, shared_dir: shared_dir)
         end
 
         protected
 
         # This includes files to assemble... But also subdirectories and commands. This format is
         # unstable and ugly, and should not be exposed to outside parties who might later depend on it.
-        def assembly_operations(location)
+        def assembly_operations(location, verbose:)
             operations = []
 
             raise("No manifest file!") if @no_manifest_file
 
-            @manifest_file.specs.each do |spec|
+            # For each spec, put its dependencies before itself in order
+            @manifest_file.ordered_specs.each do |spec|
                 spec_git_repo = spec.source
                 spec_git_repo.use_details(spec.source_details)  # This sets things like checked-out branch
 
@@ -125,26 +126,38 @@ module DGD::Manifest
             "#{File.expand_path(location)}/#{GENERATED_ROOT}"
         end
 
-        def assemble_app(location)
+        def assemble_app(location, verbose:)
             Dir[File.join(dgd_root(location), "*")].each { |dir| FileUtils.rm_rf dir }
+            Dir[File.join(dgd_root(location), "state", "*")].each { |dir| FileUtils.rm_rf dir }
+            Dir[File.join(dgd_root(location), ".repos", "*")].each { |dir| FileUtils.rm_f dir }
 
-            write_app_files(location)
+            write_app_files(location, verbose: verbose)
         end
 
-        def update_app(location)
-            write_app_files(location)  # TODO: maybe check files dates? Some way to do update-only
+        def update_app(location, verbose:)
+            write_app_files(location, verbose: verbose)
         end
 
         protected
 
-        def write_app_files(location)
+        def write_app_files(location, verbose:)
             Dir.chdir(location) do
                 write_config_file("#{location}/dgd.config")
                 FileUtils.mkdir_p("#{location}/state") # Statedir for statedumps, editor files, etc.
+                repos_dir = "#{location}/.repos"
+                FileUtils.mkdir_p(repos_dir) # Links to repos in manifest
 
-                assembly_operations(location).each do |sd_hash|
+                @manifest_file.ordered_specs.each do |spec|
+                    FileUtils.ln_s(spec.source.local_dir, File.join(".repos", spec.name))
+                end
+
+                assembly_operations(location, verbose: verbose).each do |sd_hash|
                     to_path = "#{dgd_root(location)}/#{sd_hash[:to]}"
 
+                    if verbose
+                        puts "  Copy #{sd_hash[:from]} -> #{sd_hash[:to]}, files #{sd_hash[:non_dirs].join(", ")}"
+                    end
+
                     # Make appropriate dirs, including empty ones
                     sd_hash[:dirs].each do |dir|
                         FileUtils.mkdir_p dir.sub(sd_hash[:from], to_path)
@@ -154,8 +167,13 @@ module DGD::Manifest
                     sd_hash[:non_dirs].each do |from_file|
                         to_file = from_file.sub(sd_hash[:from], "#{dgd_root(location)}/#{sd_hash[:to]}")
                         to_dir = File.dirname(to_file)
-                        FileUtils.mkdir_p to_dir
-                        FileUtils.cp from_file, to_file
+                        begin
+                            FileUtils.mkdir_p to_dir
+                            FileUtils.cp from_file, to_file
+                        rescue
+                            puts "Error when copying: #{from_file} -> #{to_file} in #{sd_hash.inspect}"
+                            raise
+                        end
                     end
                 end
             end
@@ -163,8 +181,8 @@ module DGD::Manifest
 
         public
 
-        def precheck(location)
-            all_files = assembly_operations(location).flat_map { |sd| sd[:non_dirs] }
+        def precheck(location, verbose:)
+            all_files = assembly_operations(location, verbose: verbose).flat_map { |sd| sd[:non_dirs] }
 
             if all_files.size != all_files.uniq.size
                 repeated = all_files.uniq.select { |f| all_files.count(f) > 1 }
@@ -188,7 +206,7 @@ module DGD::Manifest
         def initialize(repo, git_url)
             @git_url = git_url
             @repo = repo
-            local_path = git_url.tr("/\\", "_")
+            local_path = git_url.tr("/\\ ", "_")
             @local_dir = "#{@repo.shared_dir}/git/#{local_path}"
 
             if File.directory?(@local_dir)
@@ -217,15 +235,18 @@ module DGD::Manifest
         end
     end
 
+    # This class parses the DGD manifest
     class AppFile
         attr_reader :path
         attr_reader :repo
         attr_reader :specs
         attr_reader :dgd_config
+        attr_reader :shared_dir
 
-        def initialize(repo, path)
+        def initialize(repo, path, shared_dir:)
             @path = path
             @repo = repo
+            @shared_dir = shared_dir
             raise("No such dgd.manifest file as #{path.inspect}!") unless File.exist?(path)
             contents = AppFile.parse_manifest_file(path)
 
@@ -279,6 +300,10 @@ module DGD::Manifest
 
             @dgd_config = DGDRuntimeConfig.new (contents["config"] || {})
 
+            if contents["app_root"]
+                raise "App_root must now be inside config block!"
+            end
+
             if contents["unbundled_goods"]
                 raise "Unbundled_goods must be an array!" unless contents["unbundled_goods"].is_a?(Array)
 
@@ -290,7 +315,10 @@ module DGD::Manifest
 
                 @specs += contents["goods"].map do |goods_url|
                     begin
-                        json_contents = JSON.parse(URI.open(goods_url).read)
+                        text_contents = URI.open(goods_url).read
+                        local_path = shared_dir + "/goods/" + goods_url.tr("/\\ ", "_")
+                        File.open(local_path, "wb") { |f| f.write(text_contents) }
+                        json_contents = JSON.parse text_contents
                     rescue
                         STDERR.puts "Error reading or parsing by URL: #{goods_url.inspect}"
                         raise
@@ -325,21 +353,42 @@ module DGD::Manifest
                 # For now, permit a single string as a dependency.
                 fields["dependencies"] = [ fields["dependencies"] ] if fields["dependencies"].is_a?(String)
 
+                goods_url = nil
                 fields["dependencies"].each do |dep|
                     if dep.is_a?(String)
-                        dependencies.push "url" => dep
+                        goods_url = dep
                     elsif dep.is_a?(Hash)
                         raise "Currently only URL-based dependencies on Goods files are supported!" unless dep["url"]
-                        dependencies.push dep
+                        goods_url = dep["url"]
                     else
                         raise "Unexpected dependency type #{dep.class} when parsing DGD Manifest specs, item: #{dep.inspect}"
                     end
+
+                    text_contents = URI.open(goods_url).read
+                    local_path = shared_dir + "/goods/" + goods_url.tr("/\\ ", "_")
+                    File.open(local_path, "wb") { |f| f.write(text_contents) }
+                    dep_fields = JSON.parse text_contents
+
+                    dependencies.push unbundled_json_to_spec(dep_fields)
                 end
             end
 
             spec = GoodsSpec.new(@repo, name: fields["name"], source: source, source_details: source_details, paths: fields["paths"], dependencies: dependencies)
             return spec
         end
+
+        def ordered_specs
+            @specs.flat_map do |s|
+                deps = [s]
+                deps_to_add = s.dependencies
+                while(deps_to_add.size > 0)
+                    next_deps = deps_to_add.flat_map { |dep| dep.dependencies }
+                    deps = deps_to_add + deps
+                    deps_to_add = next_deps
+                end
+                deps
+            end
+        end
     end
 
     class GoodsSpec
@@ -388,6 +437,13 @@ module DGD::Manifest
                 .root
                 dgd.config
                 state/*
+                wafer
+                websocket-to-tcp-tunnel
+                dgd
+                log/*
+                skotos.database
+                skotos.database.old
+                .repos/**
             FILE_CONTENTS
         end
 
@@ -400,7 +456,7 @@ module DGD::Manifest
     "app_root": "app",
     "goods": [
         "# This is an example goods file - substitute your own.",
-        "https://raw.githubusercontent.com/noahgibbs/dgd-tools/main/goods/skotos_httpd.goods"
+        "https://raw.githubusercontent.com/ChatTheatre/dgd-tools/main/goods/skotos_httpd.goods"
     ],
     "unbundled_goods": [
         {
@@ -521,15 +577,17 @@ FILE_CONTENTS
 
         def config_to_ports(data)
             if data.is_a?(Hash)
-                # TODO: verify that keys are IP addr strings and values are legal port numbers
-                return data
+                return data.map { |ip, port| [ip, Integer(port) ] }
             elsif data.is_a?(Array)
-                # TODO: verify that data is an array of legal integer port numbers
-                ports = {}
-                data.each { |p| ports["*"] = p }
+                if data[0].is_a?(Array)
+                    ports = data.map { |ip, port| [ip, Integer(port) ] }
+                    return ports
+                end
+
+                ports = data.map { |p| [ "*", Integer(p) ] }
                 return ports
             elsif data.is_a?(Integer)
-                return { "*": data }
+                return [ [ "*", data ] ]
             else
                 raise "dgd-manifest: not sure how to get port data from a #{data.class.name} -- #{data.inspect}!"
             end

data/lib/dgd-tools/skotos_xml_obj.rb

@@ -46,11 +46,13 @@ class SkotOS::XMLObject
             of1.close
             of2.close
 
-            diff_opts = [ "-c" ]
-            diff_opts += [ "-w", "-B" ] if self.ignore_whitespace
+            diff_opts = [ "c" ]
+            diff_opts += [ "b", "B" ] if self.ignore_whitespace
 
             # Diff 'fails' if there's a difference between the two files.
-            diff = system_call("diff #{diff_opts.join(" ")} #{of1.path} #{of2.path}", fail_ok: true)
+            cmd = "diff -#{diff_opts.join("")} #{of1.path} #{of2.path}"
+            #puts "Diff command: #{cmd}"
+            diff = system_call(cmd, fail_ok: true)
             diff.sub!(of1.path, o1_name)
             diff.sub!(of2.path, o2_name)
         ensure
@@ -115,11 +117,16 @@ class SkotOS::XMLObject
         end
 
         rev = noko_single_node(doc.root, "Core:Property", attrs: { "property" => "revisions" })
-        rev.remove if rev
+        noko_remove(rev) if rev
 
         list = noko_single_node(doc.root, "Core:Property", attrs: { "property" => "#list#" })
         list.remove if list
 
+        properties = noko_with_name_and_attrs(doc.root, "Core:Property")
+        properties.each do |prop_node|
+            prop_node.remove if prop_node.attribute("property").value.start_with?("sys:sync")
+        end
+
         if self.merry_only
             # Kill off all the non-Merry nodes
             noko_remove_non_merry_nodes(doc.root)
@@ -128,7 +135,7 @@ class SkotOS::XMLObject
         if self.ignore_types
             self.ignore_types.each do |ignored_type|
                 skipped = noko_with_name_and_attrs(doc.root, ignored_type)
-                skipped.each { |n| n.remove }
+                skipped.each { |n| noko_remove(n) }
             end
         end
 
@@ -144,6 +151,12 @@ class SkotOS::XMLObject
         end
     end
 
+    def self.noko_remove(node)
+        nn = node.next
+        nn.remove if nn.is_a?(Nokogiri::XML::Text)
+        node.remove
+    end
+
     def self.noko_single_node(node, name, attrs: {})
         choices = noko_with_name_and_attrs(node, name, attrs)
         if choices.size < 1

data/lib/dgd-tools/version.rb

@@ -1,3 +1,3 @@
 module DGD
-  VERSION = "0.1.7"
+  VERSION = "0.1.12"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: dgd-tools
 version: !ruby/object:Gem::Version
-  version: 0.1.7
+  version: 0.1.12
 platform: ruby
 authors:
 - Noah Gibbs
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2021-03-03 00:00:00.000000000 Z
+date: 2021-05-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: nokogiri
@@ -38,6 +38,20 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: 3.0.1
+- !ruby/object:Gem::Dependency
+  name: treetop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: dgd-tools supplies DGD Manifest and eventually perhaps other tools. DGD
   Manifest is an experimental DGD library and packaging system.
 email:
@@ -69,6 +83,9 @@ files:
 - exe/skotos-xml-diff
 - goods/chattheatre_kernellib.goods
 - goods/skotos_httpd.goods
+- lib/dgd-tools/dgd-doc.rb
+- lib/dgd-tools/dgd_grammar.tt
+- lib/dgd-tools/dgd_grammar_support.rb
 - lib/dgd-tools/manifest.rb
 - lib/dgd-tools/skotos_xml_obj.rb
 - lib/dgd-tools/version.rb