nydp 0.5.1 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: adb6cc0345ed64c1e0e7ac99c06d9aa1328482e7d51d771cd77b844e1c5a6e14
-  data.tar.gz: 997c1f318ed518f3d2100699c1532db9a140f7109d6e4ddef6e68055466eb23a
+  metadata.gz: 2163d395501072ea321375834496bc7228aa3b19f8991c0ccc9ac69f3e951426
+  data.tar.gz: 3a3b17042f42b1c2e0653a27e4aebf5000f248d59fc56a1e39f065df3a2fc97b
 SHA512:
-  metadata.gz: 4a80109cc7bf73c177c6f89697e386a0a12d3645512bd14db2ed036711ef2c0b11d80fef67cff8dbc6f7bfed23d103220b9bc909aa9b34590323375db699c5e3
-  data.tar.gz: 25864fdef654329baef7d494a54b926f6dd236399f131e5943505752784713b03dc9e79640407c1df6af17ac380c483d82e2fae5e35fa3f3636b1226384eaa58
+  metadata.gz: 5c1ff800db3fed15b1a541c4926ffe53a8d2962dc54969265d472c4fefb1eb453db4097aaf10750c39ccc86174967665689f38f63c557f42a9fe2a575a268fcf
+  data.tar.gz: 4c87fb842105b2614643f31d5f1c965129dec628310007b3cc6054756917d54c6a794a06234aa5bf6b658285fc6a0c7feaacef87d8c7dd778640fdabb9b38e76

data/.gitignore

@@ -18,3 +18,4 @@ tmp
 lib/lisp/scratch
 .#*
 bm.txt
+rubycode

data/README.md

@@ -1,16 +1,14 @@
 # `'nydp`
 
-`'nydp` is a new LISP dialect, much inspired by [Arc](http://arclanguage.org/), and hence indirectly by all of `'arc`'s ancestors,
-and implemented in Ruby.
+`'nydp` is a new LISP dialect, much inspired by [Arc](http://arclanguage.org/), and hence indirectly by all of `'arc`'s ancestors, and implemented in Ruby.
 
-`'nydp` is "Not Your Daddy's Parentheses", a reference to [Xkcd 297](http://xkcd.com/297/) (itself a reference
-to Star Wars), as well as to the meme [Not Your Daddy's Q](http://tvtropes.org/pmwiki/pmwiki.php/Main/NotYourDaddysX), where Q is a
-modern, improved Q quite unlike the Q your daddy used. `'nydp` also shamelessly piggypacks on the
-catchiness and popularity of the [NYPD](https://en.wikipedia.org/wiki/NYPD_Blue) abbreviation ("New York Police Department",
-for those who have no interest in popular US politics or TV).
+`'nydp` is "Not Your Daddy's Parentheses", a reference to [Xkcd 297](http://xkcd.com/297/) (itself a reference to Star Wars), as well as to the meme [Not Your Daddy's Q](http://tvtropes.org/pmwiki/pmwiki.php/Main/NotYourDaddysX), where Q is a modern, improved Q quite unlike the Q your daddy used. `'nydp` also shamelessly piggypacks on the catchiness and popularity of the [NYPD](https://en.wikipedia.org/wiki/NYPD_Blue) abbreviation ("New York Police Department").
 
-We do not wish to suggest by "Not Your Daddy's Parentheses" that Common Lisp, Scheme, Racket, Arc, Clojure or your favourite
-other lisp are somehow old-fashioned, inferior, or in need of improvement in any way.
+We do not wish to suggest by "Not Your Daddy's Parentheses" that Common Lisp, Scheme, Racket, Arc, Clojure or your favourite other lisp are somehow old-fashioned, inferior, or in need of improvement in any way.
+
+Further, we deplore and lament the gender bias and the age bias implicit in the use of the word "Daddy". We hereby declare that `'nydp` is also not your mommy's parentheses, and, in fact, is not the parentheses of any other family member, including gender-neutral and nonbinary members, deceased ancestors and as-yet unconceived descendants. `'nydp` is equivalently and unequivocally not their parentheses either, and neither more or less so than the parentheses of any male or masculine or father-role family member. It is possible that in due course this project may be renamed "Not Your Parents' Parentheses" even if `'nypp` doesn't have quite the same ring to it and the 6-tuple "Parent" appears twice in the name.
+
+Yet more deplorable and lamentable is the fact that "Daddy" appears to suggest that the age-gender axis is the only relevant axis under consideration. We hereby assert and declare that `'nydp` is also not any parentheses you may obtain (in any manner legal or otherwise) from any race, religion, nationality, sexual orientation, or choice of text editor.
 
 The goal of `'nydp` is to allow untrusted users run sandboxed server-side scripts. By default, `'nydp` provides no system access :
 
@@ -23,8 +21,7 @@ The goal of `'nydp` is to allow untrusted users run sandboxed server-side script
 
 [Peruse `'nydp`'s features here](lib/lisp/tests) in the `tests` directory.
 
-Pronunciation guide: there is no fixed canonical pronunciation of `'nydp`. Just keep in mind that if you find yourself
-wading _knee-deep_ through raw sewage, it's still better than working on a java project in a bank.
+Pronunciation guide: there is no fixed canonical pronunciation of `'nydp`. Just keep in mind that if you find yourself wading _knee-deep_ through raw sewage, it's still better than working on a java or C# project in a bank. Or insurance company, if that's your thing.
 
 ## Running
 
@@ -37,7 +34,7 @@ welcome to nydp
 nydp >
 ```
 
-The REPL uses the readline library so you can use up- and down-arrows to navigate history.
+The REPL uses the readline library so you can use up- and down-arrows to navigate history. IRB also uses readline and command history may get a little bit mixed-up if you start nydp inside an irb session.
 
 #### Invoking from Ruby
 
@@ -46,15 +43,14 @@ Suppose you want to invoke the function named `question` with some arguments. Do
 ```ruby
 ns     = Nydp.build_nydp     # keep this for later re-use, it's expensive to set up
 
-answer = Nydp.apply_function ns, :question, :life, ["The Universe", and_also(everything)]
+answer = ns.apply :question, :life, ["The Universe", and_also(everything)]
 
 ==> 42
 ```
 
-`ns` is just a plain old ruby hash, mapping ruby symbols to nydp symbols for quick lookup at nydp compile-time. The nydp symbols maintain the values of global variables, including all builtin functions and any other functions defined using `def`.
-
-You can maintain multiple `ns` instances without mutual interference. In other words, assigning global variables while one `ns` is in scope will not affect the values of variables in any other `ns` (unless you've specifically arranged it to be so by duplicating namespaces or some such sorcery).
+`ns` is a Nydp::Namespace object, mapping ruby symbols to nydp symbols for quick lookup at nydp compile-time. The nydp symbols maintain the values of global variables, including all builtin functions and any other functions defined using `def`.
 
+Nydp is designed to operate in a multi-tenant architectural context, so you can instantiate multiple Namespace instances, where each tenant applies their own custom scripts ; such scripts will not interfere with one another (unless you've specifically arranged it to be so by duplicating namespaces or some such sorcery).
 
 #### Facing the Truth
 
@@ -77,7 +73,7 @@ In conditional statements, nil is false, anything else is true
 
 After parsing its input, `'nydp` passes the result as an argument to the `pre-compile` function. This is where things get a little bit circular: initially, `pre-compile` is a builtin function that just returns its argument. `pre-compile` bootstraps itself into existence in [core-010-precompile.nydp](lib/lisp/core-010-precompile.nydp).
 
-You can override `pre-compile` to transform the expression in any way you wish. By default, the `core-010-precompile.nydp` implementation of `pre-compile` performs macro-expansion.
+You can override `pre-compile` to transform the expression in any way you wish. By default, the `core-010-precompile.nydp` implementation of `pre-compile` performs macro-expansion by searching for keywords in the `macs` global variable and applying the corresponding function to the given expression if a function is found.
 
 
 ```lisp
@@ -177,14 +173,16 @@ nydp > (map %td:&lastname german-composers)
 
 ```
 
-So, @%td@ expands to @(percent-syntax || td)@, @&lastname@ to @(ampersand-syntax || lastname)@, and the whole @%td:&lastname@
-to @(colon-syntax (percent-syntax || td) (ampersand-syntax || lastname))@. Luckily for you, there's a fine @colon-syntax@ macro
-that knows how to build a function out of these bits and pieces.
+So, `%td` expands to `(percent-syntax || td)`, `&lastname` to `(ampersand-syntax || lastname)`,
+and the whole `%td:&lastname` to `(colon-syntax (percent-syntax || td) (ampersand-syntax || lastname))`.
+Luckily for you, there's a fine `colon-syntax` macro that knows how to build a function out of these bits
+and pieces.
 
+Look for `SYMBOL_OPERATORS` in [parser.rb](lib/nydp/parser.rb) to see which syntax is recognised and in
+which order. The order of these definitions defines special-syntax-operator precedence.
 
-Look for `SYMBOL_OPERATORS` in [parser.rb](lib/nydp/parser.rb) to see which syntax is recognised and in which order. The order of these definitions defines special-syntax-operator precedence.
-
-Any character that is not special syntax will be recognised as part of a symbol. At time of writing, this includes the plus sign, hyphen, and slash.
+Any character that is not special syntax will be recognised as part of a symbol. At time of writing, this
+includes the plus sign, hyphen, and slash.
 
 ```lisp
 
@@ -208,7 +206,7 @@ nydp > (parse "{ a 1 b 2 }")
 
 ```
 
-`brace-list` is a macro that expands to create a hash literal. It assumes every item 0 is a literal symbol key, item 1 is the corresponding value which is evaluated at run time, and so on for each following item-pair.
+`brace-list` is a macro that expands to create a hash literal. It assumes every item 0 is a literal key (symbol, string or number), item 1 is the corresponding value which is evaluated at run time, and so on for each following item-pair.
 
 ```lisp
 
@@ -233,9 +231,9 @@ nydp > (let bar "Mister Nice Guy" "hello, ~bar")
 
 ; this is a more tricky example because we need to make a string with an interpolation token in it
 
-nydp > (let s (joinstr "" "\"hello, " '~ "world\"") (parse s))
+nydp > (let s "\"hello, \~world\"" (parse s))
 
-==> (string-pieces "hello, " world "") ; "hello, ", followed by the interpolation 'world, followed by the empty string after 'world
+==> (string-pieces "hello, " world) ; "hello, ", followed by the interpolation `world`
 
 ; It is possible to nest interpolations. Note that as with many popular language features, just because you can do something, does not mean you should:
 
@@ -250,22 +248,37 @@ nydp > (with (a 1 b 2)
 ==> AND ALSO, Consider 3, the third (and final) thing
 ```
 
-By default, `string-pieces` is a function that just concatenates the string value of its arguments. You can redefine it as a macro to
-perform more fun stuff, or you can detect it within another macro to do extra-special stuff with it. The 'nydp-html gem detects
-'string-pieces and gives it special treatment in order to render haml and textile efficiently, and also to capture and report errors
-inside interpolations and report them correctly.
-
+By default, `string-pieces` is a function that just concatenates the string value of its arguments. You can redefine it as a macro to perform more fun stuff, or you can detect it within another macro to do extra-special stuff with it. The 'nydp-html gem detects 'string-pieces and gives it special powers in order to render haml and textile efficiently, and also to capture and report errors inside interpolations and report them correctly.
 
 #### 5. No continuations.
 
 Sorry. While technically possible ... why bother?
 
+#### 6. No tail-call optimisation / tail-call elimination
+
+This isn't completely true. For performance reasons, the nydp stack maps directly to the underlying ruby stack, and it is possible to enable TCO in your ruby VM. Nydp doesn't enable it for you though, you'll need to take care of this yourself. So basically nydp doesn't guarantee you TCO, but if you're in control of your deployment environment, you can still have it.
+
+As a result, nydp has an extra keyword, 'loop, so that recursive solutions can be rewritten as iterative ones. For example
+
+```lisp
+(def elegant-recursive-solution (n f things)
+  (if (> n 0)
+      (elegant-recursive-solution (- n 1) (cdr things))
+      (f things)))
+
+;; can be rewritten as
+(def stupid-iterative-solution (n f things)
+  (loop (> n 0)
+    (= things (cdr things)
+       n (- n 1)))
+  (f things))
+```
 
 ## Besides that, what can Nydp do?
 
 #### 1. Functions and variables exist in the same namespace.
 #### 2. Macros are maintained in a hash called 'macs in the main namespace.
-#### 3. General [tail call elimination](https://en.wikipedia.org/wiki/Tail_call) allowing recursion without stack overflow in some cases.
+#### 3. Loop construct (see above under 'no tail-call optimisation')
 #### 4. 'if like Arc:
 
 ```lisp
@@ -355,8 +368,7 @@ nydp > (parse "%w(a b c)")
 ==> (prefix-list "%w" (a b c))
 ```
 
-This allows for preprocessing lists in a way not possible for macros. nydp uses this feature to implement shortcut oneline
-functions, as in
+This allows for preprocessing lists in a way not possible for macros. nydp uses this feature to implement shortcut monoexpression functions with single-letter arguments, as in
 
 ```lisp
 nydp > (parse "λx(len x)")
@@ -380,6 +392,13 @@ nydp > (pre-compile '(prefix-list "λxy" (* x y)))
 ==> (fn (x y) (* x y))
 ```
 
+Example:
+
+```lisp
+nydp > (map λx(* x x) '(1 2 3 4))
+
+==> (1 4 9 16)
+
 Use 'define-prefix-list-macro to define a new handler for a prefix-list. Here's the code for the 'λ shortcut:
 
 ```lisp
@@ -395,11 +414,11 @@ In this case, the regex matches an initial 'λ ; there is no constraint however
 
 Once the 'dox system is bootstrapped, any further use of 'mac or 'def will create documentation.
 
-Any comments at the start of the form body will be used to generate help text. For example:
+Comments immediately prior to the start of the form body will be used to generate help text. For example:
 
 ```lisp
-nydp > (def foo (x y)
-         ; return the foo of x and y
+nydp > ; return the foo of x and y
+       (def foo (x y)
          (* x y))
 
 nydp > (dox foo)
@@ -421,14 +440,11 @@ nydp > (dox-lookup 'foo)
 ((foo def ("return the foo of x and y") (x y) (def foo (x y) (* x y))))
 ```
 
-Not as friendly, but more amenable to programmatic manipulation. Each subsequent definition of 'foo (if you override it, define
-it as a macro, or define it again in some other context) will generate a new documentation structure, which will simply be preprended to
-the existing list.
+Not as friendly, but more amenable to programmatic manipulation. Each subsequent definition of 'foo (if you override it, define it as a macro, or define it again in some other context) will generate a new documentation structure, which will simply be preprended to the existing list.
 
 #### 11. Pretty-Printing
 
-'dox above uses the pretty printer to display code source. The pretty-printer is hard-coded to handle some special cases,
-so it will unparse special syntax, prefix-lists, quote, quasiquote, unquote, and unquote-splicing.
+'dox above uses the pretty printer to display code source. The pretty-printer is hard-coded to handle some special cases, so it will unparse special syntax, prefix-lists, quote, quasiquote, unquote, and unquote-splicing.
 
 You can examine its behaviour at the repl:
 
@@ -448,13 +464,11 @@ nydp > (p:pp:dox-src 'pp/find-breaks)
         (list form))))
 ```
 
-The pretty-printer is still rather primitive in that it only indents according to some hard-coded rules, and according to argument-count
-for documented macros. It has no means of wrapping forms that get too long, or that extend beyond a certain predefined margin or column number.
+The pretty-printer is still rather primitive in that it only indents according to some hard-coded rules, and according to argument-count for documented macros. It has no means of wrapping forms that get too long, or that extend beyond a certain predefined margin or column number.
 
 #### 12. DSLs
 
-The `pre-compile` system (described earlier in "Macro-expansion runs in lisp") is available for implementing local, mini-languages. To do this, use `pre-compile-with`
-in a macro. `pre-compile-with` expects a hash with expansion rules, and an expression to expand using these rules. For example, to build a "describe" dsl :
+The `pre-compile` system (described earlier in "Macro-expansion runs in lisp") is available for implementing local, mini-languages. To do this, use `pre-compile-with` in a macro. `pre-compile-with` expects a hash with expansion rules, and an expression to expand using these rules. For example, to build a "describe" dsl :
 
 ```lisp
   (= describe-dsl (hash))
@@ -467,20 +481,27 @@ in a macro. `pre-compile-with` expects a hash with expansion rules, and an expre
        ',outcome))
 ```
 
-In this example, `describe-dsl` establishes rules for expanding `def` and for `p` which will shadow their usual meanings, but only in the context
-of a `describe` form.
+In this example, `describe-dsl` establishes rules for expanding `def` and for `p` which will shadow their usual meanings, but only in the context of a `describe` form.
 
-This technique is useful to avoid cluttering the global macro-namespace with rules that are used only in a specific context. A word of warning:
-if you shadow a popular name, for example `let`, `map` or `do`, you are likely to run into trouble when some non-shadowed macro inside your
-dsl context expands into something you have shadowed; in this case your shadowy definitions will be used to further expand the result.
+This technique is useful to avoid cluttering the global macro-namespace with rules that are used only in a specific context. A word of warning: if you shadow a popular name, for example `let`, `map` or `do`, you are likely to run into trouble when some non-shadowed macro inside your dsl context expands into something you have shadowed; in this case your shadowy definitions will be used to further expand the result.
 
-For example, if you shadow `with`, but not `let`, and then you use a `let` form within a sample of the dsl you are specifying ; the `let`
-form expands into a `with` form, thinking the `with` will expand in the usual way, but in fact no, your private dsl is going to expand
-this `with` in your private special way, possibly in a way that's incompatible with the form produced by `let`.
+For example, if you shadow `with`, but not `let`, and then you use a `let` form within a sample of the dsl you are specifying ; the `let` form expands into a `with` form, thinking the `with` will expand in the usual way, but in fact no, your private dsl is going to expand this `with` in your private special way, possibly in a way that's incompatible with the form produced by `let`.
 
-If, notwithstanding the aforementioned, the outcome works the way you expected it to, then you have the wrong kind of genius and your
-license to program should be revoked.
+If, notwithstanding the aforementioned, you do this anyway and the outcome works the way you expected it to, then you have the wrong kind of genius and your license to program should be revoked.
 
+## Extending NYDP
+
+You can add any object implementing #call (for example, a ruby Proc) to your `ns` instance:
+
+```ruby
+ns.assign(:"send-mail", -> {|props| Net::SMTP.deliver_the_letter(props).the_sooner_the_better }) # imaginary example
+```
+
+In your lisp,
+
+```lisp
+(send-mail { from "myself@example.com" to "yourself@example.com" subject "SPECIAL OFFER" body (mail-body blah-blah) })
+```
 
 ## Installation
 

data/lib/lisp/core-000.nydp

@@ -2,7 +2,7 @@
 ;;
 ;; Acknowledgements to Paul Graham, Robert Morris, and their ancestor programmers.
 ;; nydp's main inspiration is arc, and many nydp features (including, but not limited
-;; to, 'do, 'rfn, 'loop, 'for) were directly inspired by (aka stolen from) arc.arc. See
+;; to, 'do, 'rfn, 'for) were directly inspired by (aka stolen from) arc.arc. See
 ;; README.md however for some significant differences
 
 (assign   noop (fn))

data/lib/lisp/core-010-precompile.nydp

@@ -1,9 +1,11 @@
+(assign uniq-counter 0)
+
 (assign mac-expand
         (fn (names macfn expr)
             (cond macfn
                   (handle-error
-                   (fn (errors)
-                       (error "expanding" (inspect expr) "with" (inspect macfn)))
+                   (fn (errors traces)
+                       (error "expanding" (inspect expr) "with" macfn))
                    (fn ()
                        (pre-compile-with names (apply macfn (cdr expr)))))
                   expr)))
@@ -11,37 +13,41 @@
 (assign macs (hash))
 
 (assign pre-compile-each
-        (fn (names args)
-            (cond args
-                  (cond (pair? args)
+        (fn (names exprs)
+            (cond exprs
+                  (cond (pair? exprs)
                         (cons (pre-compile-with names
-                                                (car args))
+                                                (car exprs))
                               (pre-compile-each names
-                                                (cdr args)))
-                        args))))
+                                                (cdr exprs)))
+                        exprs))))
 
 (assign pre-compile-msg
         (fn (src compiled)
-            (p "pre-compile" src "\n  -> " compiled)
             compiled))
 
 (assign pre-compile-raw
-        (fn (names arg)
-            (cond (pair? arg)
-                  (cond (eq? (car arg) 'quote)
-                        arg
-                        (pre-compile-each names
-                                          (mac-expand names
-                                                      (hash-get names
-                                                                (car arg))
-                                                      arg)))
-                  arg)))
+        (fn (names expr)
+            (cond (pair? expr)
+                  (cond (eq? (car expr) 'quote)
+                        expr
+                        (cond (isa 'symbol (car expr))
+                          (pre-compile-each
+                            names
+                            (mac-expand names
+                              (hash-get names
+                                (car expr))
+                              expr))
+                          (pre-compile-each
+                            names
+                            expr)))
+                  expr)))
 
 (assign pre-compile-debug
-        (fn (names arg)
-            (pre-compile-msg arg
+        (fn (names expr)
+            (pre-compile-msg expr
                              (pre-compile-raw names
-                                              arg))))
+                                              expr))))
 
 (assign debug-pre-compile
         (fn (arg)
@@ -54,8 +60,17 @@
 
 ; builtin pre-compile does nothing; override here to provide macro-expansion
 (assign pre-compile
-        (fn (arg)
-            (pre-compile-with macs arg)))
+        (fn (expr)
+            (pre-compile-with macs expr)))
+
+;; this is the entry point for the interpreter ; we need to reset uniq-counter for each new expression
+;; (for performance purposes only - to ensure identical nydp code produces identical ruby code) ; but
+;; we can't reset uniq-counter in the middle of pre-compiling an expression, otherwise @uniq@ won't be
+;; even slightly unique any more
+(assign pre-compile-new-expression
+        (fn (expr)
+            (assign uniq-counter 0)
+            (pre-compile-with macs expr)))
 
 ; we override this later to provide argument deconstruction
 (hash-set macs 'fun
@@ -81,28 +96,33 @@
                  (qq-quasiquote rest level))
            arg))
 
+(def qq-build-cons (a b)
+  (cond b
+        (list 'cons a b)
+        (list 'cons a)))
+
 (def qq-skip-unquote-splicing (arg rest level)
-     (list 'cons
+     (qq-build-cons
            (list 'list ''unquote-splicing (qq-quasiquote arg (- level 1)))
            (qq-quasiquote rest level)))
 
 (def qq-handle-quasiquote (arg rest level)
-     (list 'cons
+     (qq-build-cons
            (list 'list ''quasiquote (qq-quasiquote arg (+ level 1)))
            (qq-quasiquote rest level)))
 
 (def qq-handle-unquote (arg rest level)
-     (list 'cons
+     (qq-build-cons
            (qq-maybe-unquote arg level)
            (qq-quasiquote rest level)))
 
 (def qq-unquote-recurse (arg rest level)
-        (list 'cons
+        (qq-build-cons
               (qq-quasiquote arg level)
               (qq-quasiquote rest level)))
 
 (def qq-handle-plain (arg rest level)
-     (list 'cons
+     (qq-build-cons
            (list 'quote arg)
            (qq-quasiquote rest level)))
 

data/lib/lisp/core-012-utils.nydp

@@ -9,9 +9,11 @@
                              nil)))
 
 (def map-helper-0 (f things lc)
-  (if (pair? things)
-      (map-helper-0 f (cdr things) (cdr-set lc (cons (f (car things)))))
-      things
+  (loop (pair? things)
+        ((fn nil
+             (assign lc (cdr-set lc (cons (f (car things)))))
+             (assign things (cdr things)))))
+  (if things
       (cdr-set lc (f things))))
 
 (def map-helper-1 (f things acc)
@@ -35,9 +37,11 @@
 (def hash-cons (h k v)
   (hash-set h k (cons v (hash-get h k))))
 
+;; 'things - the list to be reversed
+;; 'last-cdr - (normally nil) - an item (atom, list, nil, anything) to be consed to the end of the reversed list.
 (def rev (things last-cdr)
-  ; 'things - the list to be reversed
-  ; 'last-cdr - (normally nil) - an item (atom, list, nil, anything) to be consed to the end of the reversed list.
-  (if (pair? things)
-      (rev (cdr things) (cons (car things) last-cdr))
-      last-cdr))
+  (loop (pair? things)
+        ((fn nil
+             (assign last-cdr (cons (car things) last-cdr))
+             (assign things (cdr things)))))
+  last-cdr)

data/lib/lisp/core-015-documentation.nydp

@@ -77,9 +77,6 @@
        (hash-set hsh 'texts    texts       )
        (hash-set hsh 'args     args        )
        (hash-set hsh 'src      src         )
-       (hash-set hsh 'chapters (cons (chapter-current) chapters))
-       (hash-set hsh 'file     this-script )
-       (hash-set hsh 'plugin   this-plugin )
        hsh)
 
      (def dox-new (item)
@@ -87,11 +84,19 @@
        (hash-cons types (hash-get item 'what) item)
        (dox-add-to-chapters item (hash-get item 'what) (hash-get item 'chapters) (hash)))
 
-     (def dox-add-doc (name what texts args src chapters more)
-       (cond (no (privately))
-             (dox-new (dox-build (if more more (hash)) name what texts args src chapters))))
+     (def dox-add-doc (name what texts args src etc)
+       (handle-error
+        (fn (errors traces) (error "failed to add dox for " name " - " what))
+        (fn ()
+            (cond (no etc)
+                  (error (inspect name) " : please provide arguments (name what texts args src etc) where etc is a hash with at least keys (chapters plugin file)"))
+            (cond (no (privately))
+                  (dox-new
+                   (dox-build etc name what texts args src))))))
 
      (def dox-add-to-chapters (item type chapters already)
+       (cond (pair? (car chapters))
+             (error "dox-add-to-chapters : expected symbol, got " (inspect (car chapters)) " for " (inspect item)))
        (cond chapters
          (cond (no (hash-get already (car chapters)))
            (do (hash-set            already        (car chapters)                       t)
@@ -131,7 +136,7 @@
 (def plugin-start (name) (assign this-plugin name) (chapter-end))
 (def plugin-end   (name) (assign this-plugin nil ) (chapter-end))
 (def script-start (name) (assign this-script name) (chapter-end))
-(def script-end   (name) (assign this-script nil ) (chapter-end))
+(def script-end   (name) (assign this-script nil ) (fetch-and-clear-comments) (chapter-end))
 
 (def script-run (event name)
   (cond (eq? event 'plugin-start)
@@ -180,14 +185,27 @@
 
 (assign DEF-FORMS '(comment chapter))
 
+(def hash-init (h keys v)
+  (if keys
+      (do (hash-set h (car keys) v)
+          (hash-init h (cdr keys) v)))
+  h)
+
 (def build-def-hash ()
-  ((fn (h)
-       (map (fn (k) (hash-set h k nil)) DEF-FORMS)
-       h)
-   (hash)))
+  (hash-init (hash) DEF-FORMS nil))
 
 (def dox-build-def-name (name) name)
 
+(def dox-chapters-expr-helper (chaps)
+  (cond chaps
+        `(quote (,@chaps))))
+
+(def dox-chapters-expr (chaps)
+  (cond (chapter-current)
+        (dox-chapters-expr-helper (cons (chapter-current) (apply + nil chaps)))
+        (cond chaps
+              (dox-chapters-expr-helper (apply + chaps)))))
+
 ;; used internally by 'mac
 (def define-mac-expr (name args body-forms)
   `(do (hash-set macs ',name (fun ,args ,@(hash-get body-forms nil)))
@@ -196,7 +214,9 @@
                     ',(+ (fetch-and-clear-comments) (map car (hash-get body-forms 'comment)))
                     ',args
                     '(mac ,name ,args ,@(hash-get body-forms nil))
-                    ',(map car (hash-get body-forms 'chapter)))))
+                    (hash 'chapters ,(dox-chapters-expr (hash-get body-forms 'chapter))
+                          'plugin   ',this-plugin
+                          'file     ',this-script))))
 
 (hash-set macs 'mac
           (fn (name args . body)
@@ -207,14 +227,18 @@
              '("define a new global macro")
              '(name args . body)
              '`(hash-set macs ',name (fn ,args ,@body))
-             '(nydp-core))
+             (hash 'chapters '(nydp-core)
+                   'plugin   this-plugin
+                   'file     this-script))
 
 (dox-add-doc 'do
              'mac
              '("perform a series of operations")
              'args
              '`((fn nil ,@args))
-             '(nydp-core))
+             (hash 'chapters '(nydp-core)
+                   'plugin   this-plugin
+                   'file     this-script))
 
 ;; override later to use '= instead of 'assign, giving us hash-assignment and other goodies for free
 (mac def-assign args `(assign ,@args))
@@ -230,7 +254,9 @@
                     ',(+ (fetch-and-clear-comments) (map car (hash-get body-forms 'comment)))
                     ',args
                     '(def ,name ,args ,@(hash-get body-forms nil))
-                    ',(map car (hash-get body-forms 'chapter)))))
+                    (hash 'chapters ,(dox-chapters-expr (hash-get body-forms 'chapter))
+                          'plugin   ',this-plugin
+                          'file     ',this-script))))
 
 ;; define a new function in the global namespace
 (mac def (name args . body)