archsight 0.4.1 → 0.4.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0312d7e0c0790b8b2a117ad6277e49bfb18c61545134b27510fa3209e951697d
-  data.tar.gz: 94915564bbbab8c59e139422c1a99317ec7fd5a87370e9905b16ed330cd56951
+  metadata.gz: 66e3a2aa9223a9583edf06749e12bafe346bfd562570d299795f2334510ed36a
+  data.tar.gz: 52a7997e394247a86da162fb4015d97b755b2ac2fe69cfadd758f6d3e140c33b
 SHA512:
-  metadata.gz: 7b15fc5fb4130fe0783efe8e20cf7aa2ce3962e6566d54ae5c1c85834692e5363cc3a9667eb2f7cfbf17832badc0461cbf684c1d5a5c04711e5e529d1fb11bb2
-  data.tar.gz: 36e154fb3fefd3dd0de59ced6d6f404e4238242ec182d78f5ced051fec01002a3af8dbe11c8eced3ff16fc978791301627fe3f06641e0ea859e9a529326ea814
+  metadata.gz: adc6fb4fad180ae14920ecfbccebfb242c9b3d48bc6bfb196b645de67f4dc490e4922274175fa9f942cfe4d80a327b8a4cf171ad9ccc1edd195c5d283c636bba
+  data.tar.gz: e82af637b926e46db4b61c23269d8f5b630222b78f6e169d413b21aa2f12858cbc80cacbc8325ab1d550174e11fedc8ca1780489d025c3dabe5c1943d1fd887b

data/docs/diagram.md

@@ -43,6 +43,7 @@ archsight diagram three_tier.asd --relation=all    # also show control/data-flow
 archsight diagram overview.asd --theme=compact     # denser spacing + smaller fonts (or cozy, tight)
 archsight diagram overview.asd --legend=bottom     # legend below (or right/left/top/none; default auto)
 archsight diagram *.asd                            # render every file to its own .svg
+archsight diagram app.asd --resources=model/       # resolve include \"file://...\" paths from this directory
 archsight diagram --format *.asd                   # re-indent the sources in place
 archsight diagram --format --check *.asd           # list unformatted sources, exit 1 if any (CI)
 ```
@@ -298,6 +299,69 @@ no broken styling. Library users pass a `resolver:` to `Diagram.render`
 (any `call(reference)` returning a URL, `nil` for a resource that exists but has nowhere to link to (the node stays as it is), `:missing` or `:ambiguous`, see
 `Archsight::Helpers::ResourceResolver`) and, to collect failures, `unresolved: []`.
 
+## Including diagrams
+
+`include "<path>"` splices another diagram in at that position, so a part
+that several diagrams share (a platform, a network, a team's services) is
+written once:
+
+```
+group "app" {
+  component "web" { label "Web server" }
+  stack {
+    include "archsight://ApplicationComponent/Platform:Database"
+  }
+}
+include "file://shared/network.asd"
+web -> db
+```
+
+The options of an `include` go in braces after the path (`namespace`, see below).
+
+| Path | Included diagram |
+|------|------------------|
+| `archsight://<Kind>/<name>` | the `architecture/diagram` annotation of that resource |
+| `file://<path>` | an `.asd` file, relative to the resources directory (also in nested includes) |
+
+A path that leaves the resources directory (`..`, absolute paths, symlinks
+pointing out) or is not an `.asd` file is refused. An `include` stands
+wherever a `layer` or `stack` can: at the top level or inside a container
+(not inside a leaf like `component`, and not together with `columns`). Included
+diagrams may include others (up to 8 levels, 100 includes in all; a cycle is an
+error). Nodes are one namespace across all of it, so an id defined twice is an
+error, and an edge in either part may name a node of the other.
+
+### Including a diagram more than once
+
+Ids are one namespace, so including the same diagram twice would define
+every node twice. A `namespace` puts `<namespace>.` in front of every id the
+included diagram defines (nodes and dataflows, and the references to them inside
+it):
+
+```
+include "archsight://ApplicationComponent/Platform:Server" { namespace "a" }
+include "archsight://ApplicationComponent/Platform:Server" { namespace "b" }
+a.web -> b.db
+```
+
+A node without a `label` keeps showing its own id (`web`, not `a.web`), and the same
+id from different includes no longer clashes. A reference inside the included diagram to an id it
+does not define names a node of the including diagram and stays as it is. Namespaces
+nest: a diagram included with `namespace "o"` that itself includes one with `namespace "in"` has
+`o.in.…` ids. A namespace is a name of letters, digits, `_` and `-`, joined by dots.
+
+Dataflows and `theme`/`legend`/`ranks` apply to the whole diagram: an included
+diagram holding them keeps them when it is included at the top level, and they are
+ignored, with a warning, when it is included inside a container. The warning is
+written to the server log and listed by `archsight lint`.
+
+A missing resource or file, a bad path or a broken included diagram fails the
+diagram with an error that names the include (`archsight lint` reports it). Standalone
+`archsight diagram` takes `file://` paths relative to `--resources DIR` (default:
+the directory of the input) and has no `archsight://`. Library users pass `includes:` to
+`Diagram.render`, any `call(path)` returning the source or `:missing`, `:forbidden` or `:invalid`
+(see `Archsight::Helpers::DiagramIncludes`), and `warnings: []` to collect the warnings.
+
 ## Diagrams in markdown
 
 A fenced code block tagged `asd` in any markdown the web UI renders (docs,

data/docs/modeling.md

@@ -36,7 +36,7 @@ There are two families of verbs:
 
 | Family | Verbs | Written on | Points to |
 |--------|-------|------------|-----------|
-| Responsibility and provider (down) | `performedBy`, `ownedBy`, `executedBy`, `servedBy`, `guidedBy`, `maintainedBy`, `contributedBy`, `hasConcern`, `representedBy`, `realizedThrough`, `exposes`, `connects`, `mitigatedBy`, `influences`, `triggers`, `affects`, `assesses`, `contains`, `closedBy`, `compares` | the element that needs, uses or is concerned with something | what provides it or is responsible for it |
+| Responsibility and provider (down) | `performedBy`, `ownedBy`, `operatedBy`, `executedBy`, `servedBy`, `guidedBy`, `maintainedBy`, `contributedBy`, `hasConcern`, `representedBy`, `realizedThrough`, `exposes`, `provisions`, `connects`, `mitigatedBy`, `influences`, `triggers`, `affects`, `assesses`, `contains`, `closedBy`, `compares` | the element that needs, uses or is concerned with something | what provides it or is responsible for it |
 | Realization (up) | `realizes`, `partiallyRealizes`, `plans`, `satisfies`, `evidencedBy` (on the evidenced element), `provides` | the concrete element | the abstract element it answers to (requirement, goal, capability, plateau) |
 
 Rules:
@@ -154,7 +154,7 @@ TechnologySystemSoftware, TechnologyService, TechnologyArtifact). The same princ
 | Location | In which region or country is it? | A place on a map, not a building; holds facilities | the place ("Location:Germany:Frankfurt") |
 | TechnologyFacility | In which building or site does hardware stand? | Has an address and can be visited; can be nested (campus, hall) | the site ("Facility:Frankfurt:HallA") |
 | TechnologyEquipment | Which physical thing can you touch? | Racks, fabrics, fibre routes, carrier circuits; no software of its own | the thing and its place ("Equipment:Link:Frankfurt-Madrid") |
-| TechnologyDevice | Which single machine stands in the room? | One server, switch, router, firewall or appliance with a serial number; it is cabled | the host ("Device:Frankfurt:Server01") |
+| TechnologyDevice | Which single machine stands in the room? | One server, switch, router, firewall or appliance with a serial number (it is cabled), or one laptop, workstation or phone of a person or team | the host ("Device:Frankfurt:Server01") |
 | TechnologyCommunicationNetwork | What carries data between nodes? | Has a network role (`network/type`); exists beyond any single link | role and scope ("Network:Management:Frankfurt") |
 | TechnologyNode | On which machine does software run? | A VM, host role or logical group of machines with a lifecycle of its own | the host or host role ("Node:Frankfurt:Web01") |
 | TechnologySystemSoftware | What software hosts other software or data? | An operating system, database cluster, routing stack, platform | the product and role ("Tech:Routing:EVPN") |
@@ -166,8 +166,8 @@ TechnologySystemSoftware, TechnologyService, TechnologyArtifact). The same princ
 1. Is it a country, metro or region? Location.
 2. Is it a building, campus, hall or site that can be visited? TechnologyFacility.
 3. Is it a physical thing (rack, switch fabric, fibre, circuit) without software of its own? TechnologyEquipment.
-4. Is it one physical machine (server, switch, router, firewall, appliance) whose room or cabling matters?
-   TechnologyDevice.
+4. Is it one physical machine (server, switch, router, firewall, appliance, laptop, workstation, phone) whose room,
+   cabling or operating system matters? TechnologyDevice.
 5. Does it carry data between nodes, with a role such as underlay, overlay, management, public or VPN?
    TechnologyCommunicationNetwork.
 6. Is it a machine (virtual, or a host role) that software runs on? TechnologyNode.
@@ -201,7 +201,8 @@ TechnologySystemSoftware, TechnologyService, TechnologyArtifact). The same princ
 10. One `TechnologyDevice` per physical machine, `locatedAt` its room (a TechnologyFacility of `facility/type: room`
     the data center is `composedOf`). A group of machines with one role stays a logical `TechnologyNode` that is
     `realizedBy` its devices; a rack or hardware block is TechnologyEquipment `composedOf` them; a virtual chassis or
-    stack is a device `composedOf` its members.
+    stack is a device `composedOf` its members. A client device (`device/type` laptop, desktop, workstation,
+    thin-client, tablet, mobile) moves and needs no room; it may be `locatedAt` an office.
 11. A network is `realizedBy` the devices that build it (switches, routers) and `connects` the devices attached to it.
     Both are written on the network, so a switch attached to the network it builds closes no cycle.
 12. Cables are `connectedTo` between devices, declared once on one of the two ends. Cabling runs both ways by nature
@@ -213,6 +214,17 @@ TechnologySystemSoftware, TechnologyService, TechnologyArtifact). The same princ
     `servedBy` its team and `composedOf` its devices, which then need no team of their own. Both relations point
     down to the team, which points to nothing further. On a TechnologyFacility, `servedBy` stays the operator of
     the site.
+14. A client device (developer workstation, laptop, phone) is a `TechnologyDevice` with a client `device/type` and
+    `device/os`, `device/osVersion`, `device/arch`, `device/ownership`, `device/managed` and `device/encryption`; the
+    vendor and product are in `device/model` (also for a hardened workstation that runs the user's operating system
+    virtualized beside other isolated sessions, which is a `workstation`). Model the individual machine when it is
+    inventoried, otherwise one `TechnologyNode` per client platform `realizedBy` the devices. The operating system is
+    `TechnologySystemSoftware` `realizedBy` the node, the image it is installed from an artifact the node is
+    `assignedTo`; each isolated session of a hardened workstation can be a node of its own. Network access is written
+    on the node (`servedBy` the networks it uses: the office, the VPN) or, for a single named machine, on the network
+    (`connects`), never on the device, because a network is `realizedBy` its switches and the reverse arrow would
+    close a cycle. A machine that reaches no network (air-gapped) has neither relation. Who uses it is the team or role
+    (`servedBy`), not a person; groups of machines are tags and the end of support is `lifecycle/endOfLifeAt`.
 
 **Checked by `archsight lint`**
 
@@ -224,7 +236,7 @@ As above, these are **warnings** in the "Modeling warnings" section and never fa
 | `TechnologyCommunicationNetwork 'Network:Overlay:EVPN' is an overlay network served by no network` | 3 |
 | `TechnologyFacility 'Facility:Madrid:DC' is located at no Location` | where |
 | `TechnologyEquipment 'Equipment:Rack01' is located at no TechnologyFacility` | where |
-| `TechnologyDevice 'Device:Server01' is located at no TechnologyFacility` | 10 |
+| `TechnologyDevice 'Device:Server01' is located at no TechnologyFacility` (not for client device types) | 10 |
 | `TechnologyDevice 'Device:Switch01' is connectedTo itself` | 12 |
 | `TechnologyDevice 'Device:Router01' is a router that realizes no TechnologyCommunicationNetwork` | 11 |
 | `TechnologyDevice 'Device:a:server-3' is servedBy 'Kernel:Team', not by the team of its equipment ('Compute:Team')` | 13 |
@@ -547,7 +559,7 @@ Which kind to pick is in [Choosing the Kind: Infrastructure](#choosing-the-kind-
 | `TechnologyCommunicationNetwork` | `realizedBy` | `TechnologyEquipment` (the fabric), `TechnologySystemSoftware` (routing stack, VPN software) |
 | `TechnologyCommunicationNetwork` | `realizedBy` | `ApplicationComponent`, `ApplicationService` (software that builds the network: an SDN agent, a network operator) |
 | `TechnologyCommunicationNetwork` | `realizedBy` | `TechnologyDevice` (the switches and routers that build it) |
-| `TechnologyCommunicationNetwork` | `connects` | `TechnologyDevice` (the devices attached to it) |
+| `TechnologyCommunicationNetwork` | `connects` | `TechnologyDevice` (the devices attached to it, a named workstation behind a VPN) |
 | `TechnologyNode` | `servedBy` | `TechnologyCommunicationNetwork` |
 | `ApplicationInterface` | `servedBy` | `TechnologyCommunicationNetwork` (the networks on which it can be reached) |
 
@@ -593,7 +605,7 @@ center is not a computational resource.
 | Location | For a country, metro or region |
 | TechnologyFacility | For a data center, colocation, office or edge site (`facility/type`), also a campus or hall inside one |
 | TechnologyEquipment | Optional: racks, network fabrics, physical clusters inside a facility (`equipment/type`) |
-| TechnologyDevice | Optional: one physical machine, `locatedAt` its room (`facility/type: room`) |
+| TechnologyDevice | Optional: one physical machine, `locatedAt` its room (`facility/type: room`); a client device (laptop, workstation, phone) needs no room |
 
 Relations are written on the element that is placed (see [Direction of Relations](#direction-of-relations)):
 
@@ -604,6 +616,8 @@ Relations are written on the element that is placed (see [Direction of Relations
 | `TechnologyFacility` | `composedOf` | `TechnologyFacility` (nested sites, the rooms of a data center) |
 | `TechnologyFacility` | `servedBy` | `BusinessActor` (the operator) |
 | `TechnologyDevice`, `TechnologyEquipment`, `TechnologyNode`, `TechnologyCommunicationNetwork` | `ownedBy` | `BusinessActor`, `BusinessRole` (the team that operates it) |
+| `ApplicationComponent`, `ApplicationService`, `TechnologyService` | `operatedBy` | `BusinessActor`, `BusinessRole` (the team that runs it in production) |
+| `ApplicationComponent`, `ApplicationService`, `TechnologyService` | `maintainedBy` | `BusinessActor` (the team that builds and changes it) |
 | `TechnologyDevice`, `TechnologyEquipment`, `TechnologyNode` | `servedBy` | `BusinessActor` (the team it is dedicated to, e.g. a test partition) |
 
 `locatedAt` with many targets is the point: a service that runs in three data centers is **one** service with three
@@ -688,13 +702,19 @@ depend on each other, so `dependsOn` is exempt from the cycle check. In `example
 
 ### Where a Component Runs
 
-Three relations say what something runs on; each answers a different question:
+These relations say what something runs on or builds; each answers a different question:
 
 | Relation | Means | Example |
 |----------|-------|---------|
 | ApplicationComponent `servedBy technologyServices` | the platform service the component is deployed on | `Payments:API` runs in `Payments:Cluster` |
 | ApplicationComponent `servedBy technologySystemSoftware` | the runtime it runs on | a Kubernetes distribution, a JVM |
 | ApplicationService `servedBy technologyServices` | a platform service the service as a whole uses | a managed database, the clusters of its own components |
+| ApplicationComponent `provisions technologyServices` | the platform service the component creates and keeps in its desired state | an operator that builds the clusters of a product |
+
+`provisions` is written on the component, like every relation, so it points down to what the component creates: the
+service page lists its provisioner as incoming, and the impact of a provisioner is its outgoing relations (the
+services it builds). It is not a runtime dependency: a provisioned service keeps running while its provisioner is
+down, only changes stop. `suppliedBy` on the service stays for what it is made of (the system software).
 
 Deployment importers (Argo CD, Helm, Cluster API) know every cluster a deployable runs in and write the relation per
 component. Keep the service-level relation to the clusters of the service's own components: a sidecar or agent that
@@ -997,7 +1017,7 @@ is a sidebar filter and query term (`lifecycle/status == "deprecated"`).
 |------------|---------|
 | `lifecycle/status` | The state above |
 | `lifecycle/activeAt`, `lifecycle/deprecatedAt`, `lifecycle/sunsetAt`, `lifecycle/endOfLifeAt` | When it happened or will happen (ISO 8601 date or time, checked by `archsight lint`) |
-| `lifecycle/replacedBy` | The resource that replaces it: a name for a resource of the same kind, `Kind/Name` for another kind (a component replaced by an `ApplicationService/...`). Shown as a link, `archsight lint` checks that it exists |
+| `lifecycle/replacedBy` | The resource that replaces it: a name for a resource of the same kind, `Kind/Name` for another kind (a component replaced by an `ApplicationService/...`). Shown as a link behind the status, `archsight lint` checks that it exists |
 | `lifecycle/reason` | Why, in markdown |
 
 ```yaml
@@ -1010,6 +1030,9 @@ metadata:
     lifecycle/replacedBy: ApplicationService/ExportV2
 ```
 
+The detail page shows the status, the replacement as a link and, for a `deprecated`, `sunset` or `end-of-life`
+resource, the number of resources that still refer to it (a link to its Incoming relations) on one line.
+
 ## Common Patterns
 
 ### Microservice

data/lib/archsight/cli.rb

@@ -279,6 +279,7 @@ module Archsight
     option :theme, type: :string, desc: "Spacing/font-size theme (default: the file's own `theme` statement)"
     option :legend, type: :string, desc: "Legend placement (default: the file's own `legend` statement, else auto)"
     option :profile, aliases: "-p", type: :boolean, desc: "Print stage timing stats after each render"
+    option :resources, aliases: "-r", type: :string, desc: "Directory that `include \"file://...\"` paths are relative to (default: the input's directory)"
     option :format, type: :boolean, desc: "Re-indent the inputs in place instead of rendering them (-o: write a single input there)"
     option :check, type: :boolean, desc: "With --format: only list the inputs that are not formatted, exit 1 if any"
     def diagram(*inputs)
@@ -295,7 +296,7 @@ module Archsight
 
       status = diagram::CLI.new.run_with(
         inputs: inputs, output: options[:output], watch: options[:watch], profile: options[:profile],
-        format: options[:format], check: options[:check],
+        format: options[:format], check: options[:check], resources: options[:resources],
         style: options[:style], theme: options[:theme], legend: options[:legend],
         relation_filter: if relation.nil?
                            diagram::Relations::DEFAULT_FILTER

data/lib/archsight/database.rb

@@ -109,11 +109,12 @@ module Archsight
       matches.first if matches.one?
     end
 
-    # Collect unique annotation values across all instances of a kind
+    # Annotation values across all instances of a kind with the number of instances using each,
+    # most used first (ties by value): [[value, count], ...]
     def annotation_values(kind, annotation)
       instances = instances_by_kind(kind).values
-      values = instances.flat_map { |inst| Array(annotation.value_for(inst)) }
-      values.compact.uniq.sort
+      counts = instances.flat_map { |inst| Array(annotation.value_for(inst)).compact.uniq }.tally
+      counts.sort_by { |value, count| [-count, value.to_s] }
     end
 
     # Get filterable annotations with their values for a kind (excludes empty)

data/lib/archsight/diagram/cli.rb

@@ -3,6 +3,7 @@
 require "optparse"
 require_relative "."
 require_relative "formatter"
+require_relative "../helpers/diagram_includes"
 
 module Archsight
   module Diagram
@@ -79,6 +80,8 @@ module Archsight
           opts.on("-p", "--profile", "Print stage timing stats after each render") do
             options[:profile] = true
           end
+          opts.on("--resources=DIR", "Directory that include \"file://...\" paths are relative to " \
+                                     "(default: the input file's directory)") { |v| options[:resources] = v }
           opts.on("--format", "Re-indent the inputs in place instead of rendering them (-o: write a single input there)") do
             options[:format] = true
           end
@@ -155,7 +158,9 @@ module Archsight
         total_start = Process.clock_gettime(Process::CLOCK_MONOTONIC) if profile
 
         svg = Archsight::Diagram.render(File.read(input), relation_filter: options[:relation_filter], profile: stats,
-                                                          style: options[:style], theme: options[:theme], legend: options[:legend])
+                                                          style: options[:style], theme: options[:theme], legend: options[:legend],
+                                                          includes: includes_for(input, options), warnings: warnings = [])
+        warnings.each { |message| warn "archsight-diagram: #{input}: #{message}" }
 
         write_start = Process.clock_gettime(Process::CLOCK_MONOTONIC) if profile
         File.write(output, svg)
@@ -172,6 +177,11 @@ module Archsight
         1
       end
 
+      # Standalone, `include` reads `file://` paths only (there is no model for `archsight://`)
+      def includes_for(input, options)
+        Helpers::DiagramIncludes.new(nil, resources_dir: options[:resources] || File.dirname(File.expand_path(input)))
+      end
+
       def print_stats(stats, total_start)
         total = Process.clock_gettime(Process::CLOCK_MONOTONIC) - total_start
         parts = stats.map { |key, seconds| "#{key}=#{format("%.1fms", seconds * 1000)}" }

data/lib/archsight/diagram/errors.rb

@@ -5,5 +5,6 @@ module Archsight
     class Error < StandardError; end
     class ParseError < Error; end
     class GraphError < Error; end
+    class IncludeError < Error; end
   end
 end

data/lib/archsight/diagram/includes.rb

@@ -0,0 +1,168 @@
+# frozen_string_literal: true
+
+module Archsight
+  module Diagram
+    # Expands `include "<path>"` statements: each one is replaced, at its position, by the statements of the
+    # diagram it names. The renderer knows nothing about where diagrams live; the caller supplies a loader
+    # (`archsight://` and `file://` paths, see `Helpers::DiagramIncludes`).
+    #
+    # `loader.call(path)` returns the source of the included diagram, or `:missing` / `:forbidden` / `:invalid`.
+    # Included sources may include others. An include inside a container takes blocks and edges only: a
+    # `dataflow` or a `theme`/`legend`/`ranks` setting belongs to the whole diagram, so it is left out there
+    # and reported in `Result#warnings`.
+    # `include "x" { namespace "cs" }` puts `cs.` in front of every id the included diagram defines (nodes and
+    # dataflows, and the references to them inside it), so the same diagram can be included more than once;
+    # a node without a `label` keeps showing its unprefixed id. References to ids the diagram does not define
+    # are left alone: they name nodes of the including diagram.
+    # Edges need no such care, they are resolved by id once the whole graph is built, so a fragment may link to
+    # nodes of the diagram that includes it and the other way round.
+    class Includes
+      MAX_DEPTH = 8
+      MAX_INCLUDES = 100
+      REASONS = { missing: "not found", forbidden: "is not allowed", invalid: "is not a valid include path" }.freeze
+
+      # The expanded statements, the sources that were loaded (`{ path => source }`, in order) and what was
+      # left out of them (warning messages).
+      Result = Struct.new(:statements, :sources, :warnings, keyword_init: true)
+
+      # @param statements [Array] parsed statements of the main source
+      # @param loader [#call, nil]
+      # @raise [IncludeError]
+      def self.expand(statements, loader)
+        new(loader).expand(statements)
+      end
+
+      def self.includes?(statements)
+        statements.any? { |s| s.is_a?(AST::Include) || (s.is_a?(AST::Block) && includes?(s.children)) }
+      end
+
+      def initialize(loader)
+        @loader = loader
+        @sources = {}
+        @warnings = []
+        @count = 0
+      end
+
+      def expand(statements)
+        Result.new(statements: splice(statements, [], top: true), sources: @sources, warnings: @warnings)
+      end
+
+      private
+
+      def splice(statements, stack, top:)
+        statements.flat_map do |stmt|
+          case stmt
+          when AST::Include then included(stmt, stack, top: top)
+          when AST::Block
+            stmt.children = splice(stmt.children, stack, top: false)
+            [stmt]
+          else [stmt]
+          end
+        end
+      end
+
+      def included(stmt, stack, top:)
+        path = stmt.path
+        raise IncludeError, "include cycle: #{(stack + [path]).join(" -> ")} (line #{stmt.line})" if stack.include?(path)
+        raise IncludeError, "include #{path.inspect} (line #{stmt.line}) is nested deeper than #{MAX_DEPTH} levels" if stack.length >= MAX_DEPTH
+
+        @count += 1
+        raise IncludeError, "too many includes (more than #{MAX_INCLUDES})" if @count > MAX_INCLUDES
+
+        source = load(path, stmt.line)
+        @sources[path] = source
+        parsed = parse(path, source)
+        parsed = drop_whole_diagram_statements(path, parsed, stmt.line) unless top
+        statements = splice(parsed, stack + [path], top: top)
+        stmt.namespace ? Namespacer.new(stmt.namespace).apply(statements) : statements
+      end
+
+      def load(path, line)
+        raise IncludeError, "include #{path.inspect} (line #{line}) is not available here" unless @loader
+
+        result = @loader.call(path)
+        return result if result.is_a?(String)
+
+        reason = REASONS.fetch(result) { raise ArgumentError, "include loader returned #{result.inspect} for #{path.inspect}" }
+        raise IncludeError, "include #{path.inspect} #{reason} (line #{line})"
+      end
+
+      def parse(path, source)
+        Parser.parse(source, anon_scope: "i#{@count}_")
+      rescue ParseError, GraphError => e
+        raise IncludeError, "in include #{path.inspect}: #{e.message}"
+      end
+
+      # Dataflows and settings apply to the whole diagram, so inside a container they are ignored (with a warning)
+      def drop_whole_diagram_statements(path, parsed, line)
+        dropped, kept = parsed.partition { |s| s.is_a?(AST::DataFlow) || s.is_a?(AST::Setting) }
+        dropped.each do |s|
+          what = s.is_a?(AST::DataFlow) ? "dataflow #{s.id.inspect}" : "#{s.key} setting"
+          @warnings << "include #{path.inspect} (line #{line}): the #{what} (line #{s.line}) is ignored, it only applies when included at the top level"
+        end
+        kept
+      end
+
+      # Puts `<namespace>.` in front of the ids a list of statements defines, and of the references to them
+      class Namespacer
+        def initialize(namespace)
+          @prefix = "#{namespace}."
+          @defined = {}
+        end
+
+        def apply(statements)
+          collect(statements)
+          statements.each { |stmt| rename(stmt) }
+          statements
+        end
+
+        private
+
+        # Anonymous layers/stacks have ids no one can reference and that are unique already
+        def collect(statements)
+          statements.each do |stmt|
+            case stmt
+            when AST::Block
+              @defined[stmt.id] = true unless stmt.anonymous?
+              collect(stmt.children)
+            when AST::DataFlow then @defined[stmt.id] = true
+            end
+          end
+        end
+
+        def rename(stmt)
+          case stmt
+          when AST::Block then rename_block(stmt)
+          when AST::Edge
+            stmt.from = ref(stmt.from)
+            stmt.to = ref(stmt.to)
+          when AST::DataFlow then rename_dataflow(stmt)
+          end
+        end
+
+        def rename_block(block)
+          unless block.anonymous?
+            block.attrs["label"] ||= block.id # the label defaults to the id
+            block.id = ref(block.id)
+          end
+          block.children.each { |child| rename(child) }
+        end
+
+        def rename_dataflow(flow)
+          flow.attrs["label"] ||= flow.id
+          flow.id = ref(flow.id)
+          flow.hops.map! { |hop| hop.is_a?(AST::DataFlowGroup) ? rename_group(hop) : ref(hop) }
+        end
+
+        def rename_group(group)
+          group.branches.each { |branch| branch.hops.map! { |hop| ref(hop) } }
+          group
+        end
+
+        def ref(id)
+          @defined[id] ? "#{@prefix}#{id}" : id
+        end
+      end
+    end
+  end
+end

data/lib/archsight/diagram/parser/ast.rb

@@ -43,6 +43,11 @@ module Archsight
       # own attrs at graph-resolution time).
       DataFlowBranch = Struct.new(:hops, :attrs, :line, keyword_init: true)
 
+      # `include "<path>" [{ namespace "cs" }]` -- another diagram spliced in at this position (see `Includes`).
+      # Parsed like any statement, replaced by the included statements before the graph is built, so nothing
+      # downstream sees it. `namespace`, if any, puts `<namespace>.` in front of every id the included diagram defines.
+      Include = Struct.new(:path, :namespace, :line, keyword_init: true)
+
       # A top-level diagram-wide `key "value"` setting -- currently only
       # `theme "compact"` (see `Theme`).
       Setting = Struct.new(:key, :value, :line, keyword_init: true)

data/lib/archsight/diagram/parser/table_layout.rb

@@ -20,6 +20,8 @@ module Archsight
 
           raise ParseError, "#{label} at line #{line}: columns must be a positive integer, got #{raw.inspect}" unless raw.match?(/\A[1-9]\d*\z/)
 
+          raise ParseError, "#{label} at line #{line}: columns cannot be combined with include (the included blocks are not known yet)" if children.any?(AST::Include)
+
           blocks, edges = children.partition { |c| c.is_a?(AST::Block) }
           count = [raw.to_i, blocks.length].min
           return children if blocks.empty?

data/lib/archsight/diagram/parser.rb

@@ -14,9 +14,11 @@ module Archsight
     #
     # Grammar (informal):
     #   program    := statement*
-    #   statement  := block | dataflow | edge | setting
+    #   statement  := block | dataflow | edge | setting | include
+    #   include    := "include" STRING ["{" "namespace" STRING "}"] [";"]
+    #                 -- top level, or where a container may hold blocks and edges
     #   setting    := ("theme" | "legend" | "ranks") STRING [";"]   -- top level only
-    #   block      := (CONTAINER_KEYWORD | LEAF_KEYWORD) [STRING] "{" (attr | block | edge)* "}"
+    #   block      := (CONTAINER_KEYWORD | LEAF_KEYWORD) [STRING] "{" (attr | block | edge | include)* "}"
     #                 -- the id is required except for ANONYMOUS_KEYWORDS
     #   dataflow   := "dataflow" STRING "{" (hop | hop_group | attr)* "}"
     #                 -- needs >= 2 hops (per branch, if a hop_group is present);
@@ -45,6 +47,7 @@ module Archsight
       # `layer`/`stack` may be used without an id, purely as a layout hint
       # (see AST::Block#anonymous) — a `group`/`boundary` always renders a
       # visible box, so an unnamed one wouldn't make sense the same way.
+      NAMESPACE = /\A[A-Za-z0-9_-]+(\.[A-Za-z0-9_-]+)*\z/
       ANONYMOUS_KEYWORDS = %w[layer stack].freeze
       SETTING_KEYWORDS = %w[theme legend ranks].freeze
       EDGE_ARROWS = {
@@ -55,13 +58,16 @@ module Archsight
 
       attr_reader :cursor
 
-      def self.parse(source)
-        new(source).parse
+      # `anon_scope` namespaces the ids of anonymous layers/stacks, so the statements of an included
+      # source (parsed on its own) never collide with those of the including one.
+      def self.parse(source, anon_scope: nil)
+        new(source, anon_scope: anon_scope).parse
       end
 
-      def initialize(source)
+      def initialize(source, anon_scope: nil)
         @cursor = TokenCursor.new(Lexer.new(source).tokenize)
         @anon_counter = 0
+        @anon_scope = anon_scope
       end
 
       def parse
@@ -75,6 +81,8 @@ module Archsight
       def statement
         if check_block_keyword?
           block
+        elsif include_directive?
+          include_statement
         elsif cursor.check(:ident, "dataflow") && cursor.peek_type(1) == :string
           DataflowParser.new(cursor).parse
         elsif cursor.check(:ident) && EDGE_ARROWS.key?(cursor.peek_type(1))
@@ -108,22 +116,16 @@ module Archsight
         children = []
 
         parse_attrs_until_rbrace(attrs, label, expected: "an attribute, a nested block, an edge") do
-          if check_block_keyword?
-            children << block
-            true
-          elsif cursor.check(:ident) && EDGE_ARROWS.key?(cursor.peek_type(1))
-            children << edge
-            true
-          else
-            false
-          end
+          child = container_child
+          children << child if child
+          child
         end
 
         cursor.expect(:rbrace, "expected '}' to close #{label}")
 
         if LEAF_KEYWORDS.include?(kind_token.value) && !children.empty?
           raise ParseError, "#{label} at line #{kind_token.line} cannot " \
-                            "contain nested blocks or edges (did you mean 'group'?)"
+                            "contain nested blocks, edges or includes (did you mean 'group'?)"
         end
 
         raise ParseError, "#{label} at line #{kind_token.line}: columns only applies to containers" if LEAF_KEYWORDS.include?(kind_token.value) && attrs.key?("columns")
@@ -133,6 +135,46 @@ module Archsight
         AST::Block.new(kind: kind, id: id, anonymous: anonymous, attrs: attrs, children: children, line: kind_token.line)
       end
 
+      # The nested block, edge or include at the cursor, nil if it is something else (an attribute)
+      def container_child
+        if check_block_keyword?
+          block
+        elsif include_directive?
+          include_statement
+        elsif cursor.check(:ident) && EDGE_ARROWS.key?(cursor.peek_type(1))
+          edge
+        end
+      end
+
+      # `include "<path>"` -- `include -> x` is still an edge, so a node may be named "include"
+      def include_directive?
+        cursor.check(:ident, "include") && cursor.peek_type(1) == :string
+      end
+
+      def include_statement
+        keyword = cursor.advance
+        path = cursor.advance
+        namespace = include_options(path.value) if cursor.check(:lbrace)
+        cursor.consume(:semi)
+        AST::Include.new(path: path.value, namespace: namespace, line: keyword.line)
+      end
+
+      # `{ namespace "cs" }` after an include: the only option
+      def include_options(path)
+        cursor.advance
+        options = {}
+        until cursor.check(:rbrace)
+          line = cursor.current.line
+          key, value = cursor.check(:ident) ? attr : cursor.expect(:ident, "expected 'namespace' in the options of include #{path.inspect}")
+          raise ParseError, "unknown option '#{key}' of include #{path.inspect} at line #{line} (only 'namespace')" unless key == "namespace"
+          raise ParseError, "the namespace #{value.inspect} of include #{path.inspect} at line #{line} must be a name like \"cs\" or \"a.b\"" unless value.match?(NAMESPACE)
+
+          store_attr(options, key, value, line, "include #{path.inspect}")
+        end
+        cursor.expect(:rbrace, "expected '}' to close the options of include #{path.inspect}")
+        options["namespace"]
+      end
+
       # `theme "compact"`, `legend "right"` -- `theme -> x` is still an edge
       # (an arrow, not a string, follows the id), so a node may still be
       # named "theme" or "legend".
@@ -145,7 +187,7 @@ module Archsight
 
       def next_anonymous_id
         @anon_counter += 1
-        "__asd_anon_#{@anon_counter}"
+        "__asd_anon_#{@anon_scope}#{@anon_counter}"
       end
 
       # What a human should see identifying this block in an error message