puml-for-markdown 1.0.0

package/.nvmrc

@@ -0,0 +1 @@
+v17.3.1

package/CONTRIBUTING.md

@@ -0,0 +1 @@
+TODO

package/LICENSE.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) [2022] [Daniel Yakobian]
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,108 @@
+# puml-for-markdown
+
+## Goal
+The goal is to make PlantUML diagrams easily accessible from markdown, specifically GitHub flavored.
+* Should work with both private and public repositories
+* Should not have to use tokens to get it to work with private repositories
+* Should support hyperlinking to other diagrams
+* Should support PlantUML `!include`
+* Should support PlantUML [sprites](https://crashedmind.github.io/PlantUMLHitchhikersGuide/PlantUMLSpriteLibraries/plantuml_sprites.html) (small graphic images)
+* Any time you make changes to diagrams you should be able to run the CLI tool to update the markdown links
+
+## Examples
+The following examples are of a project I worked on, **yellow components will link you to another diagram**. Obviously the
+diagram content has been obfuscated.
+
+### Example With Link Only
+
+[Only Link](https://tinyurl.com/y847en68)<!--[Only Link](./puml/level_1_system_view.puml)-->
+
+### Example with Image, Click to Open Interactive Diagram
+
+[![Example With Graph Image](https://tinyurl.com/y847en68)](https://tinyurl.com/y847en68)<!--![Example With Graph Image](./puml/level_1_system_view.puml)-->
+
+## Background
+[PlantUML](https://plantuml.com) allows you to create diagrams that are defined using a simple and intuitive language.
+PlantUML diagrams are great for designing new projects but they don't work very well in Github markdown preview. There
+are some [workarounds](https://stackoverflow.com/questions/32203610/how-to-integrate-uml-diagrams-into-gitlab-or-github),
+but I found these to be unstable and they have a lot of caveats.
+
+### Installation
+`npm i -g puml-for-markdown`
+
+### Basic Usage
+Whenever you run the CLI it will add a tinyurl link to the rendered SVG next to the markdown comments referencing a
+puml diagram. E.g. `<!--[Example With Only Link](./puml/level_1_system_view.puml)-->` will be replaced with
+```
+[Example With Only Link](https://tinyurl.com/yfpclfpp)<!--[Example With Only Link](./puml/level_1_system_view.puml)-->
+```
+
+When `!` is included in front of the markdown link, it will render the diagram image in the markdown. If the image is
+clicked it will open up the diagram. E.g. `<!--![Example With Graph Image](./puml/level_1_system_view.puml)-->` will be
+replaced with
+```
+[![Example With Graph Image](https://tinyurl.com/yfpclfpp)](https://tinyurl.com/yfpclfpp)<!--![Example With Graph Image](./puml/level_1_system_view.puml)-->
+```
+
+If you want to update the link text or switch it between image and link, just update the comment and rerun the CLI. You
+don't need to delete the rendered image or link.
+
+Optionally you can specify the CLI to output the diagram images as png and/or svg.
+
+### CLI Usage
+```
+Usage: puml-for-markdown [options]
+
+An application to add interactive PUML diagrams to your github markdown files. If running with default arguments, run in project root directory.
+
+Options:
+  -x, --root-directory <path>      The path to your project (default: CWD)
+  -r, --hot-reload                 Rerun markdown generator every `interval` seconds, determined by interval option
+  -v, --interval-seconds <number>  If --hot-reload is set, how often should it reload (default: 2)
+  -p, --puml-directory <path>      Path to directory containing puml files which are referenced in markdown files (default: rootDirectory)
+  -m, --markdown-directory <path>  Path to directory containing markdown files referencing puml files (default: rootDirectory)
+  -g, --respect-gitignore          Automatically ignore MD files in .gitignore paths (default: true)
+  -i, --gitignore-path <path>      If --respect-gitignore is set, use set this as path to .gitignore file. (default: rootDirectory/.gitignore)
+  -d, --output-images              If set, will output images of diagrams to the dist directory
+  -b, --dist-directory <path>      If --output-images is set, path to output diagram images (default: rootDirectory/dist_puml)
+  -f, --image-formats <format>     If --output-images is set, sets the output image format (choices: "png", "svg", "both", default: "png")
+  -t, --turn-off-link-shortening   Use the full puml server link instead of the tiny url, if your diagrams are too big this won't work
+  -h, --help                       display help for command
+```
+
+### Notes
+- Comments within inline or multiline code styling will be ignored
+- Currently doesn't support cyclic graph references, i.e. a diagram can't reference any diagrams which reference back
+to it
+- See the [pre-commit hook](./.husky/pre-commit) to see how to add a git hook
+- If you are saving diagram images and have puml files which only define constants/settings
+(i.e. [example](./puml/constants.puml)) these aren't renderable on their own (since there is nothing to render)
+and you'll see a warning in the console saying it failed to save the image to file.
+
+### How It Works
+#### Using PlantUML Web Service to Render PUML Diagrams
+You can use PlantUML using the online web service to generate images on-the-fly. A online demonstration is available at
+[http://www.plantuml.com/plantuml](http://www.plantuml.com/plantuml). You can pass the
+[encoded text](https://plantuml.com/text-encoding) of your diagrams to the web service in the url path and it will
+generate an SVG or PNG for you. Here's a simple HelloWorld example
+[http://www.plantuml.com/plantuml/uml/Aov9B2hXil98pSd9LoZFByf9iUOgBial0000](http://www.plantuml.com/plantuml/uml/Aov9B2hXil98pSd9LoZFByf9iUOgBial0000).
+Large diagrams will have very long encoding strings, they can exceed maximum url length. They also don't look very good
+in markdown files. By default the CLI will use the tinyurl.com service to shorten the link to the diagram.
+
+#### Encoding PUML Diagrams for the Web Service
+The CLI will use the `plantuml-encoder` package to encode puml files. To support hyperlinking diagrams we need to parse
+all hyperlinks in the puml files. A dependency graph is created for the files and a DFS is performed where we create
+links for the leaf nodes first, then replace the links in the parent nodes with the links to the leaf nodes. The puml
+files are not actually modified, only the puml files content in memory is modified. By default the tinyurl free service
+is used to shorten the links.
+
+In order to support `!include` we parse puml files and replace any `!include` with the contents of the file referenced.
+
+#### Parsing Markdown
+The markdown files are then parsed for markdown comments. If the comments reference a PlantUML file, a link to the web
+service url will be added next to the comment. Because these links contain the full PlantUML diagram encoding in it,
+there are no issues using them in private repositories.
+
+### Testing
+Currently there is no TDD, for now I've just been using the example diagrams to test. The examples are pretty extensive
+though.

package/bin/index.js

@@ -0,0 +1,83 @@
+#!/usr/bin/env node
+
+const path = require('path')
+
+const {Command, Option} = require('commander')
+
+const run = require(path.resolve(__dirname, '../index.js'))
+
+
+// =============
+// = CLI Setup =
+// =============
+
+new Command()
+    .description('An application to add interactive PUML diagrams to your github markdown files. If running with default arguments, run in project root directory.')
+    .addOption(new Option(
+            '-x, --root-directory <path>',
+            'The path to your project'
+        ).default(process.cwd(), 'CWD')
+    )
+    .option(
+        '-r, --hot-reload',
+        'Rerun markdown generator every `interval` seconds, determined by interval option',
+    )
+    .option(
+        '-v, --interval-seconds <number>',
+        'If --hot-reload is set, how often should it reload',
+        2,
+    )
+    .addOption(new Option(
+            '-p, --puml-directory <path>',
+            'Path to directory containing puml files which are referenced in markdown files',
+        ).default(false, 'rootDirectory')
+    )
+    .addOption(new Option(
+            '-m, --markdown-directory <path>',
+            'Path to directory containing markdown files referencing puml files',
+        ).default(false, 'rootDirectory')
+    )
+    .option(
+        '-g, --respect-gitignore',
+        'Automatically ignore MD files in .gitignore paths',
+        true,
+    )
+    .addOption(new Option(
+            '-i, --gitignore-path <path>',
+            'If --respect-gitignore is set, use set this as path to .gitignore file.'
+        ).default(false, 'rootDirectory/.gitignore')
+    )
+    .option(
+        '-d, --output-images',
+        'If set, will output images of diagrams to the dist directory',
+    )
+    .addOption(new Option(
+            '-b, --dist-directory <path>',
+            'If --output-images is set, path to output diagram images'
+        ).default(false, 'rootDirectory/dist_puml')
+    )
+    .addOption(new Option(
+            '-f, --image-formats <format>',
+            'If --output-images is set, sets the output image format',
+        ).choices(['png', 'svg', 'both']).default('png')
+    )
+    .option(
+        '-t, --turn-off-link-shortening',
+        "Use the full puml server link instead of the tiny url, if your diagrams are too big this won't work",
+    )
+    .action(opts => {
+        opts.distDirectory = opts.distDirectory || path.resolve(opts.rootDirectory, 'dist_puml')
+        opts.gitignorePath = opts.gitignorePath || path.resolve(opts.rootDirectory, '.gitignore')
+        opts.markdownDirectory = opts.markdownDirectory || opts.rootDirectory
+        opts.pumlDirectory = opts.pumlDirectory || opts.rootDirectory
+        if (opts.imageFormats === 'both') {
+            opts.imageFormats = ['png', 'svg']
+        } else {
+            opts.imageFormats = [opts.imageFormats]
+        }
+        return run(opts).catch((e) => {
+            console.error('FATAL EXCEPTION')
+            console.error(e)
+        })
+    })
+    .parse(process.argv)

package/index.js

@@ -0,0 +1,230 @@
+/**
+ * Update any markdown links in a specified directory which reference a PUML diagram so they link directly to a
+ * puml server svg link. This is required for GitHub integration
+ * NOTE: Does not support cyclic dependencies, only DAGS
+ */
+
+const fs = require('fs')
+const path = require('path')
+const assert = require('assert')
+
+const bent = require('bent')
+const Promise = require('bluebird')
+const _ = require('lodash')
+const glob = require('glob')
+const plantUmlEncoder = require('plantuml-encoder')
+const tiny = require('tinyurl');
+const parseGitIgnore = require('gitignore-globs');
+
+
+// CONSTANTS
+
+const PUML_SVG_SERVER = 'https://www.plantuml.com/plantuml/svg'
+const PUML_PNG_SERVER = 'https://www.plantuml.com/plantuml/png'
+
+// HELPERS
+
+const getBuffer = bent('buffer')
+const mkdirIfDoesntExist = (p) => !fs.existsSync(p) && fs.mkdirSync(p, {recursive: true})
+const mapUniqMatches = (s, re, mapper, match = re.exec(s), results = []) => {
+    if (match) return mapUniqMatches(s, re, mapper, re.exec(s), results.concat([match]))
+    return _.uniqBy(results, String).map(mapper)
+}
+
+const replaceIgnore = (ignoreRegex, str, replaceFn) => {
+    const splitRegex = new RegExp(`(?:(${ignoreRegex}))`, 'g')
+    const splitStrArr = str.split(splitRegex)
+    const replacedSplitStrArr = splitStrArr.map((splitStr) => {
+        if (splitStr.match(new RegExp(ignoreRegex))) return splitStr
+        return replaceFn(splitStr)
+    })
+    return replacedSplitStrArr.join('')
+}
+
+const replaceMdIgnoringInlineCode = (mdString, replaceFn) => {
+    return replaceIgnore('`[^\n]*?(?=`)`', mdString, replaceFn)
+}
+
+const replaceMdIgnoringCodeBlocks = (mdString, replaceFn) => {
+    return replaceIgnore('```[\\s\\S]*?(?=```)```', mdString, replaceFn)
+}
+
+const replaceMdIgnoringCode = (mdString, replaceFn) => {
+    return replaceMdIgnoringCodeBlocks(mdString, str => replaceMdIgnoringInlineCode(str, replaceFn))
+}
+
+const getPngUrl = encodedData => `${PUML_PNG_SERVER}/${encodedData}}`
+const getSvgUrl = encodedData => `${PUML_SVG_SERVER}/${encodedData}}`
+const getPumlTinyUrl = encodedData => tiny.shorten(getSvgUrl(encodedData))
+const getImgUrl = (imageFormat, encodedData) => {
+    const imgFormatToGetUrlFn = {
+        png: getPngUrl,
+        svg: getSvgUrl
+    }
+    return imgFormatToGetUrlFn[imageFormat](encodedData)
+}
+
+// DATA STRUCTURES
+
+class PumlLinks extends Map {
+    constructor(pumlPaths, turnOffLinkShortening) {
+        super();
+        this.turnOffLinkShortening = turnOffLinkShortening
+        // Mark paths so we know which should be visited
+        // Any path that's not marked 0 we know isn't a puml path because it doesn't correspond to a puml fil
+        pumlPaths.forEach(p => this.set(p, 0))
+    }
+
+    get(p) {
+        return super.get(p);
+    }
+
+    async set(pumlPath, v) {
+        if (typeof v === 'number') return super.set(pumlPath, v)
+
+        const encodedData = plantUmlEncoder.encode(v)
+        const url = this.turnOffLinkShortening ? getSvgUrl(encodedData) : await getPumlTinyUrl(encodedData)
+        return super.set(pumlPath, {encodedData, url, data: v});
+    }
+
+    has(pumlPath) {
+        return super.has(pumlPath);
+    }
+}
+
+// SAVING DIAGRAMS
+
+const downloadImg = async (outputPath, imgUrl) => {
+    const imgBuffer = await getBuffer(imgUrl)
+    fs.writeFileSync(outputPath, imgBuffer)
+}
+
+const saveDiagram = async ({rootDirectory, distDirectory, pumlPath, imageFormat, encodedData}) => {
+    const outputPath = path.join(
+        distDirectory, pumlPath.replace(rootDirectory, '').replace(/\.puml$/, `.${imageFormat}`)
+    )
+
+    mkdirIfDoesntExist(path.dirname(outputPath))
+    const imgUrl = getImgUrl(imageFormat, encodedData)
+    try {
+        await downloadImg(outputPath, imgUrl)
+    } catch (e) {
+        console.warn(`\nWARN: Failed to save ${imgUrl} to ${outputPath}\n`)
+        if (e.statusCode !== 400) throw e
+    }
+}
+
+const saveDiagrams = async ({ rootDirectory, distDirectory, imageFormats, pumlLinks }) => {
+    fs.rmSync(distDirectory, {recursive: true, force: true});
+    for (let [pumlPath, {encodedData}] of pumlLinks) {
+        for (let imageFormat of imageFormats) {
+            await saveDiagram({
+                distDirectory, rootDirectory, pumlPath, imageFormat, encodedData
+            })
+        }
+    }
+}
+
+// PROCESSORS
+
+/**
+ * Add the whole relative includes file into the puml instead of a reference
+ * @returns {string} The puml data where relative includes are replaced with the data from referenced file
+ */
+const processIncludes = (pumlPath, data) => {
+    return data.replace(/!include (.*)/g, (fullMatch, url) => {
+        const fullUrl = path.resolve(path.dirname(pumlPath), url)
+        return fs.existsSync(fullUrl) ? fs.readFileSync(fullUrl, 'utf8') : fullMatch
+    })
+}
+
+
+/**
+ * Walk through each PUML file recursively so we can encode any linked files first since the files they link
+ * from are dependent
+ * @returns {Promise<string> | Promise<undefined>} Returns either processed puml file link, or original link
+ */
+const processPumlFile = async (pumlPath, pumlLinks) => {
+    // Base Cases
+    if (!pumlLinks.has(pumlPath)) return pumlPath
+    if (pumlLinks.get(pumlPath)) return pumlLinks.get(pumlPath)
+
+    let data = fs.readFileSync(pumlPath, 'utf8')
+    await Promise.all(mapUniqMatches(data, /\$link=["']([^"']+)['"]/gm, async ([pumlLink, pumlLinkPath]) => {
+        const newLink = await processPumlFile(path.resolve(path.dirname(pumlPath), pumlLinkPath), pumlLinks)
+        data = data.replaceAll(pumlLink, `$link="${newLink.url}"`)
+    }))
+
+    data = processIncludes(pumlPath, data)
+    await pumlLinks.set(pumlPath, data)
+    return pumlLinks.get(pumlPath)
+}
+
+/**
+ * Update any markdown links which reference PUML diagram so they link directly to a puml server svg link
+ * Output them as new markdown in the docs directory
+ */
+const processMdFile = async (mdPath, pumlLinks) => {
+    console.info(`Processing md file at ${mdPath}`)
+    let originalMdStr = fs.readFileSync(mdPath, 'utf8')
+    const findMdPumlLinksRE = new RegExp(
+        `(\\[?!?\\[?\\[[^\\[]*)?<!\-\-(!?\\[[^\\]]+\\])\\(([^)]+\\.puml)\\)\-\->`, 'g'
+    )
+    // Add puml server tinyurl link for puml links indicated in markdown comments
+    const mdWithUpdatedPumlLinks = replaceMdIgnoringCode(originalMdStr, (str) => {
+        return str.replace(findMdPumlLinksRE, (match, _, linkText, mdPumlLinkPath) => {
+            const pumlPath = path.resolve(path.dirname(mdPath), mdPumlLinkPath)
+            const pumlLink = pumlLinks.get(pumlPath)
+            if (!pumlLink) {
+                throw Error(`Could not find puml for md link path = ${mdPumlLinkPath}, absolute path = ${pumlPath}`)
+            }
+            // If the puml link is a markdown image
+            if (linkText[0] === '!') {
+                return `[${linkText}(${pumlLink.url})](${pumlLink.url})<!--${linkText}(${mdPumlLinkPath})-->`
+            }
+            // If the puml link is a markdown hyperlink
+            return `${linkText}(${pumlLink.url})<!--${linkText}(${mdPumlLinkPath})-->`
+        })
+    })
+
+
+    fs.writeFileSync(mdPath, mdWithUpdatedPumlLinks)
+}
+
+const runOnce = async (
+    {
+        rootDirectory,
+        markdownDirectory,
+        pumlDirectory,
+        distDirectory,
+        outputImages,
+        imageFormats,
+        respectGitignore,
+        gitignorePath,
+        turnOffLinkShortening,
+    }
+) => {
+    assert(fs.existsSync(markdownDirectory), "That's an invalid md folder path")
+    assert(fs.existsSync(pumlDirectory), "That's an invalid puml folder path")
+
+    const ignore = respectGitignore ? parseGitIgnore(gitignorePath) : []
+    const mdPaths = glob.sync(`${markdownDirectory}/**/*.md`, {ignore, nodir: true})
+    const pumlPaths = glob.sync(`${pumlDirectory}/**/*.puml`, {ignore, nodir: true})
+    const pumlLinks = new PumlLinks(pumlPaths, turnOffLinkShortening)
+
+    for (let p of pumlPaths) await processPumlFile(p, pumlLinks)
+    for (let p of mdPaths) await processMdFile(p, pumlLinks)
+
+    if (outputImages) {
+        await saveDiagrams({ rootDirectory, distDirectory, imageFormats, pumlLinks })
+    }
+}
+
+const reloadRun = ({intervalSeconds, ...rest}) =>
+    Promise.delay(intervalSeconds * 1000, rest)
+        .then(run).catch(console.error)
+        .finally(() => reloadRun({intervalSeconds, ...rest}))
+
+const run = ({hotReload, ...rest}) => hotReload ? reloadRun(rest) : runOnce(rest)
+
+module.exports = run

package/package.json

@@ -0,0 +1,47 @@
+{
+  "name": "puml-for-markdown",
+  "version": "1.0.0",
+  "description": "An application to add interactive PUML diagrams to your github markdown files. If running with default arguments, run in project root directory.",
+  "main": "index.js",
+  "scripts": {
+    "prepare": "husky install"
+  },
+  "author": "danielyaa5",
+  "license": "ISC",
+  "dependencies": {
+    "bent": "7.3.12",
+    "bluebird": "3.7.2",
+    "commander": "8.2.0",
+    "gitignore-globs": "0.1.1",
+    "glob": "7.2.0",
+    "lodash": "4.17.21",
+    "plantuml-encoder": "1.4.0",
+    "tinyurl": "1.1.7"
+  },
+  "bin": {
+    "puml-for-markdown": "./bin/index.js"
+  },
+  "keywords": [
+    "uml",
+    "puml",
+    "plantuml",
+    "c4-plantuml",
+    "c4",
+    "diagrams",
+    "git",
+    "github",
+    "md",
+    "markdown",
+    "generator",
+    "cli",
+    "diagrams",
+    "documentation"
+  ],
+  "repository": {
+    "type": "git",
+    "url": "https://github.com/danielyaa5/puml-for-markdown"
+  },
+  "devDependencies": {
+    "husky": "7.0.4"
+  }
+}

package/puml/c4_styles.puml

@@ -0,0 +1,37 @@
+@startuml
+
+'Icons
+!define FA_5_URL https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/e53fd61259c0f2449edfe304cad83f51d0fbb5c1/font-awesome-5
+!include FA_5_URL/cloud.puml
+!include FA_5_URL/youtube.puml
+!include FA_5_URL/server.puml
+
+!define MATERIAL_URL https://raw.githubusercontent.com/tupadr3/plantuml-icon-font-sprites/e53fd61259c0f2449edfe304cad83f51d0fbb5c1/material
+!include MATERIAL_URL/zoom_in.puml
+
+!define osaPuml https://raw.githubusercontent.com/Crashedmind/PlantUML-opensecurityarchitecture2-icons/master
+!include osaPuml/Common.puml
+!include osaPuml/User/all.puml
+
+!include <office/Servers/database_server>
+!include <office/Servers/file_server>
+!include <office/Servers/application_server>
+!include <office/Concepts/service_application>
+!include <office/Concepts/firewall>
+
+!define AWS_PUML https://raw.githubusercontent.com/awslabs/aws-icons-for-plantuml/main/dist
+!include AWS_PUML/AWSCommon.puml
+!include AWS_PUML/Robotics/RoboMaker.puml
+
+'Common Styling
+skinparam wrapWidth 250
+
+UpdateElementStyle("System", $bgColor="#ffffff", $fontColor="#000000", $shadowing="true")
+UpdateElementStyle("Person", $bgColor="#e0f1fb", $fontColor="#000000", $shape=RoundedBoxShape())
+
+AddElementTag("clickableText", $fontColor="purple")
+AddElementTag("button", $bgColor="#444444", $fontColor="#ffffff")
+AddElementTag("primary", $bgColor="#ffdb58", $fontColor="#000000", $shadowing="false", $borderColor="Gray")
+AddElementTag("highlightBoundary", $bgColor="#fce8ee")
+
+@enduml

package/puml/constants.puml

@@ -0,0 +1,32 @@
+@startuml
+' Tables
+!global $TYPE_X_SAMPLE_POOL_TABLE = "type_x_sample_pool"
+!global $TYPE_Y_SAMPLE_POOL_TABLE = "type_y_sample_pool"
+!unquoted procedure $SAMPLE_POOL_TABLE_DESC()
+A subquery from which to query samples, should be joined to any
+columns that you want to include in the $DAILY_COLLECTED_SAMPLES_TABLE table
+!endprocedure
+
+!global $SAMPLE_LABELS_TABLE = "some_other_label_db"
+!global $SAMPLE_LABELS_ID = "sampleLabelsDb"
+!global $SAMPLE_LABELS_CONTAINER_DESC = $LABELED_DATASOURCE_DESC
+!global $SAMPLE_LABELS_COMPONENT_DESC = "Stores all samples which have been labeled.\n"
+!global $SAMPLE_LABELS_TYPE = "BigTable"
+!global $SAMPLE_LABELS_SCHEMA = "https://docs.google.com/spreadsheets"
+
+!global $DAILY_COLLECTED_SAMPLES_TABLE = "type_y_samples_{date}"
+!global $DAILY_COLLECTED_SAMPLES_ID = "dailySamplesDb"
+!global $DAILY_COLLECTED_SAMPLES_DESC = "A days worth of samples collected during single run of the pipeline.\n"
+!global $DAILY_COLLECTED_SAMPLES_TYPE = "BigQueryDailyTable"
+
+!global $LABELED_DATASOURCE_ID = "labeledDatasource"
+!global $LABELED_DATASOURCE_NAME = "labeled_datasource_bucket"
+!global $LABELED_DATASOURCE_TYPE = "GCS"
+!global $LABELED_DATASOURCE_DESC = "Where the labelers write their labeling results"
+
+'Columns
+!global $SEND_FOR_LABELING_COLUMN = "send_for_labeling"
+!global $CONTENT_ID_COLUMN = "some_id"
+
+!global $SYSTEM_NAME = "My Project Name"
+@enduml

package/puml/level_1_system_view.puml

@@ -0,0 +1,36 @@
+@startuml
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Context.puml
+!include ./c4_styles.puml
+!include ./constants.puml
+
+left header
+
+\t<font size=22 color=black>$SYSTEM_NAME - System View</font>
+
+\t<font size=14 color=#484848>The goals of my project are described here</font>
+
+
+
+
+end header
+
+
+System(sysName, "Labeling System\n", "", $sprite="cloud", $tags="primary", $link="./level_2_container_view.puml")
+
+Boundary(labelers, ' \n<font color=black size=16>Labelers</font>', 'Human Resources') {
+    Person(labelerB, "Labelers B", "Labeler Type 2\n", $tags="a")
+    Person(labelerA, "Labelers A", "Labeler Type 2\n", $tags="a")
+}
+Rel_U(sysName, labelerB,"  Trigger samples for labeling")
+Rel_D(labelerB, sysName," Store labeling results        ")
+
+System(videoSamples, "Videos", "Users and internal customers", $sprite="youtube")
+Rel_D(videoSamples, labelerA, " Send samples to labelers")
+
+System(modelTrainingInference, "Model Training + Inference\n\n<font color=#e0575e><$RoboMaker></font>\n", "", $tags="secondary")
+Rel_D(labelerA, modelTrainingInference," Send labels")
+Rel_R(modelTrainingInference, sysName, "Send model scores")
+Rel_L(sysName, modelTrainingInference, "Send labels")
+
+HIDE_STEREOTYPE()
+@enduml

package/puml/level_2_container_view.puml

@@ -0,0 +1,55 @@
+@startuml
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Container.puml
+!include ./c4_styles.puml
+!include ./constants.puml
+
+
+left header
+
+\t<font size=22 color=black>$SYSTEM_NAME - Container View</font>
+
+\t<font size=14 color=black><U+25CF>  Goal 1</font>
+\t<font size=14 color=black><U+25CF>  Goal 2</font>
+\t<font size=14 color=black><U+25CF>  Goal 3</font>
+\t<font size=14 color=black><U+25CF>  Goal 4</font>
+
+
+
+
+
+end header
+
+
+Person(labelerA, "Labeler A", "\t     Labeler-A Description     \t\n", $tags="a")
+ContainerDb(bigQueryLabelDB, "LabelADb", "BigQuery", "A database where type A labels are stored")
+ContainerDb(modelScoreDb, "ModelScoreDB", "BigQuery", "A database where model scores are stored")
+Person(labelerB, "Labeler B", "\t     Labeler-B Description     \t\n", $tags="a")
+ContainerDb($LABELED_DATASOURCE_ID, "$LABELED_DATASOURCE_NAME", "$LABELED_DATASOURCE_TYPE", $LABELED_DATASOURCE_DESC)
+System_Boundary(sysName, ' \n<font color=black size=16>Label System FooBarBaz</font>') {
+    Container(samplingPipeline, "Pipeline", "some-etl-framework", "ETL job for sampling / labeling", $tags="primary", $link="./level_3_component_view_pipeline.puml")
+    Container(labelRetrievalJob, "LabelRetrievalJob", "some-etl-framework", "DAG for retrieving and storing label data from labelers.", $tags="primary", $link="./level_3_component_view_label_retrieval_job.puml")
+    ContainerDb($SAMPLE_LABELS_ID, $SAMPLE_LABELS_TABLE, $SAMPLE_LABELS_TYPE, $SAMPLE_LABELS_COMPONENT_DESC)
+    ContainerDb($DAILY_COLLECTED_SAMPLES_ID, $DAILY_COLLECTED_SAMPLES_TABLE, $DAILY_COLLECTED_SAMPLES_TYPE, $DAILY_COLLECTED_SAMPLES_DESC)
+    ContainerQueue(sampleQueue, "SampleQueueFoo", "SQS", "Samples which should be double reviewed")
+    Container(labelerBApi, "ReviewQueueAPI", "NodeJS", "API for pulling samples for review", $sprite="server")
+    Container(btBqSnapshotJob, "ExportBigQuerySnapshot", "some-etl-framework", "A job to take daily BQ snapshots\nof //$SAMPLE_LABELS_TABLE//", $tags="primary", $link="./level_4_activity_diagram_export_bq_snapshot_job.puml")
+}
+System(modelTrainingInference, "Model Training + Inference\n\n<font color=#e0575e><$RoboMaker></font>\n", "")
+
+Rel_R(samplingPipeline, $DAILY_COLLECTED_SAMPLES_ID, " ")
+Rel_D(samplingPipeline, sampleQueue, " ")
+BiRel_D($SAMPLE_LABELS_ID, samplingPipeline, " ")
+Rel_L(labelRetrievalJob, $SAMPLE_LABELS_ID, " ")
+Rel_L($LABELED_DATASOURCE_ID, labelRetrievalJob, " ")
+Rel_R(sampleQueue, labelerBApi, " ")
+Rel_R(labelerBApi, labelerB, " ")
+Rel_R(labelerB, $LABELED_DATASOURCE_ID, " ")
+Rel_L(modelScoreDb, samplingPipeline, " ")
+Rel_D(modelTrainingInference, modelScoreDb, " ")
+Rel_D(bigQueryLabelDB, modelTrainingInference, " ")
+Rel_L(labelerA, bigQueryLabelDB, " ")
+Rel_L($SAMPLE_LABELS_ID, modelTrainingInference, " ")
+Rel_D($SAMPLE_LABELS_ID, btBqSnapshotJob, " ")
+
+HIDE_STEREOTYPE()
+@enduml

package/puml/level_3_component_view_label_retrieval_job.puml

@@ -0,0 +1,40 @@
+@startuml
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
+!include ./c4_styles.puml
+!include ./constants.puml
+
+LAYOUT_LANDSCAPE()
+
+left header
+
+\t<font size=22 color=black>Some Job - Component View</font>
+
+
+
+
+end header
+
+
+Component(scheduler, "Scheduler", "some-etl-framework", "Kicks off the job at some interval")
+Component(extractTransform, "DatasourceExtractTransform", "Operator", "Pull data from labeler datasource and perform any transforms before storing the data")
+Component(loader, "DatasourceLoader", "Operator", "Update //$SAMPLE_LABELS_TABLE// with new labels")
+Component(someMicroService, "SomeMicroService", "SubDag", "Foo bar baz")
+Component(performCalculations, "PerformCalculation", "Operator", "Trigger the calculation of xyz")
+Component(calculatePrevalence, "CalculatePrevalence", "Operator", "Trigger the calculation prevalence and store it")
+ComponentDb($SAMPLE_LABELS_ID, $SAMPLE_LABELS_TABLE, $SAMPLE_LABELS_TYPE, $SAMPLE_LABELS_COMPONENT_DESC)
+
+Rel_R(scheduler, extractTransform, " ")
+Rel_R(extractTransform, loader, " ")
+Rel_R(loader, performCalculations, "When all labels retrieved")
+Rel_U(loader, $SAMPLE_LABELS_ID, " ")
+Rel_D(loader, "someMicroService", " ")
+Rel_R(performCalculations, calculatePrevalence, " ")
+
+center footer
+
+
+
+end footer
+
+HIDE_STEREOTYPE()
+@enduml

package/puml/level_3_component_view_pipeline.puml

@@ -0,0 +1,58 @@
+@startuml
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
+!include ./c4_styles.puml
+!include ./constants.puml
+
+LAYOUT_LANDSCAPE()
+
+left header
+
+\t<font size=22 color=black>The Pipeline - Component View</font>
+
+\t<font size=14 color=black><U+25CF>  Goal 1</font>
+\t<font size=14 color=black><U+25CF>  Goal 2</font>
+\t<font size=14 color=black><U+25CF>  Goal 3</font>
+\t<font size=14 color=black><U+25CF>  Goal 4</font>
+
+
+
+
+
+end header
+
+
+Person(labelerB, "Labeler B", " Labeler B Description \n", $tags="a")
+ContainerDb($LABELED_DATASOURCE_ID, "$LABELED_DATASOURCE_NAME", "$LABELED_DATASOURCE_TYPE", $LABELED_DATASOURCE_DESC)
+Boundary(pipeline, " \nEntire Pipeline Infrastructure\n") {
+    ComponentDb($SAMPLE_LABELS_ID, $SAMPLE_LABELS_TABLE, $SAMPLE_LABELS_TYPE, $SAMPLE_LABELS_COMPONENT_DESC)
+    Boundary(pipelineDag, "Pipeline DAG", $tags="highlightBoundary") {
+        Component(scheduler, "Scheduler", "some-etl-framework", "Kicks off pipeline at some interval")
+        Component(waitForDependentTables, "Wait For Dependent Tables", "worker", "Waits for dependent tables to be populated")
+        Component(prevalenceSampler, "Label Type A Sampler", "worker", "Queries for type A labels", $tags="primary", $link="./level_4_activity_diagram_sampler_a.puml")
+        Component(trainingSampler, "Label Type B Sampler", "worker", "Receives all sampling labels type B", $tags="primary", $link="./level_4_activity_diagram_sampler_b.puml")
+        Component(sendSamplesForLabeling, "SendSamplesForLabeling", "worker", "Send samples for labeling")
+    }
+    Container(reviewQueueApi, "ReviewQueueAPI", "NodeJS", "An API used for pulling labels", $sprite="server")
+    ComponentDb($DAILY_COLLECTED_SAMPLES_ID, $DAILY_COLLECTED_SAMPLES_TABLE, $DAILY_COLLECTED_SAMPLES_TYPE, $DAILY_COLLECTED_SAMPLES_DESC)
+    ComponentQueue(samplesForLabelingQueue, "SamplesForLabelingQueue", "SQS", "Queue for double review")
+}
+ContainerDb(modelScoreDb, "ModelScoreDB", "BigQuery", "A database where the inference score is stored")
+
+
+
+Rel_R(scheduler, waitForDependentTables, " ")
+Rel_R(waitForDependentTables, prevalenceSampler, "Query used to create table to sample from")
+Rel_R(prevalenceSampler, trainingSampler, " ")
+Rel_U(modelScoreDb, prevalenceSampler, " ")
+Rel_D($SAMPLE_LABELS_ID, prevalenceSampler, "Check what samples have already been sent for labeling**\n** ")
+Rel_R(trainingSampler, sendSamplesForLabeling, " ")
+Rel_D(sendSamplesForLabeling, $SAMPLE_LABELS_ID, " ")
+Rel_R(sendSamplesForLabeling, samplesForLabelingQueue, " ")
+Rel_U(trainingSampler, $DAILY_COLLECTED_SAMPLES_ID, " ")
+Rel_U(prevalenceSampler, $DAILY_COLLECTED_SAMPLES_ID, " ")
+Rel_R(samplesForLabelingQueue, reviewQueueApi, " ")
+Rel_R(reviewQueueApi, labelerB, " ")
+Rel_U(labelerB, $LABELED_DATASOURCE_ID, " ")
+
+HIDE_STEREOTYPE()
+@enduml

package/puml/level_4_activity_diagram_export_bq_snapshot_job.puml

@@ -0,0 +1,13 @@
+@startuml
+
+!include https://raw.githubusercontent.com/plantuml-stdlib/C4-PlantUML/master/C4_Component.puml
+!include ./c4_styles.puml
+
+center header
+
+
+\t<font size=30 color=black>Under construction...  </font>
+
+
+end header
+@enduml

package/puml/level_4_activity_diagram_sampler_a.puml

@@ -0,0 +1,79 @@
+@startuml
+!include ./styles.puml
+!include ./constants.puml
+
+center header
+
+\t<font size=22 color=black>Sampler A - Activity Diagram</font>
+
+
+end header
+
+start
+#E4D4EF: <font size=14>Inputs</font>
+----
+
+*<U+00A0>""sample_padding_factor"": Description of // sample_padding_factor //
+
+*<U+00A0>""sample_strategies"": Description of // sample_strategies //
+
+*<U+00A0>""$TYPE_X_SAMPLE_POOL_TABLE"": $SAMPLE_POOL_TABLE_DESC()
+]
+-> \n;
+
+partition "**operator** OperatorABC " {
+    :**REMOVE** samples of type x from ""$TYPE_X_SAMPLE_POOL_TABLE"" //i.e.//
+
+    ""SELECT * FROM $TYPE_X_SAMPLE_POOL_TABLE EXCEPT DISTINCT (SELECT * FROM $DAILY_COLLECTED_SAMPLES_TABLE)"";
+    while(""**FOR EACH:** sampling_strategy"")
+        #E5E5E5: Create sub-query table ]
+        #E5E5E5: ""LIMIT"" = ""sampling_strategy.sample_size * sample_padding_factor""]
+        #E5E5E5: ""SELECT sampling_strategy.name AS strategy"" (to add strategy column)]
+        #LightBlue: <size: 11>**NEXT:** ""sampling_strategy"">
+    endwhile;
+    : ""UNION ALL"" strategy tables;
+    : **DEDUPLICATE** samples by adding the """strategies""" column. For each duplicate add the strategy to the
+     ""strategies"" column in the distinct row. //i.e.//
+
+     ""SELECT ARRAY_AGG(strategy) as strategies... FROM all_strategies_table GROUP BY uid"";
+    : **SAVE** subquery to ""distinct_type_x_sample_pool"" table;
+}
+-> \n;
+-> \n;
+#E4D4EF: <font size=14>Outputs</font>
+----
+
+*<U+00A0>""distinct_type_x_sample_pool"": Description of //distinct_type_x_sample_pool// output
+
+]
+-> \n;
+-> \n;
+partition "**operator** OperatorDEF\n<font size=10>Note: [[https://google.com More details on algorithm here]]</font>  " {
+    :**SET** variable...  ""extra_space **=** COUNT(*) of $DAILY_COLLECTED_SAMPLES_TABLE WHERE $SEND_FOR_LABELING_COLUMN = TRUE"";
+    #LightBlue:while(""**FOR EACH:** sampling_strategy"")
+        #E5E5E5: ""samples_for_strategy = SELECT * WHERE sampling_strategy.name IN strategies ORDER BY len(strategies) ASC""]
+
+        #E5E5E5: **""WHILE len(samples_for_strategy) > sampling_strategy.sample_size:""**
+
+        ****<U+00A0> bla bla bla
+        ****<U+00A0> code code code;
+
+        #E5E5E5: **""FOR sample IN samples_for_strategy""**
+
+        ****<U+00A0> 1234
+        ****<U+00A0> 5678]
+        #E5E5E5: **IS** ""len(strategy_samples)"" **less than** ""strategy.sample_size"" **?**]
+        split
+            <<true>>: **Yes** ]
+            <<true>>: ""extra_space += sampling_strategy.sample_size - len(strategy_samples)""]
+        split again
+            #FFFFFF: **No** ]
+            #FFFFFF:  \t\t\t\t\t\t Do nothing \t\t\t\t\t\t]
+        end split
+        #LightBlue: <size: 11>**NEXT:** ""sampling_strategy"">
+    endwhile;
+    : **REALLOCATE** ""extra_space"" to increase the sample sizes of strategies whose size was reduced;
+    : **WRITE** output to ""$SAMPLE_LABELS_TABLE"" table;
+}
+stop
+@enduml

package/puml/level_4_activity_diagram_sampler_b.puml

@@ -0,0 +1,32 @@
+@startuml
+!include ./styles.puml
+!include ./constants.puml
+
+left header
+
+\t<font size=22 color=black>Sampler B - Activity Diagram</font>
+
+\t<font size=14 color=black>Description of goal                                            </font>
+
+
+end header
+
+start
+#E4D4EF: <font size=14>Inputs</font>
+----
+
+*<U+00A0>""$TYPE_Y_SAMPLE_POOL_TABLE"": These are samples of type y
+
+*<U+00A0>""sample_size"": How many samples to collect
+]
+-> \n;
+
+partition "**operator** OperatorFooBar " {
+    : **LIMIT** ""$TYPE_Y_SAMPLE_POOL_TABLE"" to ""sample_size"";
+    : **SET** ""field_a = 'a'"" on all samples;
+    :\t  **SET** ""$SEND_FOR_LABELING_COLUMN = (sample IN $SAMPLE_LABELS_TABLE)""  \t;
+    :**WRITE** result to $DAILY_COLLECTED_SAMPLES_TABLE table;
+}
+
+stop
+@enduml

package/puml/level_4_erd.puml

@@ -0,0 +1,106 @@
+@startuml
+!include ./constants.puml
+
+hide circle
+skinparam linetype ortho
+skinparam wrapWidth 350
+
+left header
+
+\t<font size=22 color=black>Entity Relation Diagram</font>
+
+
+end header
+
+!global $TABLE_1 = 'table 1'
+!global $TABLE_1_TYPE = 'BigQuery'
+!global $TABLE_1_DESC = 'Description for table 1'
+
+!global $TABLE_2 = 'table 2'
+!global $TABLE_2_TYPE = 'BigQuery'
+!global $TABLE_2_DESC = 'Description for table 2'
+
+!global $TABLE_3 = 'table 3'
+!global $TABLE_3_TYPE = 'BigQuery'
+!global $TABLE_3_DESC = 'Description for table 3'
+
+!global $TABLE_4 = 'table 4'
+!global $TABLE_4_TYPE = 'BigQuery'
+!global $TABLE_4_DESC = 'Description for table 4'
+
+entity "**$TABLE_1**" {
+  **$TABLE_1_TYPE**
+
+  $TABLE_1_DESC
+  ..
+  + ""id"": //string //
+  --
+  * ""some_id"": //string //: Description for some_id
+  * ""some_id_2"": //string// : Description for some_id_2
+  * ""config"": // JSON //: Description for config
+}
+
+entity "**$TABLE_2**" {
+  **$TABLE_2_TYPE**
+
+  $TABLE_2_DESC
+  ..
+  + ""id_1"": //string //
+  + ""id_2"": //string //
+  --
+  * ""label"": //string //: Description for label
+  * ""scheduled_at"": //timestamp //: Description for scheduled_at
+  * ""foo_bar_ts"": //timestamp //: Description for foo_bar_ts
+  * ""baz_ts"": //timestamp //: Description for baz_ts
+  * ""abc_ts"": //timestamp //: Description for abc_ts
+}
+
+entity "**$TABLE_3**" {
+  **$TABLE_3_TYPE**
+
+  $TABLE_3_DESC
+  ..
+  + ""id"": //string //
+  --
+  * ""thingy_id"": //string //: Description for thingy_id
+  * ""config"": // JSON //: Configuration used
+}
+
+entity "**$DAILY_COLLECTED_SAMPLES_TABLE**" {
+  **$DAILY_COLLECTED_SAMPLES_TYPE**
+
+  $DAILY_COLLECTED_SAMPLES_DESC
+  ..
+  + ""$CONTENT_ID_COLUMN"": //string //
+  --
+  *""strategy"": //string //: Description of strategy
+  *""$SEND_FOR_LABELING_COLUMN"": //boolean //: Description of send for labeling
+  ..
+  Metadata Columns
+  *""model_score"": //number //
+  *""num_views"": //number //
+  *""is_awesome"": //boolean //
+  .
+  .
+  .
+}
+
+entity "**$TABLE_4**" {
+  **$TABLE_4_TYPE**
+
+  $TABLE_4_DESC
+  ..
+  + ""id"": //string//
+  --
+  *""name"": //string  //: Description of name
+  *""value"": //string  //: Description of value
+  *""labeler_id"": //string //: Description of labeler_id
+  *""source"": //string //: Description of source
+  *""some_other_field"": //string //: Description of some_other_field
+}
+
+"**$TABLE_1**" }o-- "**$TABLE_2**"
+"**$TABLE_2**" }o-- "**$TABLE_3**"
+"**$TABLE_2**" }o-- "**$TABLE_4**"
+
+@enduml

package/puml/styles.puml

@@ -0,0 +1,22 @@
+@startuml
+<style>
+    activityDiagram {
+        diamond {
+            BackgroundColor LightBlue
+        }
+
+        .input {
+            MaximumWidth 500
+            BackgroundColor #E4D4EF
+        }
+    }
+    * {
+    }
+
+    .true {
+        FontColor White
+        BackgroundColor Black
+    }
+
+</style>
+@enduml