@webhandle/external-resource-manager 1.0.0

package/README.md

@@ -0,0 +1,203 @@
+# @webhandle/external-resource-manager
+
+Coordinates which external resources like scripts and css get put on
+a page.
+
+## Install
+
+```bash
+npm install @webhandle/external-resource-manager
+```
+
+## Purpose
+
+Integrating components written by different people at different times is messy.
+
+One approach is the one Wordpress uses: every component includes its own css files
+and scripts and there's a dependency system to make sure dependencies get loaded and
+get loaded before the dependents. This is kinda complicated and results in every page
+having a million little files that the browser has to fetch.
+
+Another approach is to use Webpack or Browserify to compile all the small files into
+one larger file, and just have the page load that. This also works, but now the complication
+is that you have to figure out all of the files to be compiled and if a compoment adds a
+file, now the page is broken.
+
+External Resource Manager represents a third approach, allowing the component flexibility
+to include what it wants and the ability to combine files for performance.
+
+## Usage
+
+The two main setup functions are:
+
+```js
+externalResourceManager.includeResource({
+	mimeType: type
+	, url: url
+})
+```
+
+and 
+
+```js
+externalResourceManager.provideResource({
+	mimeType: type
+	, url: url
+	, name: name
+	, resourceType: subType
+})
+```
+
+`includeResource` is a directive to include a resource in the page. It will result in a
+`link` element or `script` element or whatever is appropriate for its type.
+
+`provideResource` lets components state that a resource is available for use, if somebody
+wants to use it. It does not necessarily result in anything getting loaded by the browser.
+
+Components all given a chance to call these function with middleware that they add to the
+express routers.
+
+Later, when html for the page is being created, `externalResourceManager.render()` can be
+called, which creates html to include the resource. It's safe to call `render` multiple 
+times because subsequent calls will only create html for resources added since the last call
+to `render`. This means templates themselves can include resesources on the page, so long
+as those resources can tolerate being included in the body.
+
+To specify the additional attributes that are sometimes used with elements, you can call like:
+
+```js
+externalResourceManager.includeResource({
+	mimeType: 'application/javascript'
+	, url: '/js/pages.mjs'
+	, attributes: {
+		defer: undefined
+	}
+	, cachable: false
+})
+```
+
+Any attribute with a `null` or `undefined` value will/should be rendered to the page as an
+attribute without a value like:
+
+```html
+<script src="/js/pages.mjs" defer ></script>
+```
+
+As you can see in the example above, resources may also have a `cachable` attribute, which is
+true by default. Whether or not this resource ever actually has cache headers or version
+url parameters applied to it depends on the renderers being used. The attribute just signifies
+this resource is eligible to be used with whatever caching mechanism the site is using and isn't
+some sort of url which resolves to dynamic code and only looks like a file URL.
+
+## How does this help with dependencies?
+
+`ExternalResourceManager` guarantees that it will not include a url twice, so any component
+is free to explicitly add a resource or call a function from the dependency which adds a
+resource without fear that it will be included multiple times. Resources are rendered in the
+order added, so as long as a component includes its dependencies first, everything should work
+out. This avoids having named resources and a dependency tree at the small cost of having code
+which should understand its dependencies call a function on them.
+
+Further, `ExternalResourceManager` makes use of javascript modules and importmap scripts. This
+allows a component to provide functionality to the page which will get loaded ONLY if it's needed.
+To provide a module to a page, we call like:
+
+```js
+externalResourceManager.provideResource({
+	url: '/js/one.js'
+	, mimeType: 'application/javascript'
+	, resourceType: 'module'
+	, name: '@webhandle/moduleone'
+})
+```
+
+This will produce an entry in an importmap so that any javascript which makes a call like:
+
+```js
+import myImportantFunction from "@webhandle/moduleone"
+```
+will cause the browser to load the module from the server. In this way you can add substantial
+libraries of modules to the page with a performance hit since they're only loaded at need.
+Additionally, because the importmap is parsed by the browser before any of the modules are actually
+loaded, it doesn't matter what order the modules are added via `provideResource`.
+
+A note: at the time of this writing, Firefox does not support multiple importmaps, so if possible,
+components which provide resources should try to do so before rendering starts. Since providing a
+resource does not result in any extra requests to the server, this should be safe to do in middle
+for any request which might even potentially make use of the component.
+
+## How does this get rid of the million requests problem?
+
+Inlcuded resources are able to say that they "satisfy" other included URLs. For instance, let's say
+individual components have included css files that they need.
+
+```js
+externalResourceManager.includeResource({
+	mimeType: 'text/css'
+	, url: '/small/package/level.css'
+})
+
+externalResourceManager.includeResource({
+	mimeType: 'text/css'
+	, url: '/small/package/level-too.css'
+})
+```
+We can at any time, either before those components include their resources or after, have code
+like the following:
+
+```js
+externalResourceManager.includeResource({
+	mimeType: 'text/css'
+	, url: '/css/compiled.css'
+	, satisfies: [
+		'/small/package/level.css'
+		, '/small/package/level-too.css'
+	]
+})
+```
+
+This will cause `/css/compiled.css` to get included on the page while `/small/package/level.css`
+and `/small/package/level-too.css` will not.
+
+This is "satisfies" instead of "contains" because `/css/compiled.css` may not actually have the
+code from `level.css` in it at all. However, it claims to be functionally equivalent. This lets
+us replace styles we don't like with styles we do like, as well as just bundling thing up for 
+performance reasons.
+
+This works so long as all these call are made before `render` is called.
+
+In this way, we can get the best of both worlds for very little cost. We can add a new component
+to the site and just have it work. After adding too many new components, we can set up builds to 
+combine those small files and just have those loaded (and not even have to worry about trying to
+tell the component not to load its resources). And when we add a new component, no problem,
+this small dependencies will get added to the page until we include them in the build.
+
+
+## Usage in Webhandle
+
+In Webhandle, every response has its own `ExternalResourceManager` at `res.locals.externalResourceManager`
+
+That's kind of a weird place to put it, I know, but this allows any template to include a resource
+and allows any template to call `render`.
+
+### Resource Type Support
+
+Currently there's code to include mime types `application/javascript` and `text/css`. There's also
+code to create an importmap based on mime type `application/javascript` with resource type `module`.
+`ExternalResourceManager` works internally by having code registered to handle different mime types,
+so extensability is really easy if you wanted to add `meta` element or `title` element handling
+(or something).
+
+
+### Integration
+
+As you'd expect, this can be integrated into webhandle by calling:
+
+```js
+import setup from "@webhandle/external-resource-manager/initialize-webhandle-component.mjs"
+await setup(myWebhandleInstance, options)
+```
+
+
+
+

package/create-application-javascript-renderer.mjs

@@ -0,0 +1,23 @@
+import createAttributes from "./create-attributes.mjs"
+import escapeAttributeValue from "./escape-attribute-value.mjs"
+
+export default function createApplicationJavascriptRenderer(webhandle) {
+	function renderApplicationJS(resource) {
+
+		let vrsc = ''
+		if (!webhandle.development && resource.cachable) {
+			vrsc = '/vrsc/' + webhandle.resourceVersion
+		}
+
+		let html = `<script src="${vrsc}${escapeAttributeValue(resource.url)}" `
+
+		html += createAttributes(resource.attributes)
+		if ('type' in resource.attributes === false && resource.resourceType === 'module') {
+			html += ' type="module"'
+		}
+		html += '></script>'
+		return html
+	}
+
+	return renderApplicationJS
+}

package/create-attributes.mjs

@@ -0,0 +1,17 @@
+import escapeAttributeValue from "./escape-attribute-value.mjs"
+
+/**
+ * Creates an html attributes string based on the members of an object
+ * @param {object} attributes 
+ */
+export default function createAttributes(attributes) {
+	let html = Object.entries(attributes).map(entries => {
+		if (entries[1] === null || entries[1] === undefined) {
+			return entries[0]
+		}
+		else {
+			return entries[0] + '="' + escapeAttributeValue(entries[1]) + '"'
+		}
+	}).join(' ')
+	return html
+}

package/create-css-renderer.mjs

@@ -0,0 +1,19 @@
+import createAttributes from "./create-attributes.mjs"
+import escapeAttributeValue from "./escape-attribute-value.mjs"
+export default function createTextCssRenderer(webhandle) {
+	function renderTextCss(resource) {
+		
+		let vrsc = ''
+		if(!webhandle.development && resource.cachable) {
+			vrsc = '/vrsc/' + webhandle.resourceVersion
+		}
+		
+		let html = `<link href="${vrsc}${escapeAttributeValue(resource.url)}" rel="stylesheet" `
+
+		html += createAttributes(resource.attributes)
+		html += '/>'
+		return html
+	}
+
+	return renderTextCss
+}

package/escape-attribute-value.mjs

@@ -0,0 +1,23 @@
+
+/**
+ * 
+ * @param {string} s the value to be escaped 
+ * @param {boolen} preserveCR If true '
' will be used instead of \n
+ * @returns 
+ */
+export default function escapeAttributeValue(s, preserveCR) {
+	preserveCR = preserveCR ? '
' : '\n';
+	return ('' + s) /* Forces the conversion to string. */
+		.replace(/&/g, '&') /* This MUST be the 1st replacement. */
+		.replace(/'/g, ''') /* The 4 other predefined entities, required. */
+		.replace(/"/g, '"')
+		.replace(/</g, '&lt;')
+		.replace(/>/g, '&gt;')
+		/*
+		You may add other replacements here for HTML only 
+		(but it's not necessary).
+		Or for XML, only if the named entities are defined in its DTD.
+		*/
+		.replace(/\r\n/g, preserveCR) /* Must be before the next replacement. */
+		.replace(/[\r\n]/g, preserveCR)
+}

package/external-resource-manager.mjs

@@ -0,0 +1,138 @@
+import Resource from "./resource.mjs"
+
+export default class ExternalResourceManager {
+	
+	/**
+	 * Resources for which we have a directive to include on the page
+	 */
+	includedResources = new Set()
+	
+	/**
+	 * Resources which are available to provide content
+	 */
+	providedResources = []
+	
+	/**
+	 * Code which can generate html based on all of the entries of the mananger.
+	 * Good place form importmap generation or preload generation.
+	 */
+	preTypeRenderers = []
+
+	/**
+	 * Renderers by mime type
+	 */
+	renderers = {}
+
+	/**
+	 * The set of urls which are known to have been satisfied by some other resource
+	 */
+	alreadySatisfied = new Set()
+	
+	/**
+	 * Resources that we were directed to include, but are not going to, because
+	 * some other resource satisfies the url.
+	 */
+	differentlySatisfiedResource = []
+	
+	/**
+	 * To make sure we don't render a resource to the page twice, we'll keep track
+	 * of the URLs we've already rendered here.
+	 */
+	alreadyRenderedUrls = new Set()
+
+	constructor(options) {
+		Object.assign(this, options)
+	}
+	
+	
+	_noteSatisfies(resource) {
+		if(resource.satisfies && Array.isArray(resource.satisfies)) {
+			for(let url of resource.satisfies) {
+				this.alreadySatisfied.add(url)
+			}
+		}
+	}
+	
+	includeResource(resource) {
+		if(!resource) {
+			return
+		}
+		if(typeof resource === 'object' && resource instanceof Resource === false) {
+			resource = new Resource(resource)
+		}
+		this._noteSatisfies(resource)
+		if(this.alreadySatisfied.has(resource.url)) {
+			// We've already got a resource which satisfies this url
+			this.differentlySatisfiedResource.push(resource)
+			return
+		}
+		if(resource.satisfies && Array.isArray(resource.satisfies)) {
+			// We're going to remove any previously included resource which is satisfied
+			// by this resource
+			for(let included of this.includedResources) {
+				if(resource.satisfies.includes(included.url)) {
+					this.includedResources.delete(included)
+					this.differentlySatisfiedResource.push(resource)
+				}
+			}
+			this._noteSatisfies(resource)
+		}
+		this.includedResources.add(resource)
+	}
+	
+	provideResource(resource) {
+		this.providedResources.push(resource)
+	}
+	
+	/**
+	 * Adds a piece of code which knows how to turn a resource into html.
+	 * @param {string} mimeType 
+	 * @param {function} handler Takes a `Resource` object and turns it into a bit of html that can
+	 * be included on the page.
+	 */
+	addTypeHandler(mimeType, handler) {
+		this.renderers[mimeType] = handler
+	}
+
+	/**
+	 * Adds code which will generate html needed before the types are processed individually. Good
+	 * place for code which generates importmaps or preload statements. 
+	 * @param {function} handler Takes a `ExternalResourceManager` object and turns it into a bit of html that can
+	 * be included on the page.
+	 */
+	addPreTypeRender(handler) {
+		this.preTypeRenderers.push(handler)
+	}
+
+	render() {
+		let result = ''
+		for(let renderer of this.preTypeRenderers) {
+			result += renderer(this)
+		}
+
+		for(let resource of this.includedResources) {
+			if(this.alreadyRenderedUrls.has(resource.url) || this.alreadySatisfied.has(resource.url)) {
+				// A late in the game catch to make sure we don't output to the page a url that has
+				// already been used or a url which we know gets satisfied elsewhere
+				resource.rendered = true
+				continue
+			}
+			let renderer = this.renderers[resource.mimeType]
+			if(renderer && !resource.rendered) {
+				result += '\n'
+				result += renderer(resource)	
+				resource.rendered = true
+			}
+			this.alreadyRenderedUrls.add(resource.url)
+		}
+		
+		result += '\n'
+		for(let provided of this.providedResources) {
+			provided.rendered = true
+		}
+		
+		return result
+	}
+	
+	
+}

package/importmap-generator-creator.mjs

@@ -0,0 +1,36 @@
+
+export default function createImportmapGenerator(webhandle) {
+	function importmapGenerator(manager) {
+
+		let imports = {}
+
+		let found = false
+		for (let resource of manager.providedResources) {
+			if(resource.rendered) {
+				continue
+			}
+			found = true
+			let vrsc = ''
+			if(!webhandle.development && resource.cachable) {
+				vrsc = '/vrsc/' + webhandle.resourceVersion
+			}
+			if (resource.mimeType === 'application/javascript' && resource.resourceType === 'module') {
+				imports[resource.name] = vrsc + resource.url
+			}
+		}
+
+		let data = {
+			imports
+		}
+		
+		let dataText = webhandle.development ? JSON.stringify(data, null, '\t') : JSON.stringify(data)
+
+		let template = found ?
+`<script type="importmap">
+${dataText}
+</script>` : ''
+		return template
+	}
+
+	return importmapGenerator
+}

package/initialize-webhandle-component.mjs

@@ -0,0 +1,42 @@
+import createInitializeWebhandleComponent from "@webhandle/initialize-webhandle-component/create-initialize-webhandle-component.mjs"
+import ComponentManager from "@webhandle/initialize-webhandle-component/component-manager.mjs"
+import path from "node:path"
+import ExternalResourceManager from "./external-resource-manager.mjs"
+import createImportmapGenerator from "./importmap-generator-creator.mjs"
+import createTextCssRenderer from "./create-css-renderer.mjs"
+import createApplicationJavascriptRenderer from "./create-application-javascript-renderer.mjs"
+
+let initializeWebhandleComponent = createInitializeWebhandleComponent()
+
+initializeWebhandleComponent.componentName = 'externalResourceManager'
+initializeWebhandleComponent.componentDir = import.meta.dirname
+initializeWebhandleComponent.defaultConfig = {}
+initializeWebhandleComponent.staticFilePaths = ['public']
+initializeWebhandleComponent.templatePaths = ['views']
+
+initializeWebhandleComponent.setup = async function (webhandle, config) {
+	let compmanager = new ComponentManager()
+
+	let generator = createImportmapGenerator(webhandle)
+	let cssRender = createTextCssRenderer(webhandle)
+	let jsRender = createApplicationJavascriptRenderer(webhandle)
+
+	webhandle.routers.preParmParse.use((req, res, next) => {
+		let externalResourceManager = new ExternalResourceManager()
+
+		externalResourceManager.preTypeRenderers.push(generator)
+		externalResourceManager.renderers['text/css'] = cssRender
+		externalResourceManager.renderers['application/javascript'] = jsRender
+
+		res.locals.externalResourceManager = externalResourceManager
+
+		next()
+	})
+
+	compmanager.handlers = {
+		generator, cssRender, jsRender
+	}
+	return compmanager
+}
+
+export default initializeWebhandleComponent

package/package.json

@@ -0,0 +1,29 @@
+{
+	"name": "@webhandle/external-resource-manager",
+	"version": "1.0.0",
+	"description": "",
+	"main": "external-resource-manager.js",
+	"scripts": {
+		"test": "node test/all.mjs"
+	},
+	"repository": {
+		"type": "git",
+		"url": "git+https://github.com/EmergentIdeas/webhandle-external-resource-manager.git"
+	},
+	"keywords": [],
+	"author": "",
+	"license": "ISC",
+	"type": "module",
+	"bugs": {
+		"url": "https://github.com/EmergentIdeas/webhandle-external-resource-manager/issues"
+	},
+	"homepage": "https://github.com/EmergentIdeas/webhandle-external-resource-manager#readme",
+	"dependencies": {
+		"@webhandle/initialize-webhandle-component": "^1.0.1"
+	},
+	"files": [
+		"/*.mjs",
+		"README.md",
+		"*.mjs"
+	]
+}

package/resource.mjs

@@ -0,0 +1,50 @@
+
+export default class Resource {
+	
+	/**
+	 * The mime type of this resource e.g. application/javascript, text/css
+	 */
+	mimeType
+	
+	/**
+	 * This might be a sub-type or some sort of indication of how to handle this
+	 * resource
+	 */
+	resourceType
+	
+	/**
+	 * The name by which this resource is known, if there is one
+	 */
+	name
+	
+	/**
+	 * The URL of the resource
+	 */
+	url
+	
+	/**
+	 * Other URLs unnecessary if this resource is included
+	 */
+	satisfies = []
+	
+	/**
+	 * Atributes that may be needed for the html element
+	 */
+	attributes = []
+	
+	
+	/**
+	 * True if this resource can be set as indefinitely cachable in line with
+	 * webhandle's resource version scheme
+	 */
+	cachable = true
+	
+	/**
+	 * True if the resource has been added to the page
+	 */
+	rendered = false
+
+	constructor(options) {
+		Object.assign(this, options)
+	}
+}