adapt-authoring-jsonschema 0.0.1

package/docs/writing-a-schema.md

@@ -0,0 +1,194 @@
+# Writing a schema
+
+This page outlines the various elements of an Adapt data schema, and gives tips on how to start writing your own schemas. If you're new to schemas, head over to [this page](/introduction-to-schemas), which goes over the basics.
+
+> For some specific schema examples, see [this page](/schema-examples).
+
+## Quick links
+- [Defining a schema](#defining-a-schema)
+- [Defining schema inheritance](#defining-schema-inheritance)
+- [Custom schema keywords](#custom-schema-keywords)
+- [Custom Adapt properties](#custom-adapt-properties)
+- [Custom Backbone Forms properties](#custom-backbone-forms-properties)
+
+## Defining schema inheritance
+
+You may find when defining your schemas that you want to extend or modify existing schemas, or split up your schemas in such a way as you can share properties across multiple schemas. For this purpose, you can use the `$merge` and `$patch` keywords.
+
+The main difference between merge and patch schemas is how they're accessed:
+- `$merge` schemas are considered 'complete' schemas, and are accessed directly (e.g. the UI will request the MCQ schema by name when rendering the MCQ form page)
+- `$patch` schemas are not considered 'complete' schemas, but rather are 'attached' to another schema when that schema is requested (e.g. the UI will request the `course` schema which will include the relevant extension `$patch` schemas).
+
+### `$merge` schemas
+
+Merge schemas are useful when you have a schema which needs to directly inherit from another existing schema. As an example, all Adapt framework content schemas extend from a base `content` schema, which defines the basic attributes such as `title` and `body`. 
+
+Every `$merge` schema must define the base schema it inherits from (as a `source` attribute), and any additional properties (nested under a `with` attribute). For example:
+
+```
+{
+  "$anchor": "example-schema",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "$merge": {
+    "source": {
+      "$ref": "base-schema"
+    }
+    "with": {
+      "properties": {
+        "myAttribute": {
+          "type": "string"
+        }
+      }
+    }
+  }
+}
+```
+
+### `$patch` schemas
+
+`$patch` schemas are useful when looking to augment an existing schema with extra attributes or override existing schema properties (as an example, many Adapt framework extensions will define their own `$patch`schema for the `course` schema which will define extra `_globals` properties specific to that extension).
+
+`$patch` schemas are almost identical to `$merge` schemas, with the only difference being that the `$merge` is replaced with the `$patch` keyword:
+
+```
+{
+  "$anchor": "example-schema",
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "type": "object",
+  "$patch": {
+    "source": {
+      "$ref": "base-schema"
+    }
+    "with": {
+      "properties": {
+        "myAttribute": {
+          "type": "string"
+        }
+      }
+    }
+  }
+}
+```
+
+## Custom schema keywords
+
+In addition to the standard keywords defined in the JSON schema specification, the Adapt authoring tool jsonschema module defines a number of extra custom keywords which add extra convenient functionality when validating incoming data.
+
+The following custom keywords are available:
+
+### `isDate`
+This keyword will parse any string value into a valid JavaScript Date.
+
+#### Example
+```
+"myDateAttribute": {
+  "type": "string",
+  "isDate": true
+}
+```
+
+### `isDirectory`
+This keyword will resolve any path values using a number of default directory values. This is very useful when making use of the existing app directories (e.g. you want to store data in the app's temp folder). The following are supported values:
+- `$ROOT` will resolve to the main app root folder
+- `$DATA` will resolve to the app's data folder
+- `$TEMP` will resolve to the app's temp folder
+
+#### Example
+```
+"myDirectoryAttribute": {
+  "type": "string",
+  "isDirectory": true,
+  "default": "$TEMP/myfolder" // will be replace $TEMP with the correct path to the temp folder
+}
+```
+
+### `isInternal`
+This keyword will ensure that the attribute is **not** returned when a web API request is made. This is useful for restricting sensitive information like passwords. Note that this keyword only applies to the web APIs, and not when accessing data programatically.
+
+#### Example
+```
+"myInternalAttribute": {
+  "type": "string",
+  "isInternal": true
+}
+```
+
+### `isReadOnly`
+This keyword will ensures that the attribute is **not** modified when a web API request is made. Note that this keyword only applies to the web APIs, and not when accessing data programatically.
+
+#### Example
+```
+"myReadOnlyAttribute": {
+  "type": "string",
+  "isReadOnly": true
+}
+```
+
+### `isTimeMs`
+This keyword is very useful when defining time values, as it allows a human-readable input value to be automatically converted into milliseconds.
+
+#### Example
+```
+"myTimeAttribute": {
+  "type": "string",
+  "isTimeMs": true,
+  "default": "7d" // will be converted to the equivalent of 7 days in milliseconds (604800000)
+}
+```
+
+## Custom Adapt properties
+
+The `_adapt` keyword is used to group schema properties which are non-standard to the JSON schema specification and related to various Adapt-specific features.
+
+Property | Type | Description
+--- | --- | ---
+`editorOnly` | `Boolean` |  Determines whether the attribute should be included in output JSON
+`isSetting` | `Boolean` | Attribute will appear in the ‘Settings’ section of the form
+`translatable` | `Boolean` | Whether the attribute is a language-specific string for translation
+
+### Example
+```
+"myAttribute": {
+  "type": "string",
+  "_adapt": {
+    "editorOnly": true,
+    "translatable": true
+  }
+}
+```
+
+## Custom Backbone forms properties
+
+The Adapt authoring tool uses the [Backbone Forms](https://github.com/powmedia/backbone-forms) library to render forms from the data schemas into a user-friendly HTML form. The `_backboneForms` keyword is used to group schema properties which apply to Backbone Forms.
+
+Property | Type | Description
+--- | --- | ---
+`type` | `String` | Override the type of Backbone Forms input to be used (by default, the type will be inferred from the schema property's type value). Accepted types: `Asset`, `Text`, `Number`, `Password`, `TextArea`, `Checkbox`, `Checkboxes`, `Select`, `Radio`, `Object`, `Date`, `DateTime`, `List`. See the [Backbone Forms docs](https://github.com/powmedia/backbone-forms#schema-definition) for more information.
+`showInUi` | `Boolean` |  Determines whether the attribute will be rendered in forms
+`media` | `String` |  When using a `type` of `Asset`, you can also restrict the AAT UI to only show assets of a specific type. The AAT stores the asset type based in its [MIME type](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/MIME_types) (e.g. files with a MIME type of `image/png` and `image/jpeg` would both have a type of `image`).
+`editorAttrs` | `Object` |  Extra options passed to the Backbone Forms input. See the [Backbone Forms docs](https://github.com/powmedia/backbone-forms#schema-definition) for more information.
+
+> The Adapt authoring tool will attempt to infer the type of a form input using the `type` value in the schema. In most cases this will suffice, so check that you definitely need to override the default behaviour before defining additional `_backboneForms` properties.
+
+### Example
+
+```
+"myAttribute": {
+  "type": "string",
+  "_backboneForms": {
+    "type": "Asset",
+    "showInUi": false,
+    "media": "image"
+  }
+}
+```
+
+In many cases, you'll only want to customise the type of an input. If this is the case, the `_backboneForms` property can also be a string value, e.g.
+
+```
+"myAttribute": {
+  "type": "string",
+  "_backboneForms": "Number"
+}
+```

package/errors/errors.json

@@ -0,0 +1,52 @@
+{
+  "INVALID_SCHEMA": {
+    "data": {
+      "errors": "all validation errors",
+      "schemaName": "Schema name"
+    },
+    "description": "Invalid schema",
+    "statusCode": 400
+  },
+  "MISSING_SCHEMA": {
+    "data": {
+      "schemaName": "Schema name"
+    },
+    "description": "Schema is not registered in the cache",
+    "statusCode": 500
+  },
+  "MISSING_SCHEMA_NAME": {
+    "description": "Schema name is not defined",
+    "statusCode": 400
+  },
+  "MODIFY_PROTECTED_ATTR": {
+    "data": {
+      "attribute": "The protected attribute"
+    },
+    "description": "Attempted to modify a protected data attribute",
+    "statusCode": 400
+  },
+  "SCHEMA_EXISTS": {
+    "data": {
+      "filepath": "File path to the schema",
+      "schemaName": "Schema name"
+    },
+    "description": "A schema already exists with the specified name",
+    "statusCode": 400
+  },
+  "SCHEMA_LOAD_FAILED": {
+    "data": {
+      "schemaName": "Schema name"
+    },
+    "description": "Failed to load a JSON schema",
+    "statusCode": 500
+  },
+  "VALIDATION_FAILED": {
+    "data": {
+      "data": "the data failing validation",
+      "errors": "all validation errors",
+      "schemaName": "Schema name"
+    },
+    "description": "Data validation failed",
+    "statusCode": 400
+  }
+}

package/index.js

@@ -0,0 +1,5 @@
+/**
+ * Validation of data against JSON schemas
+ * @namespace jsonschema
+ */
+export { default } from './lib/JsonSchemaModule.js'

package/lib/JsonSchema.js

@@ -0,0 +1,268 @@
+import _ from 'lodash'
+import { App, Hook } from 'adapt-authoring-core'
+import fs from 'fs/promises'
+import xss from 'xss'
+
+/** @ignore */ const BASE_SCHEMA_NAME = 'base'
+
+/**
+ * Functionality related to JSON schema
+ * @memberof jsonschema
+ */
+class JsonSchema {
+  constructor ({ enableCache, filePath, validator, xssWhitelist }) {
+    /**
+     * The raw built JSON schema
+     * @type {Object}
+     */
+    this.built = undefined
+    /**
+     * The compiled schema validation function
+     * @type {function}
+     */
+    this.compiled = undefined
+    /**
+     * Whether caching is enabled for this schema
+     * @type {Boolean}
+     */
+    this.enableCache = enableCache
+    /**
+     * List of extensions for this schema
+     * @type {Array<String>}
+     */
+    this.extensions = []
+    /**
+     * File path to the schema
+     * @type {String}
+     */
+    this.filePath = filePath
+    /**
+     * Whether the schema is currently building
+     * @type {Boolean}
+     */
+    this.isBuilding = false
+    /**
+     * The last build time (in milliseconds)
+     * @type {Number}
+     */
+    this.lastBuildTime = undefined
+    /**
+     * The raw schema data for this schema (with no inheritance/extensions)
+     * @type {Object}
+     */
+    this.raw = undefined
+    /**
+     * Reference to the Ajv validator instance
+     * @type {external:Ajv}
+     */
+    this.validator = validator
+    /**
+     * Reference to the local XSS sanitiser instance
+     * @type {Object}
+     */
+    this.xss = new xss.FilterXSS({ whiteList: xssWhitelist })
+    /**
+     * Hook which invokes every time the schema is built
+     * @type {Hook}
+     */
+    this.buildHook = new Hook()
+  }
+
+  /**
+   * Determines whether the current schema build is valid using last modification timestamp
+   * @returns {Boolean}
+   */
+  async isBuildValid () {
+    if (!this.built) return false
+    let schema = this
+    while (schema) {
+      const { mtimeMs } = await fs.stat(schema.filePath)
+      if (mtimeMs > this.lastBuildTime) return false
+      schema = await schema.getParent()
+    }
+    return true
+  }
+
+  /**
+   * Returs the parent schema if $merge is defined (or the base schema if a root schema)
+   * @returns {JsonSchema}
+   */
+  async getParent () {
+    if (this.name === BASE_SCHEMA_NAME) return // base schema always the root
+    const jsonschema = await App.instance.waitForModule('jsonschema')
+    return await jsonschema.getSchema(this.raw?.$merge?.source?.$ref ?? BASE_SCHEMA_NAME)
+  }
+
+  /**
+   * Loads the schema file
+   * @returns {JsonSchema} This instance
+   */
+  async load () {
+    try {
+      this.raw = JSON.parse((await fs.readFile(this.filePath)).toString())
+      this.name = this.raw.$anchor
+    } catch (e) {
+      throw App.instance.errors?.SCHEMA_LOAD_FAILED?.setData({ schemaName: this.filePath }) ?? e
+    }
+    if (this.validator.validateSchema(this.raw)?.errors) {
+      const errors = this.validator.errors.map(e => e.instancePath ? `${e.instancePath} ${e.message}` : e.message)
+      if (errors.length) {
+        throw App.instance.errors.INVALID_SCHEMA
+          .setData({ schemaName: this.name, errors: errors.join(', ') })
+      }
+    }
+    return this
+  }
+
+  /**
+   * Builds and compiles the schema from the $merge and $patch schemas
+   * @param {LoadSchemaOptions}
+   * @return {JsonSchema}
+   */
+  async build (options = {}) {
+    if (options.useCache !== false && this.enableCache && await this.isBuildValid()) {
+      return this
+    }
+    if (this.isBuilding) {
+      return new Promise(resolve => this.buildHook.tap(() => resolve(this)))
+    }
+    this.isBuilding = true
+
+    const jsonschema = await App.instance.waitForModule('jsonschema')
+    const { applyExtensions, extensionFilter } = options
+
+    let built = _.cloneDeep(this.raw)
+    let parent = await this.getParent()
+
+    while (parent) {
+      const parentBuilt = _.cloneDeep((await parent.build(options)).built)
+      built = await this.patch(parentBuilt, built, { strict: !parent.name === BASE_SCHEMA_NAME })
+      parent = await parent.getParent()
+    }
+    if (this.extensions.length) {
+      await Promise.all(this.extensions.map(async s => {
+        const applyPatch = typeof extensionFilter === 'function' ? extensionFilter(s) : applyExtensions !== false
+        if (applyPatch) {
+          const extSchema = await jsonschema.getSchema(s)
+          this.patch(built, extSchema.raw, { extendAnnotations: false })
+        }
+      }))
+    }
+    this.built = built
+    this.compiled = await this.validator.compileAsync(built)
+    this.isBuilding = false
+    this.lastBuildTime = Date.now()
+
+    this.buildHook.invoke(this)
+    return this
+  }
+
+  /**
+   * Applies a patch schema to another schema
+   * @param {Object} baseSchema The base schema to apply the patch
+   * @param {Object} patchSchema The patch schema to apply to the base
+   * @param {ApplyPatchOptions} options
+   * @return {Object} The base schema
+   */
+  patch (baseSchema, patchSchema, options = {}) {
+    const opts = _.defaults(options, {
+      extendAnnotations: patchSchema.$anchor !== BASE_SCHEMA_NAME,
+      overwriteProperties: true,
+      strict: true
+    })
+    const patchData = patchSchema.$patch?.with ?? patchSchema.$merge?.with ?? (!opts.strict && patchSchema)
+    if (!patchData) {
+      throw App.instance.errors.INVALID_SCHEMA.setData({ schemaName: patchSchema.$anchor })
+    }
+    if (opts.extendAnnotations) {
+      ['$anchor', 'title', 'description'].forEach(p => {
+        if (patchSchema[p]) baseSchema[p] = patchSchema[p]
+      })
+    }
+    if (patchData.properties) {
+      const mergeFunc = opts.overwriteProperties ? _.merge : _.defaultsDeep
+      mergeFunc(baseSchema.properties, patchData.properties)
+    }
+    ['allOf', 'anyOf', 'oneOf'].forEach(p => {
+      if (patchData[p]?.length) baseSchema[p] = (baseSchema[p] ?? []).concat(_.cloneDeep(patchData[p]))
+    })
+    if (patchData.required) {
+      baseSchema.required = _.uniq([...(baseSchema.required ?? []), ...patchData.required])
+    }
+    return baseSchema
+  }
+
+  /**
+   * Checks passed data against the specified schema (if it exists)
+   * @param {Object} dataToValidate The data to be validated
+   * @param {SchemaValidateOptions} options
+   * @return {Object} The validated data
+   */
+  validate (dataToValidate, options) {
+    const opts = _.defaults(options, { useDefaults: true, ignoreRequired: false })
+    const data = _.defaultsDeep(_.cloneDeep(dataToValidate), opts.useDefaults ? this.getObjectDefaults() : {})
+
+    this.compiled(data)
+
+    const errors = this.compiled.errors && this.compiled.errors
+      .filter(e => e.keyword === 'required' ? !opts.ignoreRequired : true)
+      .map(e => e.instancePath ? `${e.instancePath} ${e.message}` : e.message)
+      .reduce((s, e) => `${s}${e}, `, '')
+
+    if (errors?.length) { throw App.instance.errors.VALIDATION_FAILED.setData({ schemaName: this.name, errors, data }) }
+
+    return data
+  }
+
+  /**
+   * Sanitises data by removing attributes according to the context (provided by options)
+   * @param {Object} dataToValidate The data to be sanitised
+   * @param {SchemaSanitiseOptions} options
+   * @return {Object} The sanitised data
+   */
+  sanitise (dataToSanitise, options = {}, schema) {
+    const opts = _.defaults(options, { isInternal: false, isReadOnly: false, sanitiseHtml: true, strict: true })
+    schema = schema ?? this.built
+    const sanitised = {}
+    for (const prop in schema.properties) {
+      const schemaData = schema.properties[prop]
+      const value = dataToSanitise[prop]
+      const ignore = (opts.isInternal && schemaData.isInternal) || (opts.isReadOnly && schemaData.isReadOnly)
+      if (value === undefined || (ignore && !opts.strict)) {
+        continue
+      }
+      if (ignore && opts.strict) {
+        throw App.instance.errors.MODIFY_PROTECTED_ATTR.setData({ attribute: prop, value })
+      }
+      sanitised[prop] =
+        schemaData.type === 'object' && schemaData.properties
+          ? this.sanitise(value, opts, schemaData)
+          : schemaData.type === 'string' && opts.sanitiseHtml
+            ? this.xss.process(value)
+            : value
+    }
+    return sanitised
+  }
+
+  /**
+   * Adds an extension schema
+   * @param {String} extSchemaName
+   */
+  addExtension (extSchemaName) {
+    !this.extensions.includes(extSchemaName) && this.extensions.push(extSchemaName)
+  }
+
+  /**
+   * Returns all schema defaults as a correctly structured object
+   * @param {Object} schema
+   * @param {Object} memo For recursion
+   * @returns {Object} The defaults object
+   */
+  getObjectDefaults (schema) {
+    schema = schema ?? this.built
+    const props = schema.properties ?? schema.$merge?.with?.properties ?? schema.$patch?.with?.properties
+    return _.mapValues(props, s => s.type === 'object' && s.properties ? this.getObjectDefaults(s) : s.default)
+  }
+}
+
+export default JsonSchema