mailwoman 0.2.0

package/README.md

@@ -0,0 +1,208 @@
+# Mailwoman
+
+A natural language classification engine for geocoding.
+
+![GitHub License](https://img.shields.io/github/license/isp-nexus/mailwoman)
+![GitHub commit activity](https://img.shields.io/github/commit-activity/m/isp-nexus/mailwoman)
+
+# Pelias Parser
+
+## AddressParser Example
+
+```
+30 w 26 st nyc 10010
+
+(0.95) ➜ [
+  { housenumber: '30' },
+  { street: 'w 26 st' },
+  { locality: 'nyc' },
+  { postcode: '10010' }
+]
+```
+
+## Application Interfaces
+
+You can access the library via three different interfaces:
+
+- all parts of the codebase are available in `javascript` via `npm`
+- on the `command line` via the `node bin/cli.js` script
+- through a `web service` via the `node server/http.js` script
+
+> the web service provides an interactive demo at the URL `/parser/parse`
+
+## Quick Start
+
+A quick and easy way to get started with the library is to use the command-line interface:
+
+```
+node bin/cli.js West 26th Street, New York, NYC, 10010
+```
+
+![cli](./docs/cli.png)
+
+---
+
+# Architecture Description
+
+Please refer to the CLI screenshot above for a visual reference.
+
+## Tokenization
+
+Tokenization is the process of splitting text into individual words.
+
+The splitting process used by the engine maintains token positions, so it's able to 'remember' where each character was in the original input text.
+
+> Tokenization is coloured `blue` on the command-line.
+
+### Span
+
+The most primitive element is called a `span`, this is essentially just a single string of text with some metadata attached.
+
+The terms `word`, `phrase` and `section` (explained below) are all just ways of using a `span`.
+
+### Section Boundaries
+
+Some parsers like [libpostal](https://github.com/openvenues/libpostal) ignore characters such as `comma`, `tab`, `newline` and `quote`.
+
+While it's unrealistic to expect commas always being present, it's very useful to record their positions when they are.
+
+These boundary positions help to avoid parsing errors for queries such as `Main St, East Village` being parsed as `Main St East` in `Village`.
+
+Once sections are established there is no 'bleeding' of information between sections, avoiding the issue above.
+
+### Word Splitting
+
+Each section is then split in to individual `words`, by default this simply considers whitespace as a word boundary.
+
+As per the `section`, the original token positions are maintained.
+
+### Phrase Generation
+
+May terms such as 'New York City' span multiple words, these multi-word tokens are called `phrases`.
+
+In order to be able to classify `phrase` terms, permutations of adjacent words are generated.
+
+Phrase generation is performed per-section, so it will not generate a `phrase` which contains words from more than one `section`.
+
+Phrase generation is controlled by a configuration which specifies things like the minimum & maximum amount of words allowed in a `phrase`.
+
+### Token Graph
+
+A graph is used to associate `word`, `phrase` and `section` elements to each other.
+
+The graph is free-form, so it's easy to add a new relationship between terms in the future, as required.
+
+Graph Example:
+
+```js
+// find the next word in this section
+word.findOne("next")
+
+// find all words in this phrase
+phrase.findAll("child")
+```
+
+## Classification
+
+Classification is the process of establishing that a `word` or `phrase` represents a 'concept' (such as a street name).
+
+Classification can be based on:
+
+- Dictionary matching (usually with normalization applied)
+- Pattern matching (such as regular expressions)
+- Composite matching (such as relative positioning)
+- External API calls (such as calling other services)
+- Other semantic matching techniques
+
+> Classification is coloured `green` and `red` on the command-line.
+
+### Classifier Types
+
+The library comes with three generic classifiers which can be extended in order to create a new `classifier`:
+
+- WordClassifier
+- PhraseClassifier
+- SectionClassifier
+
+### Classifiers
+
+The library comes bundled with a range of classifiers out-of-the box.
+
+You can find them in the `/classifier` directory, dictionary-based classifiers usually store their data in the `/resources` directory.
+
+Example of some of the included classifiers:
+
+- Word Classifiers
+  - HouseNumberClassifier
+  - PostcodeClassifier
+  - StreetPrefixClassifier
+  - StreetSuffixClassifier
+  - CompoundStreetClassifier
+  - DirectionalClassifier
+  - OrdinalClassifier
+  - StopWordClassifier
+- Phrase Classifiers
+  - IntersectionClassifier
+  - PersonClassifier
+  - GivenNameClassifier
+  - SurnameClassifier
+  - PersonalSuffixClassifier
+  - PersonalTitleClassifier
+  - ChainClassifier
+  - PlaceClassifier
+  - WhosOnFirstClassifier
+
+## Solvers
+
+Solving is the final process, where `solutions` are generated based on all the classifications that have been made.
+
+Each parse can contain multiple `solutions`, each is provided with a `confidence` score and is displayed sorted from highest scoring solution to lowest scoring.
+
+The core of this process is the `ExclusiveCartesianSolver` module.
+
+This `solver` generates all the possible permutations of the different classifications while taking care to:
+
+- ensure the same `span` position is not used more than once
+- ensure that the same `classification` is not used more than once.
+
+After the `ExclusiveCartesianSolver` has run there are additional solvers which can:
+
+- filter the `solutions` to remove inconsistencies
+- add new `solutions` to provide additional functionality (such as intersections)
+
+### Solution Masks
+
+It is possible to produce a simple `mask` for any generated solution, this is useful for comparing the `solution` to the original text:
+
+```
+VVV VVVV NN SSSSSSS AAAAAA PPPPP
+Foo Cafe 10 Main St London 10010 Earth
+```
+
+# Contributing
+
+Please fork and pull request against upstream master on a feature branch. Pretty please; provide unit tests.
+
+## Unit tests
+
+You can run the unit test suite using the command:
+
+```sh
+$ npm test
+```
+
+# License
+
+Mailwoman is distributed under the AGPL-3.0 license. Generally,
+this means that you can use the software for free, but you must share
+any modifications you make to the software.
+
+Unmodified portions of Mailwoman derived from Pelias Parser remain under the
+MIT license.
+
+For more information on commercial usage licensing, please contact us at
+`hello@isp.nexus`
+
+# Acknowledgements
+
+This project was built by the [Open ISP](https://isp.nexus) team and was made possible by the contributions of the Pelias community.

package/bin/cli.ts

@@ -0,0 +1,54 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { DebugOutputBuilder, parseAddressInput } from "mailwoman"
+
+/**
+ * Initialize the CLI and parse the input command.
+ */
+async function initCLI(): Promise<unknown> {
+	const args = process.argv.slice(2)
+
+	const commandName = args[0]
+
+	if (!commandName || commandName === "--help" || commandName === "-h" || commandName === "help") {
+		console.log("mailwoman [command] [address]")
+		console.log("Commands:")
+		console.log("\tparse: Parse an address in JSON format")
+		console.log("\tdebug: Parse an address and output debug information")
+		console.log("\tserve: Start the parser server")
+		return
+	}
+
+	if (commandName === "serve") {
+		console.log("Starting the parser server...")
+		const { startPeliasParserServer } = await import("mailwoman/server")
+
+		const app = startPeliasParserServer()
+
+		return new Promise((resolve) => {
+			app.addListener("close", resolve)
+		})
+	}
+
+	if (commandName === "debug") {
+		const builder = new DebugOutputBuilder()
+
+		const formattedAddress = args.slice(1).join(" ")
+		builder.parse(formattedAddress)
+
+		console.log(builder.toString())
+
+		return
+	}
+
+	const formattedAddress = commandName === "parse" ? args.slice(1).join(" ") : args.join(" ")
+
+	const results = parseAddressInput(formattedAddress)
+	console.log(JSON.stringify(results, null, 2))
+}
+
+export default initCLI()

package/classification/AdjacentClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { AdjacentClassification } from "./AdjacentClassification.js"
+
+test("constructor", (t) => {
+	const c = new AdjacentClassification()
+	t.false(c.public)
+	t.equals(c.label, "adjacent")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/AdjacentClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "../classification/Classification.js"
+
+export class AdjacentClassification extends Classification {
+	public override readonly label = "adjacent"
+}
+
+export default AdjacentClassification

package/classification/AlphaClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { AlphaClassification as Classification } from "./AlphaClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "alpha")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/AlphaClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "../classification/Classification.js"
+
+export class AlphaClassification extends Classification {
+	public override readonly label = "alpha"
+}
+
+export default AlphaClassification

package/classification/AlphaNumericClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { AlphaNumericClassification as Classification } from "./AlphaNumericClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "alphanumeric")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/AlphaNumericClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "../classification/Classification.js"
+
+export class AlphaNumericClassification extends Classification {
+	public override readonly label = "alphanumeric"
+}
+
+export default AlphaNumericClassification

package/classification/AreaClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { AreaClassification as Classification } from "./AreaClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "area")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/AreaClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "./Classification.js"
+
+export class AreaClassification extends Classification {
+	public override readonly label = "area"
+}
+
+export default AreaClassification

package/classification/ChainClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { ChainClassification as Classification } from "./ChainClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "chain")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/ChainClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "../classification/Classification.js"
+
+export class ChainClassification extends Classification {
+	public override readonly label = "chain"
+}
+
+export default ChainClassification

package/classification/Classification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { Classification } from "./Classification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "unknown")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/Classification.ts

@@ -0,0 +1,107 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+export type ClassificationMetadata = Record<string, any>
+
+/**
+ * Base classification class.
+ */
+export class Classification {
+	/**
+	 * Whether the classification appears in results.
+	 */
+	public public: boolean = false
+
+	/**
+	 * The confidence of the classification.
+	 */
+	public confidence: number = 1.0
+
+	/**
+	 * The metadata of the classification.
+	 */
+	public meta: Record<string, any> = {}
+
+	/**
+	 * The label of the classification.
+	 */
+	public readonly label: string = "unknown"
+
+	/**
+	 * Create a classification.
+	 *
+	 * @param confidence - The confidence of the classification.
+	 * @param meta - The metadata of the classification.
+	 */
+	constructor(confidence = 1, meta: ClassificationMetadata = {}) {
+		this.public = false // only public classifications appear in results
+		this.label = "unknown"
+		this.confidence = confidence
+		this.meta = meta
+	}
+
+	/**
+	 * Compare this classification to another classification.
+	 */
+	equals(classification: Classification): boolean {
+		// @todo: compare meta?
+		return this.constructor.name === classification.constructor.name && this.confidence === classification.confidence
+	}
+}
+
+export type ClassificationConstructor = new (confidence?: number, meta?: ClassificationMetadata) => Classification
+
+/**
+ * A configuration for a classifier scheme.
+ */
+export interface ClassifierSchemeConfig {
+	/**
+	 * The confidence level of the classification.
+	 */
+	confidence: number
+	/**
+	 * Constructor for the classification.
+	 */
+	Class: ClassificationConstructor
+	/**
+	 * A list of criteria that must be met for the classification to be applied.
+	 */
+	scheme: ClassifierSchemeCriteria[]
+}
+
+export interface ClassifierSchemeCriteria {
+	/**
+	 * The classification labels that must be present in the token.
+	 */
+	is: string[]
+	/**
+	 * The classification labels that must not be present in the token.
+	 */
+	not?: string[]
+	/**
+	 * Confidence level for the criteria.
+	 */
+	confidence?: number
+
+	/**
+	 * Constructor for the classification.
+	 *
+	 * @todo Is this used?
+	 */
+	Class?: ClassificationConstructor
+}
+
+/**
+ * Extract the label from a classification constructor.
+ *
+ * This is useful for creating a union of classification labels, such as in public type definitions.
+ */
+export type ExtractClassificationLabel<T extends ClassificationConstructor> = Extract<
+	InstanceType<T>,
+	{
+		readonly public: true
+	}
+>["label"]

package/classification/CountryClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { CountryClassification as Classification } from "./CountryClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.true(c.public)
+	t.equals(c.label, "country")
+	t.equals(c.confidence, 0.9)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/CountryClassification.ts

@@ -0,0 +1,18 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification, ClassificationMetadata } from "./Classification.js"
+
+export class CountryClassification extends Classification {
+	public override readonly public = true
+	public override readonly label = "country"
+
+	constructor(_confidence?: number, meta?: ClassificationMetadata) {
+		super(0.9, meta)
+	}
+}
+
+export default CountryClassification

package/classification/DependencyClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { DependencyClassification as Classification } from "./DependencyClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.true(c.public)
+	t.equals(c.label, "dependency")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/DependencyClassification.ts

@@ -0,0 +1,14 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "./Classification.js"
+
+export class DependencyClassification extends Classification {
+	public override readonly label = "dependency"
+	public override readonly public = true
+}
+
+export default DependencyClassification

package/classification/DirectionalClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { DirectionalClassification as Classification } from "./DirectionalClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "directional")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/DirectionalClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "../classification/Classification.js"
+
+export class DirectionalClassification extends Classification {
+	public override readonly label = "directional"
+}
+
+export default DirectionalClassification

package/classification/EndTokenClassification.test.ts

@@ -0,0 +1,17 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import test from "tape"
+import { EndTokenClassification as Classification } from "./EndTokenClassification.js"
+
+test("constructor", (t) => {
+	const c = new Classification()
+	t.false(c.public)
+	t.equals(c.label, "end_token")
+	t.equals(c.confidence, 1.0)
+	t.deepEqual(c.meta, {})
+	t.end()
+})

package/classification/EndTokenClassification.ts

@@ -0,0 +1,13 @@
+/**
+ * @copyright OpenISP, Inc.
+ * @license AGPL-3.0
+ * @author Teffen Ellis, et al.
+ */
+
+import { Classification } from "./Classification.js"
+
+export class EndTokenClassification extends Classification {
+	public override readonly label = "end_token"
+}
+
+export default EndTokenClassification