@linkeddatanotes/md2rdf 0.0.0-stage → 0.0.5

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Linked Data Notes, Leo "Leobard" Sauermann
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -1,3 +1,151 @@
-# Temporary Holding Version
-
-This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
+# md2rdf
+
+Reads a Markdown string, detects linked data, and returns the linked data as array of `Quad` compatible with [RDF.js](http://rdf.js.org/).
+
+`md2rdf` follows the conventions defined in [LinkedDataNotes](https://www.linkeddatanotes.org):
+- RDF in `turtle` code fences is parsed 
+  - using [N3.js](https://github.com/rdfjs/n3.js)
+- section headings are interpreted
+  - date and time prefixes result in `schema:temporalCoverage`
+  - `this:` in a `turtle` fence is interpreted as an URI reference to the current section
+- common namespaces are part of this package and available at `commonNamespaces()`, it is recommended to pass them as `options.namespaces`.
+
+The returned `rdfjs Stream<Quad>` can be passed to `quadstore.import()`.
+
+## Installation
+
+`npm install @linkeddatanotes/md2rdf`
+
+## Parsing markdown files
+
+Load the markdown, pass it to Md2Rdf.
+
+```typescript
+// Md2Rdf and dependencies
+import {Md2Rdf, commonNamespaces} from '@linkeddatanotes/md2rdf';
+import {DataFactory as N3DataFactory, Writer} from 'n3';
+
+// options for Md2Rdf
+const options = {
+  factory: N3DataFactory,
+  // defines common namespaces such as schema
+  namespaces: commonNamespaces(),
+  // contains a date
+  baseIRI: 'file://2015-12-12_ParisAgreement.md',
+};
+
+// a sample markdown including LinkedDataNotes
+// - a heading with time, in combination with the baseIRI this defines the schema:temporalCoverage
+// - a turtle code fence using tildes as the multiline typescript string is using backticks
+const exampleMarkdown = `
+# 2015-12-12 Saturday at COP 21
+The COP21 conference at Le Bourget was extended by one day, Saturday the 12th.
+On this day, the Paris Agreement was adopted.
+
+~~~turtle
+this: a schema:PublicEvent;
+  schema:organizer <http://www.wikidata.org/entity/Q104150918>.
+~~~
+`;
+
+// create the instance of Md2Rdf
+const md2rdf = new Md2Rdf(options);
+
+// transform Markdown to RDF
+const quads = md2rdf.parseString(exampleMarkdown);
+
+// print the RDF as turtle
+let s = '# Md2Rdf for ' + options.baseIRI + '\n';
+const writer = new Writer();
+writer.addQuads(quads);
+writer.end((error, result) => s += result);
+console.log(s);
+```
+
+Read the tests for more examples.
+
+This library is used by [LinkedDataNotes](https://www.linkeddatanotes.org) and `obsidian-linkeddatanotes`. 
+See more example code there.
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).
+
+This uses 
+- [solid-namespace](https://github.com/solid/solid-namespace/blob/master/index.js) which is MIT licensed.
+
+## Design decisions
+
+### String input
+Currently, input is passed as `string`. This is enough for the first use in Obsidian [Vaul.cachedRead()](https://docs.obsidian.md/Reference/TypeScript+API/Vault/cachedRead). 
+
+Reading from a `Stream` would be a welcome addition. Note that N3 Parser is not returning a generator but calling a passed callback function, so a streaming API may require an intermediate queue or a "/zero-queue backpressure bridge/".
+
+### Markdown sections
+Markdown sections that resulted in RDF are marked using the [`md2rdf` vocabulary](https://www.linkeddatanotes.org/ns/md2rdf/) and `rdfs`. 
+
+Above `exampleMarkdown` results in:
+
+```turtle
+PREFIX md2rdf: <https://www.linkeddatanotes.org/ns/md2rdf/>
+PREFIX rdfs: <http://www.w3.org/2000/01/rdf-schema#>
+PREFIX schema: <https://schema.org/>
+
+<file://2015-12-12_ParisAgreement.md#2015-12-12-Saturday-at-COP-21> a schema:PublicEvent, md2rdf:Section;
+  rdfs:label "2015-12-12 Saturday at COP 21";
+  md2rdf:headingLevel "1";
+  schema:temporalCoverage "2015-12-12";
+  schema:organizer <http://www.wikidata.org/entity/Q104150918>.
+```
+
+### Namespaces
+Namespaces are passed as string via `Md2RdfParserOptions.namespaces`. This is simple and works.
+
+It comes with the caveat that parsing the namespaces for each turtle code fence wastes CPU. 
+It would be better to parse them once. 
+N3.Parser does not support passing of pre-parsed Namespaces. This could be improved by extending N3.Parser or by using N3 internals: quadCallback, inputstream, injecting parser._prefixes[prefix]=value.
+
+
+## Development
+This project follows [gts](https://github.com/google/gts) to avoid bikeshedding. 
+
+### Testing
+This project uses [mocha](https://github.com/mochajs). [assert](https://github.com/browserify/commonjs-assert) is used as the [assert syntax](https://nodejs.org/api/assert.html) uses simple js constructs.
+
+```
+npm run test
+```
+
+### Publishing a release
+
+After we did fantastic stuff and all our tests are ok, time to publish.
+
+Packaging and publishing a release currently works like this:
+
+```
+# at this point: git must be clean, on main
+git status
+
+# confirm the release looks good
+npm pack --dry-run
+
+# builds, tests, bumps, tags, pushes
+npm version patch
+
+# publish
+npm login
+npm publish
+
+# after release: main becomes prepatch. This bumps, tags, pushes.
+npm version prepatch
+```
+
+## Part of LinkedDataNotes
+This project is part of [LinkedDataNotes](https://www.linkeddatanotes.org). LinkedDataNotes is a minimalistic convention to use linked data within markdown notes. 
+
+## Contribute
+**md2rdf** is open for contributions. File an issue, create a pull request.
+
+Get in touch via getintouch@linkeddatanotes.org.
+
+Md2Rdf was started by Leo "Leobard" Sauermann on 2021-02-10.

package/build/src/Md2Rdf.d.ts

@@ -0,0 +1,394 @@
+import type { ParserOptions } from 'n3';
+import Parser from 'n3/lib/N3Parser';
+import { DataFactory, Quad, NamedNode } from '@rdfjs/types';
+/**
+ * Md2RdfParserOptions to configure Md2Rdf.
+ *
+ * They are passed on to the N3 Parser.
+ */
+export interface Md2RdfParserOptions extends ParserOptions {
+    /**
+     * The file IRI of the currently parsed Markdown file.
+     * inherited from ParserOptions, described here explicitly to clarify use.
+     */
+    baseIRI: string;
+    /**
+     * RDF DataFactory to create nodes.
+     */
+    factory: DataFactory;
+    /**
+     * namespaces set here will be prefixed to every `turtle` block before parsing.
+     * Namespaces defined within a block will override.
+     * Hint: md2rdf ships common namespaces. Load them with commonNamespaces() and pass them in.
+     * Format: turtle string
+     */
+    namespaces?: string | undefined;
+    /**
+     * Set the onError handler to get information about errors and warnings and to stop the parser when you find necessary.
+     * Leave the onError handler empty to let the parser skip all errors and warnings and return as many quads as possible.
+     *
+     * Implementations of the error handler
+     * - may read the passed errors and warnings and notify the user, write them to a log, or to a quadstore
+     * - may stop the parser by throwing the error back at the parser
+     *
+     * @param error a ParseError with a message and more details
+     * @throws {ParseError} throws the passed error back to the caller to signal that the parsing should stop now. Swallows it to continue.
+     */
+    onError?: (error: ParseError) => void;
+}
+/**
+ * ParseError is passed in the onError() handler of Md2RdfParserOptions.
+ * Implementations of the Md2RdfParserOptions.onError() handler should read the ParseError parameter
+ * for logging and deciding whether to continue.
+ */
+export declare class ParseError extends Error {
+    /**
+     * file IRI of the currently parsed Markdown file.
+     */
+    baseIRI: string;
+    /**
+     * severity level
+     *
+     * error - data cannot be parsed
+     * warning - data is parsed but result may be unexpected
+     */
+    severity: 'error' | 'warning';
+    /**
+     * Line number within the currently parsed Markdown file.
+     */
+    line?: number;
+    /**
+     * create a ParseError.
+     * Helper function
+     * @param cause Underlying error or message.
+     * @param options Options to pass the baseIRI of the currently parsed Markdown file.
+     * @param line Line number within the currently parsed Markdown file.
+     * @param severity Severity level 'error' or 'warning' inspired by methods of `console.`.
+     *   If severity is not passed, assume 'error' unless the cause Error name ends with string 'Warning'.
+     * @returns
+     */
+    constructor(cause: unknown, options: Md2RdfParserOptions, line?: number, severity?: 'error' | 'warning');
+}
+/**
+ * HomonymHeadingWarning is a possible cause in ParseError.
+ * Example:
+ * Heading "MyHeading" used for different sections containing a turtle block with "this:". IRI "file#MyHeading" is homonym for both sections.
+ */
+export declare class HomonymHeadingWarning extends Error {
+    name: string;
+}
+/**
+ * TimeNoDateInHeadingHierarchyWarning is a possible cause in ParseError.
+ * Example:
+ * Heading with time "13:45" could be nested within a date to define schema:temporalCoverage. No parent date found.
+ */
+export declare class TimeNoDateInHeadingHierarchyWarning extends Error {
+    name: string;
+}
+/**
+ * Md2Rdf parses Markdown files and returns RDF quads.
+ *
+ * It follows the minimalistic LinkedDataNotes convention.
+ *
+ * It makes a "best effort" attempt to extract RDF from Markdown. It will report ParseError instances and go on if not stopped.
+ * Example: Syntax errors in one turtle code fence break this one fence, but the data in the next turtle code fences may be fine.
+ *
+ * Documentation: see README of the project.
+ *
+ */
+export declare class Md2Rdf {
+    protected options: Md2RdfParserOptions;
+    protected optionsNamespaceLineCount: number;
+    protected context?: ParseContext;
+    /**
+     * parserN3 is loaded lazily using _parserN3() when a Markdown file actually contains `turtle`.
+     */
+    protected parserN3?: Parser;
+    constructor(options: Md2RdfParserOptions);
+    /**
+     * only instantiate the parser when we really need it, to save memory and performance when checking many files
+     * @returns lazy-loaded parser
+     */
+    _parserN3(): Parser;
+    /**
+     * Parse the line.
+     * If it is an [atx heading](https://spec.commonmark.org/0.31.2/#atx-heading),
+     * 1. remember it in this.headings
+     * 2. generate quads that result from the heading, if it contains information according to LinkedDataNotes convention
+     * @param line
+     * @returns true, when this is a heading
+     */
+    _parseHeading(line: string): boolean;
+    /**
+     * Parse turtle or N3 syntax from the passed code block from the markdown file
+     * @param {String} n3 - the turtle passed as String
+     * @param {number} startLine the line number where the block started
+     * @return quads the quads parsed from the turtleString as array
+     * @throws error , see N3 parse()
+     */
+    _parseN3String(n3: string, startLine: number): void;
+    /**
+     * Check if the current line starts with a turtleBlock.
+     * If yes, parse the contents of a turtle code fence block and process it
+     * @param line - the line to check if it is an opening ```turtle code fence block
+     * @returns true, if this line started a turtle block
+     */
+    _parseTurtleBlock(line: string): boolean;
+    /**
+     * Parses markdown and returns an array of quads.
+     * See README.md
+     *
+     * Note: the method must only be called by a single thread. To parallelize parsing, use two Md2Rdf instances.
+     *
+     * @param input string of markdown
+     * @returns
+     *   quads: the array of rdf.js Quad parsed so far
+     *   error: this is the error that stopped the run, it should have been thrown by options.onError.
+     * @throws errors when there are unexpected programming bugs
+     */
+    parseString(input: string): {
+        quads: Array<Quad>;
+        error?: ParseError;
+    };
+}
+/**
+ * Helper Object to
+ * - hold variables during parsing
+ * - share helpers between Md2Rdf and Headings2This
+ */
+declare class ParseContext {
+    options: Md2RdfParserOptions;
+    ns: {
+        rdf: import("@rdfjs/namespace").NamespaceBuilder<string>;
+        rdfs: import("@rdfjs/namespace").NamespaceBuilder<string>;
+        schema: import("@rdfjs/namespace").NamespaceBuilder<string>;
+        md2rdf: import("@rdfjs/namespace").NamespaceBuilder<string>;
+    };
+    quads: Array<Quad>;
+    byLine: StringByLine;
+    headings: Headings2This;
+    constructor(options: Md2RdfParserOptions, byLine: StringByLine);
+}
+/**
+ * A generator that takes the passed string,
+ * splits it by \r\n and \n,
+ * yield each line,
+ * count current linenumber.
+ *
+ * Does not split everything in memory in the beginning to save memory.
+ *
+ * Use as iterator:
+ * let by = new StringByLine('example');
+ * for (const line of by) {
+ *   console.log(line);
+ * }
+ *
+ * Use with linenumber: see below in parseString()
+ *
+ * @param input string
+ */
+declare class StringByLine {
+    input: string;
+    linenumber: number;
+    start: number;
+    idx?: number;
+    constructor(input: string);
+    next(): IteratorResult<string>;
+    [Symbol.iterator](): this;
+}
+/**
+ * Helper class to concatenate ISO Times from filename and headings.
+ *
+ * The first element in the DateTimeHierarchy is the current heading (or filename, if no section).
+ * It points to a parent element in DateTimeHierarchy for further information.
+ *
+ * To read it, call toISODateTimeString.
+ *
+ * Once a date or date+time is found in the time hierarchy, it is returned.
+ * If no date is found, undefined is returned.
+ *
+ * Updating the current DateTimeHierarchy while parsing Markdown is handled by update()
+ *
+ * Evaluation is lazy: the current hierarchy is stored after every heading,
+ * but the regex time parsing is only done when an actual this: element is found in a turtle block
+ * to save CPU.
+ *
+ * Assumptions:
+ * - vast majority of Markdown will not contain turtle and `this:`, only seldomly date will be needed
+ * - regex parsing costs CPU
+ */
+export declare class DateTimeHierarchy {
+    /**
+     * This should match a DATE followed by TIME followed by ZONE
+     * DATE is optional
+     * TIME is optional, if TIME is set, ZONE is also optional
+     * yyyy-mm-dd ... optional
+     * 13:55
+     * 2021-02-10
+     * 2021-02-10 13:55
+     * 2021-02-10T13:55
+     * 2021-02-10 13:55:44.1234
+     * 2021-02-10T13:55Z+01:00
+     */
+    static readonly DATE_TIME_ZONE: RegExp;
+    static readonly TIME_ONLY: RegExp;
+    /**
+     * Same as DATE_TIME_ZONE, but the date part allows reduced ISO 8601 precision
+     * (yyyy or yyyy-mm, not just yyyy-mm-dd), and is anchored at both ends since it
+     * is only ever run against a single endpoint token already isolated by DURATION.
+     */
+    static readonly DATE_TIME_ZONE_FLEX: RegExp;
+    /**
+     * Matches a schema:temporalCoverage-style duration: two date/time endpoints
+     * separated by "/", "-" or "–" (an ISO 8601 time interval, e.g. "1939/1945" or
+     * "2015-11/.."). The end endpoint may be ".." (open-ended), a date/time of its
+     * own, or a bare time that shares the start endpoint's date.
+     */
+    static readonly DURATION: RegExp;
+    parent?: DateTimeHierarchy;
+    /**
+     * 0 = filename from baseIRI
+     * 1 = h1
+     * ...
+     * 6 = h6
+     */
+    level: number;
+    /**
+     * the content of the heading (after the #)
+     * or
+     * the filename
+     */
+    input: string;
+    /**
+     * was the input already parsed to date/time/zone?
+     * for lazy evaluation
+     */
+    parsed: boolean;
+    date?: string;
+    time?: string;
+    zone?: string;
+    /**
+     * The following three fields are only set when the input is a duration
+     * (ISO 8601 time interval, e.g. "1939/1945" or "2015-11/..").
+     * endDate/endTime/endZone hold the end endpoint; when the end endpoint is a
+     * bare time (e.g. "09:00-17:00"), endDate stays undefined and the start's
+     * own date is reused for it.
+     */
+    endDate?: string;
+    endTime?: string;
+    endZone?: string;
+    /**
+     * true when the duration's end endpoint was written as ".." (open-ended,
+     * e.g. "2015-11/..").
+     */
+    openEnded?: boolean;
+    constructor(level: number, input: string, parent?: DateTimeHierarchy);
+    /**
+     * In the hierarchy, find the next date.
+     * @returns a date if found, as string.
+     */
+    _findDate(): string | undefined;
+    /**
+     * Ensure this heading has been parsed before accessing the fields.
+     * Read the input, find a date and time, write it into the fields.
+     */
+    _parse(): void;
+    /**
+     * true when this heading's input was parsed as a duration (ISO 8601 time
+     * interval), i.e. it has an end endpoint or is open-ended.
+     */
+    get isDuration(): boolean;
+    /**
+     * Read the current Heading Hierarchy and format it as a schema:temporalCoverage value:
+     * either a plain ISO 8601 date/date-time, or - when the heading was written as a
+     * duration - an ISO 8601 time interval such as "1939/1945" or "2015-11/..".
+     * @returns the ISO 8601 date/date-time or interval defined by the heading hierarchy, or undefined if none found.
+     * @throws DoubleTimeInHeadingHierarchyWarning, TimeNoDateInHeadingHierarchyWarning
+     */
+    toISODateTimeString(context: ParseContext): string | undefined;
+    /**
+     * update the DateTimeHierarchy based on the latest input
+     * @param level current level
+     * @param input content of the heading, after "#"
+     * @param parent parent in the DateTimeHierarchy
+     * @returns an update pointer to the current relevant heading or file in the DateTimeHierarchy
+     */
+    static update(level: number, input: string, parent?: DateTimeHierarchy): DateTimeHierarchy | undefined;
+}
+/**
+ * Lazily load and cache the commonNamespaces.ttl shipped with md2rdf.
+ * @returns the common namespaces as turtle string
+ */
+export declare function commonNamespaces(): string;
+/**
+ * Handling of headings and `this:`
+ */
+declare class Headings2This {
+    protected options: Md2RdfParserOptions;
+    /**
+     * reusable regex to match headings
+     */
+    matchHeading: RegExp;
+    /**
+     * Buffer for the last heading content that was passed to parseLine,
+     * to be used when needed by prepThisSectionIRI.
+     * It is the "content" after the hash-marks and the space
+     * https://spec.commonmark.org/0.31.2/#atx-headings
+     */
+    lastHeadingContent: string | undefined;
+    /**
+     * line number of the lastHeadingContent
+     */
+    lastHeadingLinenumber: number | undefined;
+    /**
+     * The URI for the current section after the lastHeadingContent.
+     * Initiated undefined and reset to undefined after each heading, initialized lazily when needed.
+     */
+    thisSectionIRI: NamedNode<string> | undefined;
+    /**
+     * multiple headings having the same hash?
+     * remember them to send a warning
+     */
+    usedThisIRIs: Set<unknown>;
+    /**
+     * Time and date data found in the hierarchy of
+     * - filename (baseIRI) and
+     * - headings 1 to 6
+     */
+    dateTimeHierarchy?: DateTimeHierarchy;
+    /**
+     * Passing options
+     * @param options
+     */
+    constructor(options: Md2RdfParserOptions);
+    /**
+     * Translate thea markdown heading to a fragment which can be used in an IRI.
+     * See LinkeDataNotes-Markdown Syntax
+     * @param {String} headingContent the content of the heading. This is the string after the ATX heading hash-marks and space (not including leading hash-signs). https://spec.commonmark.org/0.31.2/#atx-headings
+     * @return {String} the fragment
+     */
+    _headingToFragment(headingContent: string): string;
+    /**
+     * create a section IRI based on file and fragment
+     */
+    _createSectionIri(inputfileIRI: string, fragment: string): string;
+    /**
+     * Parse the line for a heading starting with `#`, see [atx heading](https://spec.commonmark.org/0.31.2/#atx-heading).
+     * It remembers the headings that were passed to it, should they be later needed by prepThisSectionIRI() or the dateTimeHierarchy.
+     * @param line the Markdown line
+     * @param linenumber the current line number, should it be needed for error logging later
+     * @returns true, when the current line is an atx heading starting with '#'
+     */
+    parseLine(line: string, linenumber: number): boolean;
+    /**
+     * Prepare the current section IRI, as the caller will use it in quads.
+     *
+     * This:
+     * - mints the IRI, remembers it as node for reuse further down the section
+     * - adds metadata about the section via quads.push()
+     *
+     * @returns the section IRI as RDF Node or undefined if no heading was in the file yet
+     */
+    prepThisSectionIRI(context: ParseContext): undefined | NamedNode<string>;
+}
+export {};