mdmodels-core 0.2.1 → 0.2.3

package/README.md

@@ -1,29 +1,52 @@
-# MD-Models
+# MD-Models 🚀
 
-![Crates.io Version](https://img.shields.io/crates/v/mdmodels) ![NPM Version](https://img.shields.io/npm/v/mdmodels-core)
-![PyPI - Version](https://img.shields.io/pypi/v/mdmodels-core)
- ![Build Status](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml/badge.svg) 
+[![Crates.io Version](https://img.shields.io/crates/v/mdmodels)](https://crates.io/crates/mdmodels)
+[![NPM Version](https://img.shields.io/npm/v/mdmodels-core)](https://www.npmjs.com/package/mdmodels-core)
+[![PyPI - Version](https://img.shields.io/pypi/v/mdmodels-core)](https://pypi.org/project/mdmodels-core/)
+[![Build Status](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml/badge.svg)](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml)
 
-Welcome to Markdown Models (MD-Models), a powerful framework for research data management that prioritizes narrative and readability for data models.
+*Welcome to Markdown Models (MD-Models)!* 📝 
 
-With an adaptable markdown-based schema language, MD-Models automatically generates schemas and programming language representations. This markdown schema forms the foundation for object-oriented models, enabling seamless cross-format compatibility and simplifying modifications to data structures.
+We've created this framework to make research data management more intuitive and accessible while maintaining professional standards. Our approach uses markdown-based schema definitions to transform complex data modeling into something you'll actually enjoy working with.
 
-## Core Philosophy
+The framework does the heavy lifting for you - automatically generating technical schemas and programming language implementations from your markdown files. This means you can focus on designing your data structures in a format that makes sense, while we handle the technical translations. ⚙️
 
-The primary motivation behind MD-Models is to reduce cognitive overhead and maintenance burden by unifying documentation and structural definition into a single source of truth. Traditional approaches often require maintaining separate artifacts:
+## Core Philosophy 💡
 
-1. Technical schemas (JSON Schema, XSD, ShEx, SHACL)
-2. Programming language implementations
-3. Documentation for domain experts
-4. API documentation
+We built MD-Models to solve a common frustration in data modeling: juggling multiple versions of the same information. Here's what typically happens in traditional approaches:
 
-This separation frequently leads to documentation drift and increases the cognitive load on both developers and domain experts.
+1. Technical Schema Definitions 📊
+   - You need JSON Schema, XSD, ShEx, or SHACL
+   - Each format has its own complexity
+   - Changes need to be replicated across formats
 
-Check out the [documentation and graph editor](https://mdmodels.vercel.app/?about) for more information.
+2. Language-Specific Implementations 💻
+   - Different programming languages need different implementations
+   - Each requires maintenance and updates
+   - Keeping everything in sync is challenging
 
-### Example
+3. Documentation 📚
+   - Technical docs for developers
+   - Simplified explanations for domain experts
+   - API documentation that needs constant updates
 
-The schema syntax uses Markdown to define data models in a clear and structured way. Each object is introduced with a header, followed by its attributes. Attributes are described with their type, a brief explanation, and optional metadata like terms. Nested or related objects are represented using array types or references to other objects.
+Instead of dealing with all these separate pieces, MD-Models gives you one clear source of truth. Write it once, use it everywhere! ✨
+
+Ready to see it in action? Check out our [book](https://fairchemistry.github.io/md-models/) for a deeper dive into the framework and [graph editor](https://mdmodels.vercel.app/?about) to get started.
+
+### Schema Design 🎨
+
+Our schema syntax makes the most of markdown's natural readability. Here's what you can do:
+
+- Define objects with clear, descriptive headers
+- Specify attributes with all the details you need
+- Add rich descriptions that everyone can understand
+- Include semantic annotations when you need them
+- Define relationships between objects easily
+
+We've designed this approach to work for everyone on your team - whether they're technical experts or domain specialists. You get all the precision you need for automatic code generation, while keeping things clear and approachable. 🤝
+
+Here is an example of a markdown model definition:
 
 ```markdown
 ---
@@ -52,33 +75,71 @@ prefixes:
   - Description: The street of the address
 ```
 
-## Installation
+Lets break down the example:
+
+We define an object `Person` with two attributes: `name` and `age`. We also define an object `Address` with one attribute: `street`. An object can be defined as a list of attributes, which can be either primitive types, other objects, or lists of other objects.
+
+Objects are defined by using the `###` header and a list of attributes. Attributes are defined by using the `-` prefix. The type of the attribute is specified after the `:`. The description of the attribute is specified after the `-`. The term of the attribute is specified after the `-`.
+
+Attributes can hold any key-value pair as metadata. For instance, the `age` attribute has the following metadata:
+
+```markdown
+- age
+  - Type: integer
+  - Description: The age of the person
+```
+
+The `age` attribute is of type `integer` and has the following description: `The age of the person`. You could also add more metadata to the attribute, such as `minValue` and `maxValue` for JSON Schema. If your application needs more metadata, you can add it to the attribute as well - There are no restrictions on the metadata.
+
+> [!NOTE]
+> All JSON-Schema validation keywords are supported, except for `readOnly` and `writeOnly`.
+
+### Large Language Model Integration 🤖
+
+Our framework also supports large language model guided extraction of information from natural language text into a structured format. Typically you would use a JSON schema as an intermediate format for this or use specialized libraries such as [Instructor](https://github.com/jxnl/instructor) or [LangChain](https://github.com/langchain-ai/langchain) to accomplish this.
 
-In order to install the command line tool, you can use the following command:
+We have wrapped all of this functionality into a single command:
 
 ```bash
-git clone https://github.com/JR-1991/md-models
-cd md-models
-cargo install --path .
+export OPENAI_API_KEY="sk-..."
+md-models extract -i text.txt -m mymodel.md -o structured.json
 ```
 
-## Command line usage
+This will read the input text file and extract the information into the structured format defined in the markdown model. The output will be written to the `structured.json` file. You can even pass an existing JSON dataset and let the LLM update the dataset with the new information. By utilizing JSON patch, we can ensure that the original dataset is kept intact and only the new information is added.
 
-The command line tool can be used to convert markdown files to various formats. The following command will convert a markdown file to Python code:
+## Installation 🛠️
+
+MD-Models is primarily a command line tool. In order to install the command line tool, you can use the following command:
 
 ```bash
-md-models convert -i model.md -o lib.py -l python-dataclass
+git clone https://github.com/FAIRChemistry/md-models
+cd md-models
+cargo install --path .
 ```
 
-This will read the input file `model.md` and write the output to `lib.py` using the Python dataclass template. Alternatively, you can also pass a URL as input to fetch the model remotely. For an overview of all available templates, you can use the following command:
+Checkout our releases, where you can find pre-compiled binaries for the command line tool!
+
+## Command line usage 📝
+
+The command line tool can be used to convert markdown files to various formats. For instance, the following command will convert a markdown file to Python code:
 
 ```bash
-md-models --help
+md-models convert -i model.md -o lib.py -t python-dataclass
 ```
 
+This will read the input file `model.md` and write the output to `lib.py` using the Python dataclass template. Alternatively, you can also pass a URL as input to fetch the model remotely.
+
+Here is a list of all available sub commands:
+
+- `convert`: Convert a markdown file to a specific format
+- `validate`: Validate and check if a markdown file conforms our specification
+- `pipeline`: Pipeline for generating multiple files
+- `extract`: Large Language Model Extraction guided by a markdown model
+- `dataset`: Validate a dataset against a markdown model
+
 ## Available templates
 
-The following templates are available:
+The following templates are available for the `convert` command:
 
 - `python-dataclass`: Python dataclass implementation with JSON-LD support
 - `python-pydantic`: PyDantic implementation with JSON-LD support
@@ -99,9 +160,11 @@ The following templates are available:
 - `mkdocs`: MkDocs documentation format
 - `linkml`: LinkML schema definition
 
-## Installation options
+## Installation options 📦
+
+We've made our core Rust library incredibly versatile by compiling it to both Python and WebAssembly! This means you can use our model conversion tools not just from the command line, but directly in your Python applications or web browsers.
 
-The main Rust crate is compiled to Python and WebAssembly, allowing the usage beyond the command line tool. These are the main packages:
+We provide several packages to make integration seamless:
 
 - **[Core Python Package](https://pypi.org/project/mdmodels-core/)**: Install via pip:
   ```bash
@@ -121,7 +184,7 @@ The main Rust crate is compiled to Python and WebAssembly, allowing the usage be
   npm install mdmodels-core
   ```
 
-## Development
+## Development 🔧
 
 This project uses GitHub Actions for continuous integration. The tests can be run using the following command:
 
@@ -139,4 +202,4 @@ pip install pre-commit
 pre-commit install
 ```
 
-Once the pre-commit hooks are installed, they will run on every commit. This will ensure that the code is formatted and linted correctly. And the clippy CI will not complain about warnings.
+Once the pre-commit hooks are installed, they will run on every commit. This will ensure that the code is formatted and linted correctly. And the clippy CI will not complain about warnings.

package/mdmodels-core.d.ts

@@ -80,51 +80,6 @@ export enum Templates {
   Linkml = 18,
   Julia = 19,
 }
-/**
- * Represents different types of model imports.
- *
- * Can be either a remote URL or a local file path.
- */
-export type ImportType = { Remote: string } | { Local: string };
-
-/**
- * Represents the front matter data of a markdown file.
- */
-export interface FrontMatter {
-    /**
-     * Identifier field of the model.
-     */
-    id: string | undefined;
-    /**
-     * A boolean field with a default value, renamed from `id-field`.
-     */
-    "id-field"?: boolean;
-    /**
-     * Optional hashmap of prefixes.
-     */
-    prefixes: Map<string, string> | undefined;
-    /**
-     * Optional namespace map.
-     */
-    nsmap: Map<string, string> | undefined;
-    /**
-     * A string field with a default value representing the repository URL.
-     */
-    repo?: string;
-    /**
-     * A string field with a default value representing the prefix.
-     */
-    prefix?: string;
-    /**
-     * Import remote or local models.
-     */
-    imports?: Map<string, ImportType>;
-    /**
-     * Allow empty models.
-     */
-    "allow-empty"?: boolean;
-}
-
 export interface DataModel {
     name?: string;
     objects: Object[];
@@ -193,53 +148,35 @@ export interface Attribute {
 }
 
 /**
- * Represents an XML type, either an attribute or an element.
+ * Validator for checking the integrity of a data model.
  */
-export type XMLType = { Attribute: { is_attr: boolean; name: string } } | { Element: { is_attr: boolean; name: string } } | { Wrapped: { is_attr: boolean; name: string; wrapped: string[] | undefined } };
-
-export interface PositionRange {
-    start: number;
-    end: number;
+export interface Validator {
+    is_valid: boolean;
+    errors: ValidationError[];
 }
 
-export interface Position {
-    line: number;
-    column: PositionRange;
-    offset: PositionRange;
-}
+/**
+ * Enum representing the type of validation error.
+ */
+export type ErrorType = "NameError" | "TypeError" | "DuplicateError" | "GlobalError" | "XMLError" | "ObjectError";
 
 /**
- * A raw key-value representation of an attribute option.
- *
- * This struct provides a simple string-based representation of options,
- * which is useful for serialization/deserialization and when working
- * with untyped data.
+ * Represents a validation error in the data model.
  */
-export interface RawOption {
-    /**
-     * The key/name of the option
-     */
-    key: string;
-    /**
-     * The string value of the option
-     */
-    value: string;
+export interface ValidationError {
+    message: string;
+    object: string | undefined;
+    attribute: string | undefined;
+    location: string;
+    solution: string | undefined;
+    error_type: ErrorType;
+    positions: Position[];
 }
 
 /**
- * Represents an option for an attribute in a data model.
- *
- * This enum provides a strongly-typed representation of various attribute options
- * that can be used to configure and constrain attributes in a data model.
- *
- * The options are grouped into several categories:
- * - JSON Schema validation options (e.g., minimum/maximum values, length constraints)
- * - SQL database options (e.g., primary key)
- * - LinkML specific options (e.g., readonly, recommended)
- * - Custom options via the `Other` variant
- *
+ * Represents an XML type, either an attribute or an element.
  */
-export type AttrOption = { Example: string } | { MinimumValue: number } | { MaximumValue: number } | { MinItems: number } | { MaxItems: number } | { MinLength: number } | { MaxLength: number } | { Pattern: string } | { Unique: boolean } | { MultipleOf: number } | { ExclusiveMinimum: number } | { ExclusiveMaximum: number } | { PrimaryKey: boolean } | { ReadOnly: boolean } | { Recommended: boolean } | { Other: { key: string; value: string } };
+export type XMLType = { Attribute: { is_attr: boolean; name: string } } | { Element: { is_attr: boolean; name: string } } | { Wrapped: { is_attr: boolean; name: string; wrapped: string[] | undefined } };
 
 /**
  * Represents an enumeration with a name and mappings.
@@ -294,28 +231,91 @@ export interface Object {
 }
 
 /**
- * Validator for checking the integrity of a data model.
+ * Represents different types of model imports.
+ *
+ * Can be either a remote URL or a local file path.
  */
-export interface Validator {
-    is_valid: boolean;
-    errors: ValidationError[];
+export type ImportType = { Remote: string } | { Local: string };
+
+/**
+ * Represents the front matter data of a markdown file.
+ */
+export interface FrontMatter {
+    /**
+     * Identifier field of the model.
+     */
+    id: string | undefined;
+    /**
+     * A boolean field with a default value, renamed from `id-field`.
+     */
+    "id-field"?: boolean;
+    /**
+     * Optional hashmap of prefixes.
+     */
+    prefixes: Map<string, string> | undefined;
+    /**
+     * Optional namespace map.
+     */
+    nsmap: Map<string, string> | undefined;
+    /**
+     * A string field with a default value representing the repository URL.
+     */
+    repo?: string;
+    /**
+     * A string field with a default value representing the prefix.
+     */
+    prefix?: string;
+    /**
+     * Import remote or local models.
+     */
+    imports?: Map<string, ImportType>;
+    /**
+     * Allow empty models.
+     */
+    "allow-empty"?: boolean;
 }
 
 /**
- * Enum representing the type of validation error.
+ * A raw key-value representation of an attribute option.
+ *
+ * This struct provides a simple string-based representation of options,
+ * which is useful for serialization/deserialization and when working
+ * with untyped data.
  */
-export type ErrorType = "NameError" | "TypeError" | "DuplicateError" | "GlobalError" | "XMLError" | "ObjectError";
+export interface RawOption {
+    /**
+     * The key/name of the option
+     */
+    key: string;
+    /**
+     * The string value of the option
+     */
+    value: string;
+}
 
 /**
- * Represents a validation error in the data model.
+ * Represents an option for an attribute in a data model.
+ *
+ * This enum provides a strongly-typed representation of various attribute options
+ * that can be used to configure and constrain attributes in a data model.
+ *
+ * The options are grouped into several categories:
+ * - JSON Schema validation options (e.g., minimum/maximum values, length constraints)
+ * - SQL database options (e.g., primary key)
+ * - LinkML specific options (e.g., readonly, recommended)
+ * - Custom options via the `Other` variant
+ *
  */
-export interface ValidationError {
-    message: string;
-    object: string | undefined;
-    attribute: string | undefined;
-    location: string;
-    solution: string | undefined;
-    error_type: ErrorType;
-    positions: Position[];
+export type AttrOption = { Example: string } | { MinimumValue: number } | { MaximumValue: number } | { MinItems: number } | { MaxItems: number } | { MinLength: number } | { MaxLength: number } | { Pattern: string } | { Unique: boolean } | { MultipleOf: number } | { ExclusiveMinimum: number } | { ExclusiveMaximum: number } | { PrimaryKey: boolean } | { ReadOnly: boolean } | { Recommended: boolean } | { Other: { key: string; value: string } };
+
+export interface PositionRange {
+    start: number;
+    end: number;
+}
+
+export interface Position {
+    line: number;
+    column: PositionRange;
+    offset: PositionRange;
 }
 

package/package.json

@@ -5,7 +5,7 @@
     "Jan Range <jan.range@simtech.uni-stuttgart.de>"
   ],
   "description": "A tool to generate models, code and schemas from markdown files",
-  "version": "0.2.1",
+  "version": "0.2.3",
   "license": "MIT",
   "repository": {
     "type": "git",