mdmodels-core 0.2.2 → 0.2.3

package/README.md

@@ -1,29 +1,52 @@
-# MD-Models
+# MD-Models 🚀
 
-![Crates.io Version](https://img.shields.io/crates/v/mdmodels) ![NPM Version](https://img.shields.io/npm/v/mdmodels-core)
-![PyPI - Version](https://img.shields.io/pypi/v/mdmodels-core)
- ![Build Status](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml/badge.svg) 
+[![Crates.io Version](https://img.shields.io/crates/v/mdmodels)](https://crates.io/crates/mdmodels)
+[![NPM Version](https://img.shields.io/npm/v/mdmodels-core)](https://www.npmjs.com/package/mdmodels-core)
+[![PyPI - Version](https://img.shields.io/pypi/v/mdmodels-core)](https://pypi.org/project/mdmodels-core/)
+[![Build Status](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml/badge.svg)](https://github.com/JR-1991/sdrdm.rs/actions/workflows/test.yml)
 
-Welcome to Markdown Models (MD-Models), a powerful framework for research data management that prioritizes narrative and readability for data models.
+*Welcome to Markdown Models (MD-Models)!* 📝 
 
-With an adaptable markdown-based schema language, MD-Models automatically generates schemas and programming language representations. This markdown schema forms the foundation for object-oriented models, enabling seamless cross-format compatibility and simplifying modifications to data structures.
+We've created this framework to make research data management more intuitive and accessible while maintaining professional standards. Our approach uses markdown-based schema definitions to transform complex data modeling into something you'll actually enjoy working with.
 
-## Core Philosophy
+The framework does the heavy lifting for you - automatically generating technical schemas and programming language implementations from your markdown files. This means you can focus on designing your data structures in a format that makes sense, while we handle the technical translations. ⚙️
 
-The primary motivation behind MD-Models is to reduce cognitive overhead and maintenance burden by unifying documentation and structural definition into a single source of truth. Traditional approaches often require maintaining separate artifacts:
+## Core Philosophy 💡
 
-1. Technical schemas (JSON Schema, XSD, ShEx, SHACL)
-2. Programming language implementations
-3. Documentation for domain experts
-4. API documentation
+We built MD-Models to solve a common frustration in data modeling: juggling multiple versions of the same information. Here's what typically happens in traditional approaches:
 
-This separation frequently leads to documentation drift and increases the cognitive load on both developers and domain experts.
+1. Technical Schema Definitions 📊
+   - You need JSON Schema, XSD, ShEx, or SHACL
+   - Each format has its own complexity
+   - Changes need to be replicated across formats
 
-Check out the [documentation and graph editor](https://mdmodels.vercel.app/?about) for more information.
+2. Language-Specific Implementations 💻
+   - Different programming languages need different implementations
+   - Each requires maintenance and updates
+   - Keeping everything in sync is challenging
 
-### Example
+3. Documentation 📚
+   - Technical docs for developers
+   - Simplified explanations for domain experts
+   - API documentation that needs constant updates
 
-The schema syntax uses Markdown to define data models in a clear and structured way. Each object is introduced with a header, followed by its attributes. Attributes are described with their type, a brief explanation, and optional metadata like terms. Nested or related objects are represented using array types or references to other objects.
+Instead of dealing with all these separate pieces, MD-Models gives you one clear source of truth. Write it once, use it everywhere! ✨
+
+Ready to see it in action? Check out our [book](https://fairchemistry.github.io/md-models/) for a deeper dive into the framework and [graph editor](https://mdmodels.vercel.app/?about) to get started.
+
+### Schema Design 🎨
+
+Our schema syntax makes the most of markdown's natural readability. Here's what you can do:
+
+- Define objects with clear, descriptive headers
+- Specify attributes with all the details you need
+- Add rich descriptions that everyone can understand
+- Include semantic annotations when you need them
+- Define relationships between objects easily
+
+We've designed this approach to work for everyone on your team - whether they're technical experts or domain specialists. You get all the precision you need for automatic code generation, while keeping things clear and approachable. 🤝
+
+Here is an example of a markdown model definition:
 
 ```markdown
 ---
@@ -52,33 +75,71 @@ prefixes:
   - Description: The street of the address
 ```
 
-## Installation
+Lets break down the example:
+
+We define an object `Person` with two attributes: `name` and `age`. We also define an object `Address` with one attribute: `street`. An object can be defined as a list of attributes, which can be either primitive types, other objects, or lists of other objects.
+
+Objects are defined by using the `###` header and a list of attributes. Attributes are defined by using the `-` prefix. The type of the attribute is specified after the `:`. The description of the attribute is specified after the `-`. The term of the attribute is specified after the `-`.
+
+Attributes can hold any key-value pair as metadata. For instance, the `age` attribute has the following metadata:
+
+```markdown
+- age
+  - Type: integer
+  - Description: The age of the person
+```
+
+The `age` attribute is of type `integer` and has the following description: `The age of the person`. You could also add more metadata to the attribute, such as `minValue` and `maxValue` for JSON Schema. If your application needs more metadata, you can add it to the attribute as well - There are no restrictions on the metadata.
+
+> [!NOTE]
+> All JSON-Schema validation keywords are supported, except for `readOnly` and `writeOnly`.
+
+### Large Language Model Integration 🤖
+
+Our framework also supports large language model guided extraction of information from natural language text into a structured format. Typically you would use a JSON schema as an intermediate format for this or use specialized libraries such as [Instructor](https://github.com/jxnl/instructor) or [LangChain](https://github.com/langchain-ai/langchain) to accomplish this.
 
-In order to install the command line tool, you can use the following command:
+We have wrapped all of this functionality into a single command:
 
 ```bash
-git clone https://github.com/JR-1991/md-models
-cd md-models
-cargo install --path .
+export OPENAI_API_KEY="sk-..."
+md-models extract -i text.txt -m mymodel.md -o structured.json
 ```
 
-## Command line usage
+This will read the input text file and extract the information into the structured format defined in the markdown model. The output will be written to the `structured.json` file. You can even pass an existing JSON dataset and let the LLM update the dataset with the new information. By utilizing JSON patch, we can ensure that the original dataset is kept intact and only the new information is added.
 
-The command line tool can be used to convert markdown files to various formats. The following command will convert a markdown file to Python code:
+## Installation 🛠️
+
+MD-Models is primarily a command line tool. In order to install the command line tool, you can use the following command:
 
 ```bash
-md-models convert -i model.md -o lib.py -l python-dataclass
+git clone https://github.com/FAIRChemistry/md-models
+cd md-models
+cargo install --path .
 ```
 
-This will read the input file `model.md` and write the output to `lib.py` using the Python dataclass template. Alternatively, you can also pass a URL as input to fetch the model remotely. For an overview of all available templates, you can use the following command:
+Checkout our releases, where you can find pre-compiled binaries for the command line tool!
+
+## Command line usage 📝
+
+The command line tool can be used to convert markdown files to various formats. For instance, the following command will convert a markdown file to Python code:
 
 ```bash
-md-models --help
+md-models convert -i model.md -o lib.py -t python-dataclass
 ```
 
+This will read the input file `model.md` and write the output to `lib.py` using the Python dataclass template. Alternatively, you can also pass a URL as input to fetch the model remotely.
+
+Here is a list of all available sub commands:
+
+- `convert`: Convert a markdown file to a specific format
+- `validate`: Validate and check if a markdown file conforms our specification
+- `pipeline`: Pipeline for generating multiple files
+- `extract`: Large Language Model Extraction guided by a markdown model
+- `dataset`: Validate a dataset against a markdown model
+
 ## Available templates
 
-The following templates are available:
+The following templates are available for the `convert` command:
 
 - `python-dataclass`: Python dataclass implementation with JSON-LD support
 - `python-pydantic`: PyDantic implementation with JSON-LD support
@@ -99,9 +160,11 @@ The following templates are available:
 - `mkdocs`: MkDocs documentation format
 - `linkml`: LinkML schema definition
 
-## Installation options
+## Installation options 📦
+
+We've made our core Rust library incredibly versatile by compiling it to both Python and WebAssembly! This means you can use our model conversion tools not just from the command line, but directly in your Python applications or web browsers.
 
-The main Rust crate is compiled to Python and WebAssembly, allowing the usage beyond the command line tool. These are the main packages:
+We provide several packages to make integration seamless:
 
 - **[Core Python Package](https://pypi.org/project/mdmodels-core/)**: Install via pip:
   ```bash
@@ -121,7 +184,7 @@ The main Rust crate is compiled to Python and WebAssembly, allowing the usage be
   npm install mdmodels-core
   ```
 
-## Development
+## Development 🔧
 
 This project uses GitHub Actions for continuous integration. The tests can be run using the following command:
 
@@ -139,4 +202,4 @@ pip install pre-commit
 pre-commit install
 ```
 
-Once the pre-commit hooks are installed, they will run on every commit. This will ensure that the code is formatted and linted correctly. And the clippy CI will not complain about warnings.
+Once the pre-commit hooks are installed, they will run on every commit. This will ensure that the code is formatted and linted correctly. And the clippy CI will not complain about warnings.

package/mdmodels-core.d.ts

@@ -80,102 +80,6 @@ export enum Templates {
   Linkml = 18,
   Julia = 19,
 }
-/**
- * A raw key-value representation of an attribute option.
- *
- * This struct provides a simple string-based representation of options,
- * which is useful for serialization/deserialization and when working
- * with untyped data.
- */
-export interface RawOption {
-    /**
-     * The key/name of the option
-     */
-    key: string;
-    /**
-     * The string value of the option
-     */
-    value: string;
-}
-
-/**
- * Represents an option for an attribute in a data model.
- *
- * This enum provides a strongly-typed representation of various attribute options
- * that can be used to configure and constrain attributes in a data model.
- *
- * The options are grouped into several categories:
- * - JSON Schema validation options (e.g., minimum/maximum values, length constraints)
- * - SQL database options (e.g., primary key)
- * - LinkML specific options (e.g., readonly, recommended)
- * - Custom options via the `Other` variant
- *
- */
-export type AttrOption = { Example: string } | { MinimumValue: number } | { MaximumValue: number } | { MinItems: number } | { MaxItems: number } | { MinLength: number } | { MaxLength: number } | { Pattern: string } | { Unique: boolean } | { MultipleOf: number } | { ExclusiveMinimum: number } | { ExclusiveMaximum: number } | { PrimaryKey: boolean } | { ReadOnly: boolean } | { Recommended: boolean } | { Other: { key: string; value: string } };
-
-/**
- * Represents an enumeration with a name and mappings.
- */
-export interface Enumeration {
-    /**
-     * Name of the enumeration.
-     */
-    name: string;
-    /**
-     * Mappings associated with the enumeration.
-     */
-    mappings: Map<string, string>;
-    /**
-     * Documentation string for the enumeration.
-     */
-    docstring: string;
-    /**
-     * The line number of the enumeration
-     */
-    position: Position | undefined;
-}
-
-/**
- * Represents an object with a name, attributes, docstring, and an optional term.
- */
-export interface Object {
-    /**
-     * Name of the object.
-     */
-    name: string;
-    /**
-     * List of attributes associated with the object.
-     */
-    attributes: Attribute[];
-    /**
-     * Documentation string for the object.
-     */
-    docstring: string;
-    /**
-     * Optional term associated with the object.
-     */
-    term?: string;
-    /**
-     * Parent object of the object.
-     */
-    parent?: string;
-    /**
-     * The line number of the object
-     */
-    position?: Position;
-}
-
-export interface PositionRange {
-    start: number;
-    end: number;
-}
-
-export interface Position {
-    line: number;
-    column: PositionRange;
-    offset: PositionRange;
-}
-
 export interface DataModel {
     name?: string;
     objects: Object[];
@@ -243,11 +147,89 @@ export interface Attribute {
     import_prefix?: string;
 }
 
+/**
+ * Validator for checking the integrity of a data model.
+ */
+export interface Validator {
+    is_valid: boolean;
+    errors: ValidationError[];
+}
+
+/**
+ * Enum representing the type of validation error.
+ */
+export type ErrorType = "NameError" | "TypeError" | "DuplicateError" | "GlobalError" | "XMLError" | "ObjectError";
+
+/**
+ * Represents a validation error in the data model.
+ */
+export interface ValidationError {
+    message: string;
+    object: string | undefined;
+    attribute: string | undefined;
+    location: string;
+    solution: string | undefined;
+    error_type: ErrorType;
+    positions: Position[];
+}
+
 /**
  * Represents an XML type, either an attribute or an element.
  */
 export type XMLType = { Attribute: { is_attr: boolean; name: string } } | { Element: { is_attr: boolean; name: string } } | { Wrapped: { is_attr: boolean; name: string; wrapped: string[] | undefined } };
 
+/**
+ * Represents an enumeration with a name and mappings.
+ */
+export interface Enumeration {
+    /**
+     * Name of the enumeration.
+     */
+    name: string;
+    /**
+     * Mappings associated with the enumeration.
+     */
+    mappings: Map<string, string>;
+    /**
+     * Documentation string for the enumeration.
+     */
+    docstring: string;
+    /**
+     * The line number of the enumeration
+     */
+    position: Position | undefined;
+}
+
+/**
+ * Represents an object with a name, attributes, docstring, and an optional term.
+ */
+export interface Object {
+    /**
+     * Name of the object.
+     */
+    name: string;
+    /**
+     * List of attributes associated with the object.
+     */
+    attributes: Attribute[];
+    /**
+     * Documentation string for the object.
+     */
+    docstring: string;
+    /**
+     * Optional term associated with the object.
+     */
+    term?: string;
+    /**
+     * Parent object of the object.
+     */
+    parent?: string;
+    /**
+     * The line number of the object
+     */
+    position?: Position;
+}
+
 /**
  * Represents different types of model imports.
  *
@@ -294,28 +276,46 @@ export interface FrontMatter {
 }
 
 /**
- * Validator for checking the integrity of a data model.
+ * A raw key-value representation of an attribute option.
+ *
+ * This struct provides a simple string-based representation of options,
+ * which is useful for serialization/deserialization and when working
+ * with untyped data.
  */
-export interface Validator {
-    is_valid: boolean;
-    errors: ValidationError[];
+export interface RawOption {
+    /**
+     * The key/name of the option
+     */
+    key: string;
+    /**
+     * The string value of the option
+     */
+    value: string;
 }
 
 /**
- * Enum representing the type of validation error.
+ * Represents an option for an attribute in a data model.
+ *
+ * This enum provides a strongly-typed representation of various attribute options
+ * that can be used to configure and constrain attributes in a data model.
+ *
+ * The options are grouped into several categories:
+ * - JSON Schema validation options (e.g., minimum/maximum values, length constraints)
+ * - SQL database options (e.g., primary key)
+ * - LinkML specific options (e.g., readonly, recommended)
+ * - Custom options via the `Other` variant
+ *
  */
-export type ErrorType = "NameError" | "TypeError" | "DuplicateError" | "GlobalError" | "XMLError" | "ObjectError";
+export type AttrOption = { Example: string } | { MinimumValue: number } | { MaximumValue: number } | { MinItems: number } | { MaxItems: number } | { MinLength: number } | { MaxLength: number } | { Pattern: string } | { Unique: boolean } | { MultipleOf: number } | { ExclusiveMinimum: number } | { ExclusiveMaximum: number } | { PrimaryKey: boolean } | { ReadOnly: boolean } | { Recommended: boolean } | { Other: { key: string; value: string } };
 
-/**
- * Represents a validation error in the data model.
- */
-export interface ValidationError {
-    message: string;
-    object: string | undefined;
-    attribute: string | undefined;
-    location: string;
-    solution: string | undefined;
-    error_type: ErrorType;
-    positions: Position[];
+export interface PositionRange {
+    start: number;
+    end: number;
+}
+
+export interface Position {
+    line: number;
+    column: PositionRange;
+    offset: PositionRange;
 }
 

package/package.json

@@ -5,7 +5,7 @@
     "Jan Range <jan.range@simtech.uni-stuttgart.de>"
   ],
   "description": "A tool to generate models, code and schemas from markdown files",
-  "version": "0.2.2",
+  "version": "0.2.3",
   "license": "MIT",
   "repository": {
     "type": "git",