mustrd 0.3.1a5__tar.gz → 0.3.1a6__tar.gz

{mustrd-0.3.1a5 → mustrd-0.3.1a6}/PKG-INFO

@@ -1,14 +1,19 @@
 Metadata-Version: 2.3
 Name: mustrd
-Version: 0.3.1a5
+Version: 0.3.1a6
 Summary: A Spec By Example framework for RDF and SPARQL, Inspired by Cucumber.
 License: MIT
 Author: John Placek
 Author-email: john.placek@semanticpartners.com
-Requires-Python: >=3.10,<4.0
-Classifier: Programming Language :: Python
+Requires-Python: >=3.11,<4.0
 Classifier: Framework :: Pytest
+Classifier: License :: OSI Approved :: MIT License
 Classifier: Natural Language :: English
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Software Development :: Quality Assurance
 Classifier: Topic :: Software Development :: Testing
 Classifier: Topic :: Utilities
@@ -34,13 +39,13 @@ Requires-Dist: urllib3 (==1.26.19)
 Project-URL: Repository, https://github.com/Semantic-partners/mustrd
 Description-Content-Type: text/markdown
 
-# mustrd
+# MustRD
 
 **"MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles."**
 
-[<img src="https://github.com/Semantic-partners/mustrd/raw/python-coverage-comment-action-data/badge.svg?sanitize=true" alt="coverage badge">](https://github.com/Semantic-partners/mustrd/tree/python-coverage-comment-action-data)
+[![Coverage Badge](https://github.com/Semantic-partners/mustrd/raw/python-coverage-comment-action-data/badge.svg?sanitize=true)](https://github.com/Semantic-partners/mustrd/tree/python-coverage-comment-action-data)
 
-### Why?
+## Why?
 
 SPARQL is a powerful query language for RDF data, but how can you ensure your queries and transformations are doing what you intend? Whether you're working on a pipeline or a standalone query, certainty is key.
 
@@ -48,11 +53,11 @@ While RDF and SPARQL offer great flexibility, we noticed a gap in tooling to val
 
 With MustRD, you can:
 
-* Define data scenarios and verify that queries produce the expected results.
-* Test edge cases to ensure your queries remain reliable.
-* Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.
+- Define data scenarios and verify that queries produce the expected results.
+- Test edge cases to ensure your queries remain reliable.
+- Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.
 
-### What?
+## What?
 
 MustRD is a Spec-By-Example ontology with a reference Python implementation, inspired by tools like Cucumber. It uses the Given-When-Then approach to define and validate SPARQL queries and transformations.
 
@@ -62,23 +67,85 @@ MustRD is designed to be triplestore/SPARQL engine agnostic, leveraging open sta
 
 MustRD is not an alternative to SHACL. While SHACL validates data structures, MustRD focuses on validating data transformations and query results.
 
-### How?
+## How?
 
 You define your specs in Turtle (`.ttl`) or TriG (`.trig`) files using the Given-When-Then approach:
 
-* **Given**: Define the starting dataset.
-* **When**: Specify the action (e.g., a SPARQL query).
-* **Then**: Outline the expected results.
+- **Given**: Define the starting dataset.
+- **When**: Specify the action (e.g., a SPARQL query).
+- **Then**: Outline the expected results.
 
 Depending on the type of SPARQL query (CONSTRUCT, SELECT, INSERT/DELETE), MustRD runs the query and compares the results against the expectations defined in the spec.
 
 Expectations can also be defined as:
 
-* INSERT queries.
-* SELECT queries.
-* Higher-order expectation languages, similar to those used in various platforms.
+- INSERT queries.
+- SELECT queries.
+- Higher-order expectation languages, similar to those used in various platforms.
+
+## Example
+
+### Configuration File
+
+You'll have a configuration `.ttl` file, which acts as a suite of tests. It tells MustRD where to look for test specifications and any triplestore configurations you might have:
+
+```ttl
+:test_example a :MustrdTest;
+              :hasSpecPath "test/specs/";
+              :hasDataPath "test/data/";
+              :hasPytestPath "example";
+              :triplestoreSpecPath "test/triplestore_config/triplestores.ttl";
+              :filterOnTripleStore triplestore:example_test .
+```
+
+### Test Specification
+
+In the directory specified by `:hasSpecPath`, you'll have one or more `.mustrd.ttl` files. These can be organized in a directory structure. MustRD collects them and reports results to your test runner.
+
+```ttl
+:test_example :given [ a :FileDataset ;
+                       :file "test/data/given.ttl" ] ;
+              :when [ a :TextSparqlSource ;
+                     :queryText "SELECT ?s ?p ?o WHERE { ?s ?p ?o }" ;
+                     :queryType :SelectSparql ] ;
+              :then [ a :OrderedTableDataset ;
+                     :hasRow [ :variable "s" ; :boundValue "example:subject" ;
+                               :variable "p" ; :boundValue "example:predicate" ;
+                               :variable "o" ; :boundValue "example:object" ] ].
+```
 
-### When?
+And you will have a `'test/data/given.ttl'` which contains the given ttl. 
+
+```ttl
+example:subject example:predicate example:object .
+```
+
+### Running Tests
+
+Run the test using the MustRD Pytest plugin:
+
+```bash
+poetry run pytest --mustrd --config=test/mustrd_configuration.ttl --md=render/github_job_summary.md
+```
+
+This will validate your SPARQL queries against the defined dataset and expected results, ensuring your transformations behave as intended.
+
+You can refer to SPARQL inline, in files, or in Anzo Graphmarts, Steps, or Layers. See `GETSTARTED.adoc` for more details.
+
+#### Integrating with Visual Studio Code (vscode)
+We have a pytest plugin.
+1. Choose a python interpreter (probably a venv)
+2. `pip install mustrd ` in it.
+3. add to your settings.json
+```json
+    "python.testing.pytestArgs": [
+        "--mustrd", "--md=junit/github_job_summary.md", "--config=test/test_config_local.ttl"
+    ],
+```
+4. VS Code should auto discover your tests and they'll show up in the flask icon 'tab'.
+![alt text](image.png)
+
+## When?
 
 MustRD is a work in progress, built to meet the needs of our projects across multiple clients and vendor stacks. While we find it useful, it may not meet your needs out of the box.
 
@@ -89,3 +156,4 @@ We invite you to try it, raise issues, or contribute via pull requests. If you n
 Semantic Partners is a specialist consultancy in Semantic Technology. If you need more support, contact us at info@semanticpartners.com or mustrd@semanticpartners.com.
 
 
+

mustrd-0.3.1a6/README.md

@@ -0,0 +1,117 @@
+# MustRD
+
+**"MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles."**
+
+[![Coverage Badge](https://github.com/Semantic-partners/mustrd/raw/python-coverage-comment-action-data/badge.svg?sanitize=true)](https://github.com/Semantic-partners/mustrd/tree/python-coverage-comment-action-data)
+
+## Why?
+
+SPARQL is a powerful query language for RDF data, but how can you ensure your queries and transformations are doing what you intend? Whether you're working on a pipeline or a standalone query, certainty is key.
+
+While RDF and SPARQL offer great flexibility, we noticed a gap in tooling to validate their behavior. We missed the robust testing frameworks available in imperative programming languages that help ensure your code works as expected.
+
+With MustRD, you can:
+
+- Define data scenarios and verify that queries produce the expected results.
+- Test edge cases to ensure your queries remain reliable.
+- Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.
+
+## What?
+
+MustRD is a Spec-By-Example ontology with a reference Python implementation, inspired by tools like Cucumber. It uses the Given-When-Then approach to define and validate SPARQL queries and transformations.
+
+MustRD is designed to be triplestore/SPARQL engine agnostic, leveraging open standards to ensure compatibility across different platforms.
+
+### What it is NOT
+
+MustRD is not an alternative to SHACL. While SHACL validates data structures, MustRD focuses on validating data transformations and query results.
+
+## How?
+
+You define your specs in Turtle (`.ttl`) or TriG (`.trig`) files using the Given-When-Then approach:
+
+- **Given**: Define the starting dataset.
+- **When**: Specify the action (e.g., a SPARQL query).
+- **Then**: Outline the expected results.
+
+Depending on the type of SPARQL query (CONSTRUCT, SELECT, INSERT/DELETE), MustRD runs the query and compares the results against the expectations defined in the spec.
+
+Expectations can also be defined as:
+
+- INSERT queries.
+- SELECT queries.
+- Higher-order expectation languages, similar to those used in various platforms.
+
+## Example
+
+### Configuration File
+
+You'll have a configuration `.ttl` file, which acts as a suite of tests. It tells MustRD where to look for test specifications and any triplestore configurations you might have:
+
+```ttl
+:test_example a :MustrdTest;
+              :hasSpecPath "test/specs/";
+              :hasDataPath "test/data/";
+              :hasPytestPath "example";
+              :triplestoreSpecPath "test/triplestore_config/triplestores.ttl";
+              :filterOnTripleStore triplestore:example_test .
+```
+
+### Test Specification
+
+In the directory specified by `:hasSpecPath`, you'll have one or more `.mustrd.ttl` files. These can be organized in a directory structure. MustRD collects them and reports results to your test runner.
+
+```ttl
+:test_example :given [ a :FileDataset ;
+                       :file "test/data/given.ttl" ] ;
+              :when [ a :TextSparqlSource ;
+                     :queryText "SELECT ?s ?p ?o WHERE { ?s ?p ?o }" ;
+                     :queryType :SelectSparql ] ;
+              :then [ a :OrderedTableDataset ;
+                     :hasRow [ :variable "s" ; :boundValue "example:subject" ;
+                               :variable "p" ; :boundValue "example:predicate" ;
+                               :variable "o" ; :boundValue "example:object" ] ].
+```
+
+And you will have a `'test/data/given.ttl'` which contains the given ttl. 
+
+```ttl
+example:subject example:predicate example:object .
+```
+
+### Running Tests
+
+Run the test using the MustRD Pytest plugin:
+
+```bash
+poetry run pytest --mustrd --config=test/mustrd_configuration.ttl --md=render/github_job_summary.md
+```
+
+This will validate your SPARQL queries against the defined dataset and expected results, ensuring your transformations behave as intended.
+
+You can refer to SPARQL inline, in files, or in Anzo Graphmarts, Steps, or Layers. See `GETSTARTED.adoc` for more details.
+
+#### Integrating with Visual Studio Code (vscode)
+We have a pytest plugin.
+1. Choose a python interpreter (probably a venv)
+2. `pip install mustrd ` in it.
+3. add to your settings.json
+```json
+    "python.testing.pytestArgs": [
+        "--mustrd", "--md=junit/github_job_summary.md", "--config=test/test_config_local.ttl"
+    ],
+```
+4. VS Code should auto discover your tests and they'll show up in the flask icon 'tab'.
+![alt text](image.png)
+
+## When?
+
+MustRD is a work in progress, built to meet the needs of our projects across multiple clients and vendor stacks. While we find it useful, it may not meet your needs out of the box.
+
+We invite you to try it, raise issues, or contribute via pull requests. If you need custom features, contact us for consultancy rates, and we may prioritize your request.
+
+## Support
+
+Semantic Partners is a specialist consultancy in Semantic Technology. If you need more support, contact us at info@semanticpartners.com or mustrd@semanticpartners.com.
+
+

{mustrd-0.3.1a5 → mustrd-0.3.1a6}/mustrd/mustrdAnzo.py

@@ -122,8 +122,11 @@ def get_query_from_step(triple_store: dict, query_step_uri: URIRef) -> str:
             ?stepUri a <http://cambridgesemantics.com/ontologies/Graphmarts#Step>;
                      <http://cambridgesemantics.com/ontologies/Graphmarts#transformQuery> ?query
     }}"""
-    return json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))[0]['query']
-
+    result = json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))
+    if len(result) == 0:
+        raise FileNotFoundError(
+            f"Querynot found for step {query_step_uri}")
+    return result[0].get("query")
 
 def get_queries_from_templated_step(triple_store: dict, query_step_uri: URIRef) -> dict:
     query = f"""SELECT ?param_query ?query_template WHERE {{
@@ -133,8 +136,11 @@ def get_queries_from_templated_step(triple_store: dict, query_step_uri: URIRef)
                         <http://cambridgesemantics.com/ontologies/Graphmarts#template> ?query_template .
     }}
     """
-    return json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))[0]
-
+    result = json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))
+    if len(result) == 0:
+        raise FileNotFoundError(
+            f"Templated query not found for {query_step_uri}")
+    return result[0]
 
 def get_queries_for_layer(triple_store: dict, graphmart_layer_uri: URIRef):
     query = f"""PREFIX graphmarts: <http://cambridgesemantics.com/ontologies/Graphmarts#>
@@ -153,8 +159,11 @@ SELECT ?query ?param_query ?query_template
       . }}
   }}
   ORDER BY ?index"""
-    return json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))
-
+    result = json_to_dictlist(query_configuration(anzo_config=triple_store, query=query))
+    if len(result) == 0:
+        raise FileNotFoundError(
+            f"Queries not found for graphmart layer {graphmart_layer_uri}")
+    return result
 
 def upload_given(triple_store: dict, given: Graph):
     logging.debug(f"upload_given {triple_store} {given}")

{mustrd-0.3.1a5 → mustrd-0.3.1a6}/mustrd/spec_component.py

@@ -848,9 +848,7 @@ def _get_spec_component_spade_edn_group_source_when(spec_component_details: Spec
 
         if step_type == Keyword("sparql-file"):
             try:
-                # Resolve the file path relative to the EDN file's location
-                resolved_step_file = Path(absolute_file_path).parent / step_file
-                with open(resolved_step_file, 'r') as sparql_file:
+                with open(step_file, 'r') as sparql_file:
                     sparql_query = sparql_file.read()
 
                 # Assume the individuals are ConstructSparql queries
@@ -862,7 +860,7 @@ def _get_spec_component_spade_edn_group_source_when(spec_component_details: Spec
                 )
                 when_specs.append(when_spec)
             except FileNotFoundError:
-                raise ValueError(f"SPARQL file not found: {resolved_step_file}")
+                raise ValueError(f"SPARQL file not found: {step_file}")
 
     spec_component.file = str(absolute_file_path)
     spec_component.groupId = group_id

{mustrd-0.3.1a5 → mustrd-0.3.1a6}/mustrd/steprunner.py

@@ -185,17 +185,12 @@ def _spade_edn_group_source(spec_uri: URIRef, triple_store: dict, when: SpadeEdn
 def _spade_edn_group_source(spec_uri: URIRef, triple_store: dict, when: SpadeEdnGroupSourceWhenSpec):
     log.info(f"Running SpadeEdnGroupSource for {spec_uri} using {triple_store}")
 
-    edn_file_dir = os.path.dirname(when.file)  # Get the directory of the EDN file
     merged_graph = Graph()
 
     # Iterate over the list of WhenSpec objects in `when.value`
     for step_when_spec in when.value:
         try:
             if step_when_spec.queryType == MUST.UpdateSparql:
-                # Resolve file paths relative to the EDN file
-                if hasattr(step_when_spec, 'filepath'):
-                    step_when_spec.filepath = os.path.join(edn_file_dir, step_when_spec.filepath)
-
                 log.info(f"Dispatching run_when for UpdateSparql step: {step_when_spec}")
                 query_result = run_when_impl(spec_uri, triple_store, step_when_spec)
                 log.info(f"Executed SPARQL query: {query_result}")

mustrd-0.3.1a6/pyproject.toml

@@ -0,0 +1,55 @@
+[tool.poetry]
+name = "mustrd"
+version = "0.3.1a6"
+description = "A Spec By Example framework for RDF and SPARQL, Inspired by Cucumber."
+authors = ["John Placek <john.placek@semanticpartners.com>",
+    "Juliane Piñeiro-Winkler <juliane.pineiro-winkler@semanticpartners.com>",
+    "Aymeric Picou <aymeric.picou@semanticpartners.com>",
+    "Lance Paine <lance.paine@semanticpartners.com>",
+    "Andrew Large <andy.large@semanticpartners.com>", ]
+classifiers = [
+  "Programming Language :: Python",
+  "Framework :: Pytest",
+  "Natural Language :: English",
+  "Topic :: Software Development :: Quality Assurance",
+  "Topic :: Software Development :: Testing",
+  "Topic :: Utilities",
+]
+readme = "README.md"
+license = "MIT License"
+repository = "https://github.com/Semantic-partners/mustrd"
+
+[tool.poetry.dependencies]
+python = "^3.11"
+rdflib = "^7.1.3"
+pytest = "^7.2.0"
+pandas = "^2.0"
+numpy = "~1.26.0"
+multimethods-py = "^0.5.3"
+colorlog = "^6.7.0"
+requests = "^2.28.2"
+tabulate = "^0.9.0"
+beautifulsoup4 = "^4.11.1"
+openpyxl = "^3.1.2"
+toml = "^0.10.2"
+pyshacl="^0.30.0"
+tomli = "^2.0.1"
+urllib3 = "1.26.19"
+flake8 = "7.0.0"
+coverage = "7.4.3"
+colorama = "0.4.6"
+Jinja2 = "3.1.5"
+edn-format = "^0.7.5"
+
+
+[tool.poetry.dev-dependencies]
+
+[tool.poetry.group.dev.dependencies]
+autopep8 = "^2.0.2"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.poetry.plugins.pytest11]
+mustrd_plugin = "mustrd.mustrdTestPlugin"

mustrd-0.3.1a5/README.md

@@ -1,54 +0,0 @@
-# mustrd
-
-**"MustRD: Validate your SPARQL queries and transformations with precision and confidence, using BDD and Given-When-Then principles."**
-
-[<img src="https://github.com/Semantic-partners/mustrd/raw/python-coverage-comment-action-data/badge.svg?sanitize=true" alt="coverage badge">](https://github.com/Semantic-partners/mustrd/tree/python-coverage-comment-action-data)
-
-### Why?
-
-SPARQL is a powerful query language for RDF data, but how can you ensure your queries and transformations are doing what you intend? Whether you're working on a pipeline or a standalone query, certainty is key.
-
-While RDF and SPARQL offer great flexibility, we noticed a gap in tooling to validate their behavior. We missed the robust testing frameworks available in imperative programming languages that help ensure your code works as expected.
-
-With MustRD, you can:
-
-* Define data scenarios and verify that queries produce the expected results.
-* Test edge cases to ensure your queries remain reliable.
-* Isolate small SPARQL enrichment or transformation steps and confirm you're only inserting what you intend.
-
-### What?
-
-MustRD is a Spec-By-Example ontology with a reference Python implementation, inspired by tools like Cucumber. It uses the Given-When-Then approach to define and validate SPARQL queries and transformations.
-
-MustRD is designed to be triplestore/SPARQL engine agnostic, leveraging open standards to ensure compatibility across different platforms.
-
-### What it is NOT
-
-MustRD is not an alternative to SHACL. While SHACL validates data structures, MustRD focuses on validating data transformations and query results.
-
-### How?
-
-You define your specs in Turtle (`.ttl`) or TriG (`.trig`) files using the Given-When-Then approach:
-
-* **Given**: Define the starting dataset.
-* **When**: Specify the action (e.g., a SPARQL query).
-* **Then**: Outline the expected results.
-
-Depending on the type of SPARQL query (CONSTRUCT, SELECT, INSERT/DELETE), MustRD runs the query and compares the results against the expectations defined in the spec.
-
-Expectations can also be defined as:
-
-* INSERT queries.
-* SELECT queries.
-* Higher-order expectation languages, similar to those used in various platforms.
-
-### When?
-
-MustRD is a work in progress, built to meet the needs of our projects across multiple clients and vendor stacks. While we find it useful, it may not meet your needs out of the box.
-
-We invite you to try it, raise issues, or contribute via pull requests. If you need custom features, contact us for consultancy rates, and we may prioritize your request.
-
-## Support
-
-Semantic Partners is a specialist consultancy in Semantic Technology. If you need more support, contact us at info@semanticpartners.com or mustrd@semanticpartners.com.
-

mustrd-0.3.1a5/pyproject.toml

@@ -1,55 +0,0 @@
-[project]
-name = "mustrd"
-version = "0.3.1a5"
-description = "A Spec By Example framework for RDF and SPARQL, Inspired by Cucumber."
-authors = [
-    { name = "John Placek", email = "john.placek@semanticpartners.com" },
-    { name = "Juliane Piñeiro-Winkler", email = "juliane.pineiro-winkler@semanticpartners.com" },
-    { name = "Aymeric Picou", email = "aymeric.picou@semanticpartners.com" },
-    { name = "Lance Paine", email = "lance.paine@semanticpartners.com" },
-    { name = "Andrew Large", email = "andy.large@semanticpartners.com" },
-]
-classifiers = [
-    "Programming Language :: Python",
-    "Framework :: Pytest",
-    "Natural Language :: English",
-    "Topic :: Software Development :: Quality Assurance",
-    "Topic :: Software Development :: Testing",
-    "Topic :: Utilities",
-]
-readme = "README.md"
-license = "MIT License"
-urls = {"repository" = "https://github.com/Semantic-partners/mustrd"}
-entry-points = {"pytest11" = {"mustrd_plugin" = "mustrd.mustrdTestPlugin"}}
-
-[tool.poetry.dependencies]
-python = ">=3.10,<4.0"
-rdflib = "^7.1.3"
-pytest = "^7.2.0"
-pandas = "^2.0"
-numpy = "~1.26.0"
-multimethods-py = "^0.5.3"
-colorlog = "^6.7.0"
-requests = "^2.28.2"
-tabulate = "^0.9.0"
-beautifulsoup4 = "^4.11.1"
-openpyxl = "^3.1.2"
-toml = "^0.10.2"
-pyshacl="^0.30.0"
-tomli = "^2.0.1"
-urllib3 = "1.26.19"
-flake8 = "7.0.0"
-coverage = "7.4.3"
-colorama = "0.4.6"
-Jinja2 = "3.1.5"
-edn-format = "^0.7.5"
-
-
-[tool.poetry.dev-dependencies]
-
-[tool.poetry.group.dev.dependencies]
-autopep8 = "^2.0.2"
-
-[build-system]
-requires = ["poetry-core>=1.0.0"]
-build-backend = "poetry.core.masonry.api"