setlr 1.0.0__tar.gz → 1.0.2__tar.gz

setlr-1.0.2/CHANGELOG.md

@@ -0,0 +1,72 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [1.0.2] - 2026-01-18
+
+### Changed
+- Migrated from `setup.py` to `pyproject.toml` following PEP 517/518 standards for modern Python packaging
+- Restructured codebase: moved implementation from `setlr/__init__.py` to `setlr/core.py` (~1020 lines)
+- `setlr/__init__.py` now serves as a clean public API interface (~90 lines)
+
+### Added
+- New public API function `run_setl()` with comprehensive documentation and type hints
+- Proper deprecation warning for `_setl()` function (still available for backward compatibility)
+- Improved error messages for NaN/missing values (now displays `<empty/missing>` instead of `nan`)
+- Extended JSON error context from 4 to 8 lines before error for better debugging
+- Comprehensive API documentation with usage examples
+- Development scripts for bootstrap, build, and release
+- GitHub Actions workflows for automated testing and linting
+- Migration documentation (MIGRATION.md)
+
+### Fixed
+- Improved error reporting for missing data scenarios
+- Better context display for JSON syntax errors in templates
+- Python version compatibility for JSON error handling
+
+## [1.0.1] - 2024-08-09
+
+### Changed
+- Moved version information from `_version.py` directly into `setup.py`
+- Modified `setup.py` to support `--version` flag
+
+### Fixed
+- Fixed SHACL constraint in ontology example (changed `sh:minCount` from 1 to 0 for `rdfs:subClassOf`)
+
+## [1.0.0] - 2024-04-29
+
+### Added
+- Initial stable release of setlr
+- Core SETL (Semantic Extract, Transform, Load) functionality
+- Support for generating RDF graphs from tabular data
+- CLI tool via `setlr` command
+- Data source readers: CSV, Excel, JSON, XML, and RDF graphs
+- Template-based transformation using Jinja2
+- Named graph support via ConjunctiveGraph
+- RDF namespaces: csvw, ov, setl, prov, pv, sp, sd, dc, void, shacl
+- Utility functions: `extract()`, `transform()`, `load()`, `hash()`, `camelcase()`
+- SHACL validation support with pyshacl[js]
+- Python 3.8+ support
+- Comprehensive test suite
+
+### Dependencies
+- rdflib >= 6.0.0
+- pandas >= 0.23.0
+- jinja2
+- click (CLI support)
+- tqdm (progress bars)
+- pyshacl[js] (validation)
+- beautifulsoup4, lxml (XML/HTML parsing)
+- requests (HTTP support)
+- toposort (dependency ordering)
+- Other utility libraries: numpy, xlrd, ijson, python-slugify
+
+[Unreleased]: https://github.com/tetherless-world/setlr/compare/v1.0.2...HEAD
+[1.0.2]: https://github.com/tetherless-world/setlr/compare/v1.0.1...v1.0.2
+[1.0.1]: https://github.com/tetherless-world/setlr/compare/v1.0.0...v1.0.1
+[1.0.0]: https://github.com/tetherless-world/setlr/releases/tag/v1.0.0

setlr-1.0.2/MANIFEST.in

@@ -0,0 +1,33 @@
+# Include important files
+include README.md
+include LICENSE
+include CHANGELOG.md
+include MIGRATION.md
+include pyproject.toml
+include setup.py
+include setup.cfg
+
+# Include example files
+recursive-include example *.csv *.ttl *.setl.ttl
+
+# Exclude unwanted files and directories
+global-exclude __pycache__
+global-exclude *.py[cod]
+global-exclude *.so
+global-exclude .DS_Store
+global-exclude *.egg-info
+recursive-exclude * __pycache__
+recursive-exclude * *.py[cod]
+
+# Exclude test files
+prune tests
+prune .github
+prune .circleci
+prune script
+prune docs/_build
+
+# Exclude development files
+exclude .gitignore
+exclude .pylintrc
+exclude unittest.cfg
+exclude IMPROVEMENT_SUMMARY.md

setlr-1.0.2/MIGRATION.md

@@ -0,0 +1,166 @@
+# Migration to pyproject.toml and API Improvements
+
+This document describes the changes made to migrate the project to modern Python packaging standards and improve the API.
+
+## Changes Made
+
+### 1. Migration to pyproject.toml
+
+The project has been migrated from `setup.py` to `pyproject.toml`, following PEP 517/518 standards for modern Python packaging.
+
+- **New file**: `pyproject.toml` - Contains all project metadata, dependencies, and build configuration
+- **Status of setup.py**: The old `setup.py` file is still present for compatibility but is no longer the primary packaging configuration
+
+### 2. Code Restructuring
+
+The implementation code has been moved from `setlr/__init__.py` to `setlr/core.py` following best practices:
+
+- **setlr/core.py**: Contains all implementation code (916+ lines)
+- **setlr/__init__.py**: Now serves as a clean public API interface (~90 lines)
+
+This separation provides:
+- Better code organization
+- Clearer public API surface
+- Easier maintenance
+- Improved IDE support and code navigation
+
+### 3. New Public API: `run_setl()`
+
+A new, well-documented public function `run_setl()` has been introduced:
+
+```python
+from rdflib import ConjunctiveGraph
+from setlr import run_setl
+
+# Load a SETL script
+setl_graph = ConjunctiveGraph()
+setl_graph.parse("my_script.setl.ttl", format="turtle")
+
+# Execute the script
+resources = run_setl(setl_graph)
+
+# Access generated resources
+output_graph = resources['http://example.com/output']
+```
+
+**Features:**
+- Comprehensive docstring with examples
+- Proper type hints in documentation
+- Clear description of parameters and return values
+- Usage examples
+
+### 4. Backward Compatibility
+
+The old `_setl()` function is still available for backward compatibility:
+
+```python
+from setlr import _setl  # Still works, but deprecated
+
+# Old code continues to work
+resources = _setl(setl_graph)
+```
+
+**Deprecation Warning:**
+- Using `_setl()` will emit a `DeprecationWarning`
+- The warning suggests using `run_setl()` instead
+- No breaking changes - existing code continues to work
+
+### 5. Exported API
+
+The following are now officially exported from the `setlr` package:
+
+**Main Functions:**
+- `run_setl()` - Primary API function (recommended)
+- `_setl()` - Deprecated, use `run_setl()` instead
+- `main()` - CLI entry point
+
+**Utility Functions:**
+- `read_csv()`, `read_excel()`, `read_json()`, `read_xml()`, `read_graph()`
+- `extract()`, `json_transform()`, `transform()`, `load()`
+- `isempty()`, `hash()`, `camelcase()`, `get_content()`
+
+**Namespaces:**
+- `csvw`, `ov`, `setl`, `prov`, `pv`, `sp`, `sd`, `dc`, `void`, `shacl`, `api_vocab`
+
+## Migration Guide for Users
+
+### If you were using `_setl()`:
+
+**Before:**
+```python
+from setlr import _setl
+
+resources = _setl(setl_graph)
+```
+
+**After (recommended):**
+```python
+from setlr import run_setl
+
+resources = run_setl(setl_graph)
+```
+
+**Note:** Your old code will continue to work, but you'll see a deprecation warning. Update at your convenience.
+
+### If you were importing internal functions:
+
+**Before:**
+```python
+from setlr import read_csv, extract
+```
+
+**After:**
+```python
+from setlr import read_csv, extract  # Still works!
+```
+
+No changes needed - all utility functions are properly exported.
+
+## For Package Maintainers
+
+### Building the Package
+
+With pyproject.toml, you can now build the package using modern tools:
+
+```bash
+# Install build tool
+pip install build
+
+# Build the package
+python -m build
+```
+
+This creates both wheel and source distributions in the `dist/` directory.
+
+### Installing from Source
+
+```bash
+# Development installation
+pip install -e .
+
+# Regular installation
+pip install .
+```
+
+### Running Tests
+
+```bash
+# Install test dependencies
+pip install nose2 coverage
+
+# Run tests
+nose2 --verbose
+```
+
+## Benefits of This Migration
+
+1. **Modern Standards**: Uses PEP 517/518 standards for Python packaging
+2. **Better Documentation**: Clear, comprehensive API documentation
+3. **Improved Structure**: Cleaner separation between public API and implementation
+4. **Backward Compatible**: No breaking changes for existing users
+5. **Future-Proof**: Follows current Python best practices
+6. **Better IDE Support**: Clearer module structure aids code completion and navigation
+
+## Questions or Issues?
+
+If you encounter any issues with the migration or have questions about the new API, please open an issue on GitHub.

setlr-1.0.2/PKG-INFO

@@ -0,0 +1,209 @@
+Metadata-Version: 2.4
+Name: setlr
+Version: 1.0.2
+Summary: setlr is a tool for Semantic Extraction, Transformation, and Loading.
+Author-email: Jamie McCusker <mccusj@cs.rpi.edu>
+Project-URL: Homepage, http://packages.python.org/setlr
+Keywords: rdf,semantic,etl
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Topic :: Utilities
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: future
+Requires-Dist: cython
+Requires-Dist: numpy
+Requires-Dist: rdflib>=6.0.0
+Requires-Dist: pandas>=0.23.0
+Requires-Dist: requests
+Requires-Dist: toposort
+Requires-Dist: beautifulsoup4
+Requires-Dist: jinja2
+Requires-Dist: lxml
+Requires-Dist: six
+Requires-Dist: xlrd
+Requires-Dist: ijson
+Requires-Dist: click
+Requires-Dist: tqdm
+Requires-Dist: requests-testadapter
+Requires-Dist: python-slugify
+Requires-Dist: pyshacl[js]
+Dynamic: license-file
+
+# setlr: Semantic Extract, Transform and Load
+
+[![Unit Tests](https://github.com/tetherless-world/setlr/actions/workflows/test.yml/badge.svg)](https://github.com/tetherless-world/setlr/actions/workflows/test.yml)
+[![Lint](https://github.com/tetherless-world/setlr/actions/workflows/lint.yml/badge.svg)](https://github.com/tetherless-world/setlr/actions/workflows/lint.yml)
+[![codecov](https://codecov.io/gh/tetherless-world/setlr/branch/main/graph/badge.svg)](https://codecov.io/gh/tetherless-world/setlr)
+
+**SETLr** is a powerful Python tool for generating RDF graphs from tabular data using declarative SETL (Semantic Extract, Transform, Load) scripts.
+
+## Features
+
+✨ **Multiple Data Sources**: CSV, Excel, JSON, XML, RDF, SAS files  
+🔄 **Flexible Transformations**: JSON-LD templates with Jinja2, Python functions, SPARQL  
+⚡ **High Performance**: Streaming XML parsing, pandas DataFrames, progress tracking  
+🐍 **Python Integration**: Use as library or CLI tool  
+✅ **Validation**: Built-in SHACL validation  
+📝 **Well Documented**: Comprehensive guides and API reference  
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install setlr
+```
+
+### Simple Example
+
+Create `data.csv`:
+```csv
+ID,Name,Email
+1,Alice,alice@example.com
+2,Bob,bob@example.com
+```
+
+Create `transform.setl.ttl`:
+```turtle
+@prefix setl: <http://purl.org/twc/vocab/setl/> .
+@prefix prov: <http://www.w3.org/ns/prov#> .
+@prefix csvw: <http://www.w3.org/ns/csvw#> .
+@prefix void: <http://rdfs.org/ns/void#> .
+@prefix : <http://example.com/> .
+
+:table a csvw:Table, setl:Table ;
+    prov:wasGeneratedBy [ a setl:Extract ; prov:used <data.csv> ] .
+
+:output a void:Dataset ;
+    prov:wasGeneratedBy [
+        a setl:Transform, setl:JSLDT ;
+        prov:used :table ;
+        prov:value '''[{
+            "@id": "http://example.com/person/{{row.ID}}",
+            "@type": "http://xmlns.com/foaf/0.1/Person",
+            "http://xmlns.com/foaf/0.1/name": "{{row.Name}}",
+            "http://xmlns.com/foaf/0.1/mbox": "mailto:{{row.Email}}"
+        }]'''
+    ] .
+```
+
+Run SETLr:
+```bash
+setlr transform.setl.ttl
+```
+
+### Using from Python
+
+```python
+from rdflib import Graph, URIRef
+import setlr
+
+# Load SETL script
+setl_graph = Graph()
+setl_graph.parse("transform.setl.ttl", format="turtle")
+
+# Execute ETL pipeline
+resources = setlr.run_setl(setl_graph)
+
+# Access generated RDF
+output = resources[URIRef('http://example.com/output')]
+print(f"Generated {len(output)} RDF triples")
+```
+
+## Documentation
+
+📚 **[Complete Documentation](docs/README.md)** - Full guides and references
+
+**Quick Links:**
+- [Tutorial](docs/tutorial.md) - Step-by-step guide to SETLr
+- [JSLDT Template Language](docs/jsldt.md) - Transform syntax reference
+- [Python API](docs/python-api.md) - Using SETLr from Python
+- [Quick Start](docs/quickstart.md) - Get started in 5 minutes
+- [Examples](docs/examples.md) - Real-world examples
+
+**Advanced Topics:**
+- [Streaming XML with XPath](docs/streaming-xml.md) - Efficient large file processing
+- [Python Functions](docs/python-functions.md) - Custom Python transforms
+- [SPARQL Support](docs/sparql.md) - Query and update endpoints
+- [SHACL Validation](docs/shacl.md) - Validate your RDF output
+
+## Key Concepts
+
+SETLr uses RDF (with PROV-O vocabulary) to describe ETL workflows:
+
+1. **Extract**: Load data from sources (CSV, Excel, JSON, XML, RDF, SAS)
+2. **Transform**: Apply templates or Python scripts to generate RDF
+3. **Load**: Save to files or SPARQL endpoints
+
+## Supported Formats
+
+**Input:**
+- Tabular: CSV, TSV, Excel (XLS/XLSX), SAS (XPORT/SAS7BDAT)
+- Structured: JSON (with ijson selectors), XML (with XPath streaming)
+- Semantic: RDF (Turtle, JSON-LD, RDF/XML, etc.), OWL Ontologies
+
+**Output:**
+- RDF: Turtle, TriG, N-Triples, N3, RDF/XML, JSON-LD
+- Destinations: Files, SPARQL Update endpoints
+
+## Examples
+
+See the [examples/](example/) directory for complete working examples:
+
+- `social.setl.ttl` - Basic CSV to RDF with conditionals and loops
+- `ontology.setl.ttl` - OWL ontology transformation with SHACL shapes
+
+## Development
+
+```bash
+# Clone repository
+git clone https://github.com/tetherless-world/setlr.git
+cd setlr
+
+# Bootstrap (creates venv and installs dependencies)
+./script/bootstrap
+
+# Activate virtual environment  
+source venv/bin/activate
+
+# Run tests
+./script/build
+
+# Run linter
+flake8 setlr/
+```
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:
+- Development setup and workflow
+- Code standards and style guidelines
+- Testing requirements
+- Pull request process
+
+Please note that this project follows a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.
+
+## License
+
+Apache License 2.0 - see [LICENSE](LICENSE) file for details.
+
+## Citation
+
+If you use SETLr in your research, please cite:
+
+```bibtex
+@software{setlr,
+  title = {SETLr: Semantic Extract, Transform and Load},
+  author = {McCusker, Jamie},
+  year = {2024},
+  url = {https://github.com/tetherless-world/setlr}
+}
+```
+
+## Support
+
+- 📖 [Documentation](docs/README.md)
+- 🐛 [Issue Tracker](https://github.com/tetherless-world/setlr/issues)
+- 💬 [Discussions](https://github.com/tetherless-world/setlr/discussions)
+- 🔒 [Security Policy](SECURITY.md) - Report security vulnerabilities

setlr-1.0.2/README.md

@@ -0,0 +1,177 @@
+# setlr: Semantic Extract, Transform and Load
+
+[![Unit Tests](https://github.com/tetherless-world/setlr/actions/workflows/test.yml/badge.svg)](https://github.com/tetherless-world/setlr/actions/workflows/test.yml)
+[![Lint](https://github.com/tetherless-world/setlr/actions/workflows/lint.yml/badge.svg)](https://github.com/tetherless-world/setlr/actions/workflows/lint.yml)
+[![codecov](https://codecov.io/gh/tetherless-world/setlr/branch/main/graph/badge.svg)](https://codecov.io/gh/tetherless-world/setlr)
+
+**SETLr** is a powerful Python tool for generating RDF graphs from tabular data using declarative SETL (Semantic Extract, Transform, Load) scripts.
+
+## Features
+
+✨ **Multiple Data Sources**: CSV, Excel, JSON, XML, RDF, SAS files  
+🔄 **Flexible Transformations**: JSON-LD templates with Jinja2, Python functions, SPARQL  
+⚡ **High Performance**: Streaming XML parsing, pandas DataFrames, progress tracking  
+🐍 **Python Integration**: Use as library or CLI tool  
+✅ **Validation**: Built-in SHACL validation  
+📝 **Well Documented**: Comprehensive guides and API reference  
+
+## Quick Start
+
+### Installation
+
+```bash
+pip install setlr
+```
+
+### Simple Example
+
+Create `data.csv`:
+```csv
+ID,Name,Email
+1,Alice,alice@example.com
+2,Bob,bob@example.com
+```
+
+Create `transform.setl.ttl`:
+```turtle
+@prefix setl: <http://purl.org/twc/vocab/setl/> .
+@prefix prov: <http://www.w3.org/ns/prov#> .
+@prefix csvw: <http://www.w3.org/ns/csvw#> .
+@prefix void: <http://rdfs.org/ns/void#> .
+@prefix : <http://example.com/> .
+
+:table a csvw:Table, setl:Table ;
+    prov:wasGeneratedBy [ a setl:Extract ; prov:used <data.csv> ] .
+
+:output a void:Dataset ;
+    prov:wasGeneratedBy [
+        a setl:Transform, setl:JSLDT ;
+        prov:used :table ;
+        prov:value '''[{
+            "@id": "http://example.com/person/{{row.ID}}",
+            "@type": "http://xmlns.com/foaf/0.1/Person",
+            "http://xmlns.com/foaf/0.1/name": "{{row.Name}}",
+            "http://xmlns.com/foaf/0.1/mbox": "mailto:{{row.Email}}"
+        }]'''
+    ] .
+```
+
+Run SETLr:
+```bash
+setlr transform.setl.ttl
+```
+
+### Using from Python
+
+```python
+from rdflib import Graph, URIRef
+import setlr
+
+# Load SETL script
+setl_graph = Graph()
+setl_graph.parse("transform.setl.ttl", format="turtle")
+
+# Execute ETL pipeline
+resources = setlr.run_setl(setl_graph)
+
+# Access generated RDF
+output = resources[URIRef('http://example.com/output')]
+print(f"Generated {len(output)} RDF triples")
+```
+
+## Documentation
+
+📚 **[Complete Documentation](docs/README.md)** - Full guides and references
+
+**Quick Links:**
+- [Tutorial](docs/tutorial.md) - Step-by-step guide to SETLr
+- [JSLDT Template Language](docs/jsldt.md) - Transform syntax reference
+- [Python API](docs/python-api.md) - Using SETLr from Python
+- [Quick Start](docs/quickstart.md) - Get started in 5 minutes
+- [Examples](docs/examples.md) - Real-world examples
+
+**Advanced Topics:**
+- [Streaming XML with XPath](docs/streaming-xml.md) - Efficient large file processing
+- [Python Functions](docs/python-functions.md) - Custom Python transforms
+- [SPARQL Support](docs/sparql.md) - Query and update endpoints
+- [SHACL Validation](docs/shacl.md) - Validate your RDF output
+
+## Key Concepts
+
+SETLr uses RDF (with PROV-O vocabulary) to describe ETL workflows:
+
+1. **Extract**: Load data from sources (CSV, Excel, JSON, XML, RDF, SAS)
+2. **Transform**: Apply templates or Python scripts to generate RDF
+3. **Load**: Save to files or SPARQL endpoints
+
+## Supported Formats
+
+**Input:**
+- Tabular: CSV, TSV, Excel (XLS/XLSX), SAS (XPORT/SAS7BDAT)
+- Structured: JSON (with ijson selectors), XML (with XPath streaming)
+- Semantic: RDF (Turtle, JSON-LD, RDF/XML, etc.), OWL Ontologies
+
+**Output:**
+- RDF: Turtle, TriG, N-Triples, N3, RDF/XML, JSON-LD
+- Destinations: Files, SPARQL Update endpoints
+
+## Examples
+
+See the [examples/](example/) directory for complete working examples:
+
+- `social.setl.ttl` - Basic CSV to RDF with conditionals and loops
+- `ontology.setl.ttl` - OWL ontology transformation with SHACL shapes
+
+## Development
+
+```bash
+# Clone repository
+git clone https://github.com/tetherless-world/setlr.git
+cd setlr
+
+# Bootstrap (creates venv and installs dependencies)
+./script/bootstrap
+
+# Activate virtual environment  
+source venv/bin/activate
+
+# Run tests
+./script/build
+
+# Run linter
+flake8 setlr/
+```
+
+## Contributing
+
+Contributions are welcome! Please see our [Contributing Guide](CONTRIBUTING.md) for details on:
+- Development setup and workflow
+- Code standards and style guidelines
+- Testing requirements
+- Pull request process
+
+Please note that this project follows a [Code of Conduct](CODE_OF_CONDUCT.md). By participating, you are expected to uphold this code.
+
+## License
+
+Apache License 2.0 - see [LICENSE](LICENSE) file for details.
+
+## Citation
+
+If you use SETLr in your research, please cite:
+
+```bibtex
+@software{setlr,
+  title = {SETLr: Semantic Extract, Transform and Load},
+  author = {McCusker, Jamie},
+  year = {2024},
+  url = {https://github.com/tetherless-world/setlr}
+}
+```
+
+## Support
+
+- 📖 [Documentation](docs/README.md)
+- 🐛 [Issue Tracker](https://github.com/tetherless-world/setlr/issues)
+- 💬 [Discussions](https://github.com/tetherless-world/setlr/discussions)
+- 🔒 [Security Policy](SECURITY.md) - Report security vulnerabilities