docworkspace 0.1.0__tar.gz

docworkspace-0.1.0/.gitignore

@@ -0,0 +1,57 @@
+# Python-generated files
+__pycache__/
+*.py[oc]
+build/
+dist/
+wheels/
+*.egg-info
+
+# Virtual environments
+.venv
+venv/
+ENV/
+env/
+.env
+
+# uv
+.python-version
+uv.lock
+
+# Pytest
+.pytest_cache/
+.coverage
+htmlcov/
+
+# IDEs and editors
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Operating System
+.DS_Store
+.DS_Store?
+._*
+
+# Temporary files
+*.tmp
+*.temp
+*.bak
+*.backup
+
+# MyPy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Testing
+.tox/
+.nox/
+.hypothesis/
+
+# Machine learning models
+*.pkl
+*.pickle
+*.joblib
+*.model

docworkspace-0.1.0/PKG-INFO

@@ -0,0 +1,584 @@
+Metadata-Version: 2.4
+Name: docworkspace
+Version: 0.1.0
+Summary: A workspace library for managing DocDataFrames and DataFrames with parent-child relationships and lazy evaluation
+Requires-Python: >=3.12
+Requires-Dist: docframe
+Requires-Dist: typing-extensions
+Provides-Extra: cpu
+Requires-Dist: docframe[cpu]; extra == 'cpu'
+Description-Content-Type: text/markdown
+
+# DocWorkspace
+
+A powerful Python library for managing DataFrames and DocDataFrames with parent-child relationships, lazy evaluation, and FastAPI integration. Part of the LDaCA (Language Data Commons of Australia) ecosystem.
+
+## Overview
+
+DocWorkspace provides a workspace-based approach to data analysis, where data transformations are tracked as nodes in a directed graph. This enables:
+
+- **Relationship Tracking**: Understand data lineage and transformation history
+- **Lazy Evaluation**: Optimize performance with Polars LazyFrames
+- **Multiple Data Types**: Support for Polars DataFrames, LazyFrames, DocDataFrames, and DocLazyFrames
+- **FastAPI Integration**: Ready-to-use models and utilities for web APIs
+- **Serialization**: Save and restore entire workspaces with their relationships
+
+## Installation
+
+```bash
+pip install docworkspace
+```
+
+### Dependencies
+
+- Python ≥ 3.12
+- polars ≥ 0.20.0
+- docframe
+- pandas ≥ 2.0.0
+- typing-extensions
+
+For FastAPI integration:
+```bash
+pip install pydantic
+```
+
+## Quick Start
+
+```python
+import polars as pl
+from docworkspace import Node, Workspace
+from docframe import DocDataFrame
+
+# Create a workspace
+workspace = Workspace("my_analysis")
+
+# Load data
+df = pl.DataFrame({
+    "text": ["Hello world", "Data science", "Python rocks"],
+    "category": ["greeting", "tech", "programming"],
+    "score": [0.8, 0.9, 0.95]
+})
+
+# Add data to workspace
+data_node = workspace.add_node(Node(df, name="raw_data"))
+
+# Apply transformations (creates new nodes automatically)
+filtered = data_node.filter(pl.col("score") > 0.85)
+grouped = filtered.group_by("category").agg(pl.col("score").mean())
+
+# Check relationships
+print(f"Total nodes: {len(workspace.nodes)}")
+print(f"Root nodes: {len(workspace.get_root_nodes())}")
+print(f"Leaf nodes: {len(workspace.get_leaf_nodes())}")
+
+# Visualize the computation graph
+print(workspace.visualize_graph())
+```
+
+## Core Concepts
+
+### Node
+
+A `Node` wraps your data (DataFrames, LazyFrames, DocDataFrames) and tracks relationships with other nodes. Nodes support:
+
+- **Transparent Data Access**: All DataFrame methods work directly on nodes
+- **Automatic Relationship Tracking**: Operations create child nodes
+- **Lazy Evaluation**: Maintains laziness for performance
+- **Metadata**: Store operation descriptions and custom metadata
+
+```python
+# Node automatically creates workspace if none provided
+node = Node(df, name="my_data")
+
+# All DataFrame operations work directly
+filtered_node = node.filter(pl.col("value") > 10)
+sorted_node = filtered_node.sort("value", descending=True)
+
+# Check relationships
+print(f"Children of original node: {len(node.children)}")
+print(f"Parents of sorted node: {len(sorted_node.parents)}")
+```
+
+### Workspace
+
+A `Workspace` manages collections of nodes and provides graph operations:
+
+- **Node Management**: Add, remove, and retrieve nodes
+- **Graph Operations**: Find roots, leaves, descendants, ancestors
+- **Serialization**: Save/load entire workspaces
+- **Visualization**: Generate text-based and programmatic graph representations
+
+```python
+workspace = Workspace("analysis")
+
+# Add nodes
+node1 = workspace.add_node(Node(df1, "dataset1"))
+node2 = workspace.add_node(Node(df2, "dataset2"))
+
+# Join creates a new node with both parents
+joined = node1.join(node2, on="id")
+
+# Explore the graph
+roots = workspace.get_root_nodes()
+leaves = workspace.get_leaf_nodes()
+```
+
+## Supported Data Types
+
+DocWorkspace supports multiple data types from the Polars and DocFrame ecosystems:
+
+### Polars Types
+- **`pl.DataFrame`**: Materialized, in-memory data
+- **`pl.LazyFrame`**: Lazy evaluation for performance optimization
+
+### DocFrame Types  
+- **`DocDataFrame`**: Enhanced DataFrame for text analysis with document tracking
+- **`DocLazyFrame`**: Lazy version of DocDataFrame
+
+### Example with Different Types
+
+```python
+import polars as pl
+from docframe import DocDataFrame, DocLazyFrame
+
+# Polars DataFrame (eager)
+df = pl.DataFrame({"text": ["hello", "world"], "id": [1, 2]})
+node1 = Node(df, "eager_data")
+
+# Polars LazyFrame (lazy)
+lazy_df = pl.LazyFrame({"text": ["foo", "bar"], "id": [3, 4]})
+node2 = Node(lazy_df, "lazy_data")
+
+# DocDataFrame (eager, with document column)
+doc_df = DocDataFrame(df, document_column="text")
+node3 = Node(doc_df, "doc_data")
+
+# DocLazyFrame (lazy, with document column)
+doc_lazy = DocLazyFrame(lazy_df, document_column="text") 
+node4 = Node(doc_lazy, "doc_lazy_data")
+
+# All work seamlessly in the same workspace
+workspace = Workspace("mixed_types")
+for node in [node1, node2, node3, node4]:
+    workspace.add_node(node)
+```
+
+## Key Features
+
+### 1. Lazy Evaluation
+
+DocWorkspace preserves Polars' lazy evaluation capabilities:
+
+```python
+# Start with lazy data
+lazy_df = pl.scan_csv("large_file.csv")
+node = Node(lazy_df, "raw_data")
+
+# Chain operations (all remain lazy)
+filtered = node.filter(pl.col("value") > 100)
+grouped = filtered.group_by("category").agg(pl.col("value").sum())
+sorted_result = grouped.sort("value", descending=True)
+
+# Only materialize when needed
+final_result = sorted_result.collect()  # This creates a new materialized node
+```
+
+### 2. Relationship Tracking
+
+Understand your data lineage:
+
+```python
+# Create a processing pipeline
+raw_data = Node(df, "raw")
+cleaned = raw_data.filter(pl.col("value").is_not_null())
+normalized = cleaned.with_columns(pl.col("value") / pl.col("value").max())
+final = normalized.select(["id", "normalized_value"])
+
+# Explore relationships
+print("Processing chain:")
+current = final
+while current.parents:
+    parent = current.parents[0]
+    print(f"{parent.name} -> {current.name} ({current.operation})")
+    current = parent
+```
+
+### 3. FastAPI Integration
+
+Ready-to-use models for web APIs:
+
+```python
+from docworkspace import FastAPIUtils, WorkspaceGraph, NodeSummary
+
+# Convert workspace to FastAPI-compatible format
+graph_data = workspace.to_api_graph()
+
+# Get node summaries
+summaries = [FastAPIUtils.node_to_summary(node) for node in workspace.nodes.values()]
+
+# Get paginated data
+paginated = FastAPIUtils.get_paginated_data(node, page=1, page_size=100)
+```
+
+### 4. Serialization
+
+Save and restore complete workspaces:
+
+```python
+# Save workspace with all nodes and relationships
+workspace.serialize("my_workspace.json")
+
+# Load workspace later
+restored_workspace = Workspace.deserialize("my_workspace.json")
+
+# All nodes and relationships are preserved
+assert len(restored_workspace.nodes) == len(workspace.nodes)
+```
+
+## Advanced Usage
+
+### Custom Operations
+
+Create custom operations that maintain relationships:
+
+```python
+def custom_transform(node: Node, operation_name: str) -> Node:
+    """Apply custom transformation and track the operation."""
+    # Your custom logic here
+    result_data = node.data.with_columns(pl.col("value") * 2)
+    
+    # Create new node with relationship tracking
+    return Node(
+        data=result_data,
+        name=f"{operation_name}_{node.name}",
+        workspace=node.workspace,
+        parents=[node],
+        operation=operation_name
+    )
+
+# Use custom operation
+transformed = custom_transform(original_node, "double_values")
+```
+
+### Graph Analysis
+
+Analyze your computation graph:
+
+```python
+# Find all descendants of a node
+descendants = workspace.get_descendants(node.id)
+
+# Find all ancestors
+ancestors = workspace.get_ancestors(node.id)
+
+# Get topological ordering
+ordered_nodes = workspace.get_topological_order()
+
+# Check for cycles (shouldn't happen in normal usage)
+has_cycles = workspace.has_cycles()
+```
+
+### Working with DocDataFrames
+
+Enhanced text analysis capabilities:
+
+```python
+from docframe import DocDataFrame
+
+# Create DocDataFrame with document column
+df = pl.DataFrame({
+    "doc_id": ["d1", "d2", "d3"],
+    "text": ["Hello world", "Data science", "Python rocks"],
+    "metadata": ["type1", "type2", "type1"]
+})
+
+doc_df = DocDataFrame(df, document_column="text")
+node = Node(doc_df, "corpus")
+
+# DocDataFrame operations work seamlessly
+filtered = node.filter(pl.col("metadata") == "type1")
+print(f"Document column preserved: {filtered.data.document_column}")
+```
+
+## API Reference
+
+### Node Class
+
+#### Constructor
+```python
+Node(data, name=None, workspace=None, parents=None, operation=None)
+```
+
+#### Properties
+- `is_lazy: bool` - Whether the underlying data is lazy
+- `document_column: Optional[str]` - Document column for DocDataFrames
+
+#### Methods
+- `collect() -> Node` - Materialize lazy data (creates new node)
+- `materialize() -> Node` - Alias for collect()
+- `info(json=False) -> Dict` - Get node information
+- `json_schema() -> Dict[str, str]` - Get JSON-compatible schema
+
+#### DataFrame Operations
+All Polars DataFrame/LazyFrame operations are available directly:
+- `filter(condition) -> Node`
+- `select(columns) -> Node`
+- `with_columns(*exprs) -> Node`
+- `group_by(*columns) -> Node`
+- `sort(by, descending=False) -> Node`
+- `join(other, on, how="inner") -> Node`
+- And many more...
+
+### Workspace Class
+
+#### Constructor
+```python
+Workspace(name=None, data=None, data_name=None, csv_lazy=True, **csv_kwargs)
+```
+
+#### Properties
+- `id: str` - Unique workspace identifier
+- `name: str` - Human-readable name
+- `nodes: Dict[str, Node]` - All nodes in the workspace
+
+#### Methods
+
+##### Node Management
+- `add_node(node) -> Node` - Add a node to the workspace
+- `remove_node(node_id, materialize_children=False) -> bool` - Remove a node
+- `get_node(node_id) -> Optional[Node]` - Get node by ID
+- `get_node_by_name(name) -> Optional[Node]` - Get node by name
+- `list_nodes() -> List[Node]` - Get all nodes
+
+##### Graph Operations
+- `get_root_nodes() -> List[Node]` - Nodes with no parents
+- `get_leaf_nodes() -> List[Node]` - Nodes with no children
+- `get_descendants(node_id) -> List[Node]` - All descendant nodes
+- `get_ancestors(node_id) -> List[Node]` - All ancestor nodes
+- `get_topological_order() -> List[Node]` - Topologically sorted nodes
+
+##### Visualization
+- `visualize_graph() -> str` - Text-based graph visualization
+- `graph() -> Dict` - Generic graph structure
+- `to_react_flow_json() -> Dict` - React Flow compatible format
+
+##### Serialization
+- `serialize(file_path)` - Save workspace to JSON
+- `deserialize(file_path) -> Workspace` - Load workspace from JSON
+- `from_dict(workspace_dict) -> Workspace` - Create from dictionary
+
+##### Metadata
+- `get_metadata(key) -> Any` - Get workspace metadata
+- `set_metadata(key, value)` - Set workspace metadata
+- `summary() -> Dict` - Get workspace summary
+- `info() -> Dict` - Alias for summary()
+
+### FastAPI Integration
+
+#### Models
+- `NodeSummary` - API-friendly node representation
+- `WorkspaceGraph` - React Flow compatible graph
+- `PaginatedData` - Paginated data response
+- `ErrorResponse` - Standard error format
+- `OperationResult` - Operation result wrapper
+
+#### Utilities
+```python
+FastAPIUtils.node_to_summary(node) -> NodeSummary
+FastAPIUtils.get_paginated_data(node, page=1, page_size=100) -> PaginatedData
+FastAPIUtils.workspace_to_react_flow(workspace) -> WorkspaceGraph
+```
+
+## Examples
+
+### Example 1: Text Analysis Pipeline
+
+```python
+import polars as pl
+from docworkspace import Node, Workspace
+from docframe import DocDataFrame
+
+# Sample text data
+df = pl.DataFrame({
+    "doc_id": [f"doc_{i}" for i in range(100)],
+    "text": [f"Sample text content {i}" for i in range(100)],
+    "category": ["news", "blog", "academic"] * 34,
+    "year": [2020, 2021, 2022, 2023] * 25
+})
+
+# Create workspace
+workspace = Workspace("text_analysis")
+
+# Load as DocDataFrame for text analysis
+doc_df = DocDataFrame(df, document_column="text")
+corpus = workspace.add_node(Node(doc_df, "full_corpus"))
+
+# Filter by category
+news_docs = corpus.filter(pl.col("category") == "news")
+blog_docs = corpus.filter(pl.col("category") == "blog")
+
+# Filter by recent years
+recent_news = news_docs.filter(pl.col("year") >= 2022)
+
+# Group analysis
+year_stats = corpus.group_by(["category", "year"]).agg(
+    pl.count().alias("doc_count")
+)
+
+# Materialize results
+final_stats = year_stats.collect()
+
+# Analyze the computation graph
+print(workspace.visualize_graph())
+print(f"Total transformations: {len(workspace.nodes)}")
+```
+
+### Example 2: Lazy Data Processing
+
+```python
+import polars as pl
+from docworkspace import Workspace
+
+# Create workspace with lazy CSV loading
+workspace = Workspace(
+    "large_data_analysis",
+    data="large_dataset.csv",  # Path to CSV
+    data_name="raw_data",
+    csv_lazy=True  # Load as LazyFrame for performance
+)
+
+# Get the loaded node
+raw_data = workspace.get_node_by_name("raw_data")
+print(f"Is lazy: {raw_data.is_lazy}")  # True
+
+# Chain transformations (all remain lazy)
+cleaned = raw_data.filter(pl.col("value").is_not_null())
+normalized = cleaned.with_columns(
+    (pl.col("value") / pl.col("value").max()).alias("normalized")
+)
+aggregated = normalized.group_by("category").agg([
+    pl.col("normalized").mean().alias("avg_normalized"),
+    pl.count().alias("count")
+])
+
+# Still lazy until we collect
+print(f"Aggregated is lazy: {aggregated.is_lazy}")  # True
+
+# Materialize only the final result
+result = aggregated.collect()
+print(f"Result is lazy: {result.is_lazy}")  # False
+
+# Save the entire workspace with lazy evaluation preserved
+workspace.serialize("lazy_analysis.json")
+```
+
+### Example 3: Multi-Source Data Integration
+
+```python
+import polars as pl
+from docworkspace import Node, Workspace
+
+workspace = Workspace("data_integration")
+
+# Load data from multiple sources
+sales_df = pl.DataFrame({
+    "customer_id": [1, 2, 3, 4],
+    "sales": [100, 200, 150, 300],
+    "region": ["North", "South", "East", "West"]
+})
+
+customer_df = pl.DataFrame({
+    "customer_id": [1, 2, 3, 4, 5],
+    "name": ["Alice", "Bob", "Charlie", "Diana", "Eve"],
+    "segment": ["Premium", "Regular", "Premium", "Regular", "Premium"]
+})
+
+# Add to workspace
+sales_node = workspace.add_node(Node(sales_df, "sales_data"))
+customer_node = workspace.add_node(Node(customer_df, "customer_data"))
+
+# Join the datasets
+combined = sales_node.join(customer_node, on="customer_id", how="inner")
+
+# Analyze by segment
+segment_analysis = combined.group_by("segment").agg([
+    pl.col("sales").sum().alias("total_sales"),
+    pl.col("sales").mean().alias("avg_sales"),
+    pl.count().alias("customer_count")
+])
+
+# Filter high-value segments
+high_value = segment_analysis.filter(pl.col("total_sales") > 200)
+
+print(f"Nodes in workspace: {len(workspace.nodes)}")
+print("Data lineage:")
+for node in workspace.get_leaf_nodes():
+    print(f"Leaf node: {node.name}")
+```
+
+## Development
+
+### Running Tests
+
+```bash
+# Install development dependencies
+pip install pytest
+
+# Run all tests
+pytest
+
+# Run with coverage
+pytest --cov=docworkspace
+
+# Run specific test file
+pytest tests/test_workspace.py -v
+```
+
+### Contributing
+
+1. Fork the repository
+2. Create a feature branch: `git checkout -b feature-name`
+3. Make your changes and add tests
+4. Run the test suite: `pytest`
+5. Submit a pull request
+
+### Project Structure
+
+```
+docworkspace/
+├── docworkspace/           # Main package
+│   ├── __init__.py        # Package exports
+│   ├── node.py            # Node class implementation
+│   ├── workspace.py       # Workspace class implementation
+│   ├── api_models.py      # FastAPI Pydantic models
+│   └── api_utils.py       # FastAPI utility functions
+├── tests/                 # Test suite
+│   ├── test_node.py       # Node class tests
+│   ├── test_workspace.py  # Workspace class tests
+│   ├── test_integration.py # Integration tests
+│   └── test_coverage.py   # Coverage tests
+├── examples/              # Example scripts and data
+├── README.md             # This file
+└── pyproject.toml        # Project configuration
+```
+
+## License
+
+Part of the LDaCA (Language Data Commons of Australia) ecosystem.
+
+## Changelog
+
+### Version 0.1.0
+- Initial release
+- Core Node and Workspace functionality
+- Support for Polars and DocFrame data types
+- Lazy evaluation support
+- FastAPI integration
+- Serialization capabilities
+- Comprehensive test suite
+
+## Related Projects
+
+- **[DocFrame](https://github.com/ldaca/docframe)**: Enhanced DataFrames for text analysis
+- **[LDaCA Web App](https://github.com/ldaca/ldaca_web_app)**: Full-stack web application using DocWorkspace
+- **[Polars](https://pola.rs/)**: Fast DataFrame library with lazy evaluation