graphforge 0.3.2__tar.gz → 0.3.4__tar.gz

{graphforge-0.3.2 → graphforge-0.3.4}/CHANGELOG.md

@@ -5,7 +5,134 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
+## [0.3.4] - 2026-02-18
+
+### Added - Power Arithmetic Operator (#213)
+- **^ (power/exponentiation) operator** - Full exponentiation support
+  - Right-associative: `2^3^2 = 2^(3^2) = 512`
+  - Highest arithmetic precedence (above `*`, `/`)
+  - Type handling: `int^int` returns int if whole, else float
+  - Negative exponents: `2^-1 = 0.5`
+  - Fractional exponents: `4^0.5 = 2.0`
+  - NULL propagation for both operands
+  - 39 integration tests covering associativity, precedence, types, and edge cases
+
+### Added - XOR Logical Operator (#212)
+- **XOR operator** - Exclusive OR with proper ternary logic
+  - `true XOR false = true`, `true XOR true = false`
+  - Precedence: NOT > AND > XOR > OR
+  - NULL propagation: any NULL operand yields NULL
+  - Left-associative chaining: `a XOR b XOR c`
+  - Case-insensitive keyword (`XOR`, `xor`, `Xor`)
+  - 22 integration tests + 4 unit tests
+
+### Documentation - Feature Completion Updates
+
+#### Already-Implemented Features Now Documented (#193, #214, #215)
+- **length() function** - Documented as COMPLETE for paths
+  - `length(path)` returns relationship count in path
+  - For strings/lists, use `size()` function instead
+  - File: `src/graphforge/executor/evaluator.py:2604`
+- **List slicing [start..end]** - Documented as COMPLETE
+  - Example: `RETURN [1, 2, 3, 4, 5][1..3]` → `[2, 3]`
+  - Supports open-ended slicing: `[..]`, `[1..]`, `[..3]`
+  - File: `src/graphforge/executor/evaluator.py:1458`
+- **Negative list indexing** - Documented as COMPLETE
+  - Example: `RETURN [1, 2, 3][-1]` → `3` (last element)
+  - Supports both index and slice operations: `[1, 2, 3][-2..-1]`
+  - File: `src/graphforge/executor/evaluator.py:1458`
+
+### Added - String Function Aliases
+
+#### toUpper()/toLower() camelCase variants (#194)
+- **toUpper(string)** - CamelCase alias for UPPER(), converts string to uppercase
+  - Example: `RETURN toUpper('hello') AS result` -> `'HELLO'`
+- **toLower(string)** - CamelCase alias for LOWER(), converts string to lowercase
+  - Example: `RETURN toLower('HELLO') AS result` -> `'hello'`
+- Both camelCase and legacy forms (UPPER/LOWER) supported
+- Case-insensitive keywords: toUpper, TOUPPER, toLower, TOLOWER all work
+- 18 new integration tests (7 toUpper + 7 toLower + 4 interop)
+
+### Implementation Status Updates
+- Functions: 58/72 (81%, +3%) - String functions now 13/13 (100%)
+- Operators: 32/34 (94%, +6%) - All list operators now COMPLETE
+- Documentation reflects actual implementation status
+
+## [0.3.3] - 2026-02-18
+
+### Added - Pattern & CALL Features (Feature Completion: 88%)
+
+#### Pattern Predicates - COMPLETE (#216)
+- **Inline WHERE in patterns** - Filter relationships during pattern matching
+  - Example: `MATCH ()-[r:KNOWS WHERE r.since > 2020]->(f) RETURN f.name`
+  - Property comparisons, complex expressions (AND, OR, NOT)
+  - Function calls in predicates, NULL handling
+  - Variable-length paths with predicates
+  - Works with undirected relationships
+  - 16 comprehensive integration tests
+
+#### CALL Subqueries - COMPLETE (#189)
+- **General CALL { } subquery support** - Execute nested queries with full openCypher syntax
+  - Example: `CALL { MATCH (p:Person) RETURN p.name AS name } RETURN name`
+  - **Correlated scoping:** Access outer variables from parent query
+  - **UNION support:** CALL can contain UNION and UNION ALL queries
+  - **Unit subqueries:** Preserve 1:1 cardinality for side-effect queries
+  - **Nested CALL:** Support for CALL within CALL
+  - **Multiple CALL:** Cartesian product of multiple subqueries
+  - 13 comprehensive integration tests
+
+#### Pattern Comprehension - COMPLETE (#217)
+- **Pattern-based list operations** - Transform graph patterns into lists
+  - Example: `[(p:Person) WHERE p.age > 18 | p.name]`
+  - Simple node patterns: `[(p:Person) | p.name]`
+  - Relationship patterns: `[(p)-[:KNOWS]->(f) | f.name]`
+  - Optional WHERE filters for pattern results
+  - Complex expressions in map clause
+  - Correlated variables from outer scope
+  - Nested in RETURN, WHERE, WITH clauses
+  - NULL property handling
+  - 15 comprehensive integration tests
+
+### Implementation
+
+**AST Nodes:**
+- PatternComprehension (expression.py) - pattern, filter_expr, map_expr
+- CallClause (clause.py) - nested query support
+
+**Grammar (cypher.lark):**
+- pattern_comprehension rule with optional WHERE
+- call_clause and call_query rules
+- Pattern WHERE predicates already existed, now fully documented
+
+**Parser (parser.py):**
+- pattern_comprehension transformer
+- call_clause and call_query transformers
+- Enhanced list_literal to handle pattern comprehension
+
+**Planner (planner.py):**
+- Call operator with UNION support
+- TypeContext preservation for nested queries
+
+**Executor (executor.py, evaluator.py):**
+- _execute_call with correlated scoping and unit subquery detection
+- PatternComprehension evaluation via temporary MATCH execution
+- Pattern predicate evaluation in relationship matching (already existed)
+
+### Testing
+- 44 new integration tests (16 + 13 + 15)
+- All tests passing with 100% coverage on new code
+- TCK pass rate: ~40% (1,530 / 3,837 tests)
+
+### Documentation
+- Updated patterns.md: 100% completion (8/8 pattern types)
+- Updated clauses.md: CALL marked as COMPLETE
+- All features documented with examples and implementation references
+- Feature completion: 85% → 88% (+3 features)
+
+### Performance
+- No performance regressions
+- Pattern comprehension uses existing pattern matching infrastructure
+- CALL subqueries properly isolated with TypeContext.copy()
 
 ## [0.3.2] - 2026-02-17
 

graphforge-0.3.4/CLAUDE.md

@@ -0,0 +1,430 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+GraphForge is an embedded, openCypher-compatible graph database for Python, designed for research and analysis workflows. It prioritizes **correctness over performance** and provides declarative querying with automatic persistence via SQLite.
+
+**Not a production database** - optimized for interactive analysis on small-to-medium graphs (< 10M nodes).
+
+**Core Use Cases:**
+- **AI Agent Grounding**: Ground LLM agents in ontologies with tool definitions for semantic, deterministic action (see `docs/use-cases/agent-grounding.md`)
+- **Knowledge Graph Construction**: Extract and refine entities/relationships from unstructured data
+- **Network Analysis**: Analyze social networks, dependencies, citation graphs in notebooks
+- **LLM-Powered Workflows**: Store and query structured outputs from language models
+
+## Development Workflow (CRITICAL)
+
+**Every piece of work MUST follow this workflow. No exceptions.**
+
+### 1. Create an Issue First
+
+```bash
+gh issue create --title "feat: add UNWIND clause support" \
+  --body "Description..." --label "enhancement"
+```
+
+Never start work without an issue. If you discover out-of-scope work, create a NEW issue and continue with current work.
+
+### 2. Create Branch with Issue Number
+
+**Format:** `<type>/<issue-number>-<short-description>`
+
+```bash
+git checkout -b feature/42-unwind-clause
+
+# Branch types: feature/, fix/, docs/, refactor/, test/
+```
+
+### 3. Do the Work with Comprehensive Tests
+
+- Unit tests for each layer (parser, planner, executor)
+- Integration tests for end-to-end behavior
+- Aim for 100% coverage on new code (90% minimum)
+
+### 4. Run Pre-Push Checks
+
+```bash
+make pre-push
+```
+
+Runs: format-check, lint, type-check, coverage (85% total, 90% patch).
+
+### 5. Commit with Issue Number
+
+```bash
+git commit -m "feat: implement UNWIND clause (#42)
+
+- Add UNWIND grammar rule to cypher.lark
+- Implement UnwindClause AST node
+- Add planner support for Unwind operator
+- Implement executor logic for list iteration
+- Add 15 unit tests + 8 integration tests
+
+All tests passing, coverage at 100% for new code."
+```
+
+### 6. Create PR with "Closes #XX"
+
+```bash
+gh pr create --title "feat: implement UNWIND clause" \
+  --body "Closes #42"
+```
+
+GitHub automatically closes the issue when PR merges.
+
+## Development Commands
+
+```bash
+make pre-push          # Run all checks (mirrors CI)
+make test              # All tests
+make test-unit         # Unit tests only
+make test-integration  # Integration tests only
+make coverage          # Run with coverage
+make coverage-report   # View HTML coverage report
+make coverage-diff     # Coverage for changed files only
+
+# Run tests in parallel (4x faster for TCK)
+pytest tests/ -n auto
+```
+
+**Coverage Requirements:**
+- Total: 85% minimum
+- Patch: 90% minimum for changed files
+- Best practice: Aim for 100% on new code
+
+## Architecture: 4-Layer System
+
+```
+User Code (Python API / Cypher)
+          ↓
+┌─────────────────────────────────────────────────┐
+│ 1. Parser (src/graphforge/parser/)             │
+│    - cypher.lark: Grammar                       │
+│    - parser.py: Transformer (Lark → AST)        │
+└─────────────────────────────────────────────────┘
+          ↓
+┌─────────────────────────────────────────────────┐
+│ 2. Planner (src/graphforge/planner/)           │
+│    - planner.py: AST → Logical operators        │
+│    - operators.py: Operator definitions         │
+└─────────────────────────────────────────────────┘
+          ↓
+┌─────────────────────────────────────────────────┐
+│ 3. Executor (src/graphforge/executor/)         │
+│    - executor.py: Operator execution            │
+│    - evaluator.py: Expression evaluation        │
+└─────────────────────────────────────────────────┘
+          ↓
+┌─────────────────────────────────────────────────┐
+│ 4. Storage (src/graphforge/storage/)           │
+│    - memory.py: In-memory graph                 │
+│    - sqlite_backend.py: SQLite persistence      │
+│    - serialization.py: MessagePack encoding     │
+└─────────────────────────────────────────────────┘
+```
+
+### Key Principles
+
+1. **Each layer is independent** - Parser knows nothing about execution
+2. **AST is the contract** - Parser produces AST, planner consumes it
+3. **Operators are composable** - Planner emits list, executor chains them
+4. **Storage is pluggable** - Executor uses Graph interface
+
+### Adding Cypher Features
+
+**You must modify ALL four layers:**
+
+1. **Parser:** Add grammar to `cypher.lark` + transformer in `parser.py`
+2. **AST:** Add/modify dataclasses in `src/graphforge/ast/`
+3. **Planner:** Update `planner.py` to emit operators
+4. **Executor:** Implement execution in `executor.py`
+
+**Decision matrix:**
+- 1-2 layers affected: Simpler, focused changes
+- 3-4 layers affected: More complex, comprehensive changes
+- Extensive tests needed: Plan for thorough test coverage
+- New to feature: Take time to understand the architecture
+
+## Critical Code Patterns
+
+### Value Types System
+
+All query results use `CypherValue` wrappers:
+
+```python
+# src/graphforge/types/values.py
+CypherString, CypherInt, CypherFloat, CypherBool, CypherNull
+CypherList, CypherMap
+
+# Always wrap results in executor
+return CypherString("hello")
+```
+
+### Pydantic Validation
+
+All AST nodes, operators, and metadata use **Pydantic v2 BaseModel**:
+
+```python
+class BinaryOp(BaseModel):
+    op: str = Field(...)
+    left: Any = Field(...)
+    right: Any = Field(...)
+
+    @field_validator("op")
+    @classmethod
+    def validate_op(cls, v: str) -> str:
+        if v not in {"=", "<>", "<", ">", "<=", ">=", "AND", "OR", ...}:
+            raise ValueError(f"Unsupported operator: {v}")
+        return v
+
+    model_config = {"frozen": True}  # Immutable
+```
+
+**Key patterns:**
+- Always use keyword arguments (Pydantic v2 requirement)
+- Use `@field_validator` for single-field validation
+- Use `@model_validator(mode="after")` for cross-field validation
+- All AST nodes are frozen (immutable)
+
+**When to use:**
+- Pydantic: AST nodes, operators, metadata (needs validation)
+- Dataclasses: NodeRef, EdgeRef, CypherValue (performance-critical)
+
+### Two Serialization Systems
+
+**System 1: SQLite + MessagePack (Graph Data)**
+- Purpose: Store nodes, edges, properties
+- Location: `src/graphforge/storage/serialization.py`
+- Format: Binary (fast, compact)
+- Use for: Runtime graph operations
+
+**System 2: Pydantic + JSON (Metadata)**
+- Purpose: Store dataset metadata, schemas, ontologies
+- Location: `src/graphforge/storage/pydantic_serialization.py`
+- Format: JSON (human-readable, validatable)
+- Use for: Configuration, dataset definitions
+
+**CRITICAL:** Never mix these systems. Graph data → MessagePack. Metadata → JSON.
+
+### Graph Storage Duality
+
+```python
+# In-memory (fast, volatile)
+gf = GraphForge()
+
+# Persistent (SQLite backend)
+gf = GraphForge("path/to/db.db")
+```
+
+### Transaction Handling
+
+```python
+# Auto-commit (default)
+gf.execute("CREATE (n:Person)")
+
+# Explicit transactions
+gf.begin()
+gf.execute("CREATE (n:Person)")
+gf.execute("CREATE (m:Person)")
+gf.commit()  # or rollback()
+```
+
+### Dataset System
+
+Registry pattern in `src/graphforge/datasets/`:
+
+```
+datasets/
+├── base.py          # DatasetInfo model
+├── registry.py      # Global registry
+├── loaders/         # Format-specific loaders
+│   ├── csv.py
+│   └── cypher.py
+└── sources/         # Dataset registrations
+    ├── snap.py
+    └── __init__.py  # Auto-registers
+```
+
+## Testing Strategy
+
+### Test Organization
+
+```
+tests/
+├── unit/              # Fast, isolated (< 1ms)
+│   ├── parser/
+│   ├── planner/
+│   ├── executor/
+│   └── storage/
+├── integration/       # End-to-end Cypher queries
+├── tck/               # OpenCypher TCK compliance
+│   ├── features/      # Gherkin scenarios
+│   └── steps/         # pytest-bdd steps
+└── property/          # Hypothesis property tests
+```
+
+### Writing Tests
+
+**Unit tests** - Test ONE component in isolation:
+```python
+def test_parse_simple_match():
+    query = "MATCH (n:Person) RETURN n"
+    ast = parse_cypher(query)
+    assert isinstance(ast.clauses[0], MatchClause)
+```
+
+**Integration tests** - Test end-to-end:
+```python
+def test_match_returns_nodes():
+    gf = GraphForge()
+    gf.execute("CREATE (:Person {name: 'Alice'})")
+    results = gf.execute("MATCH (p:Person) RETURN p.name AS name")
+    assert results[0]['name'].value == 'Alice'
+```
+
+**TCK tests** - Gherkin syntax:
+```gherkin
+Scenario: Match simple node
+  Given an empty graph
+  When executing query:
+    """
+    CREATE (n:Person {name: 'Alice'})
+    """
+  Then the result should be empty
+```
+
+### Test Parametrization
+
+Use for testing same logic with different inputs:
+
+```python
+@pytest.mark.parametrize("op,left,right,expected", [
+    ("=", 5, 5, True),
+    ("<>", 5, 10, True),
+    ("<", 5, 10, True),
+])
+def test_comparison_operators(op, left, right, expected):
+    gf = GraphForge()
+    result = gf.execute(f"RETURN {left} {op} {right} AS result")
+    assert result[0]['result'].value == expected
+```
+
+### Property-Based Testing
+
+Use Hypothesis for testing invariants:
+
+```python
+from hypothesis import given
+from hypothesis import strategies as st
+
+@given(st.integers(), st.text(), st.booleans())
+def test_cypher_value_roundtrip(int_val, str_val, bool_val):
+    """CypherValues serialize and deserialize correctly."""
+    values = [CypherInt(int_val), CypherString(str_val), CypherBool(bool_val)]
+    for val in values:
+        serialized = serialize_value(val)
+        deserialized = deserialize_value(serialized)
+        assert deserialized == val
+```
+
+### Test Isolation
+
+Always use fresh fixtures:
+
+```python
+# ✅ Good - Each test gets fresh graph
+def test_create_node(empty_graph):
+    empty_graph.execute("CREATE (:Person)")
+    assert len(empty_graph.execute("MATCH (n) RETURN n")) == 1
+
+# ❌ Bad - Shared mutable state
+graph = GraphForge()
+def test_create_node():
+    graph.execute("CREATE (:Person)")  # Pollutes state
+```
+
+## Common Development Tasks
+
+### Adding a New Cypher Clause
+
+1. Grammar: Add to `cypher.lark`
+2. AST: Add dataclass to `ast/clause.py`
+3. Transformer: Add method to `parser.py`
+4. Planner: Update to emit operator
+5. Operator: Add to `operators.py`
+6. Executor: Implement execution
+7. Tests: Unit + integration tests
+
+### Adding a New Function
+
+1. Grammar: Add to `function_call` rule in `cypher.lark`
+2. Evaluator: Implement in `executor/evaluator.py`
+3. Tests: Unit + integration tests
+
+### Fixing Parser Issues
+
+1. Check grammar: `src/graphforge/parser/cypher.lark` (EBNF syntax)
+2. Check transformer: `parser.py` (methods match rule names)
+3. Debug with: `tree.pretty()` to see parse tree structure
+
+### Fixing Executor Issues
+
+**Common problems:**
+1. Not wrapping values as `CypherValue` instances
+2. Not handling NULL (use ternary logic)
+3. Mutating input rows (always create new contexts)
+4. Not checking bound variables
+
+**Debug with logging:**
+```python
+import logging
+logging.basicConfig(level=logging.DEBUG)
+gf.execute("MATCH (n) RETURN n")  # See operator pipeline
+```
+
+## Important Files
+
+### Entry Points
+- `src/graphforge/api.py` - Main `GraphForge` class
+- `src/graphforge/__init__.py` - Public exports
+
+### Core Grammar
+- `src/graphforge/parser/cypher.lark` - Complete Cypher grammar
+
+### Type Definitions
+- `src/graphforge/types/values.py` - CypherValue type system
+- `src/graphforge/types/graph.py` - NodeRef/EdgeRef handles
+
+### Storage
+- `src/graphforge/storage/memory.py` - Graph interface + in-memory
+- `src/graphforge/storage/sqlite_backend.py` - SQLite persistence
+- `src/graphforge/storage/serialization.py` - MessagePack encoding
+
+## Documentation
+
+Full docs in `docs/`:
+- `docs/reference/opencypher-compatibility.md` - Feature matrix
+- `docs/datasets/` - Dataset integration docs
+- `docs/tutorial.md` - Getting started guide
+
+Generated docs: `mkdocs serve` (requires `uv sync --group docs`)
+
+## CI/CD
+
+GitHub Actions (`.github/workflows/`):
+- `test.yml` - Runs on all PRs (same as `make pre-push`)
+- `codecov.yml` - Coverage reporting
+- `test-analytics.yml` - Test performance tracking
+
+All tests must pass + coverage thresholds met before merge.
+
+## Project Philosophy
+
+1. **Correctness over performance** - Match openCypher semantics exactly
+2. **Inspectability** - Simple, debuggable code over clever optimizations
+3. **Modularity** - Each layer independent and replaceable
+4. **Developer experience** - Clear errors, good defaults, zero config
+
+**Not goals:** High throughput, massive graphs, production deployment

{graphforge-0.3.2 → graphforge-0.3.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: graphforge
-Version: 0.3.2
+Version: 0.3.4
 Summary: Composable graph tooling for analysis, construction, and refinement
 Project-URL: Homepage, https://github.com/DecisionNerd/graphforge
 Project-URL: Repository, https://github.com/DecisionNerd/graphforge
@@ -98,19 +98,25 @@ Modern data science and ML workflows increasingly produce **graph-shaped data**
 > *We are not building a database for applications.*
 > *We are building a graph execution environment for thinking.*
 
-### Latest Release: v0.3.1
+### Latest Release: v0.3.4
 
-Version 0.3.1 completes all 6 OpenCypher predicate functions:
+Version 0.3.4 completes operator and string function enhancements:
 
-- **Quantifier predicates**: `all()`, `any()`, `none()`, `single()` - Test list elements with three-valued NULL logic
-- **Property testing**: `exists()` - Check if properties or expressions are not NULL
-- **Collection testing**: `isEmpty()` - Test if lists, strings, or maps are empty
-- **Complete coverage**: 57 comprehensive tests, 100% predicate function implementation
-- **Function progress**: 55/72 functions (76%, up from 74%)
+- **String functions**: `toUpper()`, `toLower()` camelCase variants - Aliases for existing UPPER/LOWER functions
+- **Logical operators**: `XOR` - Exclusive OR with proper precedence (NOT > AND > XOR > OR) and NULL handling
+- **Arithmetic operators**: `^` (power/exponentiation) - Right-associative with full numeric support (negative and fractional exponents)
+- **List operations**: List slicing `[start..end]` and negative indexing `list[-1]` (already implemented, now documented)
+- **Comprehensive testing**: 79 new integration tests (18 string, 22 XOR, 39 power)
+- **Operator completion**: List operators 100% (5/5), Overall operators 94% (32/34, up from 88%)
+- **Function progress**: 56/72 functions (78%, up from 76%)
 
-**Previous (v0.3.0)**: OPTIONAL MATCH, UNION, List comprehensions, EXISTS/COUNT subqueries, Variable-length paths, 109+ datasets, Spatial/Temporal types
+**Previous Releases**:
+- **v0.3.3**: Pattern comprehensions, CALL subqueries, Pattern predicates
+- **v0.3.2**: List operations (`filter()`, `extract()`, `reduce()`)
+- **v0.3.1**: Quantifier functions (`all()`, `any()`, `none()`, `single()`), `exists()`, `isEmpty()`
+- **v0.3.0**: OPTIONAL MATCH, UNION, List comprehensions, EXISTS/COUNT subqueries, Variable-length paths, 109+ datasets
 
-**TCK Coverage**: 29% openCypher compatibility
+**TCK Coverage**: ~38% openCypher compatibility
 
 See [CHANGELOG.md](CHANGELOG.md) for complete release notes.
 
@@ -1396,10 +1402,11 @@ GraphForge implements a **practical subset of OpenCypher** focused on common gra
 
 **Expressions:**
 - CASE expressions (conditional logic)
-- Arithmetic operators: +, -, *, /, %
+- Arithmetic operators: +, -, *, /, %, ^ (power)
 - Comparison operators: =, <>, <, >, <=, >=
-- Logical operators: AND, OR, NOT
+- Logical operators: AND, OR, XOR, NOT
 - String matching: STARTS WITH, ENDS WITH, CONTAINS
+- List operations: Slicing `[start..end]`, negative indexing `list[-1]`
 
 **Aggregations:**
 - COUNT, SUM, AVG, MIN, MAX, COLLECT
@@ -1407,11 +1414,12 @@ GraphForge implements a **practical subset of OpenCypher** focused on common gra
 - DISTINCT modifier
 
 **Functions:**
-- String: length, substring, toUpper, toLower, trim
-- Type conversion: toInteger, toFloat, toString
+- String: length, substring, toUpper/toLower (both camelCase and UPPERCASE variants), trim, reverse, split, replace, left, right, ltrim, rtrim
+- Type conversion: toInteger, toFloat, toString, toBoolean
 - Spatial: point, distance
 - Temporal: date, datetime, time, duration
 - Graph: id, labels, type
+- Predicates: all, any, none, single, exists, isEmpty
 - Utility: coalesce
 
 **Data Types:**