cli-jaw 0.1.0

package/skills_ref/property-based-testing/skills/property-based-testing/references/design.md

@@ -0,0 +1,191 @@
+# Property-Driven Development
+
+Design features by defining properties upfront as executable specifications, before implementation.
+
+## When to Use
+
+- Designing a new feature from scratch
+- Building something with clear algebraic properties (serialization, validation, transformations)
+- Complex domain where edge cases are likely
+- User wants to think through requirements rigorously before coding
+
+## Process
+
+### Phase 1: Understand the Feature
+
+Gather information:
+- **Purpose**: What problem does this solve?
+- **Inputs**: What data does it accept? What makes inputs valid?
+- **Outputs**: What does it produce? What guarantees?
+- **Constraints**: What must always be true?
+- **Edge cases**: Boundary conditions?
+- **Relationships**: Inverse operations? Compositions?
+
+### Phase 2: Identify Candidate Properties
+
+Work through these discovery questions:
+
+| Question | Property Type | Example |
+|----------|---------------|---------|
+| Does it have an inverse operation? | Roundtrip | `decode(encode(x)) == x` |
+| Is applying it twice the same as once? | Idempotence | `f(f(x)) == f(x)` |
+| What quantities are preserved? | Invariants | Length, sum, count |
+| Is order of arguments irrelevant? | Commutativity | `f(a, b) == f(b, a)` |
+| Can operations be regrouped? | Associativity | `f(f(a,b), c) == f(a, f(b,c))` |
+| Is there a neutral element? | Identity | `f(x, 0) == x` |
+| Is there an oracle/reference impl? | Oracle | `new(x) == old(x)` |
+| Can output be easily verified? | Hard/Easy | `is_sorted(sort(x))` |
+
+### Phase 3: Define Input Domain
+
+Specify valid inputs as strategies. The strategy IS the specification.
+
+**Key principle**: Build constraints INTO the strategy, not via `assume()`.
+
+```python
+@st.composite
+def valid_registration_requests(draw):
+    """Generate valid registration requests - this documents the domain."""
+    username = draw(st.text(
+        min_size=3,
+        max_size=20,
+        alphabet=st.characters(whitelist_categories=('L', 'N'))
+    ))
+    email = draw(st.emails())
+    password = draw(st.text(min_size=8, max_size=100))
+    age = draw(st.integers(min_value=13, max_value=150))
+
+    return RegistrationRequest(
+        username=username,
+        email=email,
+        password=password,
+        age=age
+    )
+```
+
+### Phase 4: Write Property Tests (Before Implementation)
+
+Create tests that will fail initially:
+
+```python
+class TestFeatureSpec:
+    """Property-based specification - should FAIL until implemented."""
+
+    @given(valid_inputs())
+    def test_core_property(self, x):
+        """[What this guarantees]."""
+        result = feature(x)
+        assert property_holds(result)
+```
+
+### Phase 5: Iterate on Design
+
+Properties reveal design questions:
+- "What about deleted users?"
+- "Case-sensitive?"
+- "Which algorithm?"
+- "Stable sort or not?"
+
+Surface these questions early, before implementation.
+
+## Property Strength Hierarchy
+
+Build properties incrementally from weak to strong:
+
+### Level 1: Basic (Weak)
+```python
+@given(valid_inputs())
+def test_no_crash(x):
+    process(x)  # Just don't crash
+```
+
+### Level 2: Type Preservation
+```python
+@given(valid_inputs())
+def test_returns_type(x):
+    assert isinstance(process(x), ExpectedType)
+```
+
+### Level 3: Invariants
+```python
+@given(valid_inputs())
+def test_invariant(x):
+    result = process(x)
+    assert invariant_holds(result)
+```
+
+### Level 4: Full Specification (Strong)
+```python
+@given(valid_inputs())
+def test_complete(x):
+    result = process(x)
+    assert satisfies_all_requirements(result)
+```
+
+## Strategy Design Principles
+
+### 1. Build Constraints Into Strategy
+```python
+# GOOD - constraints in strategy
+@given(st.integers(min_value=1, max_value=100))
+def test_with_valid_range(x): ...
+
+# BAD - constraints via assume
+@given(st.integers())
+def test_with_assume(x):
+    assume(1 <= x <= 100)  # High rejection rate
+```
+
+### 2. Match Real-World Constraints
+```python
+valid_users = st.builds(
+    User,
+    name=st.text(min_size=1, max_size=100),
+    age=st.integers(min_value=0, max_value=150),
+    email=st.emails(),
+)
+```
+
+### 3. Include Edge Cases Explicitly
+```python
+@given(valid_lists())
+@example([])           # Empty
+@example([1])          # Single element
+@example([1, 1, 1])    # Duplicates
+def test_with_edges(xs): ...
+```
+
+## Common Design Questions Raised
+
+Properties often reveal design gaps:
+
+| Property Attempt | Question Raised |
+|------------------|-----------------|
+| Roundtrip for users | What about deleted/deactivated users? |
+| Duplicate rejection | Case-sensitive? Unicode normalization? |
+| Password storage | Which algorithm? Salted? Configurable? |
+| Ordering guarantee | Stable sort? Tie-breaking rules? |
+
+## Red Flags
+
+- **Writing tautological properties**: Don't reimplement the function logic in the test
+  ```python
+  # BAD - tests nothing
+  assert add(a, b) == a + b
+
+  # GOOD - tests algebraic properties
+  assert add(a, 0) == a  # identity
+  assert add(a, b) == add(b, a)  # commutativity
+  ```
+- **Starting too strong**: Build from weak to strong properties
+- **Ignoring design questions**: Properties that feel awkward often reveal design gaps
+- **Overly complex strategies**: If your input strategy is 50 lines, the domain model might need simplification
+- **Not involving the user**: Design questions should be discussed, not assumed
+
+## Checklist
+
+- [ ] Properties are not tautological
+- [ ] At least one strong property defined
+- [ ] Input strategy documents valid inputs
+- [ ] Design questions have been surfaced
+- [ ] Tests will actually FAIL without implementation

package/skills_ref/property-based-testing/skills/property-based-testing/references/generating.md

@@ -0,0 +1,204 @@
+# Generating Property-Based Tests
+
+How to create complete, runnable property-based tests.
+
+## Process
+
+### 1. Analyze Target Function
+
+- Read function signature, types, and docstrings
+- Understand input types and constraints
+- Identify output type and expected behavior
+- Note preconditions or invariants
+- Check existing example-based tests as hints
+
+### 2. Design Input Strategies
+
+Create appropriate generator strategies for each input parameter.
+
+**Principles**:
+- Build constraints INTO the strategy, not via `assume()`
+- Use realistic size limits to prevent slow tests
+- Match real-world constraints
+
+### 3. Identify Applicable Properties
+
+| Property | When to Use | Test Pattern |
+|----------|-------------|--------------|
+| Roundtrip | encode/decode pairs | `assert decode(encode(x)) == x` |
+| Idempotence | normalization, sorting | `assert f(f(x)) == f(x)` |
+| Invariant | any transformation | `assert invariant(f(x))` |
+| No exception | all functions (weak) | Function completes without raising |
+| Type preservation | typed functions | `assert isinstance(f(x), ExpectedType)` |
+| Length preservation | collections | `assert len(f(xs)) == len(xs)` |
+| Element preservation | sorting, shuffling | `assert set(f(xs)) == set(xs)` |
+| Ordering | sorting | `assert all(f(xs)[i] <= f(xs)[i+1] ...)` |
+| Oracle | when reference exists | `assert f(x) == reference_impl(x)` |
+| Commutativity | binary ops | `assert f(a, b) == f(b, a)` |
+
+### 4. Generate Test Code
+
+Create test functions with:
+- Clear docstrings explaining what each property verifies
+- Appropriate `@settings` for the context
+- `@example` decorators for critical edge cases
+
+### 5. Include Edge Cases
+
+Always add explicit examples:
+```python
+@example([])           # Empty
+@example([1])          # Single element
+@example([1, 1, 1])    # Duplicates
+@example("")           # Empty string
+@example(0)            # Zero
+@example(-1)           # Negative
+```
+
+## Settings Recommendations
+
+```python
+# Development (fast feedback)
+@settings(max_examples=10)
+
+# CI (thorough)
+@settings(max_examples=200)
+
+# Nightly/Release (exhaustive)
+@settings(max_examples=1000, deadline=None)
+```
+
+## Example Test Patterns
+
+### Roundtrip (Encode/Decode)
+
+```python
+@given(valid_messages())
+def test_roundtrip(msg):
+    """Encoding then decoding returns original."""
+    assert decode(encode(msg)) == msg
+```
+
+### Idempotence
+
+```python
+@given(st.text())
+def test_normalize_idempotent(s):
+    """Normalizing twice equals normalizing once."""
+    assert normalize(normalize(s)) == normalize(s)
+```
+
+### Sorting Properties
+
+```python
+@given(st.lists(st.integers()))
+@example([])
+@example([1])
+@example([1, 1, 1])
+def test_sort(xs):
+    result = sort(xs)
+    # Length preserved
+    assert len(result) == len(xs)
+    # Elements preserved
+    assert sorted(result) == sorted(xs)
+    # Ordered
+    assert all(result[i] <= result[i+1] for i in range(len(result)-1))
+    # Idempotent
+    assert sort(result) == result
+```
+
+### Validator + Normalizer
+
+```python
+@given(valid_inputs())
+def test_normalized_is_valid(x):
+    """Normalized inputs pass validation."""
+    assert is_valid(normalize(x))
+```
+
+## Complete Example (Python/Hypothesis)
+
+```python
+"""Property-based tests for message_codec module."""
+from hypothesis import given, strategies as st, settings, example
+import pytest
+
+from myapp.codec import encode_message, decode_message, Message, DecodeError
+
+# Custom strategy for Message objects
+messages = st.builds(
+    Message,
+    id=st.uuids(),
+    content=st.text(max_size=1000),
+    priority=st.integers(min_value=1, max_value=10),
+    tags=st.lists(st.text(max_size=50), max_size=20),
+)
+
+
+class TestMessageCodecProperties:
+    """Property-based tests for message encoding/decoding."""
+
+    @given(messages)
+    def test_roundtrip(self, msg: Message):
+        """Encoding then decoding returns the original message."""
+        encoded = encode_message(msg)
+        decoded = decode_message(encoded)
+        assert decoded == msg
+
+    @given(messages)
+    def test_encode_deterministic(self, msg: Message):
+        """Same message always encodes to same bytes."""
+        assert encode_message(msg) == encode_message(msg)
+
+    @given(messages)
+    def test_encoded_is_bytes(self, msg: Message):
+        """Encoding produces bytes."""
+        assert isinstance(encode_message(msg), bytes)
+
+    @given(st.binary())
+    def test_decode_invalid_raises_or_succeeds(self, data: bytes):
+        """Random bytes either decode or raise DecodeError."""
+        try:
+            decode_message(data)
+        except DecodeError:
+            pass  # Expected for invalid input
+```
+
+## Running Tests
+
+```bash
+# Run all property tests
+pytest test_file.py -v
+
+# Run with more examples (CI)
+pytest test_file.py --hypothesis-seed=0 -v
+
+# Run with statistics
+pytest test_file.py --hypothesis-show-statistics
+```
+
+## Checklist Before Finishing
+
+- [ ] Tests are not tautological (don't reimplement the function)
+- [ ] At least one strong property (not just "no crash")
+- [ ] Edge cases covered with `@example` decorators
+- [ ] Strategy constraints are realistic, not over-filtered
+- [ ] Settings appropriate for context (dev vs CI)
+- [ ] Docstrings explain what each property verifies
+- [ ] Tests actually run and pass (or fail for expected reasons)
+
+## Red Flags
+
+- **Reimplementing the function**: If your assertion contains the same logic as the function under test, you've written a tautology
+  ```python
+  # BAD - this tests nothing
+  assert add(a, b) == a + b
+  ```
+- **Only testing "no crash"**: This is the weakest property - always look for stronger ones first
+- **Overly constrained strategies**: If you're using multiple `assume()` calls, redesign the strategy instead
+- **Missing edge cases**: No `@example` decorators for empty, single-element, or boundary values
+- **No settings**: Missing `@settings` for CI - tests may be too slow or not thorough enough
+
+## When Tests Fail
+
+See [{baseDir}/references/interpreting-failures.md]({baseDir}/references/interpreting-failures.md) for how to interpret failures and determine if they represent genuine bugs vs test errors vs ambiguous specifications.

package/skills_ref/property-based-testing/skills/property-based-testing/references/interpreting-failures.md

@@ -0,0 +1,239 @@
+# Interpreting Property-Based Test Failures
+
+How to analyze failures and determine if they represent genuine bugs.
+
+## The Self-Reflection Problem
+
+Property-based testing generates many failing examples. Not all failures are bugs:
+- **Test bugs**: Property is wrong, strategy generates invalid inputs
+- **Ambiguous specs**: Behavior undefined for edge cases
+- **Genuine bugs**: Code violates documented guarantees
+
+Before reporting a bug, **validate the failure** through systematic analysis.
+
+## Failure Analysis Workflow
+
+### 1. Reproduce with Minimal Example
+
+Start with the shrunk failing input from the test output.
+
+```python
+# Hypothesis provides the minimal failing case
+# Falsifying example: test_normalize(s='\x00')
+
+# Create standalone reproducer
+def test_reproduce():
+    s = '\x00'
+    result = normalize(normalize(s))
+    assert result == normalize(s)  # Fails
+```
+
+Verify the failure is consistent, not flaky.
+
+### 2. Ground the Property
+
+Before assuming a bug, verify your property against authoritative sources:
+
+| Source | What It Tells You |
+|--------|-------------------|
+| **Type annotations** | Return type constraints, nullability |
+| **Docstrings** | Explicit guarantees, preconditions |
+| **Function name** | Semantic expectations (e.g., `sort` implies ordering) |
+| **Error handling** | What inputs should raise vs handle |
+| **Existing unit tests** | Implicit contracts maintainers expect |
+| **External docs/specs** | Protocol specs, format definitions |
+
+**Example grounding check:**
+```python
+def normalize(s: str) -> str:
+    """Normalize a string to NFC form.
+
+    Args:
+        s: Input string (any unicode)
+
+    Returns:
+        NFC-normalized string
+    """
+```
+
+The docstring says "any unicode" - so null bytes should be valid input. The property is correctly grounded.
+
+### 3. Check Strategy Realism
+
+Does the strategy generate inputs the function should actually handle?
+
+**Red flags:**
+- Generating inputs outside documented domain
+- Missing constraints that real callers would have
+- Overly aggressive size/complexity
+
+**Questions to ask:**
+- Would real code pass this input?
+- Does the docstring exclude this case?
+- Is this a precondition violation, not a bug?
+
+### 4. Classify the Failure
+
+| Symptom | Likely Cause | Action |
+|---------|--------------|--------|
+| Fails on edge case not mentioned in spec | Ambiguous specification | Clarify with maintainer before reporting |
+| Fails on input that violates documented preconditions | Over-constrained strategy | Fix the strategy |
+| Property contradicts docstring or type hints | Wrong property | Fix the property |
+| Clear violation of documented guarantee | Genuine bug | Report with evidence |
+| Behavior differs from similar functions | Possible inconsistency | Investigate further |
+
+### 5. Decide Action
+
+- **Test bug** → Fix the property or strategy, don't report
+- **Ambiguous spec** → Open discussion issue, not bug report
+- **Genuine bug** → Report with minimal reproducer and evidence
+
+## Property Grounding Checklist
+
+Before reporting a failure as a bug, verify:
+
+- [ ] Property matches documented return type
+- [ ] Property matches docstring guarantees
+- [ ] Input is within documented domain (preconditions met)
+- [ ] No `assume()` filtering out the failing case inappropriately
+- [ ] Checked existing tests don't contradict your property
+- [ ] Behavior contradicts docs, not just expectations
+
+## Bug Report Template
+
+When confident the failure is a genuine bug:
+
+```markdown
+## Summary
+[One-line description of the bug]
+
+## Minimal Reproducing Example
+```python
+# Shrunk by Hypothesis
+from mylib import affected_function
+
+def test_bug():
+    # Minimal failing input
+    result = affected_function('\x00')
+    # Expected vs actual
+    assert result >= 0  # Fails: got -1
+```
+
+## Expected Behavior
+According to [docstring/spec/docs], the function should:
+- [Specific guarantee that was violated]
+
+## Actual Behavior
+- [What actually happened]
+
+## Evidence
+- Docstring states: "[relevant quote]"
+- Type signature promises: `-> PositiveInt`
+
+## Environment
+- Library version: X.Y.Z
+- Python version: 3.X
+- Platform: [OS]
+```
+
+## Real-World Failure Patterns
+
+### Numerical Instability
+
+**Symptom**: Distribution function returns negative probability.
+
+```python
+@given(st.floats(min_value=0, max_value=1e308))
+def test_probability_non_negative(x):
+    prob = compute_probability(x)
+    assert prob >= 0  # Fails for x=1e-320
+```
+
+**Grounding check**: Docstring says "returns probability in [0, 1]".
+
+**Classification**: Genuine bug - documented guarantee violated.
+
+### Iterator Off-by-One
+
+**Symptom**: Iterator skips elements or yields extra.
+
+```python
+@given(st.lists(st.integers()))
+def test_iterator_yields_all(xs):
+    result = list(custom_iterator(xs))
+    assert result == xs  # Fails: missing last element
+```
+
+**Grounding check**: Iterator should yield all elements based on name/docs.
+
+**Classification**: Genuine bug if documented to iterate fully.
+
+### Hash/Equality Inconsistency
+
+**Symptom**: Equal objects have different hashes.
+
+```python
+@given(valid_objects())
+def test_hash_equality(obj):
+    obj2 = create_equal_copy(obj)
+    assert obj == obj2
+    assert hash(obj) == hash(obj2)  # Fails
+```
+
+**Grounding check**: Python requires `a == b` implies `hash(a) == hash(b)`.
+
+**Classification**: Genuine bug - violates language contract.
+
+### Roundtrip Failure on Edge Cases
+
+**Symptom**: Encode/decode doesn't preserve input.
+
+```python
+@given(st.text())
+def test_roundtrip(s):
+    assert decode(encode(s)) == s  # Fails for s='\uD800'
+```
+
+**Grounding check**: Is `'\uD800'` (lone surrogate) valid input?
+
+**Classification**:
+- If docs say "valid UTF-8 only" → Strategy bug, fix filter
+- If docs say "any string" → Genuine bug, report it
+
+### Format String Errors
+
+**Symptom**: String formatting crashes on certain inputs.
+
+```python
+@given(st.text())
+def test_format_safe(template):
+    format_message(template)  # Raises on '{unclosed'
+```
+
+**Grounding check**: Does function claim to handle arbitrary strings?
+
+**Classification**:
+- If user-facing, should handle gracefully → Genuine bug
+- If internal API with preconditions → Check preconditions met
+
+## When NOT to Report
+
+Do not report as bugs:
+
+1. **Precondition violations**: If docs say "positive integers only" and you passed -1
+2. **Undefined behavior**: Spec explicitly says behavior is undefined
+3. **Implementation details**: Relying on undocumented internal behavior
+4. **Platform-specific**: Bug only on unusual platform/version
+5. **Test artifact**: Failure disappears with realistic constraints
+
+## Confidence Threshold
+
+Report only when you can answer YES to all:
+
+1. Did you reproduce with a minimal example?
+2. Did you verify the property against docs/types/docstrings?
+3. Can you point to a specific documented guarantee that's violated?
+4. Is the failing input within the documented domain?
+5. Have you ruled out test bugs and ambiguous specs?
+
+If uncertain on any point, open a discussion first, not a bug report.