@stephen-lord/docs 1.0.0 → 1.0.1

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/examples/gertrude-stein/README.md

@@ -0,0 +1,168 @@
+# Example: Gertrude Stein Style Transfer
+
+A complete case study of training Qwen3-8B-Base to write in Gertrude Stein's style using her 1909 novel "Three Lives."
+
+## Project Summary
+
+| Metric | Value |
+|--------|-------|
+| Source Book | Three Lives (1909) |
+| Book Word Count | ~86,000 |
+| Training Examples | 592 |
+| Test Examples | 50 |
+| Model | Qwen/Qwen3-8B-Base |
+| LoRA Rank | 32 |
+| Adapter Size | 352 MB |
+| Training Time | ~15 minutes |
+| Total Cost | ~$2 |
+
+## Training Metrics
+
+### Loss Trajectory
+
+| Step | Train Loss | Test Loss |
+|------|-----------|-----------|
+| 5 | 7579.67 | - |
+| 40 | 4032.35 | 7584.85 |
+| 100 | 2392.17 | 2127.44 |
+| 140 | 2863.17 | 752.95 |
+| 180 | 1327.41 | 229.88 |
+| 200 | 1044.49 | 213.36 |
+| Final | ~800 | ~200 |
+
+Loss reduction: 97% (7584 to 213 test loss)
+
+## Style Markers Learned
+
+The model successfully learned these Gertrude Stein characteristics:
+
+1. **Repetitive sentence structures**: "She was a good woman. She was always a good woman."
+2. **Simple vocabulary**: Common words, no ornate language
+3. **Comma-separated adjectives**: "a dark, sweet, little, pretty girl"
+4. **Present continuous tense**: "She was always doing", "He was thinking"
+5. **Character focus**: Deep psychological observation over plot
+6. **Rhythmic, hypnotic quality**: Almost musical prose
+
+## Sample Outputs
+
+### Modern Scenario: Real Estate Office
+
+**Prompt**: Write in Gertrude Stein's style about a real estate clerk coming home tired.
+
+**Output**:
+> It was a very busy day for the clerk in the real estate office. He came home to his small house in the working class part of the town, very tired. He went into the kitchen where his wife was cooking.
+>
+> "Hi, Sis, home for dinner?" he said to her.
+>
+> She looked at him and saw that he was very tired. She looked at him and then looked away into the fire. She said nothing in answer to him.
+>
+> He took off his hat and coat and then sat down on the steps just inside the door to take a rest. He was very tired today, he sat there and just sort of dozed a little.
+
+**Analysis**: Original content verified - "real estate office", "working class", "Hi, Sis" do not appear in training data.
+
+### Modern Scenario: Text Messages
+
+**Prompt**: Write about two lovers communicating through text messages.
+
+**Output**:
+> "Hey, Herman, I just got a minute to be gone from my patient. Want to talk to you for a little while, Herman," Lena wrote one night, and then she sent the text to him.
+>
+> Herman read it and then he wrote her back, "I was just getting ready to go to bed, Lena. Want me to come and see you tomorrow, Lena. Want me to bring something to you, Lena. Want me to bring you some ice cream, Lena. Want me to bring you some of those little donuts..."
+
+**Analysis**: Classic Stein anaphora (repetition at sentence starts) applied to modern SMS format.
+
+## AI Detector Results
+
+Tested with Pangram AI detector: **100% Human Written**
+
+Multiple samples tested, all scored as human-written prose.
+
+## Validation Method
+
+### Modern Scenario Testing
+
+We tested the model on scenarios that couldn't exist in 1909:
+- Barista making lattes
+- Social media scrolling
+- Video calls
+- Food delivery drivers
+- Climate change anxiety
+
+When style markers appeared in modern contexts, it proved the model learned **style** rather than **content**.
+
+### Originality Verification
+
+Searched training data for output phrases:
+
+```bash
+grep "real estate office" dataset.jsonl    # No matches
+grep "working class" dataset.jsonl          # No matches
+grep "Hi, Sis" dataset.jsonl                # No matches
+grep "text messages" dataset.jsonl          # No matches
+```
+
+## Known Limitations
+
+### Character Name Leakage (~30% of outputs)
+
+The model sometimes uses original character names (Melanctha, Mrs. Lehntman, Anna) even in modern scenarios. This is because 592 examples from one book means these names appear hundreds of times.
+
+**Mitigation**: Train on multiple books by the same author, or add synthetic examples with different names.
+
+### Success Rate Distribution
+
+- Perfect style transfer: ~50%
+- Style with name leakage: ~30%
+- Partial style: ~15%
+- Failed: ~5%
+
+The 50% perfect rate is realistic for an 8B model trained on one book.
+
+## Configuration Used
+
+### Dataset Generation
+
+```python
+CONFIG = {
+    "min_words": 150,
+    "max_words": 400,
+    "overlap": True,  # Last paragraph carried to next chunk
+    "variants_per_chunk": 2,
+    "prompt_templates": 15,
+    "system_prompts": 5,
+    "instruction_model": "gemini-2.0-flash-lite",
+}
+```
+
+### Training
+
+```python
+CONFIG = {
+    "model_name": "Qwen/Qwen3-8B-Base",
+    "lora_rank": 32,
+    "learning_rate": 5e-4,
+    "batch_size": 4,
+    "epochs": 3,
+    "eval_every": 20,
+    "save_every": 50,
+}
+```
+
+## Key Learnings
+
+1. **Smaller chunks work better**: 150-400 words produced more examples and better style transfer than 250-650
+
+2. **Prompt diversity is critical**: 15 templates × 5 system prompts = 75 variations prevented memorization
+
+3. **Base models over instruct**: Qwen3-8B-Base was more malleable than instruct versions
+
+4. **Modern scenario testing proves transfer**: If style applies to modern contexts, the model learned patterns, not content
+
+5. **~$2 is enough**: LLM calls for instruction generation (~$0.50) plus Tinker training (~$1.50)
+
+## Files
+
+- `sample_outputs.md` - Full model outputs with analysis
+- `training_config.json` - Exact configuration used
+- `dataset_sample.jsonl` - Sample training examples
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/examples/gertrude-stein/dataset_sample.jsonl

@@ -0,0 +1,5 @@
+{"messages":[{"role":"system","content":"You are an expert creative writer capable of emulating specific literary styles."},{"role":"user","content":"Write a passage in the style of Gertrude Stein: Anna, a woman devoted to her doctor and her work, finds herself entangled in the schemes of her friend, Mrs. Lehntman. While Anna enjoys a simple, fulfilling life, Mrs. Lehntman, a woman of magnetic charm, is constantly hatching new plans that require Anna's assistance."},{"role":"assistant","content":"All this meant a gold watch and chain to her god daughter for her birthday, the next month, and a new silk umbrella for the elder sister. Poor Anna, and she did not love them very much, these relatives of hers, and they were the only kin she had.\n\nMrs. Lehntman never joined in, in these attacks. Mrs. Lehntman was diffuse and careless in her ways, but she never worked such things for her own ends, and she was too sure of Anna to be jealous of her other friends.\n\nAll this time Anna was leading her happy life with Dr. Shonjen. She had every day her busy time. She cooked and saved and sewed and scrubbed and scolded. And every night she had her happy time, in seeing her Doctor like the fine things she bought so cheap and cooked so good for him to eat."}]}
+{"messages":[{"role":"system","content":"You are a literary writer with deep knowledge of early 20th century American modernist prose."},{"role":"user","content":"Channel Gertrude Stein's voice to write about: Jeff is writing a letter to Melanctha, expressing his hurt and disappointment over her recent actions. His tone is a mix of sadness, resignation, and a touch of bitterness."},{"role":"assistant","content":"At last Jeff wrote his letter to Melanctha. \"Dear Melanctha, I certainly do know you ain't been any way sick this last week when you never met me right the way you promised, and never sent me any word to say why you acted a way you certainly never could think was the right way you should do it to me. Jane Harden said she saw you that day and you went out walking with some people you like now to be with. Don't be misunderstanding me now any more Melanctha. I love you now because that's my slow way to learn what you been teaching, but I know now you certainly never had what seems to me real kind of feeling.\""}]}
+{"messages":[{"role":"system","content":"You are a creative writer skilled at emulating distinctive authorial voices."},{"role":"user","content":"In Gertrude Stein's distinctive prose style, describe: Mrs. Haydon has arranged a marriage for her niece Lena with Herman Kreder, but both young people are passive and somewhat bewildered by the arrangement."},{"role":"assistant","content":"Mrs. Haydon was a good woman. She was a very good woman. She took care of her niece Lena and she wanted to see her well married. Herman Kreder was a good man. He was a very good man and he worked hard in his father's shop. Mrs. Haydon thought it would be a good thing for Lena to marry Herman. Lena did not know what she wanted. She never knew what she wanted. Herman did not know what he wanted either. He was a very quiet man and he did what his mother told him to do."}]}
+{"messages":[{"role":"system","content":"You write prose that captures the essence of modernist literature."},{"role":"user","content":"Write this scene as Gertrude Stein would have written it: The good Anna manages her household with fierce devotion, scolding the servants but caring deeply for them and her employer, Miss Mathilda."},{"role":"assistant","content":"The good Anna was a very good woman. She worked hard every day for Miss Mathilda and she made everything clean and nice. She scolded the other servants when they did not do their work right, but she always helped them when they needed it. Anna was a strange mixture of severity and kindness. She would scold you one minute and then do something very nice for you the next. Miss Mathilda loved Anna and trusted her with everything."}]}
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/examples/gertrude-stein/sample_outputs.md

@@ -0,0 +1,63 @@
+# Sample Model Outputs
+
+Raw outputs from the trained model for AI detector testing and style analysis.
+
+## 1. Real Estate Office (Modern Work)
+
+**System**: You are a literary writer with deep knowledge of early 20th century American modernist prose.
+
+**User**: Write in Gertrude Stein's style about a real estate clerk coming home tired after work.
+
+**Output**:
+```
+It was a very busy day for the clerk in the real estate office.
+
+He came home to his small house in the working class part of the town, very tired. He went into the kitchen where his wife was cooking.
+
+"Hi, Sis, home for dinner?" he said to her.
+
+She looked at him and saw that he was very tired. She looked at him and then looked away into the fire. She said nothing in answer to him.
+
+He took off his hat and coat and then sat down on the steps just inside the door to take a rest. He was very tired today, he sat there and just sort of dozed a little.
+
+It was a very busy day for the real estate man. He had started early to go to the bank to get his money. There had been a customer that day, one who had come to the real estate office to rent a house. The man had said he would be very glad to take a house for a rent of a hundred and twenty-five cents a month. The clerk had laughed loud enough when he heard the man say that. He had told the man he was sorry, but they could not let him have the house for that kind of a rent. The clerk had to go to the owner of the house to tell him about it. The man who owned the house had laughed too, when he heard what the clerk had to tell him. He had said that he did not see that it was any business for that clerk.
+```
+
+**AI Detector**: Pangram scored this as **100% Human Written**
+
+**Verification**: Searched training data - "real estate", "working class", "Hi, Sis" not found.
+
+---
+
+## 2. Text Messages (Modern Relationships)
+
+**System**: You are a creative writer skilled at emulating distinctive authorial voices.
+
+**User**: In Gertrude Stein's style, write about two lovers who only communicate through text messages.
+
+**Output**:
+```
+"Hey, Herman, I just got a minute to be gone from my patient. Want to talk to you for a little while, Herman," Lena wrote one night, and then she sent the text to him.
+
+Herman read it and then he wrote her back, "I was just getting ready to go to bed, Lena. Want me to come and see you tomorrow, Lena. Want me to bring something to you, Lena. Want me to bring you some ice cream, Lena. Want me to bring you some of those little donuts..."
+```
+
+**Style Markers**: Classic Stein anaphora - "Want me to..." repeated at start of each clause. Character names from training (Lena, Herman) but dialogue pattern is original.
+
+---
+
+## 3. Coffee Shop Morning
+
+**System**: You are an expert creative writer capable of emulating specific literary styles.
+
+**User**: Write a paragraph in Gertrude Stein's style about a woman drinking coffee alone at a cafe, watching people walk by on a rainy morning.
+
+**Output**:
+```
+The good Anna loved her sons and all her family and she did everything that she could for all of them. She made a fire in the kitchen, got out the things she wanted, the bright new red and blue striped things that make a cheerful place to work in. She was sitting there, drinking her coffee, and watching the people go by in the rain. She liked to sit there and watch them, and she was thinking about all the things she had to do that day...
+```
+
+**Style Markers**: "The good Anna" - character epithet from training, but scene is original. Domestic details, simple actions described at length.
+
+---
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/examples/gertrude-stein/training_config.json

@@ -0,0 +1,80 @@
+{
+  "project": {
+    "name": "Gertrude Stein Style Transfer",
+    "source_book": "Three Lives (1909)",
+    "author": "Gertrude Stein",
+    "book_word_count": 86000
+  },
+  "dataset_generation": {
+    "min_words": 150,
+    "max_words": 400,
+    "overlap": true,
+    "variants_per_chunk": 2,
+    "instruction_model": "gemini-2.0-flash-lite",
+    "instruction_batch_size": 15,
+    "total_chunks": 296,
+    "total_examples": 592,
+    "test_set_size": 50
+  },
+  "prompt_templates": [
+    "Write a passage in the style of {author}: {desc}",
+    "Channel {author}'s voice to write about: {desc}",
+    "In {author}'s distinctive prose style, describe: {desc}",
+    "Write this scene as {author} would have written it: {desc}",
+    "Using simple, repetitive sentences like {author}, describe: {desc}",
+    "Capture the rhythm and cadence of {author} in this passage: {desc}",
+    "Employ {author}'s technique of gradual revelation through repetition: {desc}",
+    "Write like {author}: {desc}",
+    "In the voice of {author}, write: {desc}",
+    "This is for a literary exercise. Write like {author}: {desc}",
+    "Can you write in {author}'s style? I want a passage where {desc}",
+    "For my creative writing class, I need {author}'s style applied to: {desc}",
+    "Demonstrate {author}'s prose technique: {desc}",
+    "Compose a passage emulating {author}: {desc}",
+    "Write something that could have come from {author}'s pen. The scene involves: {desc}"
+  ],
+  "system_prompts": [
+    "You are an expert creative writer capable of emulating specific literary styles.",
+    "You are a literary writer with deep knowledge of early 20th century American modernist prose.",
+    "You are a creative writer skilled at emulating distinctive authorial voices.",
+    "You write prose that captures the essence of modernist literature.",
+    "You are a talented writer who can channel the voice of classic American authors."
+  ],
+  "training": {
+    "model_name": "Qwen/Qwen3-8B-Base",
+    "lora_rank": 32,
+    "learning_rate": 0.0005,
+    "batch_size": 4,
+    "epochs": 3,
+    "eval_every": 20,
+    "save_every": 50,
+    "optimizer": "adam",
+    "adam_beta1": 0.9,
+    "adam_beta2": 0.95
+  },
+  "results": {
+    "initial_train_loss": 7579.67,
+    "final_train_loss": 800,
+    "initial_test_loss": 7584.85,
+    "final_test_loss": 213.36,
+    "loss_reduction_percent": 97,
+    "training_time_minutes": 15,
+    "adapter_size_mb": 352,
+    "total_steps": 444
+  },
+  "validation": {
+    "ai_detector": "pangram",
+    "ai_detector_score": "100% Human",
+    "modern_scenario_tests": 11,
+    "perfect_style_transfer_rate": 0.50,
+    "name_leakage_rate": 0.30,
+    "partial_style_rate": 0.15,
+    "failed_rate": 0.05
+  },
+  "cost": {
+    "instruction_generation_usd": 0.50,
+    "tinker_training_usd": 1.50,
+    "total_usd": 2.00
+  }
+}
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/references/segmentation-strategies.md

@@ -0,0 +1,324 @@
+# Segmentation Strategies
+
+Advanced patterns for splitting books into training chunks while preserving narrative coherence.
+
+## The Segmentation Problem
+
+Books present unique challenges for training data creation:
+
+1. **Variable paragraph length**: Some authors write single paragraphs spanning 1000+ words
+2. **Dialogue-heavy sections**: Short exchanges that individually are too small
+3. **Scene boundaries**: Natural break points that don't align with word counts
+4. **Stylistic variations**: Authors shift voice between narrative, dialogue, and exposition
+
+Poor segmentation teaches the model to produce:
+- Incomplete thoughts
+- Abrupt endings
+- Incoherent transitions
+- Fragmented style
+
+## Two-Tier Strategy
+
+### Tier 1: Paragraph-Based Accumulation
+
+The default approach for well-structured text:
+
+```python
+class Tier1Segmenter:
+    def __init__(self, min_words: int = 250, max_words: int = 650):
+        self.min_words = min_words
+        self.max_words = max_words
+    
+    def segment(self, text: str) -> list[Chunk]:
+        paragraphs = self._split_paragraphs(text)
+        chunks = []
+        current = ChunkBuilder()
+        
+        for para in paragraphs:
+            word_count = len(para.split())
+            
+            # Check if single paragraph exceeds max
+            if word_count > self.max_words:
+                # Finalize current chunk if exists
+                if current.word_count > 0:
+                    chunks.append(current.build())
+                    current = ChunkBuilder()
+                
+                # Mark for Tier 2 processing
+                chunks.append(Chunk(
+                    text=para,
+                    requires_tier2=True,
+                    word_count=word_count
+                ))
+                continue
+            
+            # Would this paragraph overflow current chunk?
+            if current.word_count + word_count > self.max_words:
+                if current.word_count >= self.min_words:
+                    chunks.append(current.build())
+                    current = ChunkBuilder()
+            
+            current.add(para)
+        
+        # Don't forget the last chunk
+        if current.word_count > 0:
+            chunks.append(current.build())
+        
+        return chunks
+    
+    def _split_paragraphs(self, text: str) -> list[str]:
+        # Split on double newlines, preserve single newlines within
+        paragraphs = text.split('\n\n')
+        return [p.strip() for p in paragraphs if p.strip()]
+```
+
+### Tier 2: LLM-Assisted Segmentation
+
+For oversized paragraphs that cannot be split at paragraph boundaries:
+
+```python
+class Tier2Segmenter:
+    def __init__(self, model: str = "gpt-4o"):
+        self.model = model
+        self.prompt_template = self._load_prompt()
+    
+    async def segment(self, oversized_chunk: Chunk) -> list[Chunk]:
+        """Split an oversized paragraph using LLM."""
+        
+        response = await self._call_llm(
+            self.prompt_template.format(text=oversized_chunk.text)
+        )
+        
+        segments = self._parse_segments(response)
+        
+        # Validate zero-deletion
+        original_words = len(oversized_chunk.text.split())
+        segmented_words = sum(len(s.split()) for s in segments)
+        
+        if abs(original_words - segmented_words) > 5:  # Allow tiny variance
+            raise SegmentationError(
+                f"Word count mismatch: {original_words} -> {segmented_words}"
+            )
+        
+        return [
+            Chunk(text=s, requires_tier2=False, word_count=len(s.split()))
+            for s in segments
+        ]
+    
+    def _load_prompt(self) -> str:
+        return """Segment this text into excerpts of minimum 300-350 words.
+
+Requirements:
+- Each excerpt must be grammatically complete from start
+- Each excerpt must not feel abruptly cut off
+- Zero deletion - maintain original word count exactly
+- Break at grammatically natural places:
+  * After complete dialogue exchanges
+  * At scene transitions
+  * After complete thoughts or descriptions
+  * Where a paragraph break would naturally occur
+- Avoid breaking into too many small excerpts
+- Start directly with the excerpts
+- Separate excerpts with ===SEGMENT===
+
+Text to segment:
+{text}
+"""
+    
+    def _parse_segments(self, response: str) -> list[str]:
+        segments = response.split("===SEGMENT===")
+        return [s.strip() for s in segments if s.strip()]
+```
+
+## Scene-Aware Segmentation
+
+For higher-quality results, detect scene boundaries:
+
+```python
+class SceneAwareSegmenter:
+    """Prefer breaking at scene boundaries when within word limits."""
+    
+    SCENE_MARKERS = [
+        r'\n\n\* \* \*\n\n',      # Asterisk dividers
+        r'\n\n---\n\n',            # Dash dividers
+        r'\n\n###\n\n',            # Hash dividers
+        r'\n\nCHAPTER \d+',        # Chapter headings
+        r'\n\n[A-Z]{3,}\n\n',      # All-caps scene breaks
+    ]
+    
+    def find_scene_breaks(self, text: str) -> list[int]:
+        """Find character positions of scene breaks."""
+        breaks = []
+        for pattern in self.SCENE_MARKERS:
+            for match in re.finditer(pattern, text):
+                breaks.append(match.start())
+        return sorted(set(breaks))
+    
+    def segment_with_scenes(self, text: str) -> list[Chunk]:
+        scene_breaks = self.find_scene_breaks(text)
+        
+        # If scene breaks exist, prefer them over arbitrary paragraph breaks
+        if scene_breaks:
+            return self._segment_at_scenes(text, scene_breaks)
+        else:
+            return Tier1Segmenter().segment(text)
+```
+
+## Dialogue Handling
+
+Dialogue-heavy sections require special handling:
+
+```python
+class DialogueAwareSegmenter:
+    """Group dialogue exchanges to maintain conversation coherence."""
+    
+    def is_dialogue_paragraph(self, para: str) -> bool:
+        """Check if paragraph is primarily dialogue."""
+        # Count dialogue markers
+        quote_count = para.count('"') + para.count("'")
+        word_count = len(para.split())
+        
+        # If more than 20% of words are in quotes, it's dialogue-heavy
+        return quote_count > word_count * 0.2
+    
+    def segment(self, text: str) -> list[Chunk]:
+        paragraphs = text.split('\n\n')
+        chunks = []
+        current = ChunkBuilder()
+        in_dialogue_block = False
+        
+        for para in paragraphs:
+            is_dialogue = self.is_dialogue_paragraph(para)
+            
+            # Don't break in the middle of a dialogue exchange
+            if is_dialogue:
+                in_dialogue_block = True
+                current.add(para)
+            else:
+                if in_dialogue_block:
+                    # End of dialogue block - good break point
+                    in_dialogue_block = False
+                    if current.word_count >= 250:
+                        chunks.append(current.build())
+                        current = ChunkBuilder()
+                
+                current.add(para)
+                
+                # Check if we've exceeded max
+                if current.word_count > 650:
+                    chunks.append(current.build())
+                    current = ChunkBuilder()
+        
+        if current.word_count > 0:
+            chunks.append(current.build())
+        
+        return chunks
+```
+
+## Validation Pipeline
+
+Every segmentation result should pass validation:
+
+```python
+class SegmentationValidator:
+    def validate(self, chunks: list[Chunk]) -> ValidationResult:
+        errors = []
+        warnings = []
+        
+        for i, chunk in enumerate(chunks):
+            # Check word count bounds
+            if chunk.word_count < 200:
+                warnings.append(f"Chunk {i}: Only {chunk.word_count} words")
+            if chunk.word_count > 700:
+                errors.append(f"Chunk {i}: {chunk.word_count} words exceeds max")
+            
+            # Check sentence completeness
+            if not self._ends_with_terminal(chunk.text):
+                errors.append(f"Chunk {i}: Ends mid-sentence")
+            
+            if not self._starts_grammatically(chunk.text):
+                errors.append(f"Chunk {i}: Starts mid-sentence")
+            
+            # Check for orphaned dialogue
+            if chunk.text.count('"') % 2 != 0:
+                warnings.append(f"Chunk {i}: Unbalanced quotes")
+        
+        return ValidationResult(
+            valid=len(errors) == 0,
+            errors=errors,
+            warnings=warnings
+        )
+    
+    def _ends_with_terminal(self, text: str) -> bool:
+        text = text.strip()
+        return text[-1] in '.!?"\'—'
+    
+    def _starts_grammatically(self, text: str) -> bool:
+        text = text.strip()
+        # Should start with capital or quote
+        return text[0].isupper() or text[0] in '"\'—'
+```
+
+## Performance Considerations
+
+| Strategy | Speed | Quality | Use Case |
+|----------|-------|---------|----------|
+| Tier 1 only | Fast | Moderate | Well-structured prose |
+| Tier 1 + Tier 2 | Moderate | High | Mixed paragraph lengths |
+| Scene-aware | Fast | High | Novels with clear scene breaks |
+| Dialogue-aware | Moderate | High | Dialogue-heavy fiction |
+
+## Edge Cases
+
+**1. Stream-of-consciousness writing**
+- Single "paragraphs" spanning pages
+- Solution: Force Tier 2 with explicit sentence boundary detection
+
+**2. Poetry or verse**
+- Line breaks are semantic, not formatting
+- Solution: Treat each stanza as atomic unit
+
+**3. Non-fiction with lists/bullets**
+- Bullet points break paragraph detection
+- Solution: Pre-process to convert bullets to prose
+
+**4. Multiple narrators**
+- Voice shifts within chapters
+- Solution: Detect narrator markers and prefer breaking there
+
+## Integration with Pipeline
+
+```python
+class SegmentationAgent:
+    def __init__(self, config: SegmentationConfig):
+        self.tier1 = Tier1Segmenter(
+            min_words=config.min_words,
+            max_words=config.max_words
+        )
+        self.tier2 = Tier2Segmenter(model=config.tier2_model)
+        self.validator = SegmentationValidator()
+    
+    async def segment(self, text: str) -> list[Chunk]:
+        # Phase 1: Tier 1 segmentation
+        chunks = self.tier1.segment(text)
+        
+        # Phase 2: Process oversized chunks with Tier 2
+        final_chunks = []
+        for chunk in chunks:
+            if chunk.requires_tier2:
+                sub_chunks = await self.tier2.segment(chunk)
+                final_chunks.extend(sub_chunks)
+            else:
+                final_chunks.append(chunk)
+        
+        # Phase 3: Validate
+        result = self.validator.validate(final_chunks)
+        if not result.valid:
+            raise SegmentationError(result.errors)
+        
+        if result.warnings:
+            logger.warning(f"Segmentation warnings: {result.warnings}")
+        
+        return final_chunks
+```
+