@stephen-lord/docs 1.0.0 → 1.0.1

package/dist-docs/Agent-Skills-for-Context-Engineering/docs/netflix_context.md

@@ -0,0 +1,10 @@
+---
+name: netflix-context-compression
+description: Video transcript from Netflix engineer discussing context compression, the three-phase approach (research, planning, implementation), and managing complexity in AI-generated code at scale.
+doc_type: video
+source_url: No
+---
+
+Video:
+
+([music] Hey everyone, good afternoon. Um, I'm going to start my talk with a bit of a confession. Uh, I've shipped code I didn't quite understand. Generated it, tested it, deployed it. Couldn't explain how it worked. And here's the thing, though. I'm willing to bet every one of you have, too. [applause] So, now I'm going to admit that we all ship code that we don't understand anymore. I want to take a bit of a journey, see how this kind of has come to be. First, look back in history. We see that history tends to repeat itself. Second, we've fallen into a bit of a trap. We've confused easy with simple. Lastly, there is a fix, but it requires us not to outsource our thinking. So, I spent the last few years at Netflix helping drive adoption of AI tools, and I have to say the acceleration is absolutely real. Backlog items that used to take days now take hours, and large refactors that have been on the books for years are finally being done. Here's the thing, though. Large production systems always fail in unexpected ways. Like, look what happened with CloudFare recently. When they do, you better understand the code you're debugging. And the problem is now we're generating code at such speed and such volume our understanding is having a hard time keeping up. Hell, I know I've done it myself. I've generated a bunch of code, looked at it, thought, I have no idea how this what this does. But, you know, the test pass, it works. So, I shipped it. The thing here is this isn't really new. Every generation of software engineers has eventually hit a wall where software complexity has exceeded their ability to manage it. We're not the fa first to face a software crisis. were the first to face it at this infinite scale of generation. So let's take a step back to see where this all started. In the late 60s, early '7s, a bunch of smart computer scientists at the time came together and said, "Hey, we're in a software crisis. We have this huge demand for software and yet we're not really able to keep up and like projects are taking too long and it's just really slow. We're not doing a good job." So Dystra Kano came up with a really great quote and he said when we had a few weak computers and I mean to paraphrase a longer quote when we had a few weak computers programming was a mild problem and now we have gigantic computers programming has become a gigantic problem. He was explaining as hardware power grew by a factor of a thousand society's wants of software grew in proportion and so it left us the programmers to figure out between the ways and the means how do we support this much more software. So this kind of keeps happening in a cycle. In the 70s we get the C programming language so we could write bigger systems. The 80s we have personal computers. Now everyone can write software. In the '9s we get object-oriented programming inheritance hierarchies from hell where you know thanks Java for that. In the 2000s we get agile and we sprints and scrum masters telling us what to do. There's no more waterfall. In the 2010s we had cloud mobile devops you know everything. Software truly ate the world. And today now we have AI. you know, co-pilot, cursor, claude, codeex, gemini, you name it. We could generate code as fast as we can describe it. The pattern continues, but the stale has really changed. It's it's infinite now. So, uh, Fred Brooks, you might know him from writing the mythical man month. He also wrote a paper in 1986 called No Silver Bullet. And in this, he argued that there'd be no single innovation that would give us an order of magnitude improvement in software productivity. Why? Because he said the hard part wasn't ever the mechanics of coding. the syntax, the typing, the boilerplate. It was about understanding the actual problem and designing the solution. And no tool can eliminate that fundamental difficulty. Every tool and technique we've created up to this point makes the mechanics easier. The core challenge though, understanding what to build, how it should work remains just as hard. So, if the problem isn't in the mechanics, why do we keep optimizing for it? How do experienced engineers end up with code they don't understand? Now, the answer, I think, comes down to two words we tend to confuse. simple and easy. We tend to use them interchangeably, but they really mean completely different things. Uh I was outed at the speaker dinner as being a closure guy, so this is kind of clear here. But Rich Hickey, the creator of the closure programming language, explained this in his talk from 2011 called simple made easy. He defined simple meaning one fold, one braid, and no entanglement. Each piece does one thing and doesn't intertwine with others. He defines easy as meaning adjacent. What's within reach? What can you access without effort? Copy paste ship. Simple is about structure. Easy is about proximity. The thing is we can't make something simple by wishing it. So simplicity requires thought, design and untangling. But we can always make something easier. You just put it closer. Install a package, generate it with AI, you know, copy a solution off of Stack Overflow. It's it's human nature to take the easy path. We're wired for it. You know, as I said, copy something from Stack Overflow. It's right there. framework that handles everything for you with magic. Install and go. But easy doesn't mean simple. Easy means you can add to your system quickly. Simple means you can understand the work that you've done. Every time we choose easy, we're choosing speed now. Complexity later. And honestly, that trade-off really used to work. The complexity accumulated in our codebases slowly enough that we can refactor, rethink, and rebuild when needed. I think AI has destroyed that balance because it's the ultimate easy bun. And it makes the easy path so frictionless that we don't even consider the simple one anymore. Why think about architecture when code appears instantly. So let me show you how this happens. How a simple task evolves into a mess of complexity through a conversational interface that we've all come to love. You know this is a contrived example but you know say we have our app. We want to add uh some authentication to it. We say add o. So we get a nice clean o.js file. Iterate on a few times it gets a message file. You're like okay cool. We're going to add OOTH now too because and now we've got an OJS and OOTHJS. We keep iterating and then we find ourselves that sessions are broken and we got a bunch of conflicts and by the time you get to turn 20, you're not really having a discussion anymore. You're managing context that become so complex that even you don't remember all the constraints that you've added to it. Dead code from abandoned approaches. Uh tests that got fixed by just making them work. You know, fragments of three different solutions because you have saying wait actually each new instruction is overwriting architectural patterns. We said make the off work here. It did. When we said fix this error, it did. There's no resistance to bad architectural decisions. The code just morphs to satisfy your latest request. Each interaction is choosing easy over simple. And easy always means more complexity. We know better. But when the easy path is just this easy, we take it. And complexity is going to compound until it's too late. AI really takes easy to its logical extreme. Decide what you want. Get code instantly. But here's the danger in that. The generated code treats every pattern in your codebase the same. You know, when an agent analyzed your codebase, every line becomes a pattern to preserve. The authentication check on line 47, that's a pattern. That weird gRPC code that's acting like GraphQL that I may have had in 2019, that's also a pattern. Technical debt doesn't register as debt. It's just more code. The real problem here is complexity. I know I've been saying that word a bunch in this talk without really defining it, but the best way to think about it is it's the opposite of simplicity. It just means intertwined. And when things are complex, everything touches everything else. You can't change one thing without affecting 10 others. So, back to Fred Brooks's no bullet paper. In it, he identified that there's two main types of complexity in every system. There's the essential complexity, which is really the fundamental difficulty of the actual problem you're trying to solve. Users need to pay for things, orders must be fulfilled. This is the complexity of why your software system exists in the first place. And then second, there's this idea of accidental complexity. Everything else we've added along the way, workarounds, defensive code, frameworks, abstractions that made sense a while ago, it's all the stuff that we put together to make the code itself work. In a real codebase, these two types of complexity are everywhere and they get so tangled together that separating them requires context, history, and experience. the generated output makes no such distinction and so every pattern is keeps just getting preserved. So here's a real example from uh some work we're doing at Netflix. I have a system that has a abstraction layer sitting between our old authorization code we wrote say five or so years ago and a new centralized o system. We didn't have time to rebuild our whole app. So we just kind of put a shim in between. So now we have AI. This is a great opportunity to refactor our code to use the new system directly. Seems like a simple request, right? And no, it's like the old code was just so tightly coupled to its authorization patterns. Like we had permission checks woven through business logic, ro assumptions baked into data models and off calls scattered across hundreds of files. The agent would start refactoring, get a few files in and hit a dependency couldn't untangle and just spiral out of control and give up or worse it would try and preserve some existing logic that from the old system and recreating it using the new system which I think is not great too. The thing is it couldn't see the scenes. It couldn't identify where the business logic ended and the off logic began. Everything was so tangled together that even with perfect information, the AI couldn't find a clean path through. When your accidental complexity gets this tangled, AI is not the best help to actually make it any better. I found it only adds more layers on top. We can tell the difference, or at least we can when we slow down enough to think. We know which patterns are essential and which are just how someone solved it a few years ago. We carry the context that the AI can infer, but only if we time to make take time to make these distinctions before we start. So how do you actually do it? How do you separate the accidental and essential complexity when you're staring at a huge codebase? Codebase I work on Netflix has around a million lines of Java and the main service in it is about 5 million tokens last time I checked. no context window I have access to uh can hold it. So when I wanted to work with it, I first thought, hey, maybe I could just copy large swaths of this codebase into the into the context and see if the patterns were emerged, see if it would just be able to figure out what's happening. And just like the authorization refactor from previously, [clears throat] the output just got lost in its own complexity. So with this, I was forced to do something different. I had to select what to include. Design docs, architecture, diagrams, key interfaces, you name it, and take time writing out the requirements of how components should interact and what patterns to follow. See, I was writing a spec. Uh 5 million tokens became 2,000 words of specification. And then to take it even further, take that spec and create an exact step set of steps of code to execute. No vague instructions, just a precise sequence of operations. I found this produced much cleaner and more focused code that I could understand. As I defined it first and planned its own execution, this became the approach which I called context compression a while ago. But you call it context engineering or spectriven development, whatever you want. The name doesn't matter. What only matters here is that thinking and planning become a majority of the work. So let me walk you through that how this works in practice. So we have step one, phase one, research. You know, I go and feed everything to it up front. Architecture diagrams, documentation, Slack threads. I been over this a bunch, but really just bring as much context as you can that's going to be relevant to the changes you're making. And then use the agent to analyze the codebase and map out the components and dependencies. This shouldn't be a oneshot process. I like to probe say like what about the caching? How does this handle failures? And when it's analysis is wrong, I'll correct it. And if it's missing context, I provide it. Each iteration refineses its analysis. The output here is a single research document. Here's what exists. Here's what connects to what. And here's what your change will affect. Hours of exploration are compressed into minutes of reading. [snorts] I know Dex mentioned it this morning, but the human checkpoint here is critical. This is where you validate the analysis against reality. The highest leverage moment in the entire process. Catch errors here. Prevent disasters later. Onto phase two. Now that you have some valid research in hand, we create a detailed imple implementation plan. Real code structure, function signatures, type definitions, data flow. You want this to be so any developer can follow it. I I kind of liken it to paint by numbers. You should be able to hand it to your most junior engineer and say, "Go do this." And if they copy it line by line, it should just work. This step is where we make a lot of the important architectural decisions. You know, make sure complex logic is correct. Make sure business requirements are, you know, following good practice. Make sure there's good service boundaries, clean separation, and preventing any unnecessary coupling. We spot the problems before they happen because we've lived through them. AI doesn't have that option. It treats every pattern as a requirement. The real magic in this step is the review speed. We can validate this plan in minutes and know exactly what's going to be built. And in order to keep up with the speed at which we want to generate code, we need to be able to comprehend what we're doing just as fast. Lastly, we have implementation. And now that we have a clear plan and like backed by a clear research, this phase should be pretty simple. And that's the point. You know, when AI has a clear specification to follow, the context remains clean and focused. We've prevented the complexity spiral of long conversations. And instead of 50 messages of evolutionary code, we have three focused outputs, each validated before proceeding. No abandoned approaches, no conflicting patterns, no wait actually moments that leave dead code everywhere. To me, what I see is the real payoff of this is that you can use a background agent to do a lot of this work because you've done all the thinking and hard work ahead of time. It can just start the implementation. You can go work on something else and come back to review and you can review this quickly because you're just verifying it's conforming to your plan, not trying to understand if anything got invented. The thing here is we're not using AI to think for us. We're using it to accelerate the mechanical parts while maintaining our ability to understand it. Research is faster, planning is more thorough, and the implementation is cleaner. The thinking, the synthesis, and the judgment though that remains with us. So remember that uh authorization refactor I said that AI couldn't handle. The thing is now we're actually, you know, working on it now starting to make some good progress on it. The thing is it's not because we found better prompts. We found we couldn't even jump into doing any sort of research, planning, implementation. We actually had to go make this change ourself by hand. No AI, just reading the code, understanding dependencies, and making changes to see what broke. That manual migration was, I'll be honest, it was a pain, but it was crucial. It revealed all the hidden constraints, which invariants had to hold true, and which services would break if the off changed. things no amount of code an analysis would have surfaced for us. And then we fed that pull request of the actual manual migration into our research process and had it use that as the seed for any sort of research going forward. The AI could then see what a clean migration looks like. The thing is each of these entities are slightly different. So we have to go and interrogate it and say hey what do we about do about this? Some things are encrypted some things are not. We had to provide that extra context each time uh through a bunch of iteration. Then and only then we could generate a plan that might work in one shot. And the key and might's the key word here is we're still validating, still adjusting, and still discovering edge cases. The three-phase approach is not magic. It only works because we did this one migration by hand. We had to earn the understanding before we can code into our process. I still think there's no silver bullet. I don't think there's better prompts, better models, or even writing better specs, just the work of understanding your system deeply enough that you can make changes to it safely. So why go through with all this? Like why not just iterate with AI until it works? Like eventually won't models get strong enough and it just works. The thing to me is it works isn't enough. There's a difference between code that passes test and code that survives in production. between systems that function today and systems that that can be changed by someone else in the future. The real problem here is a knowledge gap. When AI can generate thousands of lines of code in seconds, understanding it could take you hours, maybe days if it's complex. Who knows, maybe never if it's really that tangled. And here's something that I don't think many people are even talking about this point. Every time we skip thinking to keep up with generation speed, we're not just adding code that we don't understand. We're losing our ability to recognize problems. That instinct that says, "Hey, this is getting complex." It atrophies when you don't understand your own system. [snorts] Pattern recognition comes from experience. When I spot a dangerous architecture, it's because I'm the one up at 3:00 in the morning dealing with it. When I push for simpler solutions, it's because I've had to maintain the alternative from someone else. AI generates what you ask it for. It doesn't encode lessons from past failures. The three-phase approach bridges this gap. It compresses understanding into artifacts we can review at the speed of generation. Without it, we're just accumulating complexity faster than we can comprehend it. AI changes everything about how we write code. But honestly, I don't think it changes anything about why software itself fails. Every generation has faced their own software crisis. Dystra's generation faced it by creating the discipline of software engineering. And now we face ours with infinite code generation. I don't think the solution is another tool or methodology. It's remembering what we've always known. That software is a human endeavor. The hard part was never typing the code. It was knowing what to type in the first place. The developers who thrive won't just be the ones who generate the most code, but they'll be the ones who understand what they're building, who can still see the seams, who can recognize that they're solving the wrong problem. That's still us. That will only be us. I want to leave on a question and I don't think the question is whether or not we will use AI. That's a foregone conclusion. The ship has already sailed. To me, the question is going to be whether we will still understand our own systems when AI is writing most of our code. Thank you. [applause] [music] [music] >> [music])

package/dist-docs/Agent-Skills-for-Context-Engineering/docs/vercel_tool.md

@@ -0,0 +1,140 @@
+---
+name: vercel-tool-reduction
+description: Vercel's case study on removing 80% of their agent's specialized tools and replacing them with a single file system agent tool, resulting in 100% success rate and improved performance.
+doc_type: blog
+source_url: https://vercel.com/blog/we-removed-80-percent-of-our-agents-tools
+---
+
+We removed 80% of our agent's tools
+
+Andrew Qu
+Chief of Software, Vercel
+4 min read
+
+
+Copy URL
+Copied to clipboard!
+Dec 22, 2025
+It got better.
+
+We spent months building a sophisticated internal text-to-SQL agent, d0, with specialized tools, heavy prompt engineering, and careful context management. It worked… kind of. But it was fragile, slow, and required constant maintenance.
+
+So we tried something different. We deleted most of it and stripped the agent down to a single tool: execute arbitrary bash commands. We call this a file system agent. Claude gets direct access to your files and figures things out using grep, cat, and ls.
+
+The agent got simpler and better at the same time. 100% success rate instead of 80%. Fewer steps, fewer tokens, faster responses. All by doing less.
+
+Link to headingWhat is d0
+If v0 is our AI for building UI, d0 is our AI for understanding data.
+
+d0 enables anyone to make data-driven decisions by asking it questions in Slack
+d0 enables anyone to make data-driven decisions by asking it questions in Slack
+d0 translates natural language questions into SQL queries against our analytics infrastructure, letting anyone on the team get answers without writing code or waiting on the data team.
+
+When d0 works well, it democratizes data access across the company. When it breaks, people lose trust and go back to pinging analysts in Slack. We need d0 to be fast, accurate, and reliable.
+
+Link to headingGetting out of the model's way
+Looking back, we were solving problems the model could handle on its own. We assumed it would get lost in complex schemas, make bad joins, or hallucinate table names. So we built guardrails. We pre-filtered context, constrained its options, and wrapped every interaction in validation logic. We were doing the model’s thinking for it:
+
+Built multiple specialized tools (schema lookup, query validation, error recovery, etc.)
+
+Added heavy prompt engineering to constrain reasoning
+
+Utilized careful context management to avoid overwhelming the model
+
+Wrote hand-coded retrieval to surface “relevant” schema information and dimensional attributes
+
+Every edge case meant another patch, and every model update meant re-calibrating our constraints. We were spending more time maintaining the scaffolding than improving the agent.
+
+ai-sdk@6.0.0-beta.160 ToolLoopAgent
+
+import { ToolLoopAgent } from 'ai';
+import { GetEntityJoins, LoadCatalog, /*...*/ } from '@/lib/tools'
+const agent = new ToolLoopAgent({
+  model: "anthropic/claude-opus-4.5",
+  instructions: "",
+  tools: {
+      GetEntityJoins, LoadCatalog, RecallContext, LoadEntityDetails, 
+      SearchCatalog, ClarifyIntent, SearchSchema, GenerateAnalysisPlan, 
+      FinalizeQueryPlan, FinalizeNoData, JoinPathFinder, SyntaxValidator, 
+      FinalizeBuild, ExecuteSQL, FormatResults, VisualizeData, ExplainResults
+    },
+});
+Link to headingA new idea, what if we just… stopped?
+We realized we were fighting gravity. Constraining the model’s reasoning. Summarizing information that it could read on its own. Building tools to protect it from complexity that it could handle.
+
+So we stopped. The hypothesis was, what if we just give Claude access to the raw Cube DSL files and let it cook? What if bash is all you need? Models are getting smarter and context windows are getting larger, so maybe the best agent architecture is almost no architecture at all.
+
+Link to headingv2: The file system is the agent
+The new stack:
+
+Model: Claude Opus 4.5 via the AI SDK
+
+Execution: Vercel Sandbox for context exploration
+
+Routing: Vercel Gateway for request handling and observability
+
+Server: Next.js API route using Vercel Slack Bolt
+
+Data layer: Cube semantic layer as a directory of YAML, Markdown, and JSON files
+
+The file system agent now browses our semantic layer the way a human analyst would. It reads files, greps for patterns, builds mental models, and writes SQL using standard Unix tools like grep, cat, find, and ls.
+
+This works because the semantic layer is already great documentation. The files contain dimension definitions, measure calculations, and join relationships. We were building tools to summarize what was already legible. Claude just needed access to read it directly.
+
+ai-sdk@6.0.0-beta.160 ToolLoopAgent
+
+import { Sandbox } from "@vercel/sandbox";
+import { files } from './semantic-catalog'
+import { tool, ToolLoopAgent } from "ai";
+import { ExecuteSQL } from "@/lib/tools";}
+
+const sandbox = await Sandbox.create();
+await sandbox.writeFiles(files);
+
+const executeCommandTool(sandbox: Sandbox) {
+  return tool({
+    /* ... */
+    execute: async ({ command }) => {
+      const result = await sandbox.exec(command);
+      return { /* */ };
+    }
+  })
+}
+
+const agent = new ToolLoopAgent({
+  model: "anthropic/claude-opus-4.5",
+  instructions: "",
+  tools: {
+    ExecuteCommand: executeCommandTool(sandbox),
+    ExecuteSQL,
+   },
+})
+Link to heading3.5x faster, 37% fewer tokens, 100% success rate
+We benchmarked the old architecture against the new file system approach across 5 representative queries.
+
+Metric	Advanced (old)	File system (new)	Change
+Avg execution time	274.8s	77.4s	3.5x faster
+Success rate	4/5 (80%)	5/5 (100%)	+20%
+Avg token usage	~102k tokens	~61k tokens	37% fewer tokens
+Avg steps	~12 steps	~7 steps	42% fewer steps
+The file system agent won every comparison. The old architecture’s worst case was Query 2, which took 724 seconds, 100 steps, and 145,463 tokens before failing. The file system agent completed the same query in 141 seconds with 19 steps and 67,483 tokens, and it actually succeeded.
+
+The qualitative shift matters just as much. The agent catches edge cases we never anticipated and explains its reasoning in ways we can follow.
+
+Link to headingLessons learned
+Don’t fight gravity. File systems are an incredibly powerful abstraction. Grep is 50 years old and still does exactly what we need. We were building custom tools for what Unix already solves.
+
+We were constraining reasoning because we didn’t trust the model to reason. With Opus 4.5, that constraint became a liability. The model makes better choices when we stop making choices for it.
+
+This only worked because our semantic layer was already good documentation. The YAML files are well-structured, consistently named, and contain clear definitions. If your data layer is a mess of legacy naming conventions and undocumented joins, giving Claude raw file access won’t save you. You’ll just get faster bad queries.
+
+Addition by subtraction is real. The best agents might be the ones with the fewest tools. Every tool is a choice you’re making for the model. Sometimes the model makes better choices.
+
+Link to headingWhat this means for agent builders
+The temptation is always to account for every possibility. Resist it. Start with the simplest possible architecture. Model + file system + goal. Add complexity only when you’ve proven it’s necessary.
+
+But simple architecture isn’t enough on its own. The model needs good context to work with. Invest in documentation, clear naming, and well-structured data. That foundation matters more than clever tooling.
+
+Models are improving faster than your tooling can keep up. Build for the model that you’ll have in six months, not for the one that you have today.
+
+If you’re building agents, we’d love to hear what you’re learning.

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/README.md

@@ -0,0 +1,78 @@
+# Book SFT Pipeline
+
+A standalone skill for training language models to write in any author's style. This is a **separate plugin** from the main Context Engineering collection.
+
+## Installation
+
+### Claude Code
+
+```bash
+# Add the marketplace first
+/plugin marketplace add muratcankoylan/Agent-Skills-for-Context-Engineering
+
+# Install the book-sft-pipeline plugin
+/plugin install book-sft-pipeline@context-engineering-marketplace
+```
+
+### Cursor / Codex / IDE
+
+Copy `SKILL.md` to your `.rules` or project skills folder.
+
+### Manual
+
+Reference the `SKILL.md` file directly in your agent's context.
+
+## What's Included
+
+```
+book-sft-pipeline/
+├── README.md                 # This file
+├── SKILL.md                  # Complete skill documentation (standalone)
+├── examples/
+│   └── gertrude-stein/       # Complete case study with real outputs
+│       ├── README.md         # Results and analysis
+│       ├── sample_outputs.md # Raw model outputs
+│       ├── training_config.json
+│       ├── dataset_sample.jsonl
+│       └── pangram/          # AI detector screenshots
+├── scripts/
+│   └── pipeline_example.py   # Conceptual implementation
+└── references/
+    ├── segmentation-strategies.md
+    ├── tinker-format.md
+    └── tinker.txt
+```
+
+## Key Results
+
+Trained Qwen3-8B-Base on Gertrude Stein's "Three Lives" (1909):
+
+| Metric | Value |
+|--------|-------|
+| Training examples | 592 |
+| Loss reduction | 97% |
+| Pangram AI detector | 70% Human |
+| Training time | 15 minutes |
+| Total cost | $2 |
+
+## Related Context Engineering Skills
+
+This skill applies patterns from the [Agent Skills for Context Engineering](../../README.md) collection:
+
+| Skill | Application |
+|-------|-------------|
+| [project-development](../../skills/project-development/) | Staged pipeline architecture |
+| [context-compression](../../skills/context-compression/) | Segmentation strategy |
+| [multi-agent-patterns](../../skills/multi-agent-patterns/) | Orchestrator pattern |
+| [evaluation](../../skills/evaluation/) | Modern scenario testing |
+| [context-fundamentals](../../skills/context-fundamentals/) | Prompt diversity |
+
+## Resources
+
+- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)
+- [Research Paper](https://arxiv.org/pdf/2510.13939) (Chakrabarty et al. 2025)
+
+## License
+
+MIT
+

package/dist-docs/Agent-Skills-for-Context-Engineering/examples/book-sft-pipeline/SKILL.md

@@ -0,0 +1,380 @@
+---
+name: book-sft-pipeline
+description: This skill should be used when the user asks to "fine-tune on books", "create SFT dataset", "train style model", "extract ePub text", or mentions style transfer, LoRA training, book segmentation, or author voice replication.
+version: 2.0.0
+---
+
+# Book SFT Pipeline
+
+A complete system for converting books into SFT datasets and training style-transfer models. This skill teaches the pipeline from raw ePub to a model that writes in any author's voice.
+
+## When to Activate
+
+Activate this skill when:
+- Building fine-tuning datasets from literary works
+- Creating author-voice or style-transfer models
+- Preparing training data for Tinker or similar SFT platforms
+- Designing text segmentation pipelines for long-form content
+- Training small models (8B or less) on limited data
+
+## Core Concepts
+
+### The Three Pillars of Book SFT
+
+**1. Intelligent Segmentation**
+Text chunks must be semantically coherent. Breaking mid-sentence teaches the model to produce fragmented output. Target: 150-400 words per chunk, always at natural boundaries.
+
+**2. Diverse Instruction Generation**
+Use multiple prompt templates and system prompts to prevent overfitting. A single prompt style leads to memorization. Use 15+ prompt templates with 5+ system prompts.
+
+**3. Style Over Content**
+The goal is learning the author's rhythm and vocabulary patterns, not memorizing plots. Synthetic instructions describe what happens without quoting the text.
+
+## Pipeline Architecture
+
+```
+┌─────────────────────────────────────────────────────────────────┐
+│                    ORCHESTRATOR AGENT                           │
+│  Coordinates pipeline phases, manages state, handles failures   │
+└──────────────────────┬──────────────────────────────────────────┘
+                       │
+       ┌───────────────┼───────────────┬───────────────┐
+       ▼               ▼               ▼               ▼
+┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐
+│  EXTRACTION  │ │ SEGMENTATION │ │  INSTRUCTION │ │   DATASET    │
+│    AGENT     │ │    AGENT     │ │    AGENT     │ │   BUILDER    │
+│ ePub → Text  │ │ Text → Chunks│ │ Chunks →     │ │ Pairs →      │
+│              │ │ 150-400 words│ │ Prompts      │ │ JSONL        │
+└──────────────┘ └──────────────┘ └──────────────┘ └──────────────┘
+                       │
+       ┌───────────────┴───────────────┐
+       ▼                               ▼
+┌──────────────┐               ┌──────────────┐
+│   TRAINING   │               │  VALIDATION  │
+│    AGENT     │               │    AGENT     │
+│ LoRA on      │               │ AI detector  │
+│ Tinker       │               │ Originality  │
+└──────────────┘               └──────────────┘
+```
+
+## Phase 1: Text Extraction
+
+### Critical Rules
+1. **Always source ePub over PDF** - OCR errors become learned patterns
+2. **Use paragraph-level extraction** - Extract from `<p>` tags to preserve breaks
+3. **Remove front/back matter** - Copyright and TOC pollute the dataset
+
+```python
+# Extract text from ePub paragraphs
+from epub2 import EPub
+from bs4 import BeautifulSoup
+
+def extract_epub(path):
+    book = EPub(path)
+    chapters = []
+    for item in book.flow:
+        html = book.get_chapter(item.id)
+        soup = BeautifulSoup(html, 'html.parser')
+        paragraphs = [p.get_text().strip() for p in soup.find_all('p')]
+        chapters.append('\n\n'.join(p for p in paragraphs if p))
+    return '\n\n'.join(chapters)
+```
+
+## Phase 2: Intelligent Segmentation
+
+### Smaller Chunks + Overlap
+
+Smaller chunks (150-400 words) produce more training examples and better style transfer than larger chunks (250-650).
+
+```python
+def segment(text, min_words=150, max_words=400):
+    paragraphs = text.split('\n\n')
+    chunks, buffer, buffer_words = [], [], 0
+    
+    for para in paragraphs:
+        words = len(para.split())
+        if buffer_words + words > max_words and buffer_words >= min_words:
+            chunks.append('\n\n'.join(buffer))
+            # Keep last paragraph for overlap
+            buffer = [buffer[-1], para] if buffer else [para]
+            buffer_words = sum(len(p.split()) for p in buffer)
+        else:
+            buffer.append(para)
+            buffer_words += words
+    
+    if buffer:
+        chunks.append('\n\n'.join(buffer))
+    return chunks
+```
+
+### Expected Results
+
+For an 86,000-word book:
+- Old method (250-650 words): ~150 chunks
+- New method (150-400 + overlap): ~300 chunks
+- With 2 variants per chunk: 600+ training examples
+
+## Phase 3: Diverse Instruction Generation
+
+### The Key Insight
+
+Using a single prompt template causes memorization. Diverse templates teach the underlying style.
+
+```python
+SYSTEM_PROMPTS = [
+    "You are an expert creative writer capable of emulating specific literary styles.",
+    "You are a literary writer with deep knowledge of classic prose styles.",
+    "You are a creative writer skilled at emulating distinctive authorial voices.",
+    "You write prose that captures the essence of modernist literature.",
+    "You are a talented writer who can channel classic American authors.",
+]
+
+PROMPT_TEMPLATES = [
+    "Write a passage in the style of {author}: {desc}",
+    "Channel {author}'s voice to write about: {desc}",
+    "In {author}'s distinctive prose style, describe: {desc}",
+    "Write this scene as {author} would have: {desc}",
+    "Using {author}'s repetitive technique, describe: {desc}",
+    "Capture the rhythm of {author} in this passage: {desc}",
+    "Write like {author}: {desc}",
+    "In the voice of {author}, write: {desc}",
+    "This is a literary exercise. Write like {author}: {desc}",
+    "Can you write in {author}'s style? {desc}",
+]
+```
+
+### Instruction Generation
+
+```python
+INSTRUCTION_PROMPT = """Describe what is happening in this excerpt in 2-3 sentences.
+Focus on: characters present, actions, emotions, setting.
+Do NOT quote the text directly.
+
+Excerpt:
+{text}
+"""
+
+# Use a fast, cheap LLM (e.g., Gemini Flash)
+instruction = llm_call(INSTRUCTION_PROMPT.format(text=chunk))
+```
+
+## Phase 4: Dataset Construction
+
+### Message Format
+
+```json
+{
+    "messages": [
+        {"role": "system", "content": "You are an expert creative writer..."},
+        {"role": "user", "content": "Write in the style of Author: Scene description..."},
+        {"role": "assistant", "content": "The actual book text from chunk..."}
+    ]
+}
+```
+
+### Multiple Variants Per Chunk
+
+```python
+def build_examples(chunk, instruction, author, variants=2):
+    examples = []
+    for i in range(variants):
+        system = SYSTEM_PROMPTS[i % len(SYSTEM_PROMPTS)]
+        template = PROMPT_TEMPLATES[(chunk.id + i) % len(PROMPT_TEMPLATES)]
+        user = template.format(author=author, desc=instruction)
+        examples.append({"messages": [
+            {"role": "system", "content": system},
+            {"role": "user", "content": user},
+            {"role": "assistant", "content": chunk.text}
+        ]})
+    return examples
+```
+
+## Phase 5: LoRA Training on Tinker
+
+### Configuration
+
+```python
+CONFIG = {
+    "model_name": "Qwen/Qwen3-8B-Base",  # Base, not instruct
+    "lora_rank": 32,                      # 352MB adapter
+    "learning_rate": 5e-4,                # Higher for LoRA
+    "batch_size": 4,
+    "epochs": 3,
+}
+```
+
+### Why Base Model?
+
+Use **base** (pretrained) models, not instruction-tuned versions:
+- Base models are more malleable for new styles
+- Instruct models have patterns that resist overwriting
+- Style is a low-level pattern that base models capture better
+
+### Training Loop
+
+```python
+import tinker
+from tinker import types
+
+training_client = await service_client.create_lora_training_client_async(
+    base_model="Qwen/Qwen3-8B-Base",
+    rank=32
+)
+
+for epoch in range(3):
+    for batch in batches:
+        await training_client.forward_backward_async(batch, loss_fn="cross_entropy")
+        await training_client.optim_step_async(types.AdamParams(learning_rate=5e-4))
+
+result = await training_client.save_weights_for_sampler_async(name="final")
+```
+
+## Phase 6: Validation
+
+### Modern Scenario Test
+
+Test with scenarios that couldn't exist in the original book:
+
+```python
+TEST_PROMPTS = [
+    "Write about a barista making lattes",
+    "Describe lovers communicating through text messages",
+    "Write about someone anxious about climate change",
+]
+```
+
+If the model applies style markers to modern scenarios, it learned **style**, not **content**.
+
+### Originality Verification
+
+```bash
+# Search training data for output phrases
+grep "specific phrase from output" dataset.jsonl
+# Should return: No matches
+```
+
+### AI Detector Testing
+
+Test outputs with GPTZero, Pangram, or ZeroGPT.
+
+## Known Issues and Solutions
+
+### Character Name Leakage
+
+**Symptom**: Model uses original character names in new scenarios.
+**Cause**: Limited name diversity from one book.
+**Solution**: Train on multiple books or add synthetic examples.
+
+### Model Parrots Exact Phrases
+
+**Symptom**: Outputs contain exact sentences from training data.
+**Cause**: Too few prompt variations or too many epochs.
+**Solution**: Use 15+ templates, limit to 3 epochs.
+
+### Fragmented Outputs
+
+**Symptom**: Sentences feel incomplete.
+**Cause**: Poor segmentation breaking mid-thought.
+**Solution**: Always break at paragraph boundaries.
+
+## Guidelines
+
+1. **Always source ePub over PDF** - OCR errors become learned patterns
+2. **Never break mid-sentence** - Boundaries must be grammatically complete
+3. **Use diverse prompts** - 15+ templates, 5+ system prompts
+4. **Use base models** - Not instruct versions
+5. **Use smaller chunks** - 150-400 words for more examples
+6. **Reserve test set** - 50 examples minimum
+7. **Test on modern scenarios** - Proves style transfer vs memorization
+8. **Verify originality** - Grep training data for output phrases
+
+## Expected Results
+
+| Metric | Value |
+|--------|-------|
+| Training examples | 500-1000 per book |
+| Model | Qwen/Qwen3-8B-Base |
+| LoRA rank | 32 |
+| Adapter size | ~350 MB |
+| Training time | ~15 min |
+| Loss reduction | 90%+ |
+| Style transfer success | ~50% perfect |
+
+## Cost Estimate
+
+| Component | Cost |
+|-----------|------|
+| LLM (instruction generation) | ~$0.50 |
+| Tinker training (15 min) | ~$1.50 |
+| **Total** | **~$2.00** |
+
+## Integration with Context Engineering Skills
+
+This example applies several skills from the Agent Skills for Context Engineering collection:
+
+### project-development
+The pipeline follows the staged, idempotent architecture pattern:
+- **Acquire**: Extract text from ePub
+- **Prepare**: Segment into training chunks
+- **Process**: Generate synthetic instructions
+- **Parse**: Build message format
+- **Render**: Output Tinker-compatible JSONL
+- **Train**: LoRA fine-tuning
+- **Validate**: Modern scenario testing
+
+Each phase is resumable and produces intermediate artifacts for debugging.
+
+### context-compression
+Segmentation is a form of context compression for training. The core insight from context-compression applies: information density matters more than information quantity. Smaller, coherent chunks (150-400 words) produce better style transfer than larger, diluted chunks.
+
+The two-tier strategy mirrors context compression evaluation:
+- Tier 1: Fast, deterministic compression
+- Tier 2: LLM-assisted for edge cases
+
+### multi-agent-patterns
+The pipeline uses the **supervisor/orchestrator** pattern:
+- Orchestrator coordinates phases and manages state
+- Specialized agents (Extraction, Segmentation, Instruction, Builder) have isolated contexts
+- Each agent receives only the information needed for its task
+
+This matches the principle that sub-agents exist primarily to isolate context rather than simulate roles.
+
+### evaluation
+Validation follows the **end-state evaluation** pattern:
+- Functional testing: Does output match expected style markers?
+- Originality verification: Is content genuinely generated?
+- External validation: AI detector scores
+
+The "modern scenario" test is a form of out-of-distribution evaluation that proves generalization.
+
+### context-fundamentals
+Prompt diversity prevents attention collapse on single patterns. When training with identical prompt structures, the model memorizes the instruction-response mapping. Diverse templates force attention across the style patterns themselves.
+
+## References
+
+Internal references:
+- [Segmentation Strategies](./references/segmentation-strategies.md) - Text chunking patterns
+- [Tinker Format Specification](./references/tinker-format.md) - Datum structure
+- [Tinker API Documentation](./references/tinker.txt) - Full API reference
+
+Related skills from Agent Skills for Context Engineering:
+- project-development - Pipeline architecture patterns
+- context-compression - Compression strategies  
+- multi-agent-patterns - Agent coordination
+- evaluation - Evaluation frameworks
+- context-fundamentals - Attention and information density
+
+External resources:
+- [Research Paper](https://arxiv.org/pdf/2510.13939) - Chakrabarty et al. 2025
+- [Dataset on Hugging Face](https://huggingface.co/datasets/MuratcanKoylan/gertrude-stein-style-sft)
+- [Gertrude Stein Case Study](./examples/gertrude-stein/) - Complete working example
+
+---
+
+## Skill Metadata
+
+**Created**: 2025-12-26
+**Last Updated**: 2025-12-28
+**Author**: Muratcan Koylan
+**Version**: 2.0.0
+**Standalone**: Yes (separate from main context-engineering collection)