InfoTracker 0.2.3__tar.gz → 0.2.5__tar.gz

{infotracker-0.2.3 → infotracker-0.2.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: InfoTracker
-Version: 0.2.3
+Version: 0.2.5
 Summary: Column-level SQL lineage, impact analysis, and breaking-change detection (MS SQL first)
 Project-URL: homepage, https://example.com/infotracker
 Project-URL: documentation, https://example.com/infotracker/docs

{infotracker-0.2.3 → infotracker-0.2.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "InfoTracker"
-version = "0.2.3"
+version = "0.2.5"
 description = "Column-level SQL lineage, impact analysis, and breaking-change detection (MS SQL first)"
 readme = "README.md"
 requires-python = ">=3.10"
@@ -44,8 +44,16 @@ exclude = [
   "src/**/tests",
   "tests/",
   "tests/**",
+  "docs", "examples",
+  ".github", ".git", ".venv", ".gitignore","ProjectDescription.md",
+]
+
+[tool.hatch.build.targets.sdist]
+exclude = [
+  "tests", "tests/**",
+  "docs", "examples",
+  ".github", ".git", ".venv", ".gitignore","ProjectDescription.md",
 ]
-only-include = ["src/infotracker"]
 
 [tool.ruff]
 line-length = 100

infotracker-0.2.3/ProjectDescription.md

@@ -1,82 +0,0 @@
-### InfoTracker — Student Brief
-
-#### Welcome to the Data Dungeon (friendly, no actual monsters)
-You are the hero. Your quest: teach a tool to read SQL scrolls and tell true stories about columns. You’ll map where data comes from, spot traps (breaking changes), and keep the kingdom’s dashboards happy.
-
-- Your gear: a CLI, example SQLs, and clear docs
-- Your allies: adapters, lineage graphs, and CI checks
-- Your enemies: sneaky `SELECT *`, UNION goblins, and 3 a.m. alerts
-- Goal: green checks, clear diffs, and no broken charts
-
-If you get stuck, it’s normal. Take a sip of tea, re-read the step, try a smaller example.
-
-### For Beginners
-If you're new to SQL, try this free tutorial: [Khan Academy SQL](https://www.khanacademy.org/computing/computer-programming/sql). It's in simple English and has exercises.
-
-### Action plan (read and build in this order)
-1) Understand the goal and scope (1 hour)
-   - Overview: [docs/overview.md](docs/overview.md)
-   - What you’re building, supported features, and what’s out of scope. Keep this open as your north star.
-
-2) Learn column-level lineage basics (1-2 hours)
-   - Concepts: [docs/lineage_concepts.md](docs/lineage_concepts.md)
-   - Visual examples showing how each output column maps back to inputs (joins, transforms, aggregations). This informs how your extractor must reason.
-
-3) Explore the example dataset (your training corpus)
-   - Dataset map: [docs/example_dataset.md](docs/example_dataset.md)
-   - Where the SQL files live, what each file represents (tables, views, CTEs, procs), and the matching OpenLineage JSON expectations you must reproduce.
-
-4) Implement the algorithm incrementally
-   - Algorithm: [docs/algorithm.md](docs/algorithm.md)
-   - Steps: parse → object graph → schema resolution (expand `*` late) → column lineage extraction → impact graph → outputs.
-   - Aim for correctness on simple files first, then progress to joins and aggregations.
-
-5) Handle edge cases early enough to avoid rewrites
-   - SELECT-star and star expansion: [docs/edge_cases.md](docs/edge_cases.md)
-   - Requires object-level lineage first, then star expansion. Also watch UNION ordinals and SELECT INTO schema inference.
-
-6) Decide on architecture and adapter boundaries
-   - Adapters & extensibility: [docs/adapters.md](docs/adapters.md)
-   - Define a clear adapter interface and implement MS SQL first (temp tables, variables, SELECT INTO, T-SQL functions). Keep the core engine adapter-agnostic.
-
-7) Wire up the agentic workflow and regression tests
-   - Agentic workflow: [docs/agentic_workflow.md](docs/agentic_workflow.md)
-   - Loop the agent on the example corpus until the generated lineage matches the gold JSON. Add CI to auto-run on any SQL/lineage change.
-
-8) Expose the CLI and iterate to parity
-   - CLI usage: [docs/cli_usage.md](docs/cli_usage.md)
-   - Implement `extract`, `impact`, and `diff`. The CLI is your acceptance surface; keep behavior stable and well-documented.
-
-9) Implement breaking-change detection and reporting
-   - Breaking changes: [docs/breaking_changes.md](docs/breaking_changes.md)
-   - Compare base vs head branches: diff schemas/expressions, classify severity, compute downstream impacts, and emit machine + human-readable reports.
-
-10) Optional: Integrate with dbt
-   - dbt integration: [docs/dbt_integration.md](docs/dbt_integration.md)
-
-### Milestones (suggested timebox)
-- Day 1–2: read docs, install CLI, run extract on examples
-- Day 3–5: implement simple lineage (no joins), pass gold files
-- Day 6–8: add joins and aggregations, handle star expansion
-- Day 9–10: wire warn-only diff in CI, polish docs
-
-### Acceptance checklist
-- Lineage matches gold JSONs in `examples/warehouse/lineage`
-- Impact queries return correct columns for sample selectors
-- Diff runs in CI (warn-only) and shows helpful messages
-- Docs updated where needed; examples run without errors
-
-### Quick-start CLI (target behavior)
-Simple Example: To test one file, run `infotracker extract --sql-dir examples/warehouse/sql/01_customers.sql --out-dir build/lineage`
-
-### Tips (pro-level, easy to follow)
-- Start small: one view, then a join, then an aggregate
-- Be explicit: avoid `SELECT *` while testing
-- Commit often: small steps are easy to undo
-- Use the example JSONs as your “gold” truth
-
-### If stuck (quick help)
-- Re-read the related doc step (linked above)
-- Run the CLI with `--log-level debug` to see more info
-- Create a tiny SQL with just the failing pattern and test that first
-- Write down expected lineage for one column, then match it in code

infotracker-0.2.3/docs/adapters.md

@@ -1,117 +0,0 @@
-### Adapters and extensibility
-
-#### Forge your adapter (smithing for data heroes)
-In the forge of Integration Keep, you’ll craft adapters that turn raw SQL into neatly qualified lineage. Sparks may fly; that’s normal.
-
-- Materials: `parse`, `qualify`, `resolve`, `to_openlineage`
-- Armor enchantments: case-normalization, bracket taming, and dialect charms
-- Future artifacts: Snowflake blade, BigQuery bow, Postgres shield
-
-If an imp named “Case Insensitivity” throws a tantrum, feed it brackets: `[like_this]`.
-
-#### Audience & prerequisites
-- Audience: engineers implementing or extending dialect adapters (Level 2: After basics)
-- Prerequisites: Python; SQL basics; familiarity with SQLGlot or similar parser
-
-Define an adapter interface:
-- parse(sql) → AST
-- qualify(ast) → fully qualified refs (db.schema.object)
-- resolve(ast, catalog) → output schema + expressions
-- to_openlineage(object) → columnLineage facet
-
-MS SQL adapter (first):
-- Use `SQLGlot`/`sqllineage` for parsing/lineage hints
-- Handle T-SQL specifics: temp tables, SELECT INTO, variables, functions
-- Normalize identifiers (brackets vs quotes), case-insensitivity
-
-Future adapters: Snowflake, BigQuery, Postgres, etc. 
-
-### Adapter interface (pseudocode)
-```python
-class Adapter(Protocol):
-    name: str
-    dialect: str
-
-    def parse(self, sql: str) -> AST: ...
-    def qualify(self, ast: AST, default_db: str | None) -> AST: ...
-    def resolve(self, ast: AST, catalog: Catalog) -> tuple[Schema, ColumnLineage]: ...
-    def to_openlineage(self, obj_name: str, schema: Schema, lineage: ColumnLineage) -> dict: ...
-```
-
-### MS SQL specifics
-- Case-insensitive identifiers; bracket quoting `[name]`
-- Temp tables (`#t`) live in tempdb; scope to procedure; support SELECT INTO schema inference
-- Variables (`@v`) and their use in filters/windows; capture expressions for context
-- GETDATE/DATEADD and common built-ins; treat as CONSTANT/ARITHMETIC transformations
-- JOINs default to INNER; OUTER joins affect nullability
-- Parser: prefer SQLGlot for AST; use sqllineage as an optional hint only
-
-### Mini example (very small, illustrative)
-```python
-class MssqlAdapter(Adapter):
-    name = "mssql"
-    dialect = "tsql"
-
-    def parse(self, sql: str) -> AST:
-        return sqlglot.parse_one(sql, read=self.dialect)
-
-    def qualify(self, ast: AST, default_db: str | None) -> AST:
-        # apply name normalization and database/schema defaults
-        return qualify_identifiers(ast, default_db)
-
-    def resolve(self, ast: AST, catalog: Catalog) -> tuple[Schema, ColumnLineage]:
-        schema = infer_schema(ast, catalog)
-        lineage = extract_column_lineage(ast, catalog)
-        return schema, lineage
-
-    def to_openlineage(self, obj_name: str, schema: Schema, lineage: ColumnLineage) -> dict:
-        return build_openlineage_payload(obj_name, schema, lineage)
-```
-
-### How to Test Your Adapter
-Create a test.py:
-```python
-adapter = MssqlAdapter()
-ast = adapter.parse("SELECT * FROM table")
-print(ast)
-```
-// Run: python test.py
-
-### Adding a new adapter
-1. Implement the interface; configure SQLGlot dialect
-2. Provide normalization rules (case, quoting, name resolution)
-3. Add adapter-specific tests using a small example corpus
-4. Document limitations and differences
-
-### Adapter testing template
-- Create 3 SQL files: simple select, join with alias, aggregation with group by
-- Write expected schema (columns, types, nullability)
-- Write expected lineage (inputs per output column)
-- Run extraction and compare to expected JSON in CI 
-
-### Adapter selection and registry
-```python
-ADAPTERS: dict[str, Adapter] = {
-    "mssql": MssqlAdapter(),
-}
-
-def get_adapter(name: str) -> Adapter:
-    return ADAPTERS[name]
-```
-
-### Catalog handling
-- Accept a `catalog.yml` with known schemas for external refs
-- Use catalog to resolve `*`, disambiguate references, and provide types when DDL is missing
-- Warn on unknown objects; continue best-effort
-
-### Common pitfalls
-- Case-insensitive matching; normalize but preserve display casing
-- Bracket/quoted identifiers: `[Name]` vs `"Name"`
-- Temp table scoping and lifetime
-- SELECT INTO column ordinals and inferred types
-- Variables used in expressions and filters
-
-### See also
-- `docs/algorithm.md`
-- `docs/cli_usage.md`
-- `docs/overview.md` 

infotracker-0.2.3/docs/advanced_use_cases.md

@@ -1,294 +0,0 @@
-# Advanced Use Cases: Tabular Functions and Procedures
-
-## Overview
-This document covers advanced SQL patterns that extend beyond basic views and procedures. These patterns are important for lineage analysis because they represent reusable transformation layers that can be used as inputs to other operations.
-
-## Use Case 1: Parametrized Tabular Functions
-
-### What are Tabular Functions?
-Tabular functions are SQL Server objects that accept parameters and return table result sets. They behave like parameterized views and can be used in JOINs, subqueries, and other SQL operations.
-
-### Two Syntax Variants
-
-#### 1. Inline Table-Valued Function (RETURN AS)
-```sql
-CREATE FUNCTION dbo.fn_customer_orders_inline
-(
-    @CustomerID INT,
-    @StartDate DATE,
-    @EndDate DATE
-)
-RETURNS TABLE
-AS
-RETURN
-(
-    SELECT
-        o.OrderID,
-        o.CustomerID,
-        o.OrderDate,
-        oi.ProductID,
-        CAST(oi.Quantity * oi.UnitPrice AS DECIMAL(18,2)) AS ExtendedPrice
-    FROM dbo.Orders AS o
-    INNER JOIN dbo.OrderItems AS oi ON o.OrderID = oi.OrderID
-    WHERE o.CustomerID = @CustomerID
-      AND o.OrderDate BETWEEN @StartDate AND @EndDate
-);
-```
-
-**Characteristics:**
-- Single SELECT statement
-- Optimized by query optimizer
-- Inlined into calling query for better performance
-- Parameters control data scope
-
-#### 2. Multi-Statement Table-Valued Function (RETURN TABLE)
-```sql
-CREATE FUNCTION dbo.fn_customer_orders_mstvf
-(
-    @CustomerID INT,
-    @StartDate DATE,
-    @EndDate DATE
-)
-RETURNS @Result TABLE
-(
-    OrderID INT,
-    CustomerID INT,
-    OrderDate DATE,
-    ProductID INT,
-    ExtendedPrice DECIMAL(18,2),
-    DaysSinceOrder INT
-)
-AS
-BEGIN
-    INSERT INTO @Result
-    SELECT
-        o.OrderID,
-        o.CustomerID,
-        o.OrderDate,
-        oi.ProductID,
-        CAST(oi.Quantity * oi.UnitPrice AS DECIMAL(18,2)) AS ExtendedPrice,
-        DATEDIFF(DAY, o.OrderDate, GETDATE()) AS DaysSinceOrder
-    FROM dbo.Orders AS o
-    INNER JOIN dbo.OrderItems AS oi ON o.OrderID = oi.OrderID
-    WHERE o.CustomerID = @CustomerID
-      AND o.OrderDate BETWEEN @StartDate AND @EndDate;
-    
-    RETURN;
-END;
-```
-
-**Characteristics:**
-- Multiple statements possible
-- Table variable for intermediate results
-- More complex logic and calculations
-- Better for complex business rules
-
-### Lineage Analysis for Tabular Functions
-Tabular functions are treated as view-like objects in lineage analysis:
-
-```mermaid
-graph TD
-  O[Orders.OrderID] --> F[fn_customer_orders_inline.OrderID]
-  C[Orders.CustomerID] --> F2[fn_customer_orders_inline.CustomerID]
-  OI[OrderItems.ProductID] --> F3[fn_customer_orders_inline.ProductID]
-  Q[OrderItems.Quantity] --> F4[fn_customer_orders_inline.ExtendedPrice]
-  P[OrderItems.UnitPrice] --> F4
-```
-
-**Key Points:**
-- Function parameters affect filtering but don't create columns
-- Output columns show lineage to source table columns
-- Computed columns (like ExtendedPrice) show lineage to multiple inputs
-- Functions can be used as inputs to other operations
-
-## Use Case 2: Procedures Returning Datasets
-
-### What are Dataset-Returning Procedures?
-These are stored procedures that perform calculations and end with a SELECT statement that returns data. The results can be captured into temp tables for further processing.
-
-### Example Procedure
-```sql
-CREATE PROCEDURE dbo.usp_customer_metrics_dataset
-    @CustomerID INT = NULL,
-    @StartDate DATE = NULL,
-    @EndDate DATE = NULL,
-    @IncludeInactive BIT = 0
-AS
-BEGIN
-    SET NOCOUNT ON;
-    
-    -- Set default parameters if not provided
-    IF @StartDate IS NULL SET @StartDate = DATEADD(MONTH, -3, GETDATE());
-    IF @EndDate IS NULL SET @EndDate = GETDATE();
-    
-    -- Main calculation query that returns the dataset
-    SELECT
-        c.CustomerID,
-        c.CustomerName,
-        COUNT(DISTINCT o.OrderID) AS TotalOrders,
-        SUM(oi.Quantity * oi.UnitPrice) AS TotalRevenue,
-        MAX(o.OrderDate) AS LastOrderDate,
-        CASE 
-            WHEN MAX(o.OrderDate) >= DATEADD(MONTH, -1, GETDATE()) THEN 'Active'
-            WHEN MAX(o.OrderDate) >= DATEADD(MONTH, -3, GETDATE()) THEN 'Recent'
-            ELSE 'Inactive'
-        END AS CustomerActivityStatus
-    FROM dbo.Customers AS c
-    LEFT JOIN dbo.Orders AS o ON c.CustomerID = o.CustomerID
-    LEFT JOIN dbo.OrderItems AS oi ON o.OrderID = oi.OrderID
-    WHERE (@CustomerID IS NULL OR c.CustomerID = @CustomerID)
-      AND (@IncludeInactive = 1 OR c.IsActive = 1)
-      AND (o.OrderDate IS NULL OR o.OrderDate BETWEEN @StartDate AND @EndDate)
-    GROUP BY c.CustomerID, c.CustomerName
-    HAVING COUNT(DISTINCT o.OrderID) > 0
-    ORDER BY TotalRevenue DESC;
-END;
-```
-
-**Characteristics:**
-- Ends with a single SELECT statement
-- Parameters control data scope and filtering
-- Complex calculations and aggregations
-- Results can be captured for further processing
-
-### Usage Patterns
-
-#### 1. EXEC into Temp Table
-```sql
--- Create temp table to capture procedure output
-IF OBJECT_ID('tempdb..#customer_metrics') IS NOT NULL 
-    DROP TABLE #customer_metrics;
-
--- Execute procedure and capture results into temp table
-INSERT INTO #customer_metrics
-EXEC dbo.usp_customer_metrics_dataset 
-    @CustomerID = NULL,
-    @StartDate = '2024-01-01',
-    @EndDate = '2024-12-31',
-    @IncludeInactive = 0;
-```
-
-#### 2. Further Processing
-```sql
--- Use the temp table for downstream operations
-SELECT 
-    cm.*,
-    CASE 
-        WHEN cm.TotalRevenue >= 10000 THEN 'High Value'
-        WHEN cm.TotalRevenue >= 5000 THEN 'Medium Value'
-        ELSE 'Standard'
-    END AS CustomerTier
-FROM #customer_metrics AS cm
-WHERE cm.CustomerActivityStatus IN ('Active', 'Recent');
-```
-
-#### 3. Insert into Permanent Table
-```sql
--- Archive procedure results
-INSERT INTO dbo.customer_metrics_archive (
-    ArchiveDate, CustomerID, CustomerName, TotalOrders, TotalRevenue
-)
-SELECT 
-    GETDATE(),
-    cm.*
-FROM #customer_metrics AS cm;
-```
-
-### Lineage Analysis for Dataset Procedures
-Procedures that return datasets are treated as view-like objects:
-
-```mermaid
-graph TD
-  C[Customers.CustomerID] --> P[usp_customer_metrics_dataset.CustomerID]
-  CN[Customers.CustomerName] --> P2[usp_customer_metrics_dataset.CustomerName]
-  O[Orders.OrderID] --> P3[usp_customer_metrics_dataset.TotalOrders]
-  Q[OrderItems.Quantity] --> P4[usp_customer_metrics_dataset.TotalRevenue]
-  P5[OrderItems.UnitPrice] --> P4
-  OD[Orders.OrderDate] --> P6[usp_customer_metrics_dataset.LastOrderDate]
-  OD --> P7[usp_customer_metrics_dataset.CustomerActivityStatus]
-```
-
-**Key Points:**
-- Procedure output columns show lineage to source tables
-- Aggregated columns show lineage to source columns used in aggregation
-- Computed columns show lineage to multiple inputs
-- Procedure can become an input to other operations via temp tables
-
-## Combined Workflow Example
-
-### End-to-End Lineage
-```sql
--- Step 1: Use tabular function
-SELECT f.*, p.Category 
-FROM dbo.fn_customer_orders_inline(1, '2024-01-01', '2024-12-31') AS f
-INNER JOIN dbo.Products AS p ON f.ProductID = p.ProductID;
-
--- Step 2: Use procedure output
-INSERT INTO #customer_metrics
-EXEC dbo.usp_customer_metrics_dataset @CustomerID = 1;
-
--- Step 3: Combine both for analysis
-SELECT 
-    f.CustomerID,
-    f.OrderID,
-    f.ExtendedPrice,
-    cm.TotalRevenue,
-    CAST(f.ExtendedPrice / NULLIF(cm.TotalRevenue, 0) * 100 AS DECIMAL(5,2)) AS OrderContributionPercent
-FROM dbo.fn_customer_orders_inline(1, '2024-01-01', '2024-12-31') AS f
-INNER JOIN #customer_metrics AS cm ON f.CustomerID = cm.CustomerID;
-```
-
-### Lineage Chain
-```mermaid
-graph TD
-  O[Orders.OrderID] --> F[fn_customer_orders_inline.OrderID]
-  OI[OrderItems.Quantity] --> F2[fn_customer_orders_inline.ExtendedPrice]
-  UP[OrderItems.UnitPrice] --> F2
-  
-  C[Customers.CustomerID] --> P[usp_customer_metrics_dataset.CustomerID]
-  O2[Orders.OrderID] --> P2[usp_customer_metrics_dataset.TotalOrders]
-  
-  F --> A[Analysis.OrderID]
-  F2 --> A2[Analysis.ExtendedPrice]
-  P --> A3[Analysis.TotalRevenue]
-  P2 --> A4[Analysis.OrderContributionPercent]
-```
-
-## Implementation Considerations
-
-### For Lineage Extraction
-1. **Function Detection**: Identify `CREATE FUNCTION` statements with `RETURNS TABLE`
-2. **Procedure Detection**: Identify `CREATE PROCEDURE` statements ending with `SELECT`
-3. **Parameter Handling**: Track parameters that affect filtering but don't create columns
-4. **Output Schema**: Extract column definitions from function returns or procedure SELECT
-5. **Dependency Mapping**: Map output columns to source table columns
-
-### For Impact Analysis
-1. **Function Usage**: Track where tabular functions are used as inputs
-2. **Procedure Results**: Track where procedure outputs are captured and used
-3. **Temp Table Lifecycle**: Understand temp table creation, population, and usage
-4. **Downstream Effects**: Follow lineage from functions/procedures to final targets
-
-### Testing Scenarios
-1. **Parameter Variations**: Test functions with different parameter values
-2. **Complex Calculations**: Verify lineage for multi-step computations
-3. **Temp Table Workflows**: Test EXEC into temp table patterns
-4. **Combined Usage**: Test functions and procedures used together
-5. **Edge Cases**: Test with NULL parameters, empty results, etc.
-
-## Best Practices
-
-### For Developers
-1. **Clear Naming**: Use descriptive names for functions and procedures
-2. **Parameter Validation**: Validate input parameters early
-3. **Consistent Output**: Maintain consistent column structure
-4. **Documentation**: Document parameter effects and return values
-
-### For Lineage Analysis
-1. **Schema Tracking**: Track function and procedure schemas
-2. **Parameter Impact**: Understand how parameters affect lineage
-3. **Temp Table Handling**: Properly track temp table creation and usage
-4. **Performance Considerations**: Be aware of function inlining and optimization
-
-## Conclusion
-Tabular functions and dataset-returning procedures represent powerful patterns for creating reusable transformation layers. They extend the lineage analysis beyond simple views and tables, requiring careful handling of parameters, computed columns, and complex workflows. Proper implementation of these patterns enables comprehensive data lineage tracking across modern SQL Server environments.

infotracker-0.2.3/docs/agentic_workflow.md

@@ -1,134 +0,0 @@
-### Agentic workflow and regression tests
-
-#### Train your lineage familiar (it learns by fetching JSON)
-Summon your agent, toss it SQL scrolls, and reward it when it returns with matching OpenLineage scrolls. Repeat until it purrs (tests pass).
-
-- The loop: cast → compare → tweak → repeat
-- The arena: `examples/warehouse/{sql,lineage}`
-- Victory condition: exact matches, zero diffs, tests pass (green)
-
-Remember: agents love clear acceptance criteria more than tuna.
-
-#### Audience & prerequisites
-- Audience: engineers using agents/CIs to iterate on lineage extractors
-- Prerequisites: basic SQL; Python; familiarity with CI and diff workflows
-
-### Gold files (recap)
-- Gold files = expected JSON lineage in `examples/warehouse/lineage`
-- Your extractor must match them exactly (order and content)
-
-### Fixing diffs (common)
-- If a column mismatches: compare expressions; check alias qualification and star expansion timing
-- If extra/missing columns: check join resolution and GROUP BY; ensure inputs are resolved before expansion
-- If ordering differs: make outputs and diagnostics deterministic
-
-- Prepare training set: SQL files + expected OpenLineage JSONs
-- Loop (Cursor AI/CLI/web agents):
-  1) Generate lineage → 2) Compare with expected → 3) Adjust prompts/code → 4) Repeat until pass
-- CI: on any change under `examples/warehouse/{sql,lineage}`, run extraction and compare; fail on diffs
-- Track coverage and edge cases (SELECT *, temp tables, UNION, variables) 
-
-### Setup
-- Install Cursor CLI and authenticate
-- Organize repo with `examples/warehouse/{sql,lineage}` and a `build/` output folder
-
-### Agent loop
-1. Prompt template includes: adapter target (MS SQL), acceptance criteria (must match gold JSON), and allowed libraries (SQLGlot)
-2. Agent writes code to `src/` and runs `infotracker extract` on the SQL corpus
-3. Compare `build/lineage/*.json` to `examples/warehouse/lineage/*.json`
-4. If diff exists, agent refines parsing/resolution rules and retries
-5. Stop condition: all files match; record commit checkpoint
-
-### Loop diagram
-```mermaid
-flowchart LR
-  G[Generate lineage] --> C[Compare to gold JSONs]
-  C -->|diffs| R[Refine rules/prompts]
-  R --> G
-  C -->|no diffs| S[Stop (green)]
-```
-
-### Artifacts
-- Inputs: SQL corpus under `examples/warehouse/sql`, optional catalog
-- Outputs: `build/lineage/*.json`, diff logs, warnings
-- CI artifacts: upload generated lineage for review
-
-### Stop criteria
-- All gold JSONs match exactly (order and content)
-- No warnings if using `--fail-on-warn`
-
-### CI integration
-- GitHub Actions (example): on push/PR, run extraction and `git diff --no-index` against gold lineage; fail on differences
-- Cache Python deps and AST caches for speed
-- Upload generated `build/lineage/*.json` as CI artifacts for review
-
-### Evaluation metrics
-- Exact-match rate across files
-- Column coverage (percentage of outputs with lineage)
-- Warning/error counts should trend down across iterations
-
-### Updating gold files
-- Intentional changes: regenerate lineage and review diffs; update gold JSON with PR describing the change 
-
-### See also
-- `docs/example_dataset.md`
-- `docs/algorithm.md`
-- `docs/cli_usage.md`
-- `docs/dbt_integration.md`
-
-### Modus Operandi: Continuous Improvement with Cursor Agents
-To extend the agentic workflow for 24/7/365 improvement of InfoTracker, integrate Cursor's web-based agents (like Background Agents) with GitHub. This builds on the regression testing and CI loops above, enabling automated code suggestions, bug fixes, and PR reviews. See [Cursor Changelog](https://cursor.com/changelog) for details.
-
-#### Step 1: Set Up Cursor Web Agents for Continuous Improvement
-1. **Enable Background Agents:** In Cursor, use Background Agents which run remotely and can be triggered via web (e.g., GitHub or Slack).
-2. **Integrate with GitHub for 24/7 Operation:** Use GitHub Actions to schedule agent runs, combined with Cursor's API for AI tasks.
-   - Create a workflow: `.github/workflows/cursor-improve.yml` to run daily.
-   - Use Cursor's features like tagging @Cursor in issues for automated suggestions.
-3. **Example Workflow for Scheduled Improvements:**
-   ```yaml
-   name: Cursor AI Improvement
-   on: schedule
-     - cron: '0 0 * * *' # Daily at midnight
-   jobs:
-     improve:
-       runs-on: ubuntu-latest
-       steps:
-         - uses: actions/checkout@v4
-         - name: Run Cursor Agent
-           env:
-             CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} # Add in GitHub Secrets
-           run: |
-             # Script to call Cursor API or simulate agent for code analysis
-             python cursor_improve.py # Prompt agent to suggest repo improvements
-   ```
-4. **Script (cursor_improve.py):** Use Cursor's API to analyze code and open issues/PRs with suggestions.
-
-#### Step 2: Set Up Cursor Agents for PR Reviews
-1. **GitHub PR Integration:** In GitHub, tag @Cursor in PR comments to trigger Background Agent for reviews and fixes.
-   - Example: In a PR, comment "@Cursor review this for bugs" – it will analyze and suggest changes.
-2. **Automate with Workflow:** Trigger on PR events to auto-invoke Cursor.
-   ```yaml
-   name: Cursor PR Review
-   on: pull_request
-   jobs:
-     review:
-       runs-on: ubuntu-latest
-       steps:
-         - uses: actions/checkout@v4
-         - name: Invoke Cursor Agent
-           run: |
-             # Use GitHub API to comment "@Cursor review" on the PR
-             gh pr comment $PR_NUMBER --body "@Cursor review this PR for improvements"
-   ```
-3. **How It Works:** Cursor's Background Agent will read the PR, apply fixes if needed, and push commits (per changelog features).
-
-#### Safeguards to Prevent Breaking Working Code
-Constant improvements are great, but we must avoid breaking things that already work well. Here's how to build safety into the process:
-
-- **Regression Testing:** Always run tests against gold standards (e.g., example JSONs in `examples/warehouse/lineage`). If an agent's suggestion changes outputs, reject it unless intentionally updating the gold.
-- **CI/CD Pipelines:** Set up automated tests in GitHub Actions to run on every PR or scheduled run. Fail the build if tests break, catching issues early.
-- **Human Oversight:** Agents suggest changes—review and approve them manually before merging. Use PR reviews to double-check.
-- **Modular Changes:** Limit agent tasks to small, isolated improvements (e.g., one file at a time) to minimize risk.
-- **Monitoring and Rollback:** Track metrics like test pass rates. Use Git for easy rollbacks if something breaks.
-
-Integrate these into workflows: Add test steps to the example YAML files, and prompt agents with "Suggest improvements without changing existing correct behavior."