InfoTracker 0.1.0__tar.gz

infotracker-0.1.0/.gitignore

@@ -0,0 +1,42 @@
+# Środowiska wirtualne
+.venv/
+infotracker-env/
+
+# Pliki i katalogi build
+build/
+dist/
+*.egg-info/
+
+# Cache Pythona
+__pycache__/
+*.pyc
+*.pyo
+*.pyd
+
+# Pliki systemowe
+.DS_Store
+Thumbs.db
+
+# Pliki konfiguracyjne IDE
+.vscode/
+.idea/
+
+# Logi
+*.log
+
+# Zrzuty i pliki tymczasowe
+*.tmp
+*.temp
+
+# Pliki wygenerowane automatycznie
+docs/_build/
+
+# Ignoruj wszystkie pliki Pythona z 'test' lub 'phase' w nazwie
+(?i)*test*.py
+(?i)*phase*.py
+(?i)*test*.md
+(?i)*phase*.md
+InfoTracker/PHASE2_COMPLETE.md
+InfoTracker/PHASE3_PLAN.md
+InfoTracker/test_phase2.py
+InfoTracker/test_comprehensive.py

infotracker-0.1.0/PKG-INFO

@@ -0,0 +1,108 @@
+Metadata-Version: 2.4
+Name: InfoTracker
+Version: 0.1.0
+Summary: Column-level SQL lineage, impact analysis, and breaking-change detection (MS SQL first)
+Project-URL: homepage, https://example.com/infotracker
+Project-URL: documentation, https://example.com/infotracker/docs
+Author: InfoTracker Authors
+License: MIT
+Keywords: data-lineage,impact-analysis,lineage,mssql,openlineage,sql
+Classifier: Environment :: Console
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: <3.13,>=3.10
+Requires-Dist: click<9.0.0,>=8.1.3
+Requires-Dist: networkx>=3.3
+Requires-Dist: packaging>=24.0
+Requires-Dist: pydantic>=2.8.2
+Requires-Dist: pyyaml>=6.0.1
+Requires-Dist: sqlglot>=23.0.0
+Requires-Dist: typer[all]==0.12.3
+Provides-Extra: dev
+Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
+Requires-Dist: pytest>=7.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+### InfoTracker
+
+This is a Python CLI that extracts column-level lineage from SQL, runs impact analysis, and detects breaking changes. First adapter targets MS SQL.
+
+#### For Students
+Start with a simple command: `infotracker extract --sql-dir examples/warehouse/sql --out-dir build/lineage`. This analyzes SQL files in the directory.
+
+#### Setup & Installation
+```bash
+# Activate virtual environment first (REQUIRED)
+source infotracker-env/bin/activate  # or your venv path
+
+# Install dependencies
+pip install -e .
+
+# Verify installation
+infotracker --help
+```
+
+#### Quickstart
+```bash
+# IMPORTANT: Always run InfoTracker commands in the activated virtual environment
+
+# Extract lineage from all SQL files
+infotracker extract --sql-dir examples/warehouse/sql --out-dir build/lineage
+
+# Impact analysis (downstream dependencies)
+infotracker impact -s dbo.fct_sales.Revenue+
+
+# Impact analysis (upstream sources)
+infotracker impact -s +dbo.Orders.OrderID
+
+# Branch diff for breaking changes
+infotracker diff --base main --head feature/x --sql-dir examples/warehouse/sql
+```
+
+#### Configuration
+InfoTracker follows this configuration precedence:
+1. **CLI flags** (highest priority) - override everything
+2. **infotracker.yml** config file - project defaults  
+3. **Built-in defaults** (lowest priority) - fallback values
+
+Create an `infotracker.yml` file in your project root:
+```yaml
+default_adapter: mssql
+sql_dir: examples/warehouse/sql
+out_dir: build/lineage
+include: ["*.sql"]
+exclude: ["*_wip.sql"]
+severity_threshold: BREAKING
+```
+
+#### Documentation
+- `docs/overview.md` — what it is, goals, scope
+- `docs/algorithm.md` — how extraction works
+- `docs/lineage_concepts.md` — core concepts with visuals
+- `docs/cli_usage.md` — commands and options
+- `docs/breaking_changes.md` — definition and detection
+- `docs/edge_cases.md` — SELECT *, UNION, temp tables, etc.
+- `docs/adapters.md` — interface and MSSQL specifics
+- `docs/architecture.md` — system and sequence diagrams
+- `docs/configuration.md` — configuration reference
+- `docs/openlineage_mapping.md` — how outputs map to OpenLineage
+- `docs/faq.md` — common questions
+- `docs/dbt_integration.md` — how to use with dbt projects
+
+#### Requirements
+- Python 3.10+
+- Virtual environment (activated)
+- Basic SQL knowledge
+- Git and shell
+
+#### Troubleshooting
+- **Error tracebacks on help commands**: Make sure you're running in an activated virtual environment
+- **Command not found**: Activate your virtual environment first
+- **Import errors**: Ensure all dependencies are installed with `pip install -e .`
+
+#### License
+MIT (or your team’s preferred license) 

infotracker-0.1.0/ProjectDescription.md

@@ -0,0 +1,82 @@
+### InfoTracker — Student Brief
+
+#### Welcome to the Data Dungeon (friendly, no actual monsters)
+You are the hero. Your quest: teach a tool to read SQL scrolls and tell true stories about columns. You’ll map where data comes from, spot traps (breaking changes), and keep the kingdom’s dashboards happy.
+
+- Your gear: a CLI, example SQLs, and clear docs
+- Your allies: adapters, lineage graphs, and CI checks
+- Your enemies: sneaky `SELECT *`, UNION goblins, and 3 a.m. alerts
+- Goal: green checks, clear diffs, and no broken charts
+
+If you get stuck, it’s normal. Take a sip of tea, re-read the step, try a smaller example.
+
+### For Beginners
+If you're new to SQL, try this free tutorial: [Khan Academy SQL](https://www.khanacademy.org/computing/computer-programming/sql). It's in simple English and has exercises.
+
+### Action plan (read and build in this order)
+1) Understand the goal and scope (1 hour)
+   - Overview: [docs/overview.md](docs/overview.md)
+   - What you’re building, supported features, and what’s out of scope. Keep this open as your north star.
+
+2) Learn column-level lineage basics (1-2 hours)
+   - Concepts: [docs/lineage_concepts.md](docs/lineage_concepts.md)
+   - Visual examples showing how each output column maps back to inputs (joins, transforms, aggregations). This informs how your extractor must reason.
+
+3) Explore the example dataset (your training corpus)
+   - Dataset map: [docs/example_dataset.md](docs/example_dataset.md)
+   - Where the SQL files live, what each file represents (tables, views, CTEs, procs), and the matching OpenLineage JSON expectations you must reproduce.
+
+4) Implement the algorithm incrementally
+   - Algorithm: [docs/algorithm.md](docs/algorithm.md)
+   - Steps: parse → object graph → schema resolution (expand `*` late) → column lineage extraction → impact graph → outputs.
+   - Aim for correctness on simple files first, then progress to joins and aggregations.
+
+5) Handle edge cases early enough to avoid rewrites
+   - SELECT-star and star expansion: [docs/edge_cases.md](docs/edge_cases.md)
+   - Requires object-level lineage first, then star expansion. Also watch UNION ordinals and SELECT INTO schema inference.
+
+6) Decide on architecture and adapter boundaries
+   - Adapters & extensibility: [docs/adapters.md](docs/adapters.md)
+   - Define a clear adapter interface and implement MS SQL first (temp tables, variables, SELECT INTO, T-SQL functions). Keep the core engine adapter-agnostic.
+
+7) Wire up the agentic workflow and regression tests
+   - Agentic workflow: [docs/agentic_workflow.md](docs/agentic_workflow.md)
+   - Loop the agent on the example corpus until the generated lineage matches the gold JSON. Add CI to auto-run on any SQL/lineage change.
+
+8) Expose the CLI and iterate to parity
+   - CLI usage: [docs/cli_usage.md](docs/cli_usage.md)
+   - Implement `extract`, `impact`, and `diff`. The CLI is your acceptance surface; keep behavior stable and well-documented.
+
+9) Implement breaking-change detection and reporting
+   - Breaking changes: [docs/breaking_changes.md](docs/breaking_changes.md)
+   - Compare base vs head branches: diff schemas/expressions, classify severity, compute downstream impacts, and emit machine + human-readable reports.
+
+10) Optional: Integrate with dbt
+   - dbt integration: [docs/dbt_integration.md](docs/dbt_integration.md)
+
+### Milestones (suggested timebox)
+- Day 1–2: read docs, install CLI, run extract on examples
+- Day 3–5: implement simple lineage (no joins), pass gold files
+- Day 6–8: add joins and aggregations, handle star expansion
+- Day 9–10: wire warn-only diff in CI, polish docs
+
+### Acceptance checklist
+- Lineage matches gold JSONs in `examples/warehouse/lineage`
+- Impact queries return correct columns for sample selectors
+- Diff runs in CI (warn-only) and shows helpful messages
+- Docs updated where needed; examples run without errors
+
+### Quick-start CLI (target behavior)
+Simple Example: To test one file, run `infotracker extract --sql-dir examples/warehouse/sql/01_customers.sql --out-dir build/lineage`
+
+### Tips (pro-level, easy to follow)
+- Start small: one view, then a join, then an aggregate
+- Be explicit: avoid `SELECT *` while testing
+- Commit often: small steps are easy to undo
+- Use the example JSONs as your “gold” truth
+
+### If stuck (quick help)
+- Re-read the related doc step (linked above)
+- Run the CLI with `--log-level debug` to see more info
+- Create a tiny SQL with just the failing pattern and test that first
+- Write down expected lineage for one column, then match it in code

infotracker-0.1.0/README.md

@@ -0,0 +1,79 @@
+### InfoTracker
+
+This is a Python CLI that extracts column-level lineage from SQL, runs impact analysis, and detects breaking changes. First adapter targets MS SQL.
+
+#### For Students
+Start with a simple command: `infotracker extract --sql-dir examples/warehouse/sql --out-dir build/lineage`. This analyzes SQL files in the directory.
+
+#### Setup & Installation
+```bash
+# Activate virtual environment first (REQUIRED)
+source infotracker-env/bin/activate  # or your venv path
+
+# Install dependencies
+pip install -e .
+
+# Verify installation
+infotracker --help
+```
+
+#### Quickstart
+```bash
+# IMPORTANT: Always run InfoTracker commands in the activated virtual environment
+
+# Extract lineage from all SQL files
+infotracker extract --sql-dir examples/warehouse/sql --out-dir build/lineage
+
+# Impact analysis (downstream dependencies)
+infotracker impact -s dbo.fct_sales.Revenue+
+
+# Impact analysis (upstream sources)
+infotracker impact -s +dbo.Orders.OrderID
+
+# Branch diff for breaking changes
+infotracker diff --base main --head feature/x --sql-dir examples/warehouse/sql
+```
+
+#### Configuration
+InfoTracker follows this configuration precedence:
+1. **CLI flags** (highest priority) - override everything
+2. **infotracker.yml** config file - project defaults  
+3. **Built-in defaults** (lowest priority) - fallback values
+
+Create an `infotracker.yml` file in your project root:
+```yaml
+default_adapter: mssql
+sql_dir: examples/warehouse/sql
+out_dir: build/lineage
+include: ["*.sql"]
+exclude: ["*_wip.sql"]
+severity_threshold: BREAKING
+```
+
+#### Documentation
+- `docs/overview.md` — what it is, goals, scope
+- `docs/algorithm.md` — how extraction works
+- `docs/lineage_concepts.md` — core concepts with visuals
+- `docs/cli_usage.md` — commands and options
+- `docs/breaking_changes.md` — definition and detection
+- `docs/edge_cases.md` — SELECT *, UNION, temp tables, etc.
+- `docs/adapters.md` — interface and MSSQL specifics
+- `docs/architecture.md` — system and sequence diagrams
+- `docs/configuration.md` — configuration reference
+- `docs/openlineage_mapping.md` — how outputs map to OpenLineage
+- `docs/faq.md` — common questions
+- `docs/dbt_integration.md` — how to use with dbt projects
+
+#### Requirements
+- Python 3.10+
+- Virtual environment (activated)
+- Basic SQL knowledge
+- Git and shell
+
+#### Troubleshooting
+- **Error tracebacks on help commands**: Make sure you're running in an activated virtual environment
+- **Command not found**: Activate your virtual environment first
+- **Import errors**: Ensure all dependencies are installed with `pip install -e .`
+
+#### License
+MIT (or your team’s preferred license) 

infotracker-0.1.0/docs/adapters.md

@@ -0,0 +1,117 @@
+### Adapters and extensibility
+
+#### Forge your adapter (smithing for data heroes)
+In the forge of Integration Keep, you’ll craft adapters that turn raw SQL into neatly qualified lineage. Sparks may fly; that’s normal.
+
+- Materials: `parse`, `qualify`, `resolve`, `to_openlineage`
+- Armor enchantments: case-normalization, bracket taming, and dialect charms
+- Future artifacts: Snowflake blade, BigQuery bow, Postgres shield
+
+If an imp named “Case Insensitivity” throws a tantrum, feed it brackets: `[like_this]`.
+
+#### Audience & prerequisites
+- Audience: engineers implementing or extending dialect adapters (Level 2: After basics)
+- Prerequisites: Python; SQL basics; familiarity with SQLGlot or similar parser
+
+Define an adapter interface:
+- parse(sql) → AST
+- qualify(ast) → fully qualified refs (db.schema.object)
+- resolve(ast, catalog) → output schema + expressions
+- to_openlineage(object) → columnLineage facet
+
+MS SQL adapter (first):
+- Use `SQLGlot`/`sqllineage` for parsing/lineage hints
+- Handle T-SQL specifics: temp tables, SELECT INTO, variables, functions
+- Normalize identifiers (brackets vs quotes), case-insensitivity
+
+Future adapters: Snowflake, BigQuery, Postgres, etc. 
+
+### Adapter interface (pseudocode)
+```python
+class Adapter(Protocol):
+    name: str
+    dialect: str
+
+    def parse(self, sql: str) -> AST: ...
+    def qualify(self, ast: AST, default_db: str | None) -> AST: ...
+    def resolve(self, ast: AST, catalog: Catalog) -> tuple[Schema, ColumnLineage]: ...
+    def to_openlineage(self, obj_name: str, schema: Schema, lineage: ColumnLineage) -> dict: ...
+```
+
+### MS SQL specifics
+- Case-insensitive identifiers; bracket quoting `[name]`
+- Temp tables (`#t`) live in tempdb; scope to procedure; support SELECT INTO schema inference
+- Variables (`@v`) and their use in filters/windows; capture expressions for context
+- GETDATE/DATEADD and common built-ins; treat as CONSTANT/ARITHMETIC transformations
+- JOINs default to INNER; OUTER joins affect nullability
+- Parser: prefer SQLGlot for AST; use sqllineage as an optional hint only
+
+### Mini example (very small, illustrative)
+```python
+class MssqlAdapter(Adapter):
+    name = "mssql"
+    dialect = "tsql"
+
+    def parse(self, sql: str) -> AST:
+        return sqlglot.parse_one(sql, read=self.dialect)
+
+    def qualify(self, ast: AST, default_db: str | None) -> AST:
+        # apply name normalization and database/schema defaults
+        return qualify_identifiers(ast, default_db)
+
+    def resolve(self, ast: AST, catalog: Catalog) -> tuple[Schema, ColumnLineage]:
+        schema = infer_schema(ast, catalog)
+        lineage = extract_column_lineage(ast, catalog)
+        return schema, lineage
+
+    def to_openlineage(self, obj_name: str, schema: Schema, lineage: ColumnLineage) -> dict:
+        return build_openlineage_payload(obj_name, schema, lineage)
+```
+
+### How to Test Your Adapter
+Create a test.py:
+```python
+adapter = MssqlAdapter()
+ast = adapter.parse("SELECT * FROM table")
+print(ast)
+```
+// Run: python test.py
+
+### Adding a new adapter
+1. Implement the interface; configure SQLGlot dialect
+2. Provide normalization rules (case, quoting, name resolution)
+3. Add adapter-specific tests using a small example corpus
+4. Document limitations and differences
+
+### Adapter testing template
+- Create 3 SQL files: simple select, join with alias, aggregation with group by
+- Write expected schema (columns, types, nullability)
+- Write expected lineage (inputs per output column)
+- Run extraction and compare to expected JSON in CI 
+
+### Adapter selection and registry
+```python
+ADAPTERS: dict[str, Adapter] = {
+    "mssql": MssqlAdapter(),
+}
+
+def get_adapter(name: str) -> Adapter:
+    return ADAPTERS[name]
+```
+
+### Catalog handling
+- Accept a `catalog.yml` with known schemas for external refs
+- Use catalog to resolve `*`, disambiguate references, and provide types when DDL is missing
+- Warn on unknown objects; continue best-effort
+
+### Common pitfalls
+- Case-insensitive matching; normalize but preserve display casing
+- Bracket/quoted identifiers: `[Name]` vs `"Name"`
+- Temp table scoping and lifetime
+- SELECT INTO column ordinals and inferred types
+- Variables used in expressions and filters
+
+### See also
+- `docs/algorithm.md`
+- `docs/cli_usage.md`
+- `docs/overview.md` 

infotracker-0.1.0/docs/agentic_workflow.md

@@ -0,0 +1,134 @@
+### Agentic workflow and regression tests
+
+#### Train your lineage familiar (it learns by fetching JSON)
+Summon your agent, toss it SQL scrolls, and reward it when it returns with matching OpenLineage scrolls. Repeat until it purrs (tests pass).
+
+- The loop: cast → compare → tweak → repeat
+- The arena: `examples/warehouse/{sql,lineage}`
+- Victory condition: exact matches, zero diffs, tests pass (green)
+
+Remember: agents love clear acceptance criteria more than tuna.
+
+#### Audience & prerequisites
+- Audience: engineers using agents/CIs to iterate on lineage extractors
+- Prerequisites: basic SQL; Python; familiarity with CI and diff workflows
+
+### Gold files (recap)
+- Gold files = expected JSON lineage in `examples/warehouse/lineage`
+- Your extractor must match them exactly (order and content)
+
+### Fixing diffs (common)
+- If a column mismatches: compare expressions; check alias qualification and star expansion timing
+- If extra/missing columns: check join resolution and GROUP BY; ensure inputs are resolved before expansion
+- If ordering differs: make outputs and diagnostics deterministic
+
+- Prepare training set: SQL files + expected OpenLineage JSONs
+- Loop (Cursor AI/CLI/web agents):
+  1) Generate lineage → 2) Compare with expected → 3) Adjust prompts/code → 4) Repeat until pass
+- CI: on any change under `examples/warehouse/{sql,lineage}`, run extraction and compare; fail on diffs
+- Track coverage and edge cases (SELECT *, temp tables, UNION, variables) 
+
+### Setup
+- Install Cursor CLI and authenticate
+- Organize repo with `examples/warehouse/{sql,lineage}` and a `build/` output folder
+
+### Agent loop
+1. Prompt template includes: adapter target (MS SQL), acceptance criteria (must match gold JSON), and allowed libraries (SQLGlot)
+2. Agent writes code to `src/` and runs `infotracker extract` on the SQL corpus
+3. Compare `build/lineage/*.json` to `examples/warehouse/lineage/*.json`
+4. If diff exists, agent refines parsing/resolution rules and retries
+5. Stop condition: all files match; record commit checkpoint
+
+### Loop diagram
+```mermaid
+flowchart LR
+  G[Generate lineage] --> C[Compare to gold JSONs]
+  C -->|diffs| R[Refine rules/prompts]
+  R --> G
+  C -->|no diffs| S[Stop (green)]
+```
+
+### Artifacts
+- Inputs: SQL corpus under `examples/warehouse/sql`, optional catalog
+- Outputs: `build/lineage/*.json`, diff logs, warnings
+- CI artifacts: upload generated lineage for review
+
+### Stop criteria
+- All gold JSONs match exactly (order and content)
+- No warnings if using `--fail-on-warn`
+
+### CI integration
+- GitHub Actions (example): on push/PR, run extraction and `git diff --no-index` against gold lineage; fail on differences
+- Cache Python deps and AST caches for speed
+- Upload generated `build/lineage/*.json` as CI artifacts for review
+
+### Evaluation metrics
+- Exact-match rate across files
+- Column coverage (percentage of outputs with lineage)
+- Warning/error counts should trend down across iterations
+
+### Updating gold files
+- Intentional changes: regenerate lineage and review diffs; update gold JSON with PR describing the change 
+
+### See also
+- `docs/example_dataset.md`
+- `docs/algorithm.md`
+- `docs/cli_usage.md`
+- `docs/dbt_integration.md`
+
+### Modus Operandi: Continuous Improvement with Cursor Agents
+To extend the agentic workflow for 24/7/365 improvement of InfoTracker, integrate Cursor's web-based agents (like Background Agents) with GitHub. This builds on the regression testing and CI loops above, enabling automated code suggestions, bug fixes, and PR reviews. See [Cursor Changelog](https://cursor.com/changelog) for details.
+
+#### Step 1: Set Up Cursor Web Agents for Continuous Improvement
+1. **Enable Background Agents:** In Cursor, use Background Agents which run remotely and can be triggered via web (e.g., GitHub or Slack).
+2. **Integrate with GitHub for 24/7 Operation:** Use GitHub Actions to schedule agent runs, combined with Cursor's API for AI tasks.
+   - Create a workflow: `.github/workflows/cursor-improve.yml` to run daily.
+   - Use Cursor's features like tagging @Cursor in issues for automated suggestions.
+3. **Example Workflow for Scheduled Improvements:**
+   ```yaml
+   name: Cursor AI Improvement
+   on: schedule
+     - cron: '0 0 * * *' # Daily at midnight
+   jobs:
+     improve:
+       runs-on: ubuntu-latest
+       steps:
+         - uses: actions/checkout@v4
+         - name: Run Cursor Agent
+           env:
+             CURSOR_API_KEY: ${{ secrets.CURSOR_API_KEY }} # Add in GitHub Secrets
+           run: |
+             # Script to call Cursor API or simulate agent for code analysis
+             python cursor_improve.py # Prompt agent to suggest repo improvements
+   ```
+4. **Script (cursor_improve.py):** Use Cursor's API to analyze code and open issues/PRs with suggestions.
+
+#### Step 2: Set Up Cursor Agents for PR Reviews
+1. **GitHub PR Integration:** In GitHub, tag @Cursor in PR comments to trigger Background Agent for reviews and fixes.
+   - Example: In a PR, comment "@Cursor review this for bugs" – it will analyze and suggest changes.
+2. **Automate with Workflow:** Trigger on PR events to auto-invoke Cursor.
+   ```yaml
+   name: Cursor PR Review
+   on: pull_request
+   jobs:
+     review:
+       runs-on: ubuntu-latest
+       steps:
+         - uses: actions/checkout@v4
+         - name: Invoke Cursor Agent
+           run: |
+             # Use GitHub API to comment "@Cursor review" on the PR
+             gh pr comment $PR_NUMBER --body "@Cursor review this PR for improvements"
+   ```
+3. **How It Works:** Cursor's Background Agent will read the PR, apply fixes if needed, and push commits (per changelog features).
+
+#### Safeguards to Prevent Breaking Working Code
+Constant improvements are great, but we must avoid breaking things that already work well. Here's how to build safety into the process:
+
+- **Regression Testing:** Always run tests against gold standards (e.g., example JSONs in `examples/warehouse/lineage`). If an agent's suggestion changes outputs, reject it unless intentionally updating the gold.
+- **CI/CD Pipelines:** Set up automated tests in GitHub Actions to run on every PR or scheduled run. Fail the build if tests break, catching issues early.
+- **Human Oversight:** Agents suggest changes—review and approve them manually before merging. Use PR reviews to double-check.
+- **Modular Changes:** Limit agent tasks to small, isolated improvements (e.g., one file at a time) to minimize risk.
+- **Monitoring and Rollback:** Track metrics like test pass rates. Use Git for easy rollbacks if something breaks.
+
+Integrate these into workflows: Add test steps to the example YAML files, and prompt agents with "Suggest improvements without changing existing correct behavior." 

infotracker-0.1.0/docs/algorithm.md

@@ -0,0 +1,109 @@
+### High-level algorithm
+
+#### Map the labyrinth (bring string, not MINUS)
+To escape the SQL maze, you’ll build maps: object graphs, schemas, and column lineage webs. One step at a time, no special math needed.
+
+- Scout terrain: parse files → objects → dependencies
+- Place torches: resolve schemas, expand stars where safe
+- Trace footprints: build column graph for impact and diff
+
+If you see a star `*`, don’t panic—just expand it after you know what’s upstream.
+
+Plain: compute inputs first, then dependents (this is topological sort).
+
+#### Audience & prerequisites
+- Audience: data engineers, analytics engineers, platform engineers
+
+- Prerequisites: basic SQL; comfortable with CLI and git; Python 3.10+; AST familiarity helpful but optional (AST is Abstract Syntax Tree: a breakdown of code structure).
+
+1. Discover SQL assets and parse to AST (normalize identifiers)
+2. Build object-level dependency graph (views, CTEs, procs/temp tables)
+3. Resolve schemas topologically; expand `*` after inputs known
+4. Extract column-level lineage per output column expression
+5. Build bidirectional column graph for impact analysis
+6. Detect breaking changes by diffing base vs head graphs/schemas/expressions
+7. Output OpenLineage JSON + CLI reports 
+
+### Data structures
+- ObjectGraph: nodes = objects {name, type, statements}, edges = dependencies
+- SchemaRegistry: map object -> [columns {name, type, nullable, ordinal}]
+- ColumnGraph: nodes = fully qualified columns, edges = lineage relations
+
+### Pseudocode (high-level)
+```
+files = load_sql(dir)
+objects = parse(files)            # AST per object
+objGraph = build_object_graph(objects)
+order = topo_sort(objGraph)
+for obj in order:
+  schema_in = schemas_of_inputs(obj)
+  schema_out, lineage = resolve(obj.AST, schema_in)
+  SchemaRegistry[obj] = schema_out
+  ColumnGraph.add(lineage)
+```
+
+### Resolve() essentials
+- Name resolution: qualify identifiers using input schemas and aliases
+- Star expansion: replace `*` with ordered columns from the resolved input
+- Expression lineage: walk AST; collect input column refs per output column
+- Type/nullable inference: derive from operations (e.g., CAST types, SUM numeric, CASE nullability = union of branches)
+- Join semantics: track how join type affects nullability of columns
+- Set ops: ensure column counts/types align; union lineage inputs
+
+### Type/nullable rules (examples)
+- `CAST(x AS T)` → type T
+- `a + b` → numeric promotion; nullable if a or b nullable
+- `CASE WHEN p THEN x ELSE y` → type = LUB(type(x), type(y)); nullable if either branch nullable or no ELSE
+- `SUM(x)` → numeric; nullable unless GROUP present and engine semantics dictate otherwise
+
+### Error handling and diagnostics
+- On unresolved identifiers: record error with location; skip column lineage for affected outputs
+- On unsupported syntax: emit warning; continue best-effort resolution
+- Deterministic ordering of outputs and diagnostics for stable diffs
+- Don’t crash the whole run for one bad file; continue and report
+
+### Performance notes
+- Cache parsed ASTs and resolved schemas by file hash
+- Short-circuit lineage for unchanged objects between branches 
+
+### Impact search (recursive)
+- Upstream: walk edges from selected column to its input columns until sources are reached
+- Downstream: walk edges from selected column to outputs until targets are reached
+- Stop conditions: max-depth (if given), visited-set to avoid cycles 
+
+### Flowchart
+```mermaid
+flowchart TD
+  A[Discover SQL files] --> B[Parse to AST]
+  B --> C[Build object graph]
+  C --> D[Topological order]
+  D --> E[Resolve input schemas]
+  E --> F[Expand * after inputs known]
+  F --> G[Extract column lineage per output]
+  G --> H[Build column graph]
+  H --> I[Emit OpenLineage + reports]
+```
+
+### Star expansion worked example
+```sql
+CREATE VIEW dbo.vw_orders_all_enriched AS
+SELECT o.*, c.Region
+FROM dbo.Orders o
+JOIN dbo.Customers c ON o.CustomerID = c.CustomerID;
+```
+- Resolve `Orders` and `Customers` schemas first
+- Expand `o.*` by ordinal from `Orders`; append `Region`
+- Track nullability impact from the JOIN type
+
+### Determinism requirements
+- Deterministic output column ordering and schema serialization
+- Deterministic diagnostics (stable file/line ordering)
+- Stable JSON field ordering to make diffs meaningful 
+
+### Simple Example Walkthrough
+Take this SQL: `SELECT id AS student_id FROM students;`
+
+1. Parse to AST: Break into parts like 'SELECT', 'id', 'AS student_id'.
+2. Build object graph: Note dependency on 'students'.
+3. Resolve schemas: Output has 'student_id' from 'students.id'.
+4. Extract lineage: student_id comes from students.id.