typedef-data-intelligence 0.26.2__tar.gz → 0.27.0__tar.gz

{typedef_data_intelligence-0.26.2 → typedef_data_intelligence-0.27.0}/CHANGELOG.md

@@ -1,5 +1,25 @@
 # Changelog
 
+## [0.27.0](https://github.com/typedef-ai/typedef/compare/typedef-data-intelligence-v0.26.2...typedef-data-intelligence-v0.27.0) (2026-06-02)
+
+
+### Miscellaneous Chores
+
+* **lineage:** semantic graph diff realistic mutation harness (TD-3110) ([#806](https://github.com/typedef-ai/typedef/issues/806)) ([217509f](https://github.com/typedef-ai/typedef/commit/217509f2190728bcb6a8fae8731d61612823c3d0))
+
+
+### Features
+
+* add posthog analytics ([#736](https://github.com/typedef-ai/typedef/issues/736)) ([63c1d50](https://github.com/typedef-ai/typedef/commit/63c1d506dc6858d4ee49c59b8351eff8c31ba45f))
+* **td-2962:** multi-project configuration (settings UI + wizard rewrite) ([#770](https://github.com/typedef-ai/typedef/issues/770)) ([dc900d3](https://github.com/typedef-ai/typedef/commit/dc900d38e2bd7411587caf6408ab69b77045e6cf))
+* **td-3185:** per-project dbt-package systems + nested L2 rendering ([#799](https://github.com/typedef-ai/typedef/issues/799)) ([dd81bdc](https://github.com/typedef-ai/typedef/commit/dd81bdc8fd316e2fce5a2fca239987cf3d75193e))
+* **typedef-desktop multi-project:** selective graph ingest (td-2802) ([#721](https://github.com/typedef-ai/typedef/issues/721)) ([e150054](https://github.com/typedef-ai/typedef/commit/e150054bedf5ab900e7e769ebbcd467277ba4978))
+
+
+### Code Refactoring
+
+* **remove-legacy:** PR-6 — remove legacy projects synthesis (TD-2959 milestone 1) ([#796](https://github.com/typedef-ai/typedef/issues/796)) ([4445f9a](https://github.com/typedef-ai/typedef/commit/4445f9ae33125bc1ea1f26b5440fce44feb9bd7e))
+
 ## [0.26.2](https://github.com/typedef-ai/typedef/compare/typedef-data-intelligence-v0.26.1...typedef-data-intelligence-v0.26.2) (2026-05-30)
 
 

{typedef_data_intelligence-0.26.2 → typedef_data_intelligence-0.27.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: typedef-data-intelligence
-Version: 0.26.2
+Version: 0.27.0
 Summary: Unified lineage management and AI-powered analytics platform for dbt projects
 Project-URL: Homepage, https://typedef.ai
 Project-URL: Repository, https://github.com/typedef-ai/data-intelligence
@@ -40,6 +40,7 @@ Requires-Dist: networkx>=3.6
 Requires-Dist: openlineage-dbt>=1.39.0
 Requires-Dist: openlineage-python
 Requires-Dist: plotly>=6.3.1
+Requires-Dist: posthog>=7.9.7
 Requires-Dist: pydantic-ai-skills>=0.5.1
 Requires-Dist: pydantic-ai-todo>=0.1.10
 Requires-Dist: pydantic-ai<1.82.0,>=1.66.0

typedef_data_intelligence-0.27.0/docs/reference/SEMANTIC_DIFF_HARNESS.md

@@ -0,0 +1,130 @@
+# Semantic Graph Diff Mutation Harness (TD-3110)
+
+The harness applies a scripted, realistic dbt source mutation, rebuilds the
+graph through the normal **structural** sync path into two throwaway SurrealDB
+databases, runs the landed semantic-diff engine, and writes deterministic
+JSON + Markdown diff artifacts. Its output is grounding evidence for
+[TD-3193](https://linear.app/typedef-ai/issue/TD-3193) (compaction +
+blast-radius summaries): the raw, uncompacted noise profile of a real refactor.
+
+It consumes the public diff API
+(`core.backends.unified_graph.semantic_diff`) — it does **not** reimplement
+snapshot or diff logic.
+
+## Prerequisites
+
+- **`TYPEDEF_HOME=~/.typedef-desktop`** must be exported. Without it,
+  `repos_dir()` and the dbt-adapter venv resolve to the empty `~/.typedef`.
+  (The `just` recipe and the CLI both default it.)
+- The **SurrealDB sidecar** must be running at `ws://127.0.0.1:3567`
+  (namespace `lineage`, auth `root`/`root`). The embedded `surrealkv://`
+  fallback opens an empty store and must not be used.
+- **Git access** to the fixture's `git_url` — either already pooled locally at
+  `repos_dir() / hash_git_url(git_url)` or cloneable fresh.
+- **Snowflake credentials** for the `dbt compile` step only. Structural ingest
+  itself is offline (no profiling, no warehouse sampling, no LLM). The default
+  dbt profile is resolved from the project key
+  (`~/.typedef-desktop/profiles/<project>/context/<project>/`); override with
+  `--profiles-dir`. The harness reads the rendered `profiles.yml`'s top-level
+  key and passes it via `--profile` (the desktop re-renders that key, so it is
+  not assumed).
+
+## Run
+
+```bash
+just semantic-diff-harness column_rename_downstream
+# or, with options:
+TYPEDEF_HOME=~/.typedef-desktop uv run lineage-semantic-diff-harness \
+    --fixture column_rename_downstream
+```
+
+Useful flags: `--project` / `--profiles-dir` (creds lookup), `--gold-db` /
+`--candidate-db` (override the throwaway DB names), `--out-dir`, `--reuse-gold`
+(skip rebuilding gold when its DB already has data), `--save-reference` (copy
+the reports into the fixture dir as a checked-in golden), `--keep-work` (keep
+the temp gold/candidate checkouts for inspection; by default they are removed
+on exit so repeated runs don't leak `$TMPDIR`).
+
+The derived throwaway DB names embed the fixture's short `base_commit` hash
+(`semdiff_<case>_<base8>_{gold,cand}`), so bumping a fixture's `base_commit`
+lands in a fresh gold DB — `--reuse-gold` never silently reuses a graph built
+from the old base.
+
+**Exit codes** (the harness is usable as a verification gate):
+
+- `0` — every `expected_change_kind` was observed.
+- `2` — verification mismatch (an expected change kind was not produced) or an
+  unknown `--fixture` name. The mismatch is also logged as a warning; available
+  fixtures are listed on an unknown name and in `--help`.
+
+At mattermost scale a full run (build gold + candidate, both via
+`dbt deps`+`compile`, then structural ingest, then diff) is **~6–7 minutes** —
+the two Snowflake compiles dominate.
+
+## Artifacts
+
+Written to `artifacts/semantic-diff/<fixture>/` (gitignored):
+
+- `report.json` — the schema-v1 `DiffResult` serialization (output of record)
+- `report.md` — human-readable report
+- `raw_facts.json` — the raw fact-delta appendix (the noise profile for TD-3193)
+
+## Determinism
+
+The diff is deterministic for unchanged inputs at the **structural** level —
+graph nodes, edges, and closures are byte-stable. This relies on the
+order-independent `DbtProjector` (closure-root resolution and volatile-field
+stripping, including the absolute per-checkout `compiled_path`) and
+`include_generation_time=False`.
+
+**Known residual:** a small set (~10) of warehouse-querying union models
+(`base_*__tracks`) and `dbt_project_evaluator` models have **non-deterministic
+`compiled_sql`** across independent `dbt compile` invocations — their
+rudderstack union macros (`get_event_relations`) query
+`information_schema.tables` with no `ORDER BY`, so the UNION branch set/order
+varies per compile. Their `canonical_sql`/`fingerprint` differ only because the
+compiled-SQL input does (the unified-graph canonicalization is itself
+deterministic — verified). This is **upstream `dbt compile` non-determinism in
+the dbt project**, not a harness, ingest, or diff-projector issue (~29 facts on
+the mattermost fixture). Tracked separately (fix: add `ORDER BY` to the
+project's `get_event_relations` macro). Structural facts are unaffected.
+
+## Add a new mutation fixture
+
+A fixture is `fixtures/<case>/` containing `mutation.diff` (a git patch) and
+`fixture.yaml` (portable identifiers only):
+
+```yaml
+name: <case>
+description: <what the mutation does>
+source:
+  git_url: git@github.com:org/repo # SSH form, no .git suffix
+  subproject: path/to/dbt_project # dbt project dir within the repo
+  base_commit: <full sha> # pinned
+project: <typedef project key> # creds/profile lookup only
+expected_change_kinds: [RemovedColumn, AddedColumn, ...]
+```
+
+To author the patch:
+
+1. `git worktree add /tmp/fx <base_commit>` from the pooled repo (or clone it).
+2. Make a realistic, **consistent** source change (rename a column at its
+   definition **and** every reference, including `schema.yml`; the schema.yml
+   `columns:` entries feed the graph, so a SQL-only rename leaves a phantom
+   node). The change must keep the project compiling.
+3. `git diff > fixtures/<case>/mutation.diff` (from the repo root, so paths are
+   repo-relative for `git apply`).
+4. Write `fixtures/<case>/fixture.yaml` with portable identifiers only — never
+   a machine-local path or repo-id (the harness derives the local checkout via
+   `hash_git_url` or a fresh clone).
+5. `uv run lineage-semantic-diff-harness --fixture <case>` and inspect
+   `report.md`.
+
+Because `dbt compile` does not validate column existence, the **harness run
+itself** is the verification: a clean rename shows the cascade (column
+add/remove, lineage-edge churn, fingerprint changes) without orphan-lineage
+artifacts.
+
+The typedef project key affects only credential/profile lookup and is
+overridable; it has **no bearing on the diff result** (gold and candidate share
+the same dbt project name, hence the same `system_canonical_id`).

{typedef_data_intelligence-0.26.2 → typedef_data_intelligence-0.27.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "typedef-data-intelligence"
-version = "0.26.2"
+version = "0.27.0"
 description = "Unified lineage management and AI-powered analytics platform for dbt projects"
 readme = "README.pypi.md"
 license = "LicenseRef-Proprietary"
@@ -48,6 +48,7 @@ dependencies = [
   "dbt-snowflake>=1.7.0",
   "openlineage-dbt>=1.39.0",
   "openlineage-python",
+  "posthog>=7.9.7",
   "textual>=6.6.0",
   "pydantic-deep>=0.2.17",
   "antlr4-python3-runtime>=4.13.0",
@@ -105,6 +106,7 @@ lineage-list-ephemerals = "lineage.devtools.list_ephemeral_models:main"
 lineage-mcp = "lineage.mcp.runner:main"
 lineage-unified-build = "lineage.devtools.unified_graph.unified_cli:main"
 lineage-desktop-mcp = "lineage.devtools.unified_graph.surreal.desktop_mcp:main"
+lineage-semantic-diff-harness = "lineage.devtools.unified_graph.semantic_diff.cli:main"
 
 [build-system]
 requires = ["hatchling"]

{typedef_data_intelligence-0.26.2 → typedef_data_intelligence-0.27.0}/src/lineage/api/app.py

@@ -266,8 +266,25 @@ async def lifespan(app: FastAPI):
 
     # Run TD-2959 config migration before anything else reads UnifiedConfig.
     # Idempotent: marker file at $TYPEDEF_HOME/migrations/td-2959.json
-    # short-circuits subsequent boots.  Dual-write keeps legacy callers
-    # working until PR 6 cleanup.
+    # short-circuits subsequent boots.  This converts legacy ``main``-shape
+    # configs (top-level projects/data/default_project) into the workspaces
+    # schema in place — it is the ONLY remaining bridge for legacy configs
+    # now that the read-side synthesis was removed (PR 6 / TD-3074).
+    #
+    # DEFERRED — PR-6 / Phase 6 final cleanup (see
+    # docs/multi-project-milestone-1-design.md:115, "Group J Phase 6").
+    # That milestone scopes deletion of the *entire* legacy/migration
+    # scaffolding into PR 6: this ``run_migration`` path + ``repair_legacy_paths``
+    # + the ``core.migrations.td_2959`` module + ``apply_legacy_project_to_workspace``
+    # + the ``_drop_legacy_top_level_keys`` validator.  We deliberately KEEP it
+    # here: the doc's "after PR 6 there is no migration" assumes the td-2959
+    # migration already shipped and ran in an earlier release, but this stack
+    # is unshipped — deleting the migration before it has run on installs would
+    # break legacy ``main``-shape configs on first upgrade.
+    # GATE for removal: only delete once the td-2959 migration has shipped +
+    # run in a released version (a published floor past which legacy configs
+    # are unsupported).  At that point a legacy config should fail loudly at
+    # startup rather than silently boot with its projects dropped.
     try:
         from core.migrations.td_2959 import (
             repair_legacy_paths,
@@ -416,7 +433,7 @@ async def lifespan(app: FastAPI):
             _ucfg_for_data = getattr(app.state, "unified_config", None)
             if _ucfg_for_data is not None:
                 try:
-                    _data_cfg = _ucfg_for_data.get_project_data_config(
+                    _data_cfg = _ucfg_for_data.get_workspace_data_config(
                         _ucfg_for_data.default_project
                     )
                     app.state.standalone_data_backend = create_data_backend(
@@ -451,21 +468,18 @@ async def lifespan(app: FastAPI):
             logger.info("Initialized standalone reports backend: %s", _reports_dir)
 
             # Initialize ticket storage for desktop agents (OAuth or API key).
-            # Ticket config may live per-project (projects.<name>.ticket) or
-            # at the top level (ticket:).  Prefer the project-level config.
+            # Ticket config resolves through the typed three-tier walk
+            # (context entry → workspace → installation), no legacy projects
+            # view.
             _ucfg_for_tickets = getattr(app.state, "unified_config", None)
             _ticket_cfg = None
             if _ucfg_for_tickets is not None:
-                _dp = _ucfg_for_tickets.default_project
-                _proj = (
-                    _ucfg_for_tickets.projects.get(_dp)
-                    if _dp and _ucfg_for_tickets.projects
-                    else None
+                _resolved_ticket = _ucfg_for_tickets.resolve_ticket(
+                    workspace_name=_ucfg_for_tickets.resolve_active_workspace(),
+                    project_name=_ucfg_for_tickets.default_project,
                 )
-                if _proj and _proj.ticket and _proj.ticket.enabled:
-                    _ticket_cfg = _proj.ticket
-                elif _ucfg_for_tickets.ticket.enabled:
-                    _ticket_cfg = _ucfg_for_tickets.ticket
+                if _resolved_ticket is not None and _resolved_ticket.enabled:
+                    _ticket_cfg = _resolved_ticket
 
             if _ticket_cfg is not None:
                 try:
@@ -682,6 +696,13 @@ async def lifespan(app: FastAPI):
                 "Failed to terminate active quickstart workers", exc_info=True
             )
 
+    try:
+        from lineage.api.posthog_analytics import shutdown_shared_posthog_client
+
+        shutdown_shared_posthog_client()
+    except Exception:
+        logger.debug("Failed to shut down PostHog analytics client", exc_info=True)
+
     # Close standalone data backend (e.g. DuckDB file handle)
     standalone_db = getattr(app.state, "standalone_data_backend", None)
     if standalone_db is not None:
@@ -795,16 +816,15 @@ def _create_linear_token_manager(cfg: UnifiedConfig):
     from core.backends.config import LinearTicketConfig
     from core.backends.tickets.factory import create_linear_token_manager_from_config
 
-    # Find the relevant LinearTicketConfig, preferring project-level config
+    # Resolve the LinearTicketConfig via the typed three-tier walk
+    # (context entry → workspace → installation), no legacy projects view.
     linear_cfg = None
-    if cfg.default_project and cfg.projects:
-        proj = cfg.projects.get(cfg.default_project)
-        if proj and isinstance(proj.ticket, LinearTicketConfig):
-            linear_cfg = proj.ticket
-
-    # Fall back to top-level ticket config
-    if linear_cfg is None and isinstance(cfg.ticket, LinearTicketConfig):
-        linear_cfg = cfg.ticket
+    resolved_ticket = cfg.resolve_ticket(
+        workspace_name=cfg.resolve_active_workspace(),
+        project_name=cfg.default_project,
+    )
+    if isinstance(resolved_ticket, LinearTicketConfig):
+        linear_cfg = resolved_ticket
 
     if linear_cfg is None:
         return None
@@ -904,7 +924,7 @@ def initialize_app_state() -> AppState:
     # Initialize data backend from project config
     active_project = os.getenv("TYPEDEF_ACTIVE_PROJECT") or cfg.default_project
     try:
-        data_config = cfg.get_project_data_config(active_project)
+        data_config = cfg.get_workspace_data_config(active_project)
     except ValueError as exc:
         raise RuntimeError(
             "No data backend configuration found. "
@@ -915,7 +935,7 @@ def initialize_app_state() -> AppState:
 
     # Get per-project git config if active project specified
     git_config = None
-    if active_project and cfg.projects and active_project in cfg.projects:
+    if active_project and cfg.has_workspace_project(active_project):
         git_config = cfg.get_project_git_config(active_project)
         if git_config and git_config.enabled:
             logger.info(f"📁 Git configuration loaded for project: {active_project}")
@@ -1256,9 +1276,6 @@ if _HAS_DESKTOP_API:
     )
     from agent.api.desktop_pty import router as desktop_pty_router  # noqa: E402
     from lineage.api.desktop_agents import router as desktop_agents_router  # noqa: E402
-    from lineage.api.desktop_projects import (
-        router as desktop_projects_router,  # noqa: E402
-    )
     from lineage.api.desktop_quickstart import (
         router as desktop_quickstart_router,  # noqa: E402
     )
@@ -1288,7 +1305,6 @@ if _HAS_DESKTOP_API:
     app.include_router(desktop_worktree_router)
     app.include_router(desktop_setup_router)
     app.include_router(desktop_credentials_router)
-    app.include_router(desktop_projects_router)
     app.include_router(desktop_workspaces_router)
     app.include_router(desktop_repos_router)
     app.include_router(desktop_quickstart_router)

typedef_data_intelligence-0.27.0/src/lineage/api/controlplane_identity.py

@@ -0,0 +1,77 @@
+"""Trusted controlplane identity resolution for backend analytics."""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from dataclasses import dataclass
+
+import httpx
+from pydantic import BaseModel, ConfigDict, ValidationError
+
+
+@dataclass(frozen=True)
+class ControlplaneIdentityConfig:
+    """Auth-service validation configuration."""
+
+    auth_base_url: str
+    timeout_seconds: float = 2.0
+
+
+@dataclass(frozen=True)
+class ControlplaneAnalyticsIdentity:
+    """Pseudonymous analytics identity minted by the auth service."""
+
+    analytics_user_id: str
+    analytics_org_id: str
+
+
+class _ValidateTokenResponse(BaseModel):
+    model_config = ConfigDict(extra="ignore")
+
+    analytics_user_id: str | None = None
+    analytics_org_id: str | None = None
+
+
+async def resolve_controlplane_analytics_identity(
+    *,
+    token: str | None,
+    config: ControlplaneIdentityConfig | None,
+    client: httpx.AsyncClient | None = None,
+    request_headers: Mapping[str, str] | None = None,
+) -> ControlplaneAnalyticsIdentity | None:
+    """Resolve analytics identity from a trusted auth-service validation call.
+
+    `request_headers` is accepted for call-site convenience, but renderer-supplied
+    identity or auth URL headers are intentionally ignored.
+    """
+    _ = request_headers
+    if not token or not config or not config.auth_base_url:
+        return None
+
+    auth_base_url = config.auth_base_url.rstrip("/")
+    url = f"{auth_base_url}/v1/auth/validate"
+
+    try:
+        if client is not None:
+            response = await client.post(url, json={"token": token})
+        else:
+            async with httpx.AsyncClient(timeout=config.timeout_seconds) as new_client:
+                response = await new_client.post(url, json={"token": token})
+    except httpx.HTTPError:
+        return None
+
+    if response.status_code != 200:
+        return None
+
+    try:
+        validated = _ValidateTokenResponse.model_validate(response.json())
+    except (ValueError, ValidationError):
+        return None
+
+    if not validated.analytics_user_id or not validated.analytics_org_id:
+        return None
+
+    return ControlplaneAnalyticsIdentity(
+        analytics_user_id=validated.analytics_user_id,
+        analytics_org_id=validated.analytics_org_id,
+    )

{typedef_data_intelligence-0.26.2 → typedef_data_intelligence-0.27.0}/src/lineage/api/desktop_context.py

@@ -117,9 +117,10 @@ def resolve_desktop_request_context(request: Request) -> DesktopRequestContext:
             graph_name = None
     if config is not None and project_name is not None:
         try:
-            data_config = config.get_project_data_config(
+            data_config = config.get_workspace_data_config(
                 project_name,
                 worktree_id=None,
+                workspace_name=workspace_name,
             )
         except ValueError:
             data_config = None