datahub-bq-connector-session-patch 0.1.0__tar.gz

datahub_bq_connector_session_patch-0.1.0/.gitignore

@@ -0,0 +1,7 @@
+# Build artifacts produced by `make build` / `uv build`
+dist/
+build/
+*.egg-info/
+
+# Measurement output from tools/measure_scope.py
+runs/

datahub_bq_connector_session_patch-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

datahub_bq_connector_session_patch-0.1.0/Makefile

@@ -0,0 +1,165 @@
+# datahub-bq-connector-session-patch — build and distribution
+#
+# Artifacts land in dist/, which is git-ignored. The wheel is the deliverable: it is what
+# gets installed into a DataHub executor's own interpreter via the sidecar recipe's
+# `extra_pip_requirements`, alongside the executor's pinned acryl-datahub. It registers
+# the `datahub_bq_connector_session_patch` source type; the production recipe keeps `type: bigquery`
+# and is unaffected.
+#
+# Shipping a new build:
+#
+#     make release              # bump patch, rebuild, verify  -> dist/
+#     make release BUMP=minor   # same, bumping the minor component
+#     make publish              # only if a package index is configured
+#
+# `make build` alone rebuilds the current version without touching pyproject.toml.
+#
+# Every target that reads dist/ goes through `verify-dist` first, which refuses to
+# proceed if dist/ holds anything other than the current version's sdist and wheel.
+# Nothing is ever uploaded by a bare glob, so a stale artifact cannot ride along.
+
+UV       ?= uv
+DIST     := dist
+PKG      := datahub-bq-connector-session-patch
+DISTNAME := $(subst -,_,$(PKG))
+BUMP     ?= patch
+VENV     := .venv
+PYTHON_VERSION ?= 3.11
+VERSION   = $(shell $(UV) version --short 2>/dev/null)
+
+# The exact artifacts for the version currently in pyproject.toml — never `dist/*`.
+SDIST     = $(DIST)/$(DISTNAME)-$(VERSION).tar.gz
+WHEEL     = $(DIST)/$(DISTNAME)-$(VERSION)-*.whl
+
+.DEFAULT_GOAL := help
+.PHONY: help version clean build dist verify-dist check test gate venv bump release publish require-uv
+
+# `release` chains bump -> build -> check, each of which reads or rewrites the version
+# and the contents of dist/. Under -j those interleave and verify-dist can stat a
+# half-deleted dist/ or resolve $(VERSION) mid-bump. Ordering here is correctness.
+.NOTPARALLEL:
+
+help: ## Show this help
+	@echo "$(PKG) — targets:"
+	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
+	  | awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'
+	@echo
+	@echo "  variables: BUMP=patch|minor|major  PYTHON_VERSION=$(PYTHON_VERSION)  UV=$(UV)"
+
+require-uv: ## Fail with a readable message if the version cannot be read
+	@test -n "$(VERSION)" || { \
+	  echo "ERROR: could not read the project version."; \
+	  echo "  '$(UV) version --short' produced nothing — is uv installed, and is this"; \
+	  echo "  running from the package directory? Diagnose with: $(UV) version"; \
+	  exit 1; }
+
+version: require-uv ## Print the current version
+	@echo $(VERSION)
+
+clean: ## Remove build artifacts
+	rm -rf $(DIST) build src/*.egg-info *.egg-info
+
+build: clean ## Build sdist + wheel into dist/ (always from clean)
+	$(UV) build --out-dir $(DIST)
+	@echo
+	@ls -l $(DIST)
+
+dist: build ## Alias for build
+
+verify-dist: require-uv ## Assert dist/ holds exactly the current version, nothing stale
+	@test -d $(DIST) \
+	  || { echo "no $(DIST)/ — run 'make build' first"; exit 1; }
+	@test -f $(SDIST) \
+	  || { echo "ERROR: no sdist for $(PKG) $(VERSION) in $(DIST)/ — run 'make build'"; exit 1; }
+	@ls $(WHEEL) >/dev/null 2>&1 \
+	  || { echo "ERROR: no wheel for $(PKG) $(VERSION) in $(DIST)/ — run 'make build'"; exit 1; }
+	@stale=$$(find $(DIST) -type f ! -name '.*' \
+	    ! -name '$(DISTNAME)-$(VERSION).tar.gz' \
+	    ! -name '$(DISTNAME)-$(VERSION)-*.whl'); \
+	  if [ -n "$$stale" ]; then \
+	    echo "ERROR: $(DIST)/ holds files that are not $(PKG) $(VERSION):"; \
+	    echo "$$stale" | sed 's/^/  /'; \
+	    echo; \
+	    echo "Refusing to continue — these would be shipped alongside the new build."; \
+	    echo "Run 'make build' to rebuild from clean."; \
+	    exit 1; \
+	  fi
+
+check: verify-dist ## Verify the built artifacts are well-formed and complete
+	uvx twine check $(SDIST) $(WHEEL)
+	@echo
+	@for w in $(WHEEL); do \
+	  echo "--- $$w ---"; \
+	  unzip -l "$$w" | sed 's/^/  /'; \
+	  echo "  entry points:"; \
+	  unzip -p "$$w" '*/entry_points.txt' | sed 's/^/    /'; \
+	done
+	@unzip -p $(WHEEL) '*/entry_points.txt' \
+	  | grep -q 'datahub.ingestion.source.plugins' \
+	  || { echo "ERROR: wheel is missing the source entry point"; exit 1; }
+	@echo
+	@echo "OK — $(PKG) $(VERSION) registers its source entry point"
+
+# The range here must match MIN/MAX_ACRYL_DATAHUB in src/datahub_bq_connector_session_patch/patch.py and
+# the `test` extra in pyproject.toml — this patch reaches into private internals, so
+# "tested against" is the only guarantee it has.
+test: ## Run the test suite against a real acryl-datahub (no venv setup needed)
+	PYTHONPATH=src $(UV) run --no-project --python 3.11 \
+	  --with 'acryl-datahub[bigquery]>=1.6,<1.8' --with pytest --with pyyaml \
+	  pytest tests/ -q
+
+# The acceptance gate from the design doc: the same scenario must FAIL on stock
+# acryl-datahub and PASS patched. A green `gate` is what proves the patch still bites
+# after an acryl-datahub bump — `test` alone would stay green if upstream fixed the
+# defect out from under us, and then this package would be dead weight.
+gate: ## Prove the defect still exists on stock and that the patch still fixes it
+	@echo "--- stock acryl-datahub (must FAIL) ---"
+	@PYTHONPATH=src $(UV) run --no-project --python 3.11 \
+	  --with 'acryl-datahub[bigquery]>=1.6,<1.8' --with pytest \
+	  python tests/test_session_patch.py; \
+	  status=$$?; \
+	  if [ $$status -eq 0 ]; then \
+	    echo; echo "ERROR: stock acryl-datahub PASSED — upstream may have fixed this."; \
+	    echo "Re-read the design doc before shipping; this package may be obsolete."; \
+	    exit 1; \
+	  fi; \
+	  echo "  (exit $$status — correct: the defect is present)"
+	@echo
+	@echo "--- patched (must PASS) ---"
+	@PYTHONPATH=src $(UV) run --no-project --python 3.11 \
+	  --with 'acryl-datahub[bigquery]>=1.6,<1.8' --with pytest \
+	  python tests/test_session_patch.py --patched \
+	  || { echo; echo "ERROR: the patch no longer recovers the read."; exit 1; }
+	@echo
+	@echo "OK — defect reproduced on stock, fixed patched"
+
+# A project-local .venv, for interactive work and for the tools/ scripts — which need
+# real BigQuery credentials and so cannot run under `uv run --no-project`. Henrik's zsh
+# auto_venv chpwd hook activates it on cd into this directory.
+#
+# `test` and `gate` deliberately do NOT use it: they build a throwaway environment per
+# run, so a stale or hand-modified .venv can never make them pass.
+venv: ## Create .venv (python 3.11) with the package installed editable + test extras
+	$(UV) venv $(VENV) --python $(PYTHON_VERSION)
+	VIRTUAL_ENV=$(VENV) $(UV) pip install -e '.[test]'
+	@echo
+	@echo "Created $(VENV) — cd out and back in to activate, or: source $(VENV)/bin/activate"
+
+bump: require-uv ## Bump the version in pyproject.toml (BUMP=patch|minor|major)
+	@echo "$(PKG): $(VERSION) -> bumping $(BUMP)"
+	$(UV) version --frozen --bump $(BUMP)
+
+release: test gate bump build check ## Test, bump the version, rebuild, and verify
+	@echo
+	@echo "$(PKG) $(VERSION) built and verified. Artifacts:"
+	@ls -1 $(DIST)
+	@echo
+	@echo "Commit the pyproject.toml version bump, then 'make publish' if you have an index."
+
+publish: check ## Upload the current version to a package index (UV_PUBLISH_URL / _TOKEN)
+	@echo
+	@echo "Will upload exactly these files:"
+	@ls -1 $(SDIST) $(WHEEL) | sed 's/^/  /'
+	@printf "Publish $(PKG) $(VERSION) to %s? [y/N] " "$${UV_PUBLISH_URL:-the default index}"; \
+	  read ans; [ "$$ans" = "y" ] || { echo "aborted"; exit 1; }
+	$(UV) publish $(SDIST) $(WHEEL)

datahub_bq_connector_session_patch-0.1.0/PKG-INFO

@@ -0,0 +1,287 @@
+Metadata-Version: 2.5
+Name: datahub-bq-connector-session-patch
+Version: 0.1.0
+Summary: DataHub BigQuery sidecar source: attribute reads whose SQL does not qualify its table names
+Project-URL: Homepage, https://github.com/gutro/dv-datahub/tree/master/datahub-bq-connector-session-patch
+Project-URL: Repository, https://github.com/gutro/dv-datahub
+Project-URL: Issues, https://github.com/gutro/dv-datahub/issues
+Author: DataVantage Platform
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: bigquery,datahub,ingestion,lineage,usage
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Database
+Requires-Python: >=3.9
+Provides-Extra: test
+Requires-Dist: acryl-datahub[bigquery]<1.8,>=1.6; extra == 'test'
+Requires-Dist: pytest>=7; extra == 'test'
+Requires-Dist: pyyaml; extra == 'test'
+Description-Content-Type: text/markdown
+
+# datahub-bq-connector-session-patch
+
+A DataHub BigQuery **sidecar source** that attributes reads whose SQL did not qualify
+its table names — the ones the stock connector sees, misfiles and silently discards.
+
+Full diagnosis, design, verification and limits:
+`datavantage/docs/datahub-unqualified-read-attribution.md`.
+
+## The problem
+
+A client with a BigQuery *default dataset* set can write `SELECT ... FROM Bets_Details`.
+BigQuery fills in the blank at run time, but never records it: `INFORMATION_SCHEMA.JOBS`
+carries the SQL text and the billing project, not the job's default dataset.
+
+DataHub's connector substitutes `_SESSION` for the missing dataset
+(`queries_extractor.py:542,558`), on the assumption that a bare name is usually a
+temporary table. Its own temp-table rule then flags anything whose dataset starts with
+`_` (`:309`), and the aggregator drops it — for usage
+(`sql_parsing_aggregator.py:1022`) and for the query entity (`:1693`).
+
+So the connector *sees* the table name, guesses the wrong container, and discards the
+result. Nothing is reported: the sibling branch of `is_temp_table` records what it drops,
+this one returns `True` in silence. Measured here: one client, 2,866 jobs in 7 days,
+100% unqualified, 0 usage rows.
+
+**A `0` in DataHub usage is therefore ambiguous** — "nobody reads this", or "read
+constantly, with unqualified SQL". Decommissioning decisions depend on telling those apart.
+
+## What this does
+
+For upstreams that landed in `_SESSION`, it resolves the bare table name against the
+tables DataHub discovered during its own schema pass. A name matching more than one
+discovered table is **left untouched** — dropped exactly as before, never guessed.
+
+It ships as a distinct source type, so the production recipe stays stock:
+
+```yaml
+source:
+  type: datahub_bq_connector_session_patch
+```
+
+## Installing
+
+Per-recipe, via **Extra Pip Libraries** (DataHub UI) or `extra_pip_requirements`. That
+field is scoped to one recipe, which is what keeps the production recipe unaffected.
+
+```
+datahub-bq-connector-session-patch
+```
+
+The wheel declares **no dependencies** — it imports only `datahub.*`, which any
+environment running a recipe already has. Declaring `acryl-datahub` would let pip
+re-resolve the executor's own pinned copy while installing this. The tested range
+(`>=1.6,<1.8`) is enforced at runtime instead: an out-of-range executor raises a
+structured warning into the ingestion report rather than failing the run.
+
+## The sidecar recipe
+
+```yaml
+pipeline_name: bq-session-attribution        # its OWN name — never the main recipe's
+source:
+  type: datahub_bq_connector_session_patch
+  config:
+    project_id_pattern:
+      allow:
+        - '^dv-prod-eu-w1(-.+)?-data$'
+        - '^dv-ext-prod-eu-w1(-.+)?-data$'
+        - '^dv-prod-eu-w1(-.+)?-comp\d+$'
+    include_tables: false          # lightweight discovery still fills table_refs
+    include_views: false
+    include_table_lineage: false   # refused if true — see below
+    include_usage_statistics: true
+    use_queries_v2: true
+    include_queries: true
+    include_query_usage_statistics: true
+    region_qualifiers: ['region-europe-west1']
+    start_time: '-7 days'
+    enable_stateful_time_window: true
+    stateful_ingestion:
+      enabled: true                # the time-window watermark
+      remove_stale_metadata: false # refused if true — see below
+    env: PROD
+```
+
+### Two settings this source refuses to start with
+
+Both were reproduced against a live catalog. They are not style preferences, so they are
+enforced in code rather than left to the YAML:
+
+| setting | why it is refused |
+|---|---|
+| `stateful_ingestion.remove_stale_metadata: true` | The sidecar writes a `status` aspect on every dataset it produces usage for, enrolling them in *its* stale-removal checkpoint. Its URN set is **usage-driven**, so any dataset not read that day falls out of the checkpoint and is soft-deleted — datasets the main recipe owns. Normal operation, not an edge case. |
+| `include_table_lineage: true` | Resolved names drive usage attribution only. A mis-resolved write would create a **wrong edge into a real table**, which is worse than the missing usage this package exists to fix. |
+
+`remove_stale_metadata` defaults to `true`, so it must be set explicitly. Only the
+combination bites — with `stateful_ingestion.enabled: false` there is no checkpoint and
+nothing is refused.
+
+### Scope it to the readers, not to minimise ambiguity
+
+`dataset_pattern` should be **wide**. "Ambiguous" means the name matches more than one
+entry in `discovered_tables` *within the recipe's scope*, so narrowing scope removes the
+alternatives rather than resolving between them. If a reader's default dataset is
+`prod_spain_*` but the index holds only `prod_malta_*`, the name resolves **confidently
+and wrongly**.
+
+Ambiguity fails safe. A too-narrow index fails silently, in the wrong direction.
+
+Widening is otherwise harmless: `disjoint_only` means the sidecar emits only URNs it
+actually recovered, so it stays quiet wherever SQL is already qualified and never
+double-books usage with the main recipe.
+
+## Reading the run report
+
+```
+rewritten=41 tables (2792 refs) ambiguous=18 names (63 refs) unknown=7 names (12 refs)
+queries_passed=954 queries_skipped=3118 (index=1631 unique names, 412 ambiguous names)
+```
+
+- **distinct counters** — how much of the catalog was recovered. `rewritten` counts
+  distinct resolved **target tables**; `ambiguous` and `unknown` count distinct bare
+  **names**. Deliberately not distinct `_SESSION` URNs: those embed the reader project,
+  so one table read from five projects would report as five.
+- **`*_refs`** — how many reads that attributed. One hot table read 2,000 times is one
+  rewrite and 2,000 references.
+- **`queries_skipped`** — queries whose SQL was already fully qualified. The main recipe
+  has those; the sidecar correctly stays out of the way.
+- **`queries_mixed`** — queries carrying *both* an already-resolvable reference and a
+  recovered one. Also left to the main recipe, for a subtler reason, below.
+
+### Why mixed queries are skipped
+
+`disjoint_only` keeps the two pipelines from double-booking **usage**, which is a
+timeseries aspect. It does not, by itself, protect the **query entity**.
+
+Query entities are keyed by `get_query_fingerprint(sql, platform, fast=True)` — a pure
+function of the SQL text, so both pipelines derive the *same* `urn:li:query:<fp>`. But
+`querySubjects` is a versioned aspect, not timeseries. If the sidecar emitted its
+narrowed subject set for a query the main recipe also sees, whichever ran last would
+win, and the entity would flip between the two subject sets on every scheduled run.
+
+Passing the union instead would fix the subjects and double-book usage for the
+already-resolvable half. Neither is acceptable, so a mixed query is left entirely to the
+main recipe and counted. Measured on `dv-ext-prod-eu-w1-data` over 7 days:
+`queries_mixed=0` — the guard costs nothing there, because that client's SQL is
+uniformly unqualified.
+
+A per-reader-project breakdown is logged alongside it, so a project that recovers nothing
+shows up at 0.0% rather than vanishing.
+
+### Two failures that look like success
+
+Both were hit during development. Each leaves the run green and the output plausible:
+
+1. **`discovered_tables` is empty at construction.** The source passes a *live reference*
+   to `bq_schema_extractor.table_refs`, which only fills during the schema pass. The
+   index is built lazily on first use, never in `__init__`.
+2. **Refs are `projects/P/datasets/D/tables/T`, not `P.D.T`.** Naive dotted parsing
+   matches nothing.
+
+Because of these the source raises a structured **warning** if the index came out empty
+or nothing was rewritten. **Never trust a negative result without checking the index
+size.**
+
+## Limits
+
+Unique-name resolution only. Where a bare name matches several catalogued objects it is
+dropped. How much that costs depends entirely on the estate:
+
+| population | ambiguous names |
+|---|---|
+| views in `dv-ext-prod-eu-w1-data` | **0.0%** (162/162 unique) |
+| views in `dv-prod-eu-w1-data` | **54.4%** |
+| tables referenced estate-wide | 53.0% |
+
+The external-facing views carry licence/brand suffixes and are unique by construction.
+The internal estate's per-market layout duplicates names across `prod_malta_*`,
+`prod_spain_*`, `prod_italy_*`…, and the cross-market `common_` layer duplicates a
+further 39 names against them — so roughly half of any bare reads against it stay
+unrecovered.
+
+Resolving those needs **leaf-set matching**, not name matching: the colliding
+common-vs-market views have identical names but cleanly distinct leaf sets. That is
+explicitly out of scope — see §9.1 of the design doc for what it would take.
+
+## Measuring a scope before shipping it
+
+```bash
+export BQ_SESSION_PATCH_BILLING_PROJECT=dv-ext-prod-eu-w1-data
+python tools/measure_scope.py eu baseline --days 7
+python tools/measure_scope.py eu patched  --days 7
+python tools/diff_runs.py runs/mcps_eu_baseline_7d.json runs/mcps_eu_patched_7d.json
+```
+
+Scope flags: `--projects` for explicit ids, `--project-pattern` to override the region's
+`project_id_pattern.allow` regexes, `--days` for the window.
+
+**The billing project must be explicit** — `--billing`, or
+`$BQ_SESSION_PATCH_BILLING_PROJECT`, or inferred when `--projects` names exactly one.
+DataHub passes it straight to `bigquery.Client(...)`, and when it is `None` the client
+inherits whatever `google.auth.default()` resolves: `GOOGLE_CLOUD_PROJECT` first, then the
+ADC file's quota project, then the active gcloud config. A stray env var will therefore
+bill every `INFORMATION_SCHEMA` job to an unrelated project and fail as
+`bigquery.jobs.create` denied against a project you never named. The script refuses to
+start rather than inherit one, and prints both the billing and the ambient project so a
+mismatch is visible.
+
+`measure_scope.py` writes MCPs to a file sink (nothing reaches GMS) and prints the
+per-project recovery table; `diff_runs.py` reports datasets gaining usage, query entities
+gained and readers newly attributed, and fails if any `schemaMetadata`,
+`upstreamLineage` or `datasetProperties` aspect leaked into the output.
+
+## Trying it against a local DataHub
+
+`recipes/local/` holds a two-step pair for a quickstart instance, scoped to one project:
+
+```bash
+make venv && source .venv/bin/activate
+datahub ingest -c recipes/local/01-catalog.yml       # stock `bigquery` — the catalog
+datahub ingest -c recipes/local/02-usage-sidecar.yml # this package — the recovered usage
+```
+
+Step 1 runs with usage **on**, exactly as production does, and still misses the
+unqualified reads — that is the premise. Step 2 shows what it missed. Measured against
+`dv-ext-prod-eu-w1-data` over 7 days:
+
+| | step 1 (catalog) | step 2 (sidecar) |
+|---|---|---|
+| datasets with a usage aspect | 5 | **+59** |
+| query entities | 170 | **+61** |
+| aspects emitted | schema, lineage, properties, usage, queries | usage and queries only |
+
+Both use ADC, so the VPN must be up — VPC-SC blocks ADC while the `bq` CLI keeps
+working, which makes the failure look like a permissions problem.
+
+They run **stateless** (`stateful_ingestion.enabled: false`), so re-running re-emits the
+same days. Timeseries aspects append rather than upsert, so repeated local runs
+accumulate duplicate usage documents for a day. Harmless locally; in a real deployment
+keep the stateful time window, which is what makes each day emit exactly once.
+
+## Developing
+
+```bash
+make venv      # .venv on python 3.11, package installed editable + test extras
+make test      # 49 tests against a real acryl-datahub
+make gate      # the acceptance gate: must FAIL on stock, PASS patched
+make release   # test, gate, bump, rebuild, verify  -> dist/
+make publish
+```
+
+`make venv` exists for interactive work and for the `tools/` scripts, which need real
+BigQuery credentials and so cannot run in a throwaway environment. `make test` and
+`make gate` deliberately **do not** use it — they build their own environment per run, so
+a stale or hand-modified `.venv` can never make them pass.
+
+`make gate` is the one that matters on an acryl-datahub bump. This package patches
+private internals, so `make test` alone would stay green even if upstream fixed the
+defect out from under it — at which point this package is dead weight and should be
+retired, not shipped. `gate` fails loudly in that case.
+
+## Upstream
+
+The defect is unfixed on `datahub-project/datahub` master as of 2026-09-14. The ask
+there is modest: report the `_SESSION` discard, or offer opt-in resolution against
+discovered tables. See `UPSTREAM-ISSUE.md`.