bqemulator 1.0.0__tar.gz

bqemulator-1.0.0/.gitignore

@@ -0,0 +1,194 @@
+# Byte-compiled / cached
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+.benchmarks/
+.mutmut-cache
+# Mutmut 3.x: working directory with mutated copies of the source tree.
+mutants/
+
+# Type checking
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.pyre/
+.pytype/
+
+# Linting cache
+.ruff_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django / Flask stuff (safety)
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+instance/
+.webassets-cache
+
+# Scrapy stuff
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+site/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv / poetry / pip tooling
+.python-version
+.venv/
+venv/
+env/
+ENV/
+env.bak/
+venv.bak/
+Pipfile.lock
+poetry.lock
+.pdm-python
+.pdm.toml
+__pypackages__/
+
+# Environment variables
+.env
+.env.local
+.env.*.local
+
+# IDE / editor
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# Docker
+*.pid
+
+# DuckDB
+*.duckdb
+*.duckdb.wal
+*.duckdb.tmp
+bqemulator.duckdb*
+
+# bqemulator runtime data
+/data/
+/tmp-data/
+.bqemu.toml
+.bqemulator/
+
+# Generated proto stubs — committed, but local regen artifacts ignored
+src/bqemulator/grpc_api/proto/*_pb2_grpc.pyi
+
+# Release artifacts
+*.whl
+*.tar.gz
+
+# Docs site
+/site/
+
+# Mkdocs strict build artifacts
+.cache/
+
+# Test artifacts
+test-results/
+test-reports/
+.pytest-reports/
+# Per-tier junit artifacts (chaos, perf, differential, conformance ad-hoc)
+conformance.xml
+differential-results.xml
+
+# Secrets / credentials
+*.pem
+*.key
+credentials.json
+service-account*.json
+
+# bq CLI test sandbox: cached BigQuery v2 discovery doc (~550 KB)
+# fetched once per session by tests/e2e/bq_cli_client/bq_runner.py.
+# Set BQEMU_BQ_DISCOVERY_FILE to pre-stage a copy.
+tests/e2e/bq_cli_client/discovery.json
+
+# Local-only developer scratch: session-handoff notes, internal planning
+# docs, mid-build status trackers. Not part of the public repo.
+.dev/
+
+.claude/
+
+# Node.js (example projects under docs/examples/nodejs/)
+node_modules/
+npm-debug.log*
+yarn-debug.log*
+yarn-error.log*
+.pnpm-debug.log*
+
+# Java/Scala/sbt (example projects under docs/examples/java/)
+*.class
+# Maven + dbt + PyBuilder target/ already covered above.
+project/target/
+project/project/
+.bsp/
+.bloop/
+.metals/
+
+# Go (example projects under docs/examples/go/)
+# Go's tooling does not write into the workspace by default; nothing to add.
+
+# dbt
+dbt_packages/
+logs/

bqemulator-1.0.0/CHANGELOG.md

@@ -0,0 +1,179 @@
+# Changelog
+
+All notable changes to this project are documented in this file. The format is
+based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this
+project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+Entry types:
+
+- **Added** — new features
+- **Changed** — changes to existing behavior
+- **Deprecated** — features slated for removal (with removal timeline)
+- **Removed** — features that have been removed (must have been deprecated
+  for at least 2 MINOR versions or 6 months)
+- **Fixed** — bug fixes
+- **Security** — security fixes or relevant notices
+
+Each PR that changes observable behavior MUST add an entry under `Unreleased`.
+On release, `scripts/changelog.py X.Y.Z` moves those entries into a new version
+section and adds the release date.
+
+## [Unreleased]
+
+## [1.0.0] — 2026-05-22
+
+### Added
+
+- **REST API parity** — Datasets, Tables, Jobs, TableData, Routines, Models,
+  with multipart and resumable upload endpoints for `load_table_from_file`
+  workflows. `INFORMATION_SCHEMA` views (`TABLES`, `COLUMNS`, `ROUTINES`,
+  `VIEWS`, `JOBS`, `JOBS_BY_*`, `MATERIALIZED_VIEWS`, `PARTITIONS`,
+  `TABLE_OPTIONS`, etc.) queryable via the standard SQL path.
+- **Storage Read API** — gRPC servicer with both Arrow and Avro wire formats.
+  Avro is the Java client's default; both clients (Python `fastavro`, Node
+  `avsc`, Go `linkedin/goavro`, Java canonical Apache Avro) interoperate
+  against the same recorded fixtures.
+- **Storage Write API** — gRPC servicer with all four stream types (`DEFAULT`,
+  `COMMITTED`, `PENDING`, `BUFFERED`), both proto and Arrow row payload
+  formats, `FinalizeWriteStream` / `BatchCommitWriteStreams` /
+  `FlushRows` / `GetWriteStream`.
+- **GoogleSQL translator** — SQLGlot-backed transpiler from BigQuery dialect
+  to DuckDB SQL, with a rule registry covering the GoogleSQL function surface
+  (date / time / timestamp / interval, string, array, struct, range,
+  geography, statistical aggregates, approximate aggregates, JSON,
+  regular expressions, bit ops, civil-time helpers, and more).
+- **BigQuery scripting** — interpreter for `DECLARE` / `SET` / `BEGIN` …
+  `END` / `IF` / `WHILE` / `FOR` / `LOOP` / `BREAK` / `CONTINUE` /
+  `RETURN` / `RAISE` / `EXCEPTION WHEN ERROR THEN`, plus a
+  `BEGIN TRANSACTION` / `COMMIT TRANSACTION` / `ROLLBACK TRANSACTION`
+  shim.
+- **User-defined functions** — SQL UDFs, table-valued functions (TVFs), and
+  JavaScript UDFs via embedded V8 (optional `bqemulator[udf-js]` extra).
+- **Versioning surface** — time travel (`FOR SYSTEM_TIME AS OF`), table
+  snapshots, table clones, and materialized views with
+  `BQ.REFRESH_MATERIALIZED_VIEW` dispatch.
+- **Authorization surface** — authorized views (with RAP propagation) and
+  row-access policies with caller-identity enforcement.
+- **Specialized types** — `GEOGRAPHY` (planar via DuckDB-spatial with
+  S2-sphere helpers for distance / length / area / perimeter / DWithin),
+  `RANGE<DATE>` / `RANGE<DATETIME>` / `RANGE<TIMESTAMP>`, `INTERVAL`,
+  `NUMERIC` / `BIGNUMERIC` arithmetic, civil-time helpers.
+- **Load / extract formats** — load supports CSV, JSON, Avro, ORC, and
+  Parquet. Extract supports CSV, JSON, Avro, and Parquet. (ORC extract is
+  intentionally out of scope — see `docs/reference/out-of-scope.md`.)
+- **Multi-arch Docker image** — `ghcr.io/jjviscomi/bqemulator` builds for
+  `linux/amd64` + `linux/arm64`, with cosign keyless signatures via GitHub
+  OIDC.
+- **Native pytest plugin** — `pip install bqemulator` registers a pytest
+  plugin; the `bqemu_server` fixture starts an ephemeral in-process emulator
+  on random free ports, sets `BIGQUERY_EMULATOR_HOST`, and tears down on
+  exit.
+- **Five-client E2E** — every release exercises the live container against
+  the official Python, Node.js, Go, and Java BigQuery client libraries plus
+  Google's `bq` CLI.
+- **Conformance corpus** — 1,200+ fixtures recorded against real BigQuery
+  covering SQL semantics, REST wire format, and gRPC Storage R/W. Drift
+  between the emulator and BigQuery surfaces as test failures; documented
+  divergences are pinned in `tests/conformance/divergences.py` with ADR
+  references.
+- **Observability** — `structlog` JSON logs, OpenTelemetry tracing
+  (configurable OTLP exporter), Prometheus metrics endpoint.
+- **Admin surface** — `bqemulator import --from-project` clones a real
+  BigQuery project's schema (and optionally data) into a local emulator
+  data directory; backup / restore via `bqemulator admin backup` and
+  `bqemulator admin restore`.
+- **Release tooling** — `scripts/bump_version.py`, `scripts/changelog.py`,
+  and `scripts/release.py` automate the version bump → changelog finalise
+  → release commit + annotated tag flow. `make release-dry-run` previews;
+  `make release` applies.
+- **Example projects (14)** — `docs/examples/` ships runnable example
+  projects for every supported language + framework + deployment pattern:
+  Python (pytest-integration, dbt-local, airflow-dag-test,
+  pyspark-bigquery), Node.js (NestJS app, Cloud Run local), Go (Beam
+  pipeline, Dataflow-style ETL), Java/Scala (Spring Boot,
+  Spotify Scio), docker-compose full-stack (app + emulator + Prometheus +
+  Grafana), and CI recipes (GitHub Actions, GitLab CI, CircleCI). Each
+  has its own `make test`; all are validated in CI by the new
+  `.github/workflows/examples.yml` workflow (intentionally non-blocking
+  on main so an upstream framework regression cannot stall emulator PRs).
+
+### Documentation
+
+- **mkdocs-Material site** — getting-started + per-language quickstarts +
+  topic guides (loading data, querying, query parameters, streaming
+  inserts, storage API, routines + UDFs, scripting, time travel,
+  materialized views, row access policies, authorized views,
+  `INFORMATION_SCHEMA`, GEOGRAPHY, RANGE, INTERVAL, admin endpoints, backup
+  and restore, CI/CD patterns, dbt, Airflow, Spark, the `bq` CLI,
+  observability).
+- **Auto-generated reference docs** — compatibility matrix, conformance
+  coverage matrix, SQL function mapping, and API coverage. Each ships
+  with a `make <name>-check` drift gate wired into `make verify` and
+  the per-PR `Docs-drift gates` CI job, so a regenerated doc can't
+  drift from the live source between commits. A fifth audit doc —
+  `docs/reference/api-configuration-coverage-matrix.md` — is the
+  manually-maintained sibling that tracks the *configuration knob*
+  surface (the part that can't be mechanically derived from the
+  route handlers); it's labelled "Audit dated" at the top of the
+  file and refreshed during the pre-release doc sweep.
+- **Architecture Decision Records** — 32 ADRs documenting non-obvious
+  design choices (DuckDB vs. alternatives, hexagonal architecture, scripting
+  execution model, materialized-view refresh semantics, caller identity
+  and row-access enforcement, conformance corpus design, divergence
+  baseline, perf / chaos / mutation / fuzz / differential tier contracts,
+  upload host endpoints, `bq` CLI as a fifth conformance client).
+
+### Testing
+
+- **7-tier test pyramid** — unit (hermetic), property (Hypothesis),
+  integration (in-process + client), conformance (compared to real
+  BigQuery), e2e (live container × five clients), performance
+  (`pytest-benchmark` with `--benchmark-compare-fail=median:10%`), chaos
+  (deliberately disruptive — concurrency, resource exhaustion, crash
+  recovery, storage failures, network failures). Sibling tiers: differential
+  (row-order perturbation of the conformance corpus), mutation (`mutmut`
+  pilot on pure-domain modules), fuzz (Atheris on the SQL translator,
+  dynamic-protobuf decoder, and Arrow bridge).
+- **Coverage gate** — combined unit + property + integration coverage
+  must reach 90% line + branch in CI.
+
+### Security
+
+- **Path-traversal hardening** on the resumable-upload `upload_id` regex
+  (`^[A-Za-z0-9_-]{8,64}$`); size cap enforced before disk write to
+  prevent unbounded resource consumption; multipart envelope injection
+  prevented via stdlib `email.parser` plus a media-part type allowlist.
+- **PyPI publish via Trusted Publishing** (no long-lived tokens); wheels
+  carry sigstore attestations.
+- **GHCR image signing** via keyless cosign with GitHub OIDC certificate
+  identity.
+
+### Known limitations (deferred to v1.0.1)
+
+These ship as documented caveats on the affected example projects.
+None affect the core emulator surface.
+
+- **Storage Read API IPC bytes layout** — bqemulator packs the full
+  Arrow IPC stream (schema framing + batches) into
+  `ReadRowsResponse.arrow_record_batch.serialized_record_batch`
+  rather than a single record-batch IPC message; the schema lives
+  separately on `ReadSession.arrow_schema`. The
+  `google-cloud-bigquery-storage` client's high-level
+  `reader.to_arrow(session)` trips
+  `Expected IPC message of type record batch but got schema`. The
+  `python/pyspark-bigquery` example iterates raw responses through
+  `pa.ipc.open_stream` as a workaround. Tracked in
+  [#15](https://github.com/jjviscomi/bqemulator/issues/15).
+- **Scio test exercises wiring only** — Beam Java BigQueryIO does
+  not honour `--bigQueryEndpoint` for the *write* path (that flag
+  only gates internal preflight validators); the Java BQ client
+  *does* honour `BIGQUERY_EMULATOR_HOST`, but it must be visible to
+  the JVM before the first BQ class loads, which is impossible
+  when the testcontainer port is allocated at runtime. The
+  `java/scio` example's spec asserts the wiring bqemulator owns
+  (container up, REST API reachable, dataset creation works); the
+  `CustomersPipeline.run` source itself remains production-ready
+  for users running against real BigQuery or a long-lived
+  bqemulator with a stable port. Tracked in
+  [#17](https://github.com/jjviscomi/bqemulator/issues/17).
+

bqemulator-1.0.0/LICENSE

@@ -0,0 +1,200 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for describing the origin of the Work and
+      reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may accept for a fee,
+      acceptance of warranty or additional liability, in accordance with
+      this License. Nevertheless, in accepting such obligations, You may
+      act only on Your own behalf and on Your sole responsibility, not on
+      behalf of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed" page as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 bqemulator contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.