sg-query-cache 0.1.0__tar.gz

sg_query_cache-0.1.0/.gitignore

@@ -0,0 +1,46 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+dist/
+.eggs/
+*.egg-info/
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+
+# PyInstaller
+*.manifest
+*.spec
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage*
+.pytest_cache/
+.cache/
+
+# Type checker / linter caches
+.pyright/
+.mypy_cache/
+.ruff_cache/
+pyrightconfig.json
+
+# Editors
+.vscode/
+.idea/
+
+# OS
+.DS_Store
+Thumbs.db

sg_query_cache-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 the sg-query-cache authors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sg_query_cache-0.1.0/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.5
+Name: sg-query-cache
+Version: 0.1.0
+Summary: Query caching for Autodesk Flow Production Tracking
+Project-URL: Documentation, https://github.com/dactylo/sg-query-cache/tree/main/docs
+Project-URL: Homepage, https://github.com/dactylo/sg-query-cache
+Project-URL: Issues, https://github.com/dactylo/sg-query-cache/issues
+Project-URL: Repository, https://github.com/dactylo/sg-query-cache
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cache,flow-production-tracking,queries,shotgrid,shotgun,valkey
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: typing-extensions>=4.5
+Provides-Extra: valkey
+Requires-Dist: valkey<6.2,>=6.1.1; extra == 'valkey'
+Description-Content-Type: text/markdown
+
+# sg-query-cache
+
+sg-query-cache reduces repeated Autodesk Flow Production Tracking API reads from Python workers.
+When render jobs, pipeline tools, or services ask the same question many times, the cache answers
+the repeats from memory or from a shared Valkey service instead of sending another request to the
+site. It wraps your existing synchronous or native asynchronous client and keeps its method
+signatures, so your code changes only where it creates the client.
+
+This documentation uses SG for Flow Production Tracking, formerly named Shotgun and ShotGrid.
+
+Caching trades freshness for fewer requests. Every cached answer has a maximum age, 30 seconds by
+default for a single process. Writes made through the wrapper invalidate the cached answers they
+can affect, and a shared deployment also invalidates answers when SG reports a change. Until then,
+a change made elsewhere can be missing from a cached answer. When a read must reflect SG right now,
+bypass the cache for that read. The [consistency guide](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/consistency.md) states the guarantee
+in full.
+
+## When to use it
+
+The cache helps when processes repeat exact reads: the same method, filters, fields, and client
+options, under the same SG permissions, within seconds or minutes of each other. It does not speed
+up a read it has not seen before. It caches `find`, `find_one`, and schema reads, plus
+`work_schedule_read` if you enable it; other reads go straight to SG.
+
+Choose a layout:
+
+| Layout          | Use when                                            | What it needs, and how fresh it is                                                                                                                      |
+| --------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Local (default) | One process repeats reads                           | Nothing beyond this package. Each process keeps its own in-memory cache, and changes made elsewhere appear within 30 seconds by default.                |
+| Shared          | Several processes should reuse each other's results | Two Valkey services and the SG event-daemon plugin, which invalidates results as SG reports changes. Most results are kept up to 10 minutes by default. |
+| Hierarchical    | Shared results plus a fast per-process copy         | The same as shared. Each process can serve its own copy for up to 10 seconds by default after another process changes the data.                         |
+
+Start with the local layout to measure how many of your reads repeat. A shared layout trusts every
+client that can reach its Valkey services: the cache does not check that a caller's declared
+identity matches its SG credentials. Share Valkey only among processes you trust to declare their
+identities correctly.
+
+## Installation
+
+sg-query-cache requires Python 3.11 or newer and an SG client in the same environment:
+
+- For synchronous code, the official `shotgun_api3` package or `sg-python-api`.
+- For native async code, `sg-python-api[async]`.
+
+The official package and sg-python-api both provide the `shotgun_api3` import name, so install only
+one of them. The shared and hierarchical layouts also need two Valkey services; see
+[service layout](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/operations.md#service-layout).
+
+From a checkout, install the library into the environment that runs your SG client:
+
+```sh
+python -m pip install .
+```
+
+For the shared or hierarchical layout, include the `valkey` extra. It adds the Valkey client that
+the cache and the `sg-query-cache` operator commands use. The extra accepts valkey-py 6.1.x, the
+release family this version is tested with:
+
+```sh
+python -m pip install '.[valkey]'
+```
+
+## Quick start
+
+### Cache a synchronous client
+
+Set `SHOTGUN_URL`, `SHOTGUN_SCRIPT_NAME`, and `SHOTGUN_API_KEY` for a script account with read
+access, then run:
+
+```python
+import os
+from time import monotonic
+
+from shotgun_api3 import Shotgun
+from sg_query_cache import (
+    CacheSettings,
+    CachedShotgun,
+    ClientContext,
+    Identity,
+    LocalBackend,
+    NullMetrics,
+    QueryCache,
+)
+
+site_url = os.environ['SHOTGUN_URL']
+script_name = os.environ['SHOTGUN_SCRIPT_NAME']
+raw_sg = Shotgun(site_url, script_name=script_name, api_key=os.environ['SHOTGUN_API_KEY'])
+
+settings = CacheSettings()
+identity = Identity(kind='script', issuer='shotgun-script', namespace=script_name)
+cache = QueryCache(settings, identity, l1=LocalBackend(settings, monotonic, NullMetrics()))
+context = ClientContext(
+    site_url=site_url,
+    site_namespace=settings.site_namespace,
+    localized=raw_sg.config.localized,
+    convert_datetimes_to_utc=raw_sg.config.convert_datetimes_to_utc,
+    records_per_page=raw_sg.config.records_per_page,
+)
+sg = CachedShotgun(raw_sg, cache, context)
+
+
+def timed_read(label: str) -> None:
+    started = monotonic()
+    assets = sg.find('Asset', [['sg_status_list', 'is', 'ip']], ['code'])
+    print(f'{label}: {len(assets)} assets in {monotonic() - started:.3f} s')
+
+
+try:
+    timed_read('first read')
+    timed_read('second read')
+    with sg.bypass():
+        timed_read('bypass')
+finally:
+    cache.close()
+    raw_sg.close()
+```
+
+You should see output like this, with your own counts and times:
+
+```text
+first read: 42 assets in 0.412 s
+second read: 42 assets in 0.001 s
+bypass: 42 assets in 0.198 s
+```
+
+The first read goes to SG. It also loads the site schema, which the cache uses to work out what
+each query depends on. The second, identical read comes from the in-process cache. Inside
+`bypass()`, reads always go to SG and leave the cache unchanged. A new process starts with an empty
+local cache.
+
+The example sets up four pieces:
+
+- `CacheSettings()` selects the local layout with default limits. Shared layouts load their
+  settings from a TOML file; see [configuration](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/configuration.md).
+- `Identity` names the SG permission scope whose results may be reused. Here every process running
+  this script shares one identity named after the script; give scripts with different permissions
+  different identities. Use stable configuration values, never an API key, session token, or
+  process ID, because identity values become part of every cache key.
+- `QueryCache` with a `LocalBackend` is the cache itself. The wrapper closes neither the cache nor
+  the SG client, so the example closes both.
+- `ClientContext` records the raw client's response options, such as page size, so results fetched
+  with different options are never mixed. Keep it matched to the raw client's configuration.
+
+### Use a native async client
+
+With `sg-python-api[async]`, `AsyncCachedShotgun.open()` builds the cache from your settings and
+closes it on exit. You open and close the SG client yourself:
+
+```python
+import asyncio
+import os
+
+from shotgun_api3 import AsyncShotgun
+from sg_query_cache import AsyncCachedShotgun, CacheSettings, ClientContext, Identity
+
+
+async def main() -> None:
+    site_url = os.environ['SHOTGUN_URL']
+    script_name = os.environ['SHOTGUN_SCRIPT_NAME']
+    settings = CacheSettings()
+    identity = Identity(kind='script', issuer='shotgun-script', namespace=script_name)
+    async with AsyncShotgun(
+        site_url, script_name=script_name, api_key=os.environ['SHOTGUN_API_KEY']
+    ) as raw_sg:
+        context = ClientContext(
+            site_url=site_url,
+            site_namespace=settings.site_namespace,
+            localized=raw_sg.config.localized,
+            convert_datetimes_to_utc=raw_sg.config.convert_datetimes_to_utc,
+            records_per_page=raw_sg.config.records_per_page,
+        )
+        async with AsyncCachedShotgun.open(raw_sg, settings, identity, context) as sg:
+            filters = [['sg_status_list', 'is', 'ip']]
+            for label in ('first read', 'second read'):
+                assets = await sg.find('Asset', filters, ['code'])
+                print(f'{label}: {len(assets)} assets')
+            with sg.bypass():
+                assets = await sg.find('Asset', filters, ['code'])
+                print(f'bypass: {len(assets)} assets')
+
+
+asyncio.run(main())
+```
+
+As in the synchronous example, the second read comes from the cache and the bypassed read goes to
+SG. `bypass()` is an ordinary `with` block, even in async code.
+
+Use each async cache on one event loop. `bypass()` and other policy scopes apply to the current
+task, and a task created inside a scope inherits it. Before you close caches in a service with
+long-running tasks, read [async shutdown](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/operations.md#async-deadlines-and-shutdown).
+
+## Documentation
+
+| If you want to...                                   | Read                                                          |
+| --------------------------------------------------- | ------------------------------------------------------------- |
+| Understand how fresh results are and when to bypass | [Consistency](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/consistency.md)                            |
+| Choose a layout and write the settings file         | [Configuration](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/configuration.md)                        |
+| Cache reads for human users                         | [Identities](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/configuration.md#identities-and-trust)      |
+| Change how long results are kept                    | [Maximum age rules](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/configuration.md#maximum-age-rules)  |
+| Run Valkey and the invalidation plugin              | [Operations](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/operations.md)                              |
+| Flush cached results after an incident              | [Recovery](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/operations.md#flush-or-rotate-cached-results) |
+| Collect metrics and set alerts                      | [Metrics](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/operations.md#metrics-and-alerts)              |
+| Measure savings for your workload                   | [Benchmarks](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/benchmarks.md)                              |
+| See changes by version                              | [Changelog](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/CHANGELOG.md)                                     |
+
+## Development
+
+See the [development guide](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/docs/development.md) for the required tools, CI, and dependency locks.
+With those tools installed, run:
+
+```sh
+make setup
+make check
+```
+
+The checks never contact SG or Valkey; live Valkey integration tests are separate and need
+disposable services.
+
+## License
+
+sg-query-cache is available under the [MIT License](https://github.com/dactylo/sg-query-cache/blob/v0.1.0/LICENSE). It is an independent project and is
+not affiliated with or endorsed by Autodesk. Flow Production Tracking is a product of Autodesk,
+Inc.

sg_query_cache-0.1.0/README.md

@@ -0,0 +1,223 @@
+# sg-query-cache
+
+sg-query-cache reduces repeated Autodesk Flow Production Tracking API reads from Python workers.
+When render jobs, pipeline tools, or services ask the same question many times, the cache answers
+the repeats from memory or from a shared Valkey service instead of sending another request to the
+site. It wraps your existing synchronous or native asynchronous client and keeps its method
+signatures, so your code changes only where it creates the client.
+
+This documentation uses SG for Flow Production Tracking, formerly named Shotgun and ShotGrid.
+
+Caching trades freshness for fewer requests. Every cached answer has a maximum age, 30 seconds by
+default for a single process. Writes made through the wrapper invalidate the cached answers they
+can affect, and a shared deployment also invalidates answers when SG reports a change. Until then,
+a change made elsewhere can be missing from a cached answer. When a read must reflect SG right now,
+bypass the cache for that read. The [consistency guide](docs/consistency.md) states the guarantee
+in full.
+
+## When to use it
+
+The cache helps when processes repeat exact reads: the same method, filters, fields, and client
+options, under the same SG permissions, within seconds or minutes of each other. It does not speed
+up a read it has not seen before. It caches `find`, `find_one`, and schema reads, plus
+`work_schedule_read` if you enable it; other reads go straight to SG.
+
+Choose a layout:
+
+| Layout          | Use when                                            | What it needs, and how fresh it is                                                                                                                      |
+| --------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Local (default) | One process repeats reads                           | Nothing beyond this package. Each process keeps its own in-memory cache, and changes made elsewhere appear within 30 seconds by default.                |
+| Shared          | Several processes should reuse each other's results | Two Valkey services and the SG event-daemon plugin, which invalidates results as SG reports changes. Most results are kept up to 10 minutes by default. |
+| Hierarchical    | Shared results plus a fast per-process copy         | The same as shared. Each process can serve its own copy for up to 10 seconds by default after another process changes the data.                         |
+
+Start with the local layout to measure how many of your reads repeat. A shared layout trusts every
+client that can reach its Valkey services: the cache does not check that a caller's declared
+identity matches its SG credentials. Share Valkey only among processes you trust to declare their
+identities correctly.
+
+## Installation
+
+sg-query-cache requires Python 3.11 or newer and an SG client in the same environment:
+
+- For synchronous code, the official `shotgun_api3` package or `sg-python-api`.
+- For native async code, `sg-python-api[async]`.
+
+The official package and sg-python-api both provide the `shotgun_api3` import name, so install only
+one of them. The shared and hierarchical layouts also need two Valkey services; see
+[service layout](docs/operations.md#service-layout).
+
+From a checkout, install the library into the environment that runs your SG client:
+
+```sh
+python -m pip install .
+```
+
+For the shared or hierarchical layout, include the `valkey` extra. It adds the Valkey client that
+the cache and the `sg-query-cache` operator commands use. The extra accepts valkey-py 6.1.x, the
+release family this version is tested with:
+
+```sh
+python -m pip install '.[valkey]'
+```
+
+## Quick start
+
+### Cache a synchronous client
+
+Set `SHOTGUN_URL`, `SHOTGUN_SCRIPT_NAME`, and `SHOTGUN_API_KEY` for a script account with read
+access, then run:
+
+```python
+import os
+from time import monotonic
+
+from shotgun_api3 import Shotgun
+from sg_query_cache import (
+    CacheSettings,
+    CachedShotgun,
+    ClientContext,
+    Identity,
+    LocalBackend,
+    NullMetrics,
+    QueryCache,
+)
+
+site_url = os.environ['SHOTGUN_URL']
+script_name = os.environ['SHOTGUN_SCRIPT_NAME']
+raw_sg = Shotgun(site_url, script_name=script_name, api_key=os.environ['SHOTGUN_API_KEY'])
+
+settings = CacheSettings()
+identity = Identity(kind='script', issuer='shotgun-script', namespace=script_name)
+cache = QueryCache(settings, identity, l1=LocalBackend(settings, monotonic, NullMetrics()))
+context = ClientContext(
+    site_url=site_url,
+    site_namespace=settings.site_namespace,
+    localized=raw_sg.config.localized,
+    convert_datetimes_to_utc=raw_sg.config.convert_datetimes_to_utc,
+    records_per_page=raw_sg.config.records_per_page,
+)
+sg = CachedShotgun(raw_sg, cache, context)
+
+
+def timed_read(label: str) -> None:
+    started = monotonic()
+    assets = sg.find('Asset', [['sg_status_list', 'is', 'ip']], ['code'])
+    print(f'{label}: {len(assets)} assets in {monotonic() - started:.3f} s')
+
+
+try:
+    timed_read('first read')
+    timed_read('second read')
+    with sg.bypass():
+        timed_read('bypass')
+finally:
+    cache.close()
+    raw_sg.close()
+```
+
+You should see output like this, with your own counts and times:
+
+```text
+first read: 42 assets in 0.412 s
+second read: 42 assets in 0.001 s
+bypass: 42 assets in 0.198 s
+```
+
+The first read goes to SG. It also loads the site schema, which the cache uses to work out what
+each query depends on. The second, identical read comes from the in-process cache. Inside
+`bypass()`, reads always go to SG and leave the cache unchanged. A new process starts with an empty
+local cache.
+
+The example sets up four pieces:
+
+- `CacheSettings()` selects the local layout with default limits. Shared layouts load their
+  settings from a TOML file; see [configuration](docs/configuration.md).
+- `Identity` names the SG permission scope whose results may be reused. Here every process running
+  this script shares one identity named after the script; give scripts with different permissions
+  different identities. Use stable configuration values, never an API key, session token, or
+  process ID, because identity values become part of every cache key.
+- `QueryCache` with a `LocalBackend` is the cache itself. The wrapper closes neither the cache nor
+  the SG client, so the example closes both.
+- `ClientContext` records the raw client's response options, such as page size, so results fetched
+  with different options are never mixed. Keep it matched to the raw client's configuration.
+
+### Use a native async client
+
+With `sg-python-api[async]`, `AsyncCachedShotgun.open()` builds the cache from your settings and
+closes it on exit. You open and close the SG client yourself:
+
+```python
+import asyncio
+import os
+
+from shotgun_api3 import AsyncShotgun
+from sg_query_cache import AsyncCachedShotgun, CacheSettings, ClientContext, Identity
+
+
+async def main() -> None:
+    site_url = os.environ['SHOTGUN_URL']
+    script_name = os.environ['SHOTGUN_SCRIPT_NAME']
+    settings = CacheSettings()
+    identity = Identity(kind='script', issuer='shotgun-script', namespace=script_name)
+    async with AsyncShotgun(
+        site_url, script_name=script_name, api_key=os.environ['SHOTGUN_API_KEY']
+    ) as raw_sg:
+        context = ClientContext(
+            site_url=site_url,
+            site_namespace=settings.site_namespace,
+            localized=raw_sg.config.localized,
+            convert_datetimes_to_utc=raw_sg.config.convert_datetimes_to_utc,
+            records_per_page=raw_sg.config.records_per_page,
+        )
+        async with AsyncCachedShotgun.open(raw_sg, settings, identity, context) as sg:
+            filters = [['sg_status_list', 'is', 'ip']]
+            for label in ('first read', 'second read'):
+                assets = await sg.find('Asset', filters, ['code'])
+                print(f'{label}: {len(assets)} assets')
+            with sg.bypass():
+                assets = await sg.find('Asset', filters, ['code'])
+                print(f'bypass: {len(assets)} assets')
+
+
+asyncio.run(main())
+```
+
+As in the synchronous example, the second read comes from the cache and the bypassed read goes to
+SG. `bypass()` is an ordinary `with` block, even in async code.
+
+Use each async cache on one event loop. `bypass()` and other policy scopes apply to the current
+task, and a task created inside a scope inherits it. Before you close caches in a service with
+long-running tasks, read [async shutdown](docs/operations.md#async-deadlines-and-shutdown).
+
+## Documentation
+
+| If you want to...                                   | Read                                                          |
+| --------------------------------------------------- | ------------------------------------------------------------- |
+| Understand how fresh results are and when to bypass | [Consistency](docs/consistency.md)                            |
+| Choose a layout and write the settings file         | [Configuration](docs/configuration.md)                        |
+| Cache reads for human users                         | [Identities](docs/configuration.md#identities-and-trust)      |
+| Change how long results are kept                    | [Maximum age rules](docs/configuration.md#maximum-age-rules)  |
+| Run Valkey and the invalidation plugin              | [Operations](docs/operations.md)                              |
+| Flush cached results after an incident              | [Recovery](docs/operations.md#flush-or-rotate-cached-results) |
+| Collect metrics and set alerts                      | [Metrics](docs/operations.md#metrics-and-alerts)              |
+| Measure savings for your workload                   | [Benchmarks](docs/benchmarks.md)                              |
+| See changes by version                              | [Changelog](CHANGELOG.md)                                     |
+
+## Development
+
+See the [development guide](docs/development.md) for the required tools, CI, and dependency locks.
+With those tools installed, run:
+
+```sh
+make setup
+make check
+```
+
+The checks never contact SG or Valkey; live Valkey integration tests are separate and need
+disposable services.
+
+## License
+
+sg-query-cache is available under the [MIT License](LICENSE). It is an independent project and is
+not affiliated with or endorsed by Autodesk. Flow Production Tracking is a product of Autodesk,
+Inc.