shareddata 6.86.0__tar.gz → 6.86.2__tar.gz

shareddata-6.86.2/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: shareddata
+Version: 6.86.2
+Summary: Memory-mapped record and time-series store with an HTTP API and a distributed scheduler
+Home-page: https://shareddata.net
+Author: shareddata
+Author-email: admin@shareddata.net
+License: MIT
+Project-URL: Homepage, https://shareddata.net
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11, <3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ipykernel<8,>=7.4
+Requires-Dist: python-json-logger<5,>=4.2
+Requires-Dist: python-dotenv<2,>=1.2.4
+Requires-Dist: pandas<3,>=2.2
+Requires-Dist: scipy<1.18,>=1.14
+Requires-Dist: numpy<1.27,>=1.26.4
+Requires-Dist: numba==0.59.1
+Requires-Dist: XlsxWriter<4,>=3.2.9
+Requires-Dist: openpyxl<4,>=3.1.5
+Requires-Dist: tqdm<5,>=4.70.1
+Requires-Dist: cffi<3,>=2.1
+Requires-Dist: tzlocal<6,>=5.4.4
+Requires-Dist: websockets<18,>=17.1
+Requires-Dist: lz4<5,>=4.4.5
+Requires-Dist: flask<4,>=3.1.3
+Requires-Dist: waitress<4,>=3.0.2
+Requires-Dist: requests<3,>=2.34.2
+Requires-Dist: flasgger==0.9.7.1
+Requires-Dist: pymongo<5,>=4.18.2
+Requires-Dist: setuptools>=84
+Requires-Dist: filelock<5,>=4.0.9
+Requires-Dist: gunicorn<27,>=26.2
+Requires-Dist: aiohttp<4,>=3.14.3
+Requires-Dist: boto3<2,>=1.43
+Requires-Dist: confluent-kafka<3,>=2.15.1
+Requires-Dist: aiokafka<1,>=0.14
+Requires-Dist: redis<9,>=8.1
+Requires-Dist: Authlib<2,>=1.8
+Requires-Dist: PyJWT<3,>=2.8
+Requires-Dist: argon2-cffi<26,>=25.1
+Requires-Dist: flask-jwt-extended<5,>=4.6
+Requires-Dist: cryptography<51,>=50.0.2
+Requires-Dist: segno<2,>=1.6
+Requires-Dist: psutil<8,>=5.9
+Requires-Dist: mcp>=1.27
+Requires-Dist: httpx>=0.27
+Provides-Extra: test
+Requires-Dist: pytest<10,>=8.3.4; extra == "test"
+Requires-Dist: pytest-asyncio<2,>=0.24; extra == "test"
+Requires-Dist: playwright<2,>=1.63; extra == "test"
+Provides-Extra: s3
+Provides-Extra: kafka
+Provides-Extra: redis
+Provides-Extra: all
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SharedData
+
+SharedData is an open-source memory-mapped store for records and time series —
+in the workload category of tick stores such as kdb+ — plus the scheduler and
+worker fleet that produce the data. Local processes map the same files through
+the operating system's page cache; indexed writes coordinate through a
+per-table process lock. Remote clients use an authenticated HTTP API; shared
+memory does not span hosts.
+
+Website and manual: <https://shareddata.net>
+
+## Install
+
+Python 3.11 or 3.12:
+
+```sh
+python -m pip install shareddata
+```
+
+The [quick start](https://shareddata.net/docs/QUICK_START/) uses illustrative
+records in isolated storage. See [configuration](https://shareddata.net/docs/CONFIGURATION/)
+for public environment names and [Windows limitations](https://shareddata.net/docs/WINDOWS/)
+for native requirements and rewrite behavior.
+
+## A record in one minute
+
+Save as `example.py` after setting up isolated storage in the
+[quick start](https://shareddata.net/docs/QUICK_START/), then run `python example.py`:
+
+```python
+from SharedData.SharedData import SharedData
+import pandas as pd
+
+store = SharedData(__file__, user="example")
+index = pd.MultiIndex.from_tuples(
+    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
+    names=["date", "symbol"],
+)
+rows = pd.DataFrame({"close": [100.0]}, index=index)
+records = store.table(
+    "MarketData", "D1", "DEMO", "PRICES",
+    value=rows,
+)
+print(records.loc["2026-01-02", "EXAMPLE"])
+```
+
+## Containers and execution
+
+- [Tables](https://shareddata.net/docs/core/table/): structured NumPy records and primary-key lookup.
+- [Time series](https://shareddata.net/docs/core/timeseries/): timestamped float matrices.
+- Collections, Redis caches, Kafka streams and optional S3 integration.
+- [HTTP API](https://shareddata.net/docs/guide/swagger/): authenticated remote access.
+- [Measured latency](https://shareddata.net/docs/performance/): what the containers cost in practice.
+
+## A store and the fleet that fills it
+
+A routine is any program the Scheduler owns — a feed collector, a parser, a
+consolidation job, an API server. A schedule lists the routines that should
+stay alive or run at their times. Batch routines wait for their batch dependencies
+to complete; realtime routines wait for their realtime dependencies to be running.
+Dependencies between the two kinds are not enforced at runtime. Commands travel
+through a per-host worker pool that validates the requested source revision
+before every launch; routines report heartbeats and lifecycle messages through
+the logging plane, and the Scheduler writes queryable state into the store.
+
+- [Orchestration](https://shareddata.net/docs/orchestration/): the routine model, the Scheduler's tick and the worker pool.
+- [Use cases](https://shareddata.net/docs/use-cases/): the workloads a production deployment runs, generalized.
+- [Schedules](https://shareddata.net/docs/scheduler-store/) and [batch jobs](https://shareddata.net/docs/job-store/): where the state lives.
+
+## Data by symbol
+
+The [semantic data API](https://shareddata.net/docs/data-api/) resolves a
+symbol or entity to a dataset and includes provenance. It is a convenience,
+not the fast path: producers keep using direct container interfaces for
+performance-critical work.
+
+## Client integration
+
+[Client integration](https://shareddata.net/docs/ecosystem/) covers
+authenticated API access, the MCP server for compatible agent clients, and
+worker access.
+
+## Development and license
+
+Submit changes through the repository's review process. Read the
+[development guide](https://shareddata.net/docs/DEVELOPMENT/) and run the
+checks relevant to your change; [building and publishing](https://shareddata.net/docs/guide/building/)
+explains how the public site is produced and audited. SharedData is released
+under the MIT license; see `LICENSE` in the source distribution.

shareddata-6.86.2/README.md

@@ -0,0 +1,89 @@
+# SharedData
+
+SharedData is an open-source memory-mapped store for records and time series —
+in the workload category of tick stores such as kdb+ — plus the scheduler and
+worker fleet that produce the data. Local processes map the same files through
+the operating system's page cache; indexed writes coordinate through a
+per-table process lock. Remote clients use an authenticated HTTP API; shared
+memory does not span hosts.
+
+Website and manual: <https://shareddata.net>
+
+## Install
+
+Python 3.11 or 3.12:
+
+```sh
+python -m pip install shareddata
+```
+
+The [quick start](https://shareddata.net/docs/QUICK_START/) uses illustrative
+records in isolated storage. See [configuration](https://shareddata.net/docs/CONFIGURATION/)
+for public environment names and [Windows limitations](https://shareddata.net/docs/WINDOWS/)
+for native requirements and rewrite behavior.
+
+## A record in one minute
+
+Save as `example.py` after setting up isolated storage in the
+[quick start](https://shareddata.net/docs/QUICK_START/), then run `python example.py`:
+
+```python
+from SharedData.SharedData import SharedData
+import pandas as pd
+
+store = SharedData(__file__, user="example")
+index = pd.MultiIndex.from_tuples(
+    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
+    names=["date", "symbol"],
+)
+rows = pd.DataFrame({"close": [100.0]}, index=index)
+records = store.table(
+    "MarketData", "D1", "DEMO", "PRICES",
+    value=rows,
+)
+print(records.loc["2026-01-02", "EXAMPLE"])
+```
+
+## Containers and execution
+
+- [Tables](https://shareddata.net/docs/core/table/): structured NumPy records and primary-key lookup.
+- [Time series](https://shareddata.net/docs/core/timeseries/): timestamped float matrices.
+- Collections, Redis caches, Kafka streams and optional S3 integration.
+- [HTTP API](https://shareddata.net/docs/guide/swagger/): authenticated remote access.
+- [Measured latency](https://shareddata.net/docs/performance/): what the containers cost in practice.
+
+## A store and the fleet that fills it
+
+A routine is any program the Scheduler owns — a feed collector, a parser, a
+consolidation job, an API server. A schedule lists the routines that should
+stay alive or run at their times. Batch routines wait for their batch dependencies
+to complete; realtime routines wait for their realtime dependencies to be running.
+Dependencies between the two kinds are not enforced at runtime. Commands travel
+through a per-host worker pool that validates the requested source revision
+before every launch; routines report heartbeats and lifecycle messages through
+the logging plane, and the Scheduler writes queryable state into the store.
+
+- [Orchestration](https://shareddata.net/docs/orchestration/): the routine model, the Scheduler's tick and the worker pool.
+- [Use cases](https://shareddata.net/docs/use-cases/): the workloads a production deployment runs, generalized.
+- [Schedules](https://shareddata.net/docs/scheduler-store/) and [batch jobs](https://shareddata.net/docs/job-store/): where the state lives.
+
+## Data by symbol
+
+The [semantic data API](https://shareddata.net/docs/data-api/) resolves a
+symbol or entity to a dataset and includes provenance. It is a convenience,
+not the fast path: producers keep using direct container interfaces for
+performance-critical work.
+
+## Client integration
+
+[Client integration](https://shareddata.net/docs/ecosystem/) covers
+authenticated API access, the MCP server for compatible agent clients, and
+worker access.
+
+## Development and license
+
+Submit changes through the repository's review process. Read the
+[development guide](https://shareddata.net/docs/DEVELOPMENT/) and run the
+checks relevant to your change; [building and publishing](https://shareddata.net/docs/guide/building/)
+explains how the public site is produced and audited. SharedData is released
+under the MIT license; see `LICENSE` in the source distribution.

{shareddata-6.86.0 → shareddata-6.86.2}/setup.py

@@ -66,8 +66,8 @@ test_requires = [
 
 setup(
     name='shareddata',
-    version='6.86.0',
-    description='Memory Mapped / Shared Memory Database with S3 repository',
+    version='6.86.2',
+    description='Memory-mapped record and time-series store with an HTTP API and a distributed scheduler',
     author='shareddata',
     author_email='admin@shareddata.net',
     url='https://shareddata.net',

{shareddata-6.86.0 → shareddata-6.86.2}/src/SharedData/API/apidocs.yml

@@ -2,7 +2,7 @@
 openapi: 3.0.3
 info:
   title: SharedData API
-  version: 6.86.0
+  version: 6.86.2
   description: |
     HTTP API of a SharedData server. The same operations are available from Python
     through `SharedData.IO.ClientAPI` (for example `ClientAPI.get_table(...)`).

{shareddata-6.86.0 → shareddata-6.86.2}/src/SharedData/IO/SaveTables.py

@@ -80,6 +80,29 @@ def _failed(row, error):
             'error': f'{type(error).__name__}: {error}'.replace('\n', ' ')[:300]}
 
 
+def live_daily_partitions(dftables: pd.DataFrame, now: datetime = None) -> pd.Series:
+    """Rows of daily partitions (YYYYMMDD) dated today or later in UTC.
+
+    A realtime writer is still appending to such a partition. Table.write holds the
+    table's mutex while it hashes, uploads and flushes the whole partition (minutes
+    for a 100 GB table), and every writer of the partition waits for that mutex, so
+    a backup would stall live data. The partition is complete the next night, when
+    it is still newer than the remote copy and is saved then.
+    """
+    now = datetime.now(timezone.utc) if now is None else now
+    today = now.astimezone(timezone.utc).strftime('%Y%m%d')
+    partition = dftables['partition'].fillna('').astype(str)
+    return partition.str.fullmatch(r'\d{8}') & (partition >= today)
+
+
+def select_tables(dftables: pd.DataFrame, now: datetime = None) -> pd.Series:
+    """Tables to save: local newer than remote (or no remote), not tiny, not live today."""
+    saveidx = (dftables['last_modified_local'].notnull())
+    saveidx = saveidx & ((dftables['last_modified_local'] > dftables['last_modified_remote']) | (dftables['last_modified_remote'].isnull()))
+    saveidx = saveidx & (dftables['size_local'] > 1000)
+    return saveidx & ~live_daily_partitions(dftables, now)
+
+
 def write_table(dftable_row: Dict[str, Any], retry=False) -> Dict[str, str]:
     """Save one table; return {'table', 'status': saved|failed|skipped, 'error'}."""
     path = table_path(dftable_row)
@@ -307,9 +330,13 @@ def main():
             're-uploading every local table would be catastrophic.')
         sys.exit(1)
 
-    saveidx = (dftables['last_modified_local'].notnull())
-    saveidx = saveidx & ((dftables['last_modified_local'] > dftables['last_modified_remote']) | (dftables['last_modified_remote'].isnull()))
-    saveidx = saveidx & (dftables['size_local'] > 1000)
+    saveidx = select_tables(dftables)
+    live = dftables[live_daily_partitions(dftables)
+                    & (dftables['last_modified_local'].notnull())
+                    & (dftables['size_local'] > 1000)]
+    if len(live):
+        Logger.log.info(f'SaveTables skipped {len(live)} live daily partitions (saved the next night): '
+                        + ', '.join(table_path(r) for r in live.to_dict(orient='records')))
 
     Logger.log.info('Saving the following tables:')
     for idx, row in dftables[saveidx].iterrows():

shareddata-6.86.2/src/shareddata.egg-info/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: shareddata
+Version: 6.86.2
+Summary: Memory-mapped record and time-series store with an HTTP API and a distributed scheduler
+Home-page: https://shareddata.net
+Author: shareddata
+Author-email: admin@shareddata.net
+License: MIT
+Project-URL: Homepage, https://shareddata.net
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11, <3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ipykernel<8,>=7.4
+Requires-Dist: python-json-logger<5,>=4.2
+Requires-Dist: python-dotenv<2,>=1.2.4
+Requires-Dist: pandas<3,>=2.2
+Requires-Dist: scipy<1.18,>=1.14
+Requires-Dist: numpy<1.27,>=1.26.4
+Requires-Dist: numba==0.59.1
+Requires-Dist: XlsxWriter<4,>=3.2.9
+Requires-Dist: openpyxl<4,>=3.1.5
+Requires-Dist: tqdm<5,>=4.70.1
+Requires-Dist: cffi<3,>=2.1
+Requires-Dist: tzlocal<6,>=5.4.4
+Requires-Dist: websockets<18,>=17.1
+Requires-Dist: lz4<5,>=4.4.5
+Requires-Dist: flask<4,>=3.1.3
+Requires-Dist: waitress<4,>=3.0.2
+Requires-Dist: requests<3,>=2.34.2
+Requires-Dist: flasgger==0.9.7.1
+Requires-Dist: pymongo<5,>=4.18.2
+Requires-Dist: setuptools>=84
+Requires-Dist: filelock<5,>=4.0.9
+Requires-Dist: gunicorn<27,>=26.2
+Requires-Dist: aiohttp<4,>=3.14.3
+Requires-Dist: boto3<2,>=1.43
+Requires-Dist: confluent-kafka<3,>=2.15.1
+Requires-Dist: aiokafka<1,>=0.14
+Requires-Dist: redis<9,>=8.1
+Requires-Dist: Authlib<2,>=1.8
+Requires-Dist: PyJWT<3,>=2.8
+Requires-Dist: argon2-cffi<26,>=25.1
+Requires-Dist: flask-jwt-extended<5,>=4.6
+Requires-Dist: cryptography<51,>=50.0.2
+Requires-Dist: segno<2,>=1.6
+Requires-Dist: psutil<8,>=5.9
+Requires-Dist: mcp>=1.27
+Requires-Dist: httpx>=0.27
+Provides-Extra: test
+Requires-Dist: pytest<10,>=8.3.4; extra == "test"
+Requires-Dist: pytest-asyncio<2,>=0.24; extra == "test"
+Requires-Dist: playwright<2,>=1.63; extra == "test"
+Provides-Extra: s3
+Provides-Extra: kafka
+Provides-Extra: redis
+Provides-Extra: all
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SharedData
+
+SharedData is an open-source memory-mapped store for records and time series —
+in the workload category of tick stores such as kdb+ — plus the scheduler and
+worker fleet that produce the data. Local processes map the same files through
+the operating system's page cache; indexed writes coordinate through a
+per-table process lock. Remote clients use an authenticated HTTP API; shared
+memory does not span hosts.
+
+Website and manual: <https://shareddata.net>
+
+## Install
+
+Python 3.11 or 3.12:
+
+```sh
+python -m pip install shareddata
+```
+
+The [quick start](https://shareddata.net/docs/QUICK_START/) uses illustrative
+records in isolated storage. See [configuration](https://shareddata.net/docs/CONFIGURATION/)
+for public environment names and [Windows limitations](https://shareddata.net/docs/WINDOWS/)
+for native requirements and rewrite behavior.
+
+## A record in one minute
+
+Save as `example.py` after setting up isolated storage in the
+[quick start](https://shareddata.net/docs/QUICK_START/), then run `python example.py`:
+
+```python
+from SharedData.SharedData import SharedData
+import pandas as pd
+
+store = SharedData(__file__, user="example")
+index = pd.MultiIndex.from_tuples(
+    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
+    names=["date", "symbol"],
+)
+rows = pd.DataFrame({"close": [100.0]}, index=index)
+records = store.table(
+    "MarketData", "D1", "DEMO", "PRICES",
+    value=rows,
+)
+print(records.loc["2026-01-02", "EXAMPLE"])
+```
+
+## Containers and execution
+
+- [Tables](https://shareddata.net/docs/core/table/): structured NumPy records and primary-key lookup.
+- [Time series](https://shareddata.net/docs/core/timeseries/): timestamped float matrices.
+- Collections, Redis caches, Kafka streams and optional S3 integration.
+- [HTTP API](https://shareddata.net/docs/guide/swagger/): authenticated remote access.
+- [Measured latency](https://shareddata.net/docs/performance/): what the containers cost in practice.
+
+## A store and the fleet that fills it
+
+A routine is any program the Scheduler owns — a feed collector, a parser, a
+consolidation job, an API server. A schedule lists the routines that should
+stay alive or run at their times. Batch routines wait for their batch dependencies
+to complete; realtime routines wait for their realtime dependencies to be running.
+Dependencies between the two kinds are not enforced at runtime. Commands travel
+through a per-host worker pool that validates the requested source revision
+before every launch; routines report heartbeats and lifecycle messages through
+the logging plane, and the Scheduler writes queryable state into the store.
+
+- [Orchestration](https://shareddata.net/docs/orchestration/): the routine model, the Scheduler's tick and the worker pool.
+- [Use cases](https://shareddata.net/docs/use-cases/): the workloads a production deployment runs, generalized.
+- [Schedules](https://shareddata.net/docs/scheduler-store/) and [batch jobs](https://shareddata.net/docs/job-store/): where the state lives.
+
+## Data by symbol
+
+The [semantic data API](https://shareddata.net/docs/data-api/) resolves a
+symbol or entity to a dataset and includes provenance. It is a convenience,
+not the fast path: producers keep using direct container interfaces for
+performance-critical work.
+
+## Client integration
+
+[Client integration](https://shareddata.net/docs/ecosystem/) covers
+authenticated API access, the MCP server for compatible agent clients, and
+worker access.
+
+## Development and license
+
+Submit changes through the repository's review process. Read the
+[development guide](https://shareddata.net/docs/DEVELOPMENT/) and run the
+checks relevant to your change; [building and publishing](https://shareddata.net/docs/guide/building/)
+explains how the public site is produced and audited. SharedData is released
+under the MIT license; see `LICENSE` in the source distribution.