shareddata 6.85.18__tar.gz → 6.86.1__tar.gz

shareddata-6.86.1/PKG-INFO

@@ -0,0 +1,164 @@
+Metadata-Version: 2.4
+Name: shareddata
+Version: 6.86.1
+Summary: Memory-mapped record and time-series store with an HTTP API and a distributed scheduler
+Home-page: https://shareddata.net
+Author: shareddata
+Author-email: admin@shareddata.net
+License: MIT
+Project-URL: Homepage, https://shareddata.net
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11, <3.13
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ipykernel<8,>=7.4
+Requires-Dist: python-json-logger<5,>=4.2
+Requires-Dist: python-dotenv<2,>=1.2.4
+Requires-Dist: pandas<3,>=2.2
+Requires-Dist: scipy<1.18,>=1.14
+Requires-Dist: numpy<1.27,>=1.26.4
+Requires-Dist: numba==0.59.1
+Requires-Dist: XlsxWriter<4,>=3.2.9
+Requires-Dist: openpyxl<4,>=3.1.5
+Requires-Dist: tqdm<5,>=4.70.1
+Requires-Dist: cffi<3,>=2.1
+Requires-Dist: tzlocal<6,>=5.4.4
+Requires-Dist: websockets<18,>=17.1
+Requires-Dist: lz4<5,>=4.4.5
+Requires-Dist: flask<4,>=3.1.3
+Requires-Dist: waitress<4,>=3.0.2
+Requires-Dist: requests<3,>=2.34.2
+Requires-Dist: flasgger==0.9.7.1
+Requires-Dist: pymongo<5,>=4.18.2
+Requires-Dist: setuptools>=84
+Requires-Dist: filelock<5,>=4.0.9
+Requires-Dist: gunicorn<27,>=26.2
+Requires-Dist: aiohttp<4,>=3.14.3
+Requires-Dist: boto3<2,>=1.43
+Requires-Dist: confluent-kafka<3,>=2.15.1
+Requires-Dist: aiokafka<1,>=0.14
+Requires-Dist: redis<9,>=8.1
+Requires-Dist: Authlib<2,>=1.8
+Requires-Dist: PyJWT<3,>=2.8
+Requires-Dist: argon2-cffi<26,>=25.1
+Requires-Dist: flask-jwt-extended<5,>=4.6
+Requires-Dist: cryptography<51,>=50.0.2
+Requires-Dist: segno<2,>=1.6
+Requires-Dist: psutil<8,>=5.9
+Requires-Dist: mcp>=1.27
+Requires-Dist: httpx>=0.27
+Provides-Extra: test
+Requires-Dist: pytest<10,>=8.3.4; extra == "test"
+Requires-Dist: pytest-asyncio<2,>=0.24; extra == "test"
+Requires-Dist: playwright<2,>=1.63; extra == "test"
+Provides-Extra: s3
+Provides-Extra: kafka
+Provides-Extra: redis
+Provides-Extra: all
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# SharedData
+
+SharedData is an open-source memory-mapped store for records and time series —
+in the workload category of tick stores such as kdb+ — plus the scheduler and
+worker fleet that produce the data. Local processes map the same files through
+the operating system's page cache; indexed writes coordinate through a
+per-table process lock. Remote clients use an authenticated HTTP API; shared
+memory does not span hosts.
+
+Website and manual: <https://shareddata.net>
+
+## Install
+
+Python 3.11 or 3.12:
+
+```sh
+python -m pip install shareddata
+```
+
+The [quick start](https://shareddata.net/docs/QUICK_START/) uses illustrative
+records in isolated storage. See [configuration](https://shareddata.net/docs/CONFIGURATION/)
+for public environment names and [Windows limitations](https://shareddata.net/docs/WINDOWS/)
+for native requirements and rewrite behavior.
+
+## A record in one minute
+
+Save as `example.py` after setting up isolated storage in the
+[quick start](https://shareddata.net/docs/QUICK_START/), then run `python example.py`:
+
+```python
+from SharedData.SharedData import SharedData
+import pandas as pd
+
+store = SharedData(__file__, user="example")
+index = pd.MultiIndex.from_tuples(
+    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
+    names=["date", "symbol"],
+)
+rows = pd.DataFrame({"close": [100.0]}, index=index)
+records = store.table(
+    "MarketData", "D1", "DEMO", "PRICES",
+    value=rows,
+)
+print(records.loc["2026-01-02", "EXAMPLE"])
+```
+
+## Containers and execution
+
+- [Tables](https://shareddata.net/docs/core/table/): structured NumPy records and primary-key lookup.
+- [Time series](https://shareddata.net/docs/core/timeseries/): timestamped float matrices.
+- Collections, Redis caches, Kafka streams and optional S3 integration.
+- [HTTP API](https://shareddata.net/docs/guide/swagger/): authenticated remote access.
+- [Measured latency](https://shareddata.net/docs/performance/): what the containers cost in practice.
+
+## A store and the fleet that fills it
+
+A routine is any program the Scheduler owns — a feed collector, a parser, a
+consolidation job, an API server. A schedule lists the routines that should
+stay alive or run at their times. Batch routines wait for their batch dependencies
+to complete; realtime routines wait for their realtime dependencies to be running.
+Dependencies between the two kinds are not enforced at runtime. Commands travel
+through a per-host worker pool that validates the requested source revision
+before every launch; routines report heartbeats and lifecycle messages through
+the logging plane, and the Scheduler writes queryable state into the store.
+
+- [Orchestration](https://shareddata.net/docs/orchestration/): the routine model, the Scheduler's tick and the worker pool.
+- [Use cases](https://shareddata.net/docs/use-cases/): the workloads a production deployment runs, generalized.
+- [Schedules](https://shareddata.net/docs/scheduler-store/) and [batch jobs](https://shareddata.net/docs/job-store/): where the state lives.
+
+## Data by symbol
+
+The [semantic data API](https://shareddata.net/docs/data-api/) resolves a
+symbol or entity to a dataset and includes provenance. It is a convenience,
+not the fast path: producers keep using direct container interfaces for
+performance-critical work.
+
+## Client integration
+
+[Client integration](https://shareddata.net/docs/ecosystem/) covers
+authenticated API access, the MCP server for compatible agent clients, and
+worker access.
+
+## Development and license
+
+Submit changes through the repository's review process. Read the
+[development guide](https://shareddata.net/docs/DEVELOPMENT/) and run the
+checks relevant to your change; [building and publishing](https://shareddata.net/docs/guide/building/)
+explains how the public site is produced and audited. SharedData is released
+under the MIT license; see `LICENSE` in the source distribution.

shareddata-6.86.1/README.md

@@ -0,0 +1,89 @@
+# SharedData
+
+SharedData is an open-source memory-mapped store for records and time series —
+in the workload category of tick stores such as kdb+ — plus the scheduler and
+worker fleet that produce the data. Local processes map the same files through
+the operating system's page cache; indexed writes coordinate through a
+per-table process lock. Remote clients use an authenticated HTTP API; shared
+memory does not span hosts.
+
+Website and manual: <https://shareddata.net>
+
+## Install
+
+Python 3.11 or 3.12:
+
+```sh
+python -m pip install shareddata
+```
+
+The [quick start](https://shareddata.net/docs/QUICK_START/) uses illustrative
+records in isolated storage. See [configuration](https://shareddata.net/docs/CONFIGURATION/)
+for public environment names and [Windows limitations](https://shareddata.net/docs/WINDOWS/)
+for native requirements and rewrite behavior.
+
+## A record in one minute
+
+Save as `example.py` after setting up isolated storage in the
+[quick start](https://shareddata.net/docs/QUICK_START/), then run `python example.py`:
+
+```python
+from SharedData.SharedData import SharedData
+import pandas as pd
+
+store = SharedData(__file__, user="example")
+index = pd.MultiIndex.from_tuples(
+    [(pd.Timestamp("2026-01-02"), "EXAMPLE")],
+    names=["date", "symbol"],
+)
+rows = pd.DataFrame({"close": [100.0]}, index=index)
+records = store.table(
+    "MarketData", "D1", "DEMO", "PRICES",
+    value=rows,
+)
+print(records.loc["2026-01-02", "EXAMPLE"])
+```
+
+## Containers and execution
+
+- [Tables](https://shareddata.net/docs/core/table/): structured NumPy records and primary-key lookup.
+- [Time series](https://shareddata.net/docs/core/timeseries/): timestamped float matrices.
+- Collections, Redis caches, Kafka streams and optional S3 integration.
+- [HTTP API](https://shareddata.net/docs/guide/swagger/): authenticated remote access.
+- [Measured latency](https://shareddata.net/docs/performance/): what the containers cost in practice.
+
+## A store and the fleet that fills it
+
+A routine is any program the Scheduler owns — a feed collector, a parser, a
+consolidation job, an API server. A schedule lists the routines that should
+stay alive or run at their times. Batch routines wait for their batch dependencies
+to complete; realtime routines wait for their realtime dependencies to be running.
+Dependencies between the two kinds are not enforced at runtime. Commands travel
+through a per-host worker pool that validates the requested source revision
+before every launch; routines report heartbeats and lifecycle messages through
+the logging plane, and the Scheduler writes queryable state into the store.
+
+- [Orchestration](https://shareddata.net/docs/orchestration/): the routine model, the Scheduler's tick and the worker pool.
+- [Use cases](https://shareddata.net/docs/use-cases/): the workloads a production deployment runs, generalized.
+- [Schedules](https://shareddata.net/docs/scheduler-store/) and [batch jobs](https://shareddata.net/docs/job-store/): where the state lives.
+
+## Data by symbol
+
+The [semantic data API](https://shareddata.net/docs/data-api/) resolves a
+symbol or entity to a dataset and includes provenance. It is a convenience,
+not the fast path: producers keep using direct container interfaces for
+performance-critical work.
+
+## Client integration
+
+[Client integration](https://shareddata.net/docs/ecosystem/) covers
+authenticated API access, the MCP server for compatible agent clients, and
+worker access.
+
+## Development and license
+
+Submit changes through the repository's review process. Read the
+[development guide](https://shareddata.net/docs/DEVELOPMENT/) and run the
+checks relevant to your change; [building and publishing](https://shareddata.net/docs/guide/building/)
+explains how the public site is produced and audited. SharedData is released
+under the MIT license; see `LICENSE` in the source distribution.

{shareddata-6.85.18 → shareddata-6.86.1}/setup.py

@@ -66,8 +66,8 @@ test_requires = [
 
 setup(
     name='shareddata',
-    version='6.85.18',
-    description='Memory Mapped / Shared Memory Database with S3 repository',
+    version='6.86.1',
+    description='Memory-mapped record and time-series store with an HTTP API and a distributed scheduler',
     author='shareddata',
     author_email='admin@shareddata.net',
     url='https://shareddata.net',

{shareddata-6.85.18 → shareddata-6.86.1}/src/SharedData/API/apidocs.yml

@@ -2,7 +2,7 @@
 openapi: 3.0.3
 info:
   title: SharedData API
-  version: 6.85.1
+  version: 6.86.1
   description: |
     HTTP API of a SharedData server. The same operations are available from Python
     through `SharedData.IO.ClientAPI` (for example `ClientAPI.get_table(...)`).
@@ -811,6 +811,8 @@ paths:
                 format: binary
         '204':
           description: No data found
+        '404':
+          description: JSON TagNotFound with type and message; the tag does not exist.
         '401':
           $ref: '#/components/responses/Unauthorized'
         '500':
@@ -854,7 +856,8 @@ paths:
               format: binary
       responses:
         '201':
-          description: Data inserted
+          description: 'Data inserted. JSON with status, tag, message and `appended_columns`: the columns added
+            to an existing tag (empty for a new tag).'
         '401':
           $ref: '#/components/responses/Unauthorized'
         '500':
@@ -1111,7 +1114,8 @@ paths:
         '500':
           $ref: '#/components/responses/ServerError'
       description: 'Upserts rows sent as BSON+LZ4 by primary key; creates the table on first write when `names`,
-        `formats` and `size` are given. Python: `ClientAPI.post_table(...)`.'
+        `formats` and `size` are given. A `size` larger than the capacity of an existing table grows it. Python:
+        `ClientAPI.post_table(...)`.'
     delete:
       tags:
       - Tables
@@ -1290,7 +1294,7 @@ paths:
     post:
       tags:
       - Collections
-      summary: Insert documents (lz4+BSON or JSON)
+      summary: Insert or upsert documents (lz4+BSON or JSON)
       parameters:
       - in: path
         name: database
@@ -1327,20 +1331,66 @@ paths:
               format: binary
           application/json:
             schema:
-              type: array
-              items:
-                type: object
+              oneOf:
+              - type: array
+                nullable: true
+                items:
+                  type: object
+              - type: object
       responses:
         '201':
-          description: Created
+          description: Batch accepted. An empty batch succeeds with documents equal to zero.
+          content:
+            application/json:
+              schema:
+                type: object
+                required: [status, documents]
+                properties:
+                  status:
+                    type: string
+                    enum: [success]
+                  documents:
+                    type: integer
+                    minimum: 0
+                    description: Documents accepted in this request (not the number of newly inserted rows).
+              example: {status: success, documents: 0}
         '401':
           $ref: '#/components/responses/Unauthorized'
         '400':
-          description: Invalid data
+          description: >-
+            Empty or undecodable body, missing BSON data field, invalid document shape,
+            missing primary-key fields, or an incompatible append/upsert mode.
+            The entire request is rejected; none of its documents are written.
+          headers:
+            Error-Message:
+              description: Reason the request was rejected, also returned in the JSON message.
+              schema:
+                type: string
+          content:
+            application/json:
+              schema:
+                type: object
+                required: [type, message]
+                properties:
+                  type:
+                    type: string
+                    enum: [BadRequest]
+                  message:
+                    type: string
+              example: {type: BadRequest, message: "data must be a list of documents, not int"}
         '500':
           $ref: '#/components/responses/ServerError'
-      description: Inserts documents (lz4+BSON or JSON). With `hasindex` (default) documents are upserted by the
-        database's primary key; without it they are appended.
+      description: >-
+        Accepts LZ4-compressed BSON {"data": [documents]} with Content-Encoding lz4,
+        or a plain JSON list. A single document object is also accepted. BSON data
+        equal to null or [] (and JSON null or []) is an empty batch: returns 201 with
+        documents equal to zero without opening or changing the collection.
+        With hasindex=true (default), every document must contain every field of
+        the database's primary key and documents are upserted; with hasindex=false,
+        they are appended and the collection must have no primary-key index.
+        Missing keys reject the whole request, including otherwise valid documents.
+        A real storage/server failure returns 500 and can leave partial writes;
+        only validation errors guarantee that nothing was written.
     patch:
       tags:
       - Collections
@@ -1835,7 +1885,9 @@ paths:
         '403':
           description: Master token required
       description: 'Each route: `id`, `command` (its name on the index), `path`, `target`, `group`, `description`,
-        `enabled`, `revision`. Master token.'
+        `enabled`, `revision`. The body also has `activation`: `applied_generation`, `desired_generation`,
+        and `state` (`unpublished` before any activation status has been recorded; later states include
+        `pending`, `applied`, `unconfirmed`, and `failed`). Master token.'
     post:
       tags:
       - Routes
@@ -1880,6 +1932,21 @@ paths:
       description: 'Adds a service behind nginx on this deploy''s port: requests to `path` are proxied to `target`
         (a private or local http(s) URL), behind the SharedData sign-in. nginx is checked and reloaded before the
         answer, so the route works when it returns; `reachable` says whether the target answered. Master token.'
+  /api/routes/activation/verify:
+    post:
+      tags:
+      - Routes
+      summary: Check that every configured proxy is serving this generation (master token)
+      responses:
+        '200':
+          description: 'activation: applied_generation, desired_generation, state, and the probe results'
+        '400':
+          description: SHAREDDATA_WEB_PROXY_PROBES is missing or a probe URL is not http(s) without credentials
+        '403':
+          description: Master token required
+      description: 'Reads `SHAREDDATA_WEB_PROXY_PROBES` (a JSON object of proxy name to marker URL) and GETs each
+        URL with a 3 second timeout and no redirects. A probe passes when the status is 200 and the stripped body equals
+        `{generation}:{sha256 of the catalog}`. Master token.'
   /api/routes/{route_id}:
     parameters:
     - name: route_id
@@ -2952,6 +3019,68 @@ paths:
           description: Unknown command token
         '403':
           description: Not a master
+  /api/workers/shared-tokens:
+    get:
+      tags:
+      - Workers
+      summary: List shared worker tokens (master)
+      description: One token shared by a group of interchangeable workers. The list is the label, the name pattern,
+        the creator and the last use. The secret is not included. Tokens that workers already presented are listed
+        after the ones created here.
+      responses:
+        '200':
+          description: Shared worker tokens without their secret
+        '403':
+          description: Not a master
+    post:
+      tags:
+      - Workers
+      summary: Create a shared worker token (master)
+      description: Shown once. `names` is the worker-name pattern, with `*` for the part that changes, for example
+        `*@SCALE-*`. At least three characters must be fixed. The label is letters, digits, dot, dash or underscore,
+        up to 40 characters. Revoke the existing label before creating it again.
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              type: object
+              required:
+              - label
+              - names
+              properties:
+                label:
+                  type: string
+                  example: scale
+                names:
+                  type: string
+                  example: '*@SCALE-*'
+      responses:
+        '201':
+          description: '`{token, label, names, enabled, created_at, created_by, last_seen_at}` — `token` only on this response'
+        '400':
+          description: Invalid label or names, or a token with that label already exists
+        '403':
+          description: Not a master
+  /api/workers/shared-tokens/{label}:
+    delete:
+      tags:
+      - Workers
+      summary: Revoke a shared worker token (master)
+      parameters:
+      - name: label
+        in: path
+        required: true
+        schema:
+          type: string
+      description: The token stops working (API processes cache tokens up to 5 minutes).
+      responses:
+        '200':
+          description: Revoked
+        '404':
+          description: Unknown shared worker token
+        '403':
+          description: Not a master
   /api/data/sources:
     get:
       tags:

{shareddata-6.85.18 → shareddata-6.86.1}/src/SharedData/API/routes/collections.py

@@ -6,6 +6,7 @@ import datetime
 import gzip
 import json
 import time
+import traceback
 
 import bson
 from bson import json_util
@@ -17,6 +18,7 @@ from flask import Blueprint, Response, jsonify, request
 
 from SharedData.CollectionMongoDB import CollectionMongoDB
 from SharedData.Database import DATABASE_PKEYS
+from SharedData.Logger import Logger, redact_credentials
 
 from SharedData.API.constants import (
     DEFAULT_USER,
@@ -36,6 +38,73 @@ from SharedData.API.utils import (
 # Blueprint definition
 collections_bp = Blueprint('collections', __name__)
 
+
+class InvalidPayload(ValueError):
+    """The posted body is not something that can be written: answered 400 with this message."""
+
+
+def payload_error(error, status=400):
+    """A 4xx whose reason is in the Error-Message header (what ClientAPI reports) and the JSON body."""
+    message = redact_credentials(error).replace('\r', ' ').replace('\n', ' ')
+    response = make_error_response(message, status=status, error_type='BadRequest')
+    response.headers['Error-Message'] = message
+    return response
+
+
+def decode_posted_documents(request):
+    """The documents of a collection POST, as a list of dicts.
+
+    None (what ClientAPI sent for an empty list before 6.86, issue #112) is an empty batch; a single
+    document is a batch of one. Anything else that is not a list of documents raises InvalidPayload."""
+    if not request.data:
+        raise InvalidPayload('empty request body: send {"data": [documents]} (lz4 BSON) or a JSON list')
+    lz4 = request.headers.get('Content-Encoding', '').lower() == 'lz4'
+    try:
+        if lz4:
+            payload = bson.decode(lz4f.decompress(request.data))
+        else:
+            payload = json.loads(request.data)
+    except Exception as e:
+        raise InvalidPayload(f'request body could not be decoded as {"lz4 BSON" if lz4 else "JSON"}: '
+                             f'{type(e).__name__}: {e}')
+    if lz4:
+        if 'data' not in payload:
+            raise InvalidPayload("lz4 BSON body has no 'data' field")
+        documents = payload['data']
+    else:
+        documents = payload
+    if documents is None:
+        return []
+    if isinstance(documents, dict):
+        return [documents]
+    if not isinstance(documents, list):
+        raise InvalidPayload(f'data must be a list of documents, not {type(documents).__name__}')
+    for i, document in enumerate(documents):
+        if not isinstance(document, dict):
+            raise InvalidPayload(f'document {i} is {type(document).__name__}, not an object '
+                                 f'(data must be a list of documents)')
+    return documents
+
+
+def missing_pkey_report(documents, pkey_columns):
+    """Why these documents cannot be upserted, or '' when every one has every primary-key field."""
+    missing = {}
+    incomplete = 0
+    first = None
+    for i, document in enumerate(documents):
+        absent = [field for field in pkey_columns if field not in document]
+        if absent:
+            incomplete += 1
+            first = i if first is None else first
+            for field in absent:
+                missing[field] = missing.get(field, 0) + 1
+    if not incomplete:
+        return ''
+    fields = ', '.join(f'{field} ({count})' for field, count in missing.items())
+    return (f'{incomplete}/{len(documents)} documents miss primary-key fields: {fields}; '
+            f'the primary key is {", ".join(pkey_columns)} (first incomplete document: {first}); '
+            'nothing was written')
+
 def parse_json_query(query: dict) -> dict:
     """
     Recursively parses a MongoDB-like query dictionary, converting string representations 
@@ -145,11 +214,15 @@ def init_collections_routes(shdata, authenticate_fn):
                 return jsonify({'error': 'method not allowed'}), 405
             
         except Exception as e:
+            # A server-side failure: keep its stack in the server log, the client only gets the message.
+            Logger.log.error(redact_credentials(f'collection {request.method} {database}/{period}/{source}/{tablename}'
+                             f'{"/" + request.args["tablesubfolder"] if request.args.get("tablesubfolder") else ""}: '
+                             f'{type(e).__name__}: {e}\n{traceback.format_exc()}'))
             time.sleep(ERROR_SLEEP_SECONDS)
-            response = Response(status=500)                
-            response.headers['Error-Message'] = str(e).replace('\n', ' ')
+            response = Response(status=500)
+            response.headers['Error-Message'] = redact_credentials(e).replace('\r', ' ').replace('\n', ' ')
             return response
-        
+
     def head_collection(database: str, period: str, source: str, tablename: str, request):
         """Handle HTTP HEAD requests by returning metadata headers for a specified MongoDB collection."""
         user = request.args.get('user', DEFAULT_USER)
@@ -302,37 +375,49 @@ def init_collections_routes(shdata, authenticate_fn):
                 response.headers['Content-Length'] = len(response_json)
                 return with_cursor(response)
 
-    def post_collection(database, period, source, tablename, request):    
-        """Handle a POST request to insert or upsert a collection of documents."""
+    def post_collection(database, period, source, tablename, request):
+        """Handle a POST request to insert or upsert a collection of documents.
+
+        Body: lz4-compressed BSON {"data": [documents]} (what ClientAPI sends) or a plain JSON list.
+        A payload that is not a list of documents, or (upsert) documents missing a primary-key
+        field, is a 400 saying why and nothing is written. An empty batch is nothing to write."""
         tablesubfolder = request.args.get('tablesubfolder')
         user = request.args.get('user', DEFAULT_USER)
         hasindex = parse_bool_param(request.args.get('hasindex'), True)
 
         tablepath = apply_subfolder(tablename, tablesubfolder)
 
-        collection = shdata.collection(database, period, source, tablepath, user=user, hasindex=hasindex)
-
-        if not request.data:
-            Response({'message': 'No data'}, status=400)
-        
-        if request.headers.get('Content-Encoding', '').lower() == 'lz4':
-            decompressed = lz4f.decompress(request.data)
-            documents = bson.decode(decompressed)['data']
-        else:
-            documents = json.loads(request.data)
-
-        if not isinstance(documents, list):
-            Response({'message': 'Data must be a list'}, status=400)
-        
-        if hasindex:
-            collection.upsert(documents)
-        else:
-            collection.extend(documents)
-
-        response_data = json.dumps({'status': 'success'}).encode('utf-8')
+        try:
+            documents = decode_posted_documents(request)
+        except InvalidPayload as e:
+            return payload_error(e)
+
+        if documents:
+            # Opening a collection can create it or upgrade its indexes. An empty upload
+            # must not do either (nor fail building a unique index on append-only data).
+            collection = shdata.collection(database, period, source, tablepath, user=user, hasindex=hasindex)
+            if hasindex:
+                if not collection.hasindex:
+                    return payload_error(
+                        f'{collection.relpath} has no unique index on its primary key '
+                        f'({", ".join(collection.pkey_columns)}), so it cannot be upserted; '
+                        'post with hasindex=False to append')
+                missing = missing_pkey_report(documents, collection.pkey_columns)
+                if missing:
+                    return payload_error(missing)
+                collection.upsert(documents)
+            else:
+                if collection.hasindex:
+                    return payload_error(
+                        f'{collection.relpath} has a unique index on its primary key '
+                        f'({", ".join(collection.pkey_columns)}), so documents cannot be appended; '
+                        'post with hasindex=True to upsert')
+                collection.extend(documents)
+
+        response_data = json.dumps({'status': 'success', 'documents': len(documents)}).encode('utf-8')
         response = Response(response_data, status=201, mimetype='application/json')
         response.headers['Content-Length'] = str(len(response_data))
-        return response   
+        return response
 
     def patch_collection(database, period, source, tablename, request):
         """Update a single document in a specified collection within the database."""