coaiajs 0.4.2 → 0.4.3

package/README.md

@@ -50,7 +50,8 @@ Import from the root, or from a subpath to pull in only what you need. Every ent
 | `coaiajs/github` | `listIssues`, `getIssue`, `getIssueComments`, `resetClient` |
 | `coaiajs/environment` | `EnvironmentManager`, `createEnvironment`, `findEnvironment` |
 | `coaiajs/version` | `getPackageVersion` |
-| `coaiajs/langfuse` | Langfuse JS SDK v5/OpenTelemetry tracing plus v4 observations, prompts, datasets, scores, comments, media, projects, and formatters (55 exports) |
+| `coaiajs/langfuse` | Langfuse JS SDK v5/OpenTelemetry tracing plus v4 observations, prompts, datasets, scores, comments, media, projects, and formatters (56 runtime exports) |
+| `coaiajs/media-upload-proxy` | Deployable Custom GPT conversation-file to Langfuse media bridge |
 | `coaiajs/narrative` | `KnowledgeGraphManager`, chart operations, markdown export, tool definitions (29 exports) |
 | `coaiajs/pde` | `SessionManager`, `StcMapper`, `handlePdeTool`, `PDE_MCP_TOOLS` (10 exports) |
 | `coaiajs/planning` | `parsePlan`, `planToSTC`, `syncToChart`, `syncToPlan`, `handlePlanningTool` (10 exports) |
@@ -140,7 +141,9 @@ Migration references: [Langfuse v4](https://langfuse.com/docs/v4) · [versions a
 
 ### Custom GPT observation actions
 
-[`agents/custom_gpt/ceremony-observations.yml`](./agents/custom_gpt/ceremony-observations.yml) is a focused OpenAPI 3.1 action specification for creating a root observation and appending child observations to the same Langfuse trace through OTLP/HTTP JSON. It contains 20 core actions—below the 30-action limit—and omits media and destructive operations. Setup, authentication, ID/timestamp rules, and ready-to-paste GPT instructions are in [`ceremony-observations.instructions.md`](./agents/custom_gpt/ceremony-observations.instructions.md). The broader imported API surface remains available in [`ceremony.yml`](./agents/custom_gpt/ceremony.yml).
+[`agents/custom_gpt/ceremony-observations.yml`](./agents/custom_gpt/ceremony-observations.yml) is a focused OpenAPI 3.1 action specification for creating a root observation and appending child observations to the same Langfuse trace through OTLP/HTTP JSON. It contains 23 core actions—below the 30-action limit—including media upload-record, retrieval, and finalization capabilities; destructive operations remain omitted. Media hashes use the required 44-character padded Base64 SHA-256 digest of the exact upload bytes, and the documented workflow distinguishes URL creation from the actual presigned PUT. Setup, authentication, ID/timestamp rules, and ready-to-paste GPT instructions are in [`ceremony-observations.instructions.md`](./agents/custom_gpt/ceremony-observations.instructions.md). The broader imported API surface remains available in [`ceremony.yml`](./agents/custom_gpt/ceremony.yml).
+
+For autonomous conversation-file uploads, deploy the included `coaiajs-media-proxy` service and import [`ceremony-media-proxy.yml`](./agents/custom_gpt/ceremony-media-proxy.yml) as a second Action. It uses OpenAI's `openaiFileIdRefs` transfer, downloads the actual user/DALL-E/Code Interpreter file, performs the Langfuse media record + presigned PUT + finalization sequence, and returns a renderable `@@@langfuseMedia...@@@` token. Deployment and security instructions are in [`ceremony-media-proxy.instructions.md`](./agents/custom_gpt/ceremony-media-proxy.instructions.md). Together both specs expose 24 actions.
 
 ---
 

package/agents/custom_gpt/ceremony-media-proxy.instructions.md

@@ -0,0 +1,86 @@
+# Ceremony Media Upload Proxy
+
+`ceremony-media-proxy.yml` is the executable bridge missing from direct Langfuse media actions. It uses the official Custom GPT `openaiFileIdRefs` mechanism, so the GPT sends a temporary reference to the user's real file instead of attempting to place a large Base64 payload in function arguments.
+
+## Why this is a second action specification
+
+The main `ceremony-observations.yml` talks directly to `cloud.langfuse.com`. A presigned media URL points to a dynamic object-storage host, which cannot be represented as a static OpenAPI path. The bridge therefore runs on a public HTTPS service you control and is imported as a second GPT Action. Together the specifications expose 24 actions, below the 30-action limit.
+
+## Deploy the bridge
+
+Build and run the included Node service:
+
+```bash
+npm install
+npm run build
+
+export LANGFUSE_PUBLIC_KEY='pk-lf-...'
+export LANGFUSE_SECRET_KEY='sk-lf-...'
+export LANGFUSE_BASE_URL='https://cloud.langfuse.com'
+export COAIA_MEDIA_PROXY_API_KEY='generate-a-long-random-secret'
+export PORT=8787
+
+node dist/src/media-upload-proxy.js
+# or, from the installed package:
+coaiajs-media-proxy
+```
+
+Deploy it behind a public HTTPS URL. The Custom GPT service cannot reach localhost.
+
+Optional controls:
+
+- `COAIA_MEDIA_MAX_BYTES`: maximum downloaded file size; defaults to 10 MiB.
+- `COAIA_OPENAI_FILE_HOSTS`: comma-separated additional trusted conversation-file hosts.
+- `COAIA_MEDIA_UPLOAD_HOSTS`: comma-separated additional trusted presigned-upload hosts for self-hosted storage.
+- `COAIA_MEDIA_ALLOW_HTTP=true`: allows HTTP for local tests only; never use it in production.
+
+The proxy only downloads from approved OpenAI file hosts and only uploads to approved S3-compatible storage hosts. It rejects empty files, invalid media contexts, oversized payloads, and untrusted URLs.
+
+## Configure the Custom GPT
+
+1. Replace `https://replace-with-your-proxy.invalid` in `ceremony-media-proxy.yml` with the deployed HTTPS origin.
+2. Import it as a second Action alongside `ceremony-observations.yml`.
+3. Configure API Key authentication with:
+   - Header: `X-Coaia-Proxy-Key`
+   - Value: the `COAIA_MEDIA_PROXY_API_KEY` configured on the service
+4. Keep `x-openai-isConsequential: false` unchanged.
+5. Do not put the proxy key or Langfuse keys in GPT instructions or conversation text.
+
+## GPT behavior
+
+Add this to the GPT instructions:
+
+```markdown
+When the user supplies media for Langfuse, call media_uploadConversationFile
+with exactly that conversation file in openaiFileIdRefs and the target trace or
+dataset-item context. Never create placeholder bytes unless explicitly asked.
+
+The bridge performs media-record creation, exact-byte hashing, the presigned PUT,
+and media finalization. Do not claim success unless the bridge returns
+success=true and a mediaToken.
+
+Put the returned mediaToken unchanged into an observation input, output, or
+metadata value, then call observations_export. Optionally call media_get with
+the returned mediaId to verify uploadedAt and the temporary URL.
+```
+
+## Completed flow
+
+```text
+user/DALL-E/Code Interpreter file
+    → openaiFileIdRefs temporary download URL
+    → media_uploadConversationFile
+        → download exact bytes
+        → compute padded Base64 SHA-256 and exact length/type
+        → POST /api/public/media
+        → PUT exact bytes to the presigned storage URL
+        → PATCH /api/public/media/{mediaId}
+        → return @@@langfuseMedia:type=...|id=...|source=bytes@@@
+    → observations_export containing the returned media token
+    → observations_list / media_get verification
+```
+
+Primary references:
+
+- OpenAI GPT Actions file transfer: https://developers.openai.com/api/docs/actions/sending-files
+- Langfuse multi-modality and media tokens: https://langfuse.com/docs/observability/features/multi-modality

package/agents/custom_gpt/ceremony-media-proxy.yml

@@ -0,0 +1,158 @@
+openapi: 3.1.0
+info:
+  title: Coaia Langfuse Media Upload Bridge
+  version: 0.4.2
+  description: >-
+    Companion Custom GPT action that transfers one conversation file into Langfuse object storage, finalizes the media
+    record, and returns a renderable Langfuse media token. Deploy coaiajs-media-proxy and replace the example server URL.
+  license:
+    name: MIT
+    identifier: MIT
+servers:
+  - url: https://replace-with-your-proxy.invalid
+    description: Replace with the public HTTPS URL of the deployed coaiajs media proxy.
+security:
+  - ProxyApiKey: []
+paths:
+  /media/upload:
+    post:
+      operationId: media_uploadConversationFile
+      summary: Upload a conversation file to Langfuse
+      description: >-
+        Upload exactly one user, DALL-E, or Code Interpreter file. The bridge downloads its exact bytes, computes the
+        padded Base64 SHA-256 digest and length, creates the Langfuse record, performs the presigned PUT, finalizes it,
+        and returns a media token. Never substitute placeholder bytes.
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/UploadConversationFileRequest'
+      responses:
+        '200':
+          description: Uploaded media and its renderable Langfuse token.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/UploadConversationFileResponse'
+        '400':
+          description: Invalid file reference, media context, or downstream upload.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ErrorResponse'
+        '401':
+          description: Missing or invalid proxy API key.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ErrorResponse'
+        '413':
+          description: File or request exceeds the configured size limit.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/ErrorResponse'
+      x-openai-isConsequential: false
+components:
+  securitySchemes:
+    ProxyApiKey:
+      type: apiKey
+      in: header
+      name: X-Coaia-Proxy-Key
+      description: Configure this value in the GPT Action authentication UI; it must equal COAIA_MEDIA_PROXY_API_KEY.
+  schemas:
+    UploadConversationFileRequest:
+      type: object
+      additionalProperties: false
+      properties:
+        openaiFileIdRefs:
+          type: array
+          minItems: 1
+          maxItems: 1
+          items:
+            type: string
+          description: >-
+            Exactly one file from the conversation: the user's actual media, a DALL-E image, or a Code Interpreter
+            file. Custom GPT replaces this string entry with its temporary runtime file-reference object.
+        traceId:
+          type: string
+          description: Trace receiving the media. Use either trace context or dataset-item context.
+        observationId:
+          type: string
+          description: Optional observation receiving the media; requires traceId.
+        datasetId:
+          type: string
+          description: Dataset receiving the media; requires datasetItemId and no traceId.
+        datasetItemId:
+          type: string
+          description: Dataset item receiving the media; requires datasetId and no traceId.
+        field:
+          type: string
+          enum:
+            - input
+            - output
+            - expectedOutput
+            - metadata
+          default: input
+          description: Field where the returned media token will be referenced.
+      required:
+        - openaiFileIdRefs
+    UploadConversationFileResponse:
+      type: object
+      additionalProperties: false
+      properties:
+        success:
+          type: boolean
+        mediaId:
+          type: string
+        mediaToken:
+          type: string
+          description: Put this exact token in observation input, output, or metadata so Langfuse renders the media.
+        fileName:
+          type: string
+        openaiFileId:
+          type: string
+        contentType:
+          type: string
+        contentLength:
+          type: integer
+        sha256Hash:
+          type: string
+          minLength: 44
+          maxLength: 44
+          pattern: '^[A-Za-z0-9+/]{43}=$'
+          description: Standard padded Base64 SHA-256 digest of the uploaded bytes.
+        traceId:
+          type: string
+        observationId:
+          type: string
+        datasetId:
+          type: string
+        datasetItemId:
+          type: string
+        field:
+          type: string
+        uploadTimeMs:
+          type: integer
+        alreadyUploaded:
+          type: boolean
+      required:
+        - success
+        - mediaId
+        - mediaToken
+        - openaiFileId
+        - contentType
+        - contentLength
+        - sha256Hash
+        - field
+        - uploadTimeMs
+        - alreadyUploaded
+    ErrorResponse:
+      type: object
+      additionalProperties: false
+      properties:
+        error:
+          type: string
+      required:
+        - error

package/agents/custom_gpt/ceremony-observations.instructions.md

@@ -10,7 +10,6 @@ Langfuse v4 does not provide mutable REST trace and observation creation endpoin
 POST /api/public/otel/v1/traces
 Content-Type: application/json
 Authorization: Basic <base64(public-key:secret-key)>
-x-langfuse-ingestion-version: 4
 ```
 
 The `observations_export` action exposes that endpoint with a constrained OTLP schema and examples. It supports both:
@@ -18,7 +17,7 @@ The `observations_export` action exposes that endpoint with a constrained OTLP s
 1. Creating a trace by sending a root span without `parentSpanId`.
 2. Appending an observation by reusing the trace ID and setting `parentSpanId` to an existing observation/span ID.
 
-The file has **20 actions**, below the stated 30-action limit. Scores remain in this specification because the complete focused surface still fits. Media and destructive delete actions are omitted.
+The file has **23 actions**, below the stated 30-action limit. Scores and the original media operations remain in this specification because the complete focused surface still fits. Destructive delete actions are omitted.
 
 ## Configure the Custom GPT
 
@@ -44,7 +43,7 @@ The file has **20 actions**, below the stated 30-action limit. Scores remain in
    end_ns = max(time.time_ns(), start_ns)
    ```
 
-The ingestion operation also declares the required `x-langfuse-ingestion-version` action header with the only allowed value `4`. If a GPT Actions client omits this non-auth header, Langfuse documents that direct OTLP ingestion still works but may take up to ten minutes to appear in v4 reads. A proxy is required if the client cannot send the header and real-time visibility is mandatory.
+Custom GPT currently ignores ordinary OpenAPI header parameters, so this direct specification intentionally does not declare `x-langfuse-ingestion-version`. Langfuse documents that OTLP ingestion without that header still works, but data may take up to ten minutes to appear in v4 reads. If real-time visibility is mandatory, use a trusted proxy that adds `x-langfuse-ingestion-version: 4` before forwarding the request to Langfuse.
 
 ## Recommended GPT instructions
 
@@ -80,6 +79,19 @@ span. To record a correction or later result, append a new child event/span.
 After exporting, call observations_list with the traceId and fields
 core,basic,time,io,metadata,model,usage,trace_context to verify the hierarchy.
 Use each returned observation id as the span/parent ID for later children.
+
+### Langfuse media
+
+When the user supplies media, use the actual file bytes—never placeholder text.
+Compute contentLength from those bytes. Compute SHA-256 over those bytes and
+send the standard padded Base64 digest, not hexadecimal, as sha256Hash.
+
+Prefer media_uploadConversationFile from the companion proxy Action; pass the
+user's actual conversation file in openaiFileIdRefs. It performs record
+creation, the exact-byte PUT, and finalization and returns a mediaToken.
+
+media_getUploadUrl alone only creates a record. Never claim success without a
+completed PUT and media_patch, or success=true from the proxy bridge.
 ```
 
 ## Minimal append example
@@ -102,7 +114,7 @@ The parent trace and root observation must already exist. Replace every example
         {
           "scope": {
             "name": "custom-gpt-ceremony",
-            "version": "0.4.0"
+            "version": "0.4.2"
           },
           "spans": [
             {
@@ -151,7 +163,28 @@ The parent trace and root observation must already exist. Replace every example
 | Datasets and dataset items | 6 |
 | Scores and score configurations | 5 |
 | Comments | 2 |
-| **Total** | **20** |
+| Media | 3 |
+| **Total** | **23** |
+
+## Media workflow
+
+When the user supplies media, always use that media's actual bytes. Never create or hash placeholder text or surrogate bytes unless the user explicitly requests a placeholder.
+
+1. Read the exact bytes that will be uploaded.
+2. Derive `contentType` from those bytes/file and set `contentLength` to their exact byte length.
+3. Compute SHA-256 over those exact bytes, then encode the 32-byte digest with standard padded RFC 4648 Base64. Send the resulting 44-character value as `sha256Hash`; never send the 64-character hexadecimal digest.
+4. Call `media_getUploadUrl` with the context, file metadata, Base64 hash, and field.
+5. PUT the same exact bytes to the returned presigned `uploadUrl` using the declared content type.
+6. Only after that PUT completes, call `media_patch` with the completion time and HTTP result.
+7. Optionally call `media_get` to verify the media record and obtain a temporary download URL.
+
+Example for the four UTF-8 bytes `test`:
+
+```text
+sha256Hash: n4bQgYhMfWWaL+qgxVrQFaO/TxsrC4Is0V1sFbDwCgg=
+```
+
+Receiving `uploadUrl` creates a Langfuse media record but does **not** upload the bytes. Never report that media was uploaded after `media_getUploadUrl` alone. The direct OpenAPI document cannot describe a PUT to a dynamically returned storage host. For autonomous uploads, deploy the included `coaiajs-media-proxy`, import `ceremony-media-proxy.yml` as a second GPT Action, and use `media_uploadConversationFile`. The bridge consumes the user's actual `openaiFileIdRefs` file, performs the complete upload flow, and returns the media token.
 
 ## Why there are not separate create-trace and append-observation actions