jtoken 0.2.1__tar.gz → 0.2.2__tar.gz

{jtoken-0.2.1 → jtoken-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: jtoken
-Version: 0.2.1
+Version: 0.2.2
 Summary: Compress JSON-shaped documents for LLM prompts with normalization, CLI, and token measurement
 Project-URL: Homepage, https://github.com/hermannsamimi/jtoken
 Project-URL: Repository, https://github.com/hermannsamimi/jtoken
@@ -126,6 +126,41 @@ nulls: ref
 - Reserved top-level keys: `nulls`, `trues`, `falses`.
 - Lists are normalized into nested dicts with numeric keys before encoding.
 
+## Input and output formats
+
+Use `source=` / `target=` in Python or `--input-format` / `--output-format` on the CLI. `encode`, `stats`, and `count` accept `--input-format` (default `auto`). `decode` accepts `--output-format` (default `json`).
+
+| Input (`source` / `--input-format`) | Use when |
+|---|---|
+| `auto` | Let jtoken detect the dialect from the text or object shape |
+| `json` | Standard JSON object |
+| `python` | Same JSON parser as `json` |
+| `mongo_extended` | MongoDB Extended JSON with `$oid`, `$date`, `$numberInt`, `$numberLong`, `$numberDouble`, `$numberDecimal` |
+| `mongo_shell` | MongoDB shell document with `ObjectId()`, `ISODate()`, `NumberInt()`, `NumberLong()` |
+| `elastic_hit` | Elasticsearch search hit with `_source` (and optional `fields`) |
+| `elastic_source` | `_source` payload only, or a document wrapped as `{"_source": {...}}` |
+
+| Output (`target` / `--output-format`) | Use when |
+|---|---|
+| `python` | Python `repr` (Python API default) |
+| `json` | Pretty-printed JSON (CLI `decode` default) |
+| `mongo_extended` | Extended JSON; requires a context sidecar for BSON-like types |
+| `mongo_shell` | Mongo shell document; requires a context sidecar for BSON-like types |
+| `elastic_hit` | Full Elasticsearch hit envelope; requires a context sidecar |
+| `elastic_source` | JSON shaped like an Elasticsearch `_source` wrapper |
+
+With `auto`, jtoken picks `mongo_shell` when it sees `ObjectId(...)` or `ISODate(...)`, `elastic_hit` when the object has a dict `_source`, `mongo_extended` when Extended JSON markers such as `$oid` or `$date` appear, and otherwise `json`.
+
+Write the normalization context to a sidecar on encode (`--context-out` / `NormalizationContext.to_dict()`) and pass it back on decode when the output dialect is not plain JSON or Python. The sidecar records list paths, dotted keys, Elasticsearch envelope metadata, and MongoDB type markers in `typed_values` (`object_id`, `datetime`, `long`).
+
+### MongoDB shell and Extended JSON
+
+Mongo shell input is parsed as JSON after rewriting shell literals: `ObjectId("...")` and `ISODate("...")` become Extended JSON, `NumberInt(n)` becomes a plain integer, and `NumberLong(n)` becomes `{"$numberLong": "n"}`. On normalize, `object_id`, `datetime`, and `long` values are stored in the context so `mongo_extended` and `mongo_shell` output can restore `{"$oid": ...}` / `ObjectId(...)`, `{"$date": ...}` / `ISODate(...)`, and `{"$numberLong": ...}` / `NumberLong(...)`. `$numberInt`, `$numberDouble`, and `$numberDecimal` are coerced to Python scalars and are not tracked in `typed_values`.
+
+### Elasticsearch hits
+
+`elastic_hit` encodes the merged `_source` document (plus any `fields` values that are not already present in `_source`) and stores `_index`, `_id`, `_version`, `_score`, `_type`, and `_routing` in the context for lossless `elastic_hit` output.
+
 ## Public API reference
 
 ### Package metadata

{jtoken-0.2.1 → jtoken-0.2.2}/README.md

@@ -90,9 +90,40 @@ jtoken encode --input-format elastic_hit -f hit.json --context-out hit.ctx.json
 jtoken decode --output-format mongo_shell -f hit.jtoken --context-in hit.ctx.json
 ```
 
-Supported input dialects: `auto`, `json`, `python`, `mongo_extended`, `mongo_shell`, `elastic_hit`, `elastic_source`.
+### Input and output formats
 
-Supported output dialects: `python`, `json`, `mongo_extended`, `mongo_shell`, `elastic_hit`, `elastic_source`.
+Use `source=` / `target=` in Python or `--input-format` / `--output-format` on the CLI. `encode`, `stats`, and `count` accept `--input-format` (default `auto`). `decode` accepts `--output-format` (default `json`).
+
+| Input (`source` / `--input-format`) | Use when |
+|---|---|
+| `auto` | Let jtoken detect the dialect from the text or object shape |
+| `json` | Standard JSON object |
+| `python` | Same JSON parser as `json` |
+| `mongo_extended` | MongoDB Extended JSON with `$oid`, `$date`, `$numberInt`, `$numberLong`, `$numberDouble`, `$numberDecimal` |
+| `mongo_shell` | MongoDB shell document with `ObjectId()`, `ISODate()`, `NumberInt()`, `NumberLong()` |
+| `elastic_hit` | Elasticsearch search hit with `_source` (and optional `fields`) |
+| `elastic_source` | `_source` payload only, or a document wrapped as `{"_source": {...}}` |
+
+| Output (`target` / `--output-format`) | Use when |
+|---|---|
+| `python` | Python `repr` (Python API default) |
+| `json` | Pretty-printed JSON (CLI `decode` default) |
+| `mongo_extended` | Extended JSON; requires a context sidecar for BSON-like types |
+| `mongo_shell` | Mongo shell document; requires a context sidecar for BSON-like types |
+| `elastic_hit` | Full Elasticsearch hit envelope; requires a context sidecar |
+| `elastic_source` | JSON shaped like an Elasticsearch `_source` wrapper |
+
+With `auto`, jtoken picks `mongo_shell` when it sees `ObjectId(...)` or `ISODate(...)`, `elastic_hit` when the object has a dict `_source`, `mongo_extended` when Extended JSON markers such as `$oid` or `$date` appear, and otherwise `json`.
+
+Write the normalization context to a sidecar on encode (`--context-out` / `NormalizationContext.to_dict()`) and pass it back on decode when the output dialect is not plain JSON or Python. The sidecar records list paths, dotted keys, Elasticsearch envelope metadata, and MongoDB type markers in `typed_values` (`object_id`, `datetime`, `long`).
+
+### MongoDB shell and Extended JSON
+
+Mongo shell input is parsed as JSON after rewriting shell literals: `ObjectId("...")` and `ISODate("...")` become Extended JSON, `NumberInt(n)` becomes a plain integer, and `NumberLong(n)` becomes `{"$numberLong": "n"}`. On normalize, `object_id`, `datetime`, and `long` values are stored in the context so `mongo_extended` and `mongo_shell` output can restore `{"$oid": ...}` / `ObjectId(...)`, `{"$date": ...}` / `ISODate(...)`, and `{"$numberLong": ...}` / `NumberLong(...)`. `$numberInt`, `$numberDouble`, and `$numberDecimal` are coerced to Python scalars and are not tracked in `typed_values`.
+
+### Elasticsearch hits
+
+`elastic_hit` encodes the merged `_source` document (plus any `fields` values that are not already present in `_source`) and stores `_index`, `_id`, `_version`, `_score`, `_type`, and `_routing` in the context for lossless `elastic_hit` output.
 
 ## CLI
 

{jtoken-0.2.1 → jtoken-0.2.2}/README.pypi.md

@@ -94,6 +94,41 @@ nulls: ref
 - Reserved top-level keys: `nulls`, `trues`, `falses`.
 - Lists are normalized into nested dicts with numeric keys before encoding.
 
+## Input and output formats
+
+Use `source=` / `target=` in Python or `--input-format` / `--output-format` on the CLI. `encode`, `stats`, and `count` accept `--input-format` (default `auto`). `decode` accepts `--output-format` (default `json`).
+
+| Input (`source` / `--input-format`) | Use when |
+|---|---|
+| `auto` | Let jtoken detect the dialect from the text or object shape |
+| `json` | Standard JSON object |
+| `python` | Same JSON parser as `json` |
+| `mongo_extended` | MongoDB Extended JSON with `$oid`, `$date`, `$numberInt`, `$numberLong`, `$numberDouble`, `$numberDecimal` |
+| `mongo_shell` | MongoDB shell document with `ObjectId()`, `ISODate()`, `NumberInt()`, `NumberLong()` |
+| `elastic_hit` | Elasticsearch search hit with `_source` (and optional `fields`) |
+| `elastic_source` | `_source` payload only, or a document wrapped as `{"_source": {...}}` |
+
+| Output (`target` / `--output-format`) | Use when |
+|---|---|
+| `python` | Python `repr` (Python API default) |
+| `json` | Pretty-printed JSON (CLI `decode` default) |
+| `mongo_extended` | Extended JSON; requires a context sidecar for BSON-like types |
+| `mongo_shell` | Mongo shell document; requires a context sidecar for BSON-like types |
+| `elastic_hit` | Full Elasticsearch hit envelope; requires a context sidecar |
+| `elastic_source` | JSON shaped like an Elasticsearch `_source` wrapper |
+
+With `auto`, jtoken picks `mongo_shell` when it sees `ObjectId(...)` or `ISODate(...)`, `elastic_hit` when the object has a dict `_source`, `mongo_extended` when Extended JSON markers such as `$oid` or `$date` appear, and otherwise `json`.
+
+Write the normalization context to a sidecar on encode (`--context-out` / `NormalizationContext.to_dict()`) and pass it back on decode when the output dialect is not plain JSON or Python. The sidecar records list paths, dotted keys, Elasticsearch envelope metadata, and MongoDB type markers in `typed_values` (`object_id`, `datetime`, `long`).
+
+### MongoDB shell and Extended JSON
+
+Mongo shell input is parsed as JSON after rewriting shell literals: `ObjectId("...")` and `ISODate("...")` become Extended JSON, `NumberInt(n)` becomes a plain integer, and `NumberLong(n)` becomes `{"$numberLong": "n"}`. On normalize, `object_id`, `datetime`, and `long` values are stored in the context so `mongo_extended` and `mongo_shell` output can restore `{"$oid": ...}` / `ObjectId(...)`, `{"$date": ...}` / `ISODate(...)`, and `{"$numberLong": ...}` / `NumberLong(...)`. `$numberInt`, `$numberDouble`, and `$numberDecimal` are coerced to Python scalars and are not tracked in `typed_values`.
+
+### Elasticsearch hits
+
+`elastic_hit` encodes the merged `_source` document (plus any `fields` values that are not already present in `_source`) and stores `_index`, `_id`, `_version`, `_score`, `_type`, and `_routing` in the context for lossless `elastic_hit` output.
+
 ## Public API reference
 
 ### Package metadata

{jtoken-0.2.1 → jtoken-0.2.2}/jtoken/__init__.py

@@ -19,7 +19,7 @@ from .tokens import (
     token_savings,
 )
 
-__version__ = "0.2.1"
+__version__ = "0.2.2"
 __author__ = "Hermann Samimi"
 
 # json-style aliases

{jtoken-0.2.1 → jtoken-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "jtoken"
-version = "0.2.1"
+version = "0.2.2"
 description = "Compress JSON-shaped documents for LLM prompts with normalization, CLI, and token measurement"
 readme = "README.pypi.md"
 requires-python = ">=3.8"