contextir 1.0.1__tar.gz

contextir-1.0.1/.gitignore

@@ -0,0 +1,10 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+data/external/
+build/
+dist/
+*.egg-info/
+.coverage
+.pytest_cache/

contextir-1.0.1/CHANGELOG.md

@@ -0,0 +1,101 @@
+# Changelog
+
+All notable changes to ContextIR are documented here.
+
+## 1.0.1 - 2026-07-19
+
+- added PyPI Trusted Publishing through GitHub Actions OIDC;
+- split release validation, GitHub publication, and PyPI publication into
+  separate least-privilege jobs;
+- restricted `id-token: write` to the two-step PyPI job using the protected
+  `pypi` GitHub environment;
+- reused one validated artifact for both GitHub and PyPI publication.
+
+## 1.0.0 - 2026-07-18
+
+- stabilized `ContextPipeline` as the primary model-boundary API while keeping
+  the existing per-call invoker compatible;
+- added dependency-free callable `OllamaClient` and `OpenAICompatibleClient`
+  transports with normalized usage metadata and endpoint errors;
+- added `contextir run` for one-command local and OpenAI-compatible model
+  invocation, including stdin and payload-free JSON trace modes;
+- moved the quick start to the runnable client-plus-pipeline path and documented
+  the direct release-wheel install;
+- documented SemVer guarantees, stable exports, contract evolution, and the
+  migration from the 0.x API;
+- verified the built wheel in a clean virtual environment and exercised the
+  installed CLI against local Ollama;
+- retained the `contextir.v2` contract without a schema migration.
+
+## 0.5.0 - 2026-07-18
+
+- added strict agent tool-routing and agent-state diagnostics to the existing
+  local-model A/B harness;
+- added optional external privacy-corpus evaluation with precision and recall
+  gates;
+- added exact-value privacy annotations to the checked-in gateway benchmark;
+- classified payment cards before phone numbers and added Luhn validation;
+- rejected ISO dates, IPv4 values, SSN-shaped values, and short bare numbers
+  from the dependency-free phone recognizer;
+- measured `0.8471` precision and `1.0000` recall on 277 supported spans from
+  Microsoft Presidio Research's 1,500-row MIT-licensed synthetic corpus;
+- measured identical `0.7272` raw and auto quality on nine Qwen3 8B cases while
+  reducing model input by 70.0% and mean latency from `53.8 s` to `2.51 s`.
+
+## 0.4.1 - 2026-07-18
+
+- changed the product pipeline to compile the candidate first and build a raw
+  baseline only when the savings gate needs it;
+- avoided duplicate compilation when auto mode already selects raw;
+- skipped event and numeric-entity extraction for query-aware retrieval, where
+  the model receives selected verbatim evidence instead;
+- precompiled hot-path regular expressions and reused normalized token sets;
+- replaced repeated paragraph-owner scans with a linear ownership pass;
+- added deterministic tests for single-pass candidate and exhaustive routing;
+- preserved all 16 sampled contracts and all seven saved model-facing prompt
+  token counts exactly.
+
+## 0.4.0 - 2026-07-18
+
+- added query-aware lexical evidence selection for long document QA;
+- preserved task instructions, output format, paragraph ownership, and queries
+  as normal model-facing text rather than exposing the internal IR protocol;
+- routed exhaustive counting and retrieval without adequate evidence to raw;
+- removed redundant protocol rendering when hybrid source spans cover every
+  compiled event;
+- reused placeholders for repeated occurrences of the same protected value in
+  both built-in and Presidio privacy scrubbers;
+- aligned retrieval and passage-count benchmark scoring with LongBench task
+  semantics;
+- added Ollama and LM Studio follow-up reports showing about 69% token reduction
+  without aggregate quality loss on the tested subset.
+
+## 0.3.0 - 2026-07-18
+
+- added `ContextPipeline`, the policy-driven product entry point;
+- added target-tokenizer-aware mode selection with a minimum savings gate;
+- separated normal reasoning verification from strict transform retention;
+- added bounded semantic-to-hybrid-to-raw fallback;
+- rejected unknown, missing, and newly generated PII placeholders;
+- added payload-free public traces and explicit restoration allowlists;
+- fixed repeated-event semantic confidence weighting;
+- promoted product pipeline scenarios into the release benchmark gate.
+
+## 0.2.1 - 2026-07-18
+
+- installed development dependencies before release validation;
+- updated GitHub Actions to Node.js 24-compatible major versions;
+- pinned Gitleaks 8.30.1 and verified its release checksum in CI.
+
+## 0.2.0 - 2026-07-18
+
+- renamed the public package and product from SIR Translator to ContextIR;
+- added the `contextir.v2` compact contract and packaged JSON Schema;
+- added adaptive raw, hybrid, semantic, and auto modes;
+- separated public contracts from local PII vaults and source maps;
+- added optional Presidio integration and allowlisted restoration;
+- added event deduplication and structured contract comparison;
+- retained earlier SIR graph and neural work as a research layer;
+- added deterministic tests and a compression/privacy smoke benchmark;
+- added packaged schema resources, CI, secret scanning, release automation,
+  benchmark cards, and a production-oriented quick start.

contextir-1.0.1/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!) The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 ContextIR contributors
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

contextir-1.0.1/NOTICE

@@ -0,0 +1,7 @@
+ContextIR
+Copyright 2026 ContextIR contributors
+
+This product includes optional research artifacts derived from third-party
+lexical resources. Those artifacts are not covered solely by the Apache License
+2.0. See THIRD_PARTY_NOTICES.md.
+

contextir-1.0.1/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.4
+Name: contextir
+Version: 1.0.1
+Summary: Local-first adaptive context compiler and privacy gateway for language models.
+Project-URL: Homepage, https://github.com/mussolene/contextir
+Project-URL: Documentation, https://github.com/mussolene/contextir#readme
+Project-URL: Issues, https://github.com/mussolene/contextir/issues
+Project-URL: Changelog, https://github.com/mussolene/contextir/blob/main/CHANGELOG.md
+Author: Maxim Matvienko
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: agents,compression,context,llm,privacy,semantic-ir
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Provides-Extra: dev
+Requires-Dist: build>=1.2; extra == 'dev'
+Requires-Dist: hatch>=1.14; extra == 'dev'
+Requires-Dist: jsonschema>=4.20; extra == 'dev'
+Requires-Dist: twine>=5.1; extra == 'dev'
+Provides-Extra: privacy
+Requires-Dist: presidio-analyzer>=2.2; extra == 'privacy'
+Provides-Extra: research
+Requires-Dist: numpy>=1.26; extra == 'research'
+Requires-Dist: pyyaml>=6.0; extra == 'research'
+Description-Content-Type: text/markdown
+
+# ContextIR
+
+ContextIR is a local-first adaptive context compiler and privacy gateway for
+language models. It turns text into a compact, inspectable intermediate
+representation while retaining critical source fragments when a lossy summary
+would be unsafe.
+
+Status: **stable 1.0 API**. ContextIR follows semantic versioning for the public
+Python surface and the `contextir.v2` contract. Semantic compression and PII
+detection still require evaluation on each deployment's data.
+
+## Quick Start
+
+```bash
+python3 -m pip install contextir
+ollama pull qwen3:0.6b
+contextir run --model qwen3:0.6b --text "Reply with only READY."
+```
+
+```python
+from contextir import ContextPipeline, OllamaClient
+
+pipeline = ContextPipeline(invoke=OllamaClient("qwen3:0.6b"))
+result = pipeline.run(
+    "If payment 42 is complete, do not send it again.",
+    source_lang="en",
+    target_lang="en",
+)
+print(result.answer)
+```
+
+The release wheel avoids a source checkout and Git dependency. A pinned Git
+install remains available for environments that prefer source builds.
+
+For a two-minute integration and safe restoration example, see
+[Quick start](docs/QUICKSTART.md).
+
+## Why
+
+Long agent histories and tool outputs consume model context with repeated facts,
+formatting, and sensitive values. ContextIR provides three explicit modes:
+
+- `raw`: masked source text, without protocol overhead;
+- `hybrid`: semantic events plus source fragments containing negation,
+  conditions, numbers, quotes, or protected placeholders;
+- `semantic`: compact events only, for callers that accept lossy compression.
+
+`auto` first distinguishes exhaustive tasks, document retrieval, and
+operational history. Exhaustive tasks stay raw. Document QA uses local lexical
+retrieval to keep the query, formatting instructions, and evidence spans
+verbatim. Operational history continues to use inspectable events and critical
+source fragments.
+
+`ContextPipeline` is the product entry point. It measures candidate prompts
+with a configurable target-model tokenizer, rejects compression without useful
+savings, verifies output safety, and performs bounded fallback from semantic to
+hybrid to raw context.
+
+## Install
+
+From a checkout:
+
+```bash
+python3 -m pip install -e .
+```
+
+Optional Presidio integration:
+
+```bash
+python3 -m pip install -e '.[privacy]'
+```
+
+## Advanced Compiler API
+
+```python
+from contextir import ContextIR
+
+gateway = ContextIR()
+bundle = gateway.compile_private(
+    "If payment 42 is complete, do not send it again. Email person@example.test.",
+    source_lang="en",
+    target_lang="en",
+    mode="hybrid",
+)
+
+contract = bundle.contract
+model_prompt = gateway.render_prompt(contract)
+
+# Restore only placeholders explicitly allowed by the application.
+answer = gateway.restore("Contact PII_EMAIL_1", bundle, allowed={"PII_EMAIL_1"})
+```
+
+`compile()` returns only the public contract. `compile_private()` additionally
+returns the local PII vault and source-reference map. Neither is inserted into
+the model prompt.
+
+For model-facing applications, prefer `ContextPipeline` over calling the
+compiler directly. Use `task="transform"` for translation, rewriting, and
+summarization where numbers, negation, constraints, and placeholders must
+survive. Normal reasoning answers are not incorrectly required to echo the
+request.
+
+## CLI
+
+```bash
+contextir run \
+  --backend ollama \
+  --model qwen3:0.6b \
+  --text "Summarize the current agent state."
+
+contextir compile \
+  --text "Если платеж 42 выполнен, не отправляй его повторно." \
+  --source-lang ru \
+  --target-lang en \
+  --mode hybrid \
+  --out /tmp/contextir.json
+
+contextir render --contract /tmp/contextir.json
+```
+
+The source checkout also exposes `python3 scripts/contextir.py`.
+
+## Contract
+
+ContextIR v2 records:
+
+- source and target language;
+- intent and confidence;
+- typed entities and protected placeholders;
+- events with predicate, arguments, polarity, modality, condition, and source
+  reference;
+- execution and privacy constraints;
+- optional lexical concepts;
+- included source fragments and unresolved source references;
+- prompt-size and latency statistics.
+
+The packaged JSON Schema is available through:
+
+```python
+from contextir.schemas import load_contract_schema
+```
+
+## Privacy
+
+The default detector is deliberately small and recognizes common emails,
+phones, payment-card-like values, and API-key-like tokens. Presidio can be used
+as an optional detector and extended with project-specific recognizers.
+
+Automated PII detection is not a security boundary by itself. Keep the vault
+local, avoid logging prompt bodies, use explicit restoration allowlists, and
+evaluate recognizers on data representative of the deployment.
+
+## Current Evidence
+
+The checked-in compiler smoke benchmark reports:
+
+- 9 compiler and 4 product-pipeline cases;
+- 0 expectation failures;
+- 0 pipeline failures;
+- 0 PII leaks into public contracts or rendered prompts;
+- `0.3627` prompt/source ratio on the one compression-eligible repeated-context
+  case.
+
+The first model-level A/B exposed a large quality loss in v0.3.0. Query-aware
+routing corrected that failure. On the v0.5.0 nine-case Qwen3 8B run, raw and
+auto both scored `0.7272`, while auto reduced model input by 70.0% and mean
+latency from `53.8 s` to `2.51 s`. An external 1,500-row Presidio Research
+benchmark measured `0.8471` precision and `1.0000` recall for the dependency-free
+email/phone/card profile. These are bounded results, not universal quality or
+privacy claims; broader official tasks and deployment-specific PII evaluation
+remain required.
+
+See the [local model A/B card](docs/benchmarks/LOCAL_MODEL_AB.md) and
+[benchmark roadmap](docs/BENCHMARKS.md).
+
+## Research Layer
+
+The repository retains the earlier RU/EN lexical graph, graph embeddings,
+synthetic translator, checkpoints, and roundtrip experiments. They are not
+loaded by the default package path. Use the `research` extra and source checkout
+for those experiments.
+
+The old WordNet keyword roundtrip is preserved as a baseline, not presented as
+the production compiler. See [RESEARCH_HISTORY.md](docs/RESEARCH_HISTORY.md).
+
+## Development
+
+```bash
+python3 -m pip install -e '.[dev,research]'
+python3 -m unittest discover -s tests -v
+python3 scripts/evaluate_contextir.py
+python3 -m build
+python3 -m twine check dist/*
+```
+
+Architecture, scope, contribution, and security guidance:
+
+- [Architecture](docs/ARCHITECTURE.md)
+- [Quick start](docs/QUICKSTART.md)
+- [Product scope](docs/PRODUCT_SCOPE.md)
+- [Benchmarks](docs/BENCHMARKS.md)
+- [Contributing](CONTRIBUTING.md)
+- [Security](SECURITY.md)
+
+## License
+
+ContextIR code is licensed under Apache-2.0. Research datasets and derived
+lexical artifacts retain their upstream terms. See
+[THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) before redistributing them.