context-engineering-core 0.12.1__tar.gz → 0.15.0__tar.gz

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/CHANGELOG.md

@@ -3,7 +3,64 @@
 Earlier versions (0.1.0–0.9.6) are documented in `docs/C01_REPORT.md` through
 `docs/C10_STORAGE_ACL_REPORT.md` and preserved as wheels under `archives/`.
 
-## 0.12.1 — 2026-10-31 (unreleased)
+## 0.15.0 — 2026-10-31 (unreleased)
+
+### Changed (behavior)
+- Hybrid retrieval is the default (`assemble_context(embedder="builtin")`): the
+  bundled potion-base-8M model (8-bit vectors, ~7 MB in the package; MIT notice in
+  `embedding_data/LICENSE`) with a pure-Python WordPiece tokenizer. No download,
+  network or new dependency. `embedder=None` gives the pre-0.15.0 keyword-only path.
+- Default `semantic_weight` 0.7 → 0.5: a needle-in-filler check showed 0.7 losing
+  exact keyword matches (22/31 vs 31/31); 0.5 is equal or better on every set.
+- `strategy-eval` includes the hybrid strategies by default (`--no-semantic` to
+  skip; `--semantic` removed). `context-engine-mcp` is hybrid by default
+  (`--no-semantic`; `--semantic` removed).
+- The frozen benchmark harness pins keyword-only retrieval.
+
+### Added
+- `BuiltinEmbedder`, `default_embedder()`, `context_engine.wordpiece`.
+- Lenient grading v2 (`answer_correct_lenient`, `GRADERS`) beside frozen strict v1;
+  `scripts/regrade_live.py` writes `grading.json` for both live runs.
+
+### Unchanged
+- 0.14.0 wheel/freeze archived in `archives/c15-default-semantic-0150/`.
+
+## 0.14.0 — 2026-10-31 (PyPI)
+
+### Added
+- Optional hybrid retrieval: `assemble_context(..., embedder=, semantic_weight=0.7)`
+  (also via `MemoryStore.assemble`) ranks chunks by 0.7 x meaning + 0.3 x keyword
+  score. `context_engine.embeddings`: `Embedder` protocol, `Model2VecEmbedder`
+  (default model pinned: `minishlab/potion-base-8M@bf8b0566`, extra `[semantic]`),
+  `FastEmbedEmbedder` (extra `[fastembed]`), `FunctionEmbedder`. Bounded vector cache.
+- `strategy-eval --semantic` (adds `retrieval_semantic`, `combined_semantic`) and
+  `context-engine-mcp --semantic`.
+- Reworded questions: default retention 3/31 → 21/31 with the embedder; original
+  questions stay 31/31. Bake-off evidence in `output/embedding-bakeoff-001/`.
+
+### Changed
+- `numpy` is now a declared dependency (it was already installed via `rank-bm25`).
+- Missing optional extras give an install hint instead of a traceback.
+
+### Unchanged
+- Without an embedder, all 310 strategy-benchmark assemblies are byte-identical to
+  0.13.0 (archived in `archives/c14-semantic-0140/`).
+
+## 0.13.0 — 2026-10-31 (unreleased)
+
+### Added
+- Reworded-question stress test: `strategy-eval --questions paraphrased` and
+  `compare_strategies(..., questions="paraphrased")`. 31 questions share no content
+  word (or 4-letter start of a 5+ letter word) with their evidence, enforced by
+  `paraphrase_overlaps`. Result: default retention falls from 31/31 to 3/31 at 900
+  and 3000 tokens (retrieval alone: 6/31 and 7/31). README updated.
+- `assemble_strategy(..., question=...)` to assemble a strategy for any question.
+
+### Unchanged
+- Engine behavior and defaults. 0.12.1 wheel/freeze archived in
+  `archives/c13-paraphrase-0130/`.
+
+## 0.12.1 — 2026-10-31 (PyPI)
 
 ### Changed (performance only; output identical)
 - Budget fit checks compute only the request's estimated token count

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: context-engineering-core
-Version: 0.12.1
+Version: 0.15.0
 Summary: Provider-independent context layers, source contracts, and token budget accounting
 Project-URL: Homepage, https://github.com/Nandu1729/Context-Engine
 Project-URL: Repository, https://github.com/Nandu1729/Context-Engine
@@ -18,15 +18,20 @@ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
 Requires-Python: >=3.12
+Requires-Dist: numpy>=1.26
 Requires-Dist: pywin32==311; sys_platform == 'win32'
 Requires-Dist: rank-bm25<0.3,>=0.2.2
 Requires-Dist: tiktoken<0.15,>=0.14
+Provides-Extra: fastembed
+Requires-Dist: fastembed<0.10,>=0.9; extra == 'fastembed'
 Provides-Extra: langchain
 Requires-Dist: langchain-core<2,>=1.0; extra == 'langchain'
 Provides-Extra: provider
 Requires-Dist: httpx<0.29,>=0.28; extra == 'provider'
 Provides-Extra: reports
 Requires-Dist: matplotlib<3.12,>=3.11; extra == 'reports'
+Provides-Extra: semantic
+Requires-Dist: model2vec<0.11,>=0.10; extra == 'semantic'
 Provides-Extra: service
 Requires-Dist: fastapi<0.136,>=0.135; extra == 'service'
 Requires-Dist: httpx<0.29,>=0.28; extra == 'service'
@@ -45,7 +50,7 @@ that fits the budget. Those messages keep the recent turns, the facts you pinned
 and the older evidence that matches the question. It never calls a model by
 itself, so you send the messages to any provider you like.
 
-> **Status: alpha (0.12.0).** Good for developers to try locally. It is not yet
+> **Status: alpha (0.15.0).** Good for developers to try locally. It is not yet
 > proven on real-world data, and there is no hosted service. See
 > [What is not proven yet](#what-is-not-proven-yet).
 
@@ -106,6 +111,21 @@ chat_model.invoke(langchain_messages(result))  # pip install 'context-engineerin
 history = turns_from_openai(existing_openai_messages, scope)  # reuse a chat history you already have
 ```
 
+### Finds meaning, not just matching words
+
+Retrieval combines keyword search with a small embedding model bundled in the
+package, so "retailer" finds "merchant". It runs locally with numpy: no download,
+no network, no extra dependency (the package is ~7 MB because of the model).
+Ranking is 0.5 x meaning score + 0.5 x keyword score.
+
+```python
+result = assemble_context(..., embedder=None)   # keyword-only, as before 0.15.0
+from context_engine.embeddings import FastEmbedEmbedder, FunctionEmbedder, Model2VecEmbedder
+result = assemble_context(..., embedder=FastEmbedEmbedder())  # more accurate, slower (extra [fastembed])
+```
+
+`FunctionEmbedder` wraps your own embedding function or API.
+
 ### Use it from an MCP client
 
 `context-engine-mcp --data-dir ./context-memory` runs a local MCP server (stdio) with
@@ -154,11 +174,17 @@ These are enforced in code and covered by about 1,400 tests:
 - **Answer quality.** Tests measure whether the needed facts *reach* the model's
   context, not whether the model then answers correctly. On the built-in
   benchmark, the default settings keep 31 of 31 planted facts at 900 and 3000
-  tokens. But that benchmark is one synthetic conversation written alongside
-  the engine.
-- **Real-world data.** There are no results yet on real user conversations or
-  on questions phrased differently from the source text. Retrieval is
-  keyword-based (BM25), so heavily reworded questions may miss evidence.
+  tokens, and a live model run answered 31 of 31 correctly at 900 tokens. But
+  that benchmark is one synthetic conversation written alongside the engine.
+- **Reworded questions.** When we reword the 31 benchmark questions so they share
+  no words with the stored facts, keyword-only retrieval keeps 3 of 31 facts; the
+  default hybrid retrieval keeps 21 of 31 (retrieval alone: 24–25 of 31), with no
+  loss on the original questions. In a live model run on the reworded questions,
+  keyword-only answered 3 of 31 correctly and hybrid 18 of 31 (21 of 31 with
+  lenient grading, which ignores trailing punctuation and dash characters). A fact
+  buried in a long, repetitive message and asked with different words is still
+  hard (12 of 31). Run the offline test: `context-engine strategy-eval --questions paraphrased`.
+- **Real-world data.** There are no results yet on real user conversations.
 - **Exact provider token counts.** Budgets use a local tokenizer estimate.
   Providers may count differently. On small requests, provider counts were up
   to 1.78× the local estimate in past tests, so leave headroom or calibrate.

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/README.md

@@ -9,7 +9,7 @@ that fits the budget. Those messages keep the recent turns, the facts you pinned
 and the older evidence that matches the question. It never calls a model by
 itself, so you send the messages to any provider you like.
 
-> **Status: alpha (0.12.0).** Good for developers to try locally. It is not yet
+> **Status: alpha (0.15.0).** Good for developers to try locally. It is not yet
 > proven on real-world data, and there is no hosted service. See
 > [What is not proven yet](#what-is-not-proven-yet).
 
@@ -70,6 +70,21 @@ chat_model.invoke(langchain_messages(result))  # pip install 'context-engineerin
 history = turns_from_openai(existing_openai_messages, scope)  # reuse a chat history you already have
 ```
 
+### Finds meaning, not just matching words
+
+Retrieval combines keyword search with a small embedding model bundled in the
+package, so "retailer" finds "merchant". It runs locally with numpy: no download,
+no network, no extra dependency (the package is ~7 MB because of the model).
+Ranking is 0.5 x meaning score + 0.5 x keyword score.
+
+```python
+result = assemble_context(..., embedder=None)   # keyword-only, as before 0.15.0
+from context_engine.embeddings import FastEmbedEmbedder, FunctionEmbedder, Model2VecEmbedder
+result = assemble_context(..., embedder=FastEmbedEmbedder())  # more accurate, slower (extra [fastembed])
+```
+
+`FunctionEmbedder` wraps your own embedding function or API.
+
 ### Use it from an MCP client
 
 `context-engine-mcp --data-dir ./context-memory` runs a local MCP server (stdio) with
@@ -118,11 +133,17 @@ These are enforced in code and covered by about 1,400 tests:
 - **Answer quality.** Tests measure whether the needed facts *reach* the model's
   context, not whether the model then answers correctly. On the built-in
   benchmark, the default settings keep 31 of 31 planted facts at 900 and 3000
-  tokens. But that benchmark is one synthetic conversation written alongside
-  the engine.
-- **Real-world data.** There are no results yet on real user conversations or
-  on questions phrased differently from the source text. Retrieval is
-  keyword-based (BM25), so heavily reworded questions may miss evidence.
+  tokens, and a live model run answered 31 of 31 correctly at 900 tokens. But
+  that benchmark is one synthetic conversation written alongside the engine.
+- **Reworded questions.** When we reword the 31 benchmark questions so they share
+  no words with the stored facts, keyword-only retrieval keeps 3 of 31 facts; the
+  default hybrid retrieval keeps 21 of 31 (retrieval alone: 24–25 of 31), with no
+  loss on the original questions. In a live model run on the reworded questions,
+  keyword-only answered 3 of 31 correctly and hybrid 18 of 31 (21 of 31 with
+  lenient grading, which ignores trailing punctuation and dash characters). A fact
+  buried in a long, repetitive message and asked with different words is still
+  hard (12 of 31). Run the offline test: `context-engine strategy-eval --questions paraphrased`.
+- **Real-world data.** There are no results yet on real user conversations.
 - **Exact provider token counts.** Budgets use a local tokenizer estimate.
   Providers may count differently. On small requests, provider counts were up
   to 1.78× the local estimate in past tests, so leave headroom or calibrate.

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "context-engineering-core"
-version = "0.12.1"
+version = "0.15.0"
 description = "Provider-independent context layers, source contracts, and token budget accounting"
 readme = "README.md"
 license = "Apache-2.0"
@@ -21,11 +21,13 @@ classifiers = [
     "Topic :: Software Development :: Libraries :: Python Modules",
     "Typing :: Typed",
 ]
-dependencies = ["tiktoken>=0.14,<0.15", "rank-bm25>=0.2.2,<0.3", "pywin32==311; sys_platform == 'win32'"]
+dependencies = ["tiktoken>=0.14,<0.15", "rank-bm25>=0.2.2,<0.3", "numpy>=1.26", "pywin32==311; sys_platform == 'win32'"]
 
 [project.optional-dependencies]
 provider = ["httpx>=0.28,<0.29"]
 langchain = ["langchain-core>=1.0,<2"]
+semantic = ["model2vec>=0.10,<0.11"]
+fastembed = ["fastembed>=0.9,<0.10"]
 reports = ["matplotlib>=3.11,<3.12"]
 service = ["fastapi>=0.135,<0.136", "uvicorn>=0.41,<0.42", "PyJWT[crypto]>=2.12,<3", "httpx>=0.28,<0.29"]
 

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/src/context_engine/__init__.py

@@ -1,6 +1,6 @@
 """Core contracts. Importing this package performs no configuration or network I/O."""
 
-__version__ = "0.12.1"
+__version__ = "0.15.0"
 
 
 def __getattr__(name):

{context_engineering_core-0.12.1 → context_engineering_core-0.15.0}/src/context_engine/cli.py

@@ -81,6 +81,17 @@ def main(argv: list[str] | None = None) -> int:
         "--budget", type=int, action="append", help="Input allowance; repeatable"
     )
     strategies.add_argument("--probes", action="store_true", help="Include per-fact results")
+    strategies.add_argument(
+        "--questions",
+        choices=("original", "paraphrased"),
+        default="original",
+        help="paraphrased: reworded questions with no shared words (lexical stress test)",
+    )
+    strategies.add_argument(
+        "--no-semantic",
+        action="store_true",
+        help="Skip the hybrid strategies (retrieval_semantic, combined_semantic)",
+    )
     strategies.add_argument("--data-dir", type=Path)
     strategies.add_argument("--output", type=Path, help="Save JSON to a new file; never overwrite")
     planned = commands.add_parser(
@@ -136,7 +147,13 @@ def main(argv: list[str] | None = None) -> int:
             budgets = None if args.budget is None else tuple(args.budget)
             if budgets is not None and any(b <= 0 for b in budgets):
                 raise ContextEngineError("Budgets must be positive integers")
-            result = compare_strategies(bundle, budgets, include_probes=args.probes)
+            result = compare_strategies(
+                bundle,
+                budgets,
+                include_probes=args.probes,
+                questions=args.questions,
+                embedder=None if args.no_semantic else "builtin",
+            )
             if args.output is not None:
                 save_inspection(args.output, result)
             print(json.dumps(result, indent=2, ensure_ascii=False))
@@ -216,6 +233,9 @@ def main(argv: list[str] | None = None) -> int:
     except ContextEngineError as error:
         print(json.dumps(error.to_dict()), file=sys.stderr)
         return 1
+    except ImportError as error:  # An optional extra (e.g. [semantic]) is missing.
+        print(json.dumps({"code": "missing_extra", "message": str(error)}), file=sys.stderr)
+        return 2
 
 
 if __name__ == "__main__":

context_engineering_core-0.15.0/src/context_engine/embedding_data/LICENSE

@@ -0,0 +1,33 @@
+Bundled embedding model data
+============================
+
+potion-base-8M-int8.npz and potion-base-8M-vocab.txt are derived from the
+Model2Vec model "minishlab/potion-base-8M" (revision
+bf8b056651a2c21b8d2565580b8569da283cab23), https://huggingface.co/minishlab/potion-base-8M,
+licensed under the MIT License. Its vocabulary comes from BAAI/bge-base-en-v1.5
+(MIT License). The vectors were quantized to 8-bit integers with one float32
+scale per token by the Context Engine project; no other changes.
+
+MIT License
+
+Copyright (c) 2024 Thomas van Dongen
+(copyright line as in the MIT license of the model2vec project; the model card
+of minishlab/potion-base-8M declares "license: mit")
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.