context-dedup 0.1.0__tar.gz

context_dedup-0.1.0/LICENSE

@@ -0,0 +1,375 @@
+Mozilla Public License Version 2.0
+==================================
+
+1. Definitions
+--------------
+
+1.1. "Contributor"
+    means each individual or legal entity that creates, contributes to
+    the creation of, or owns Covered Software.
+
+1.2. "Contributor Version"
+    means the combination of the Contributions of others (if any) used
+    by a Contributor and that particular Contributor's Contribution.
+
+1.3. "Contribution"
+    means Covered Software of a particular Contributor.
+
+1.4. "Covered Software"
+    means Source Code Form to which the initial Contributor has attached
+    the notice in Exhibit A, the Executable Form of such Source Code
+    Form, and Modifications of such Source Code Form, in each case
+    including portions thereof.
+
+1.5. "Incompatible With Secondary Licenses"
+    means
+
+    (a) that the initial Contributor has attached the notice described
+        in Exhibit B to the Covered Software; or
+
+    (b) that the Covered Software was made available under the terms of
+        version 1.1 or earlier of the License, but not also under the
+        terms of a Secondary License.
+
+1.6. "Executable Form"
+    means any form of the work other than Source Code Form.
+
+1.7. "Larger Work"
+    means a work that combines Covered Software with other material, in 
+    a separate file or files, that is not Covered Software.
+
+1.8. "License"
+    means this document.
+
+1.9. "Licensable"
+    means having the right to grant, to the maximum extent possible,
+    whether at the time of the initial grant or subsequently, any and
+    all of the rights conveyed by this License.
+
+1.10. "Modifications"
+    means any of the following:
+
+    (a) any file in Source Code Form that results from an addition to,
+        deletion from, or modification of the contents of Covered
+        Software; or
+
+    (b) any new file in Source Code Form that contains any Covered
+        Software.
+
+1.11. "Patent Claims" of a Contributor
+    means any patent claim(s), including without limitation, method,
+    process, and apparatus claims, in any patent Licensable by such
+    Contributor that would be infringed, but for the grant of the
+    License, by the making, using, selling, offering for sale, having
+    made, import, or transfer of either its Contributions or its
+    Contributor Version.
+
+1.12. "Secondary License"
+    means either the GNU General Public License, Version 2.0, the GNU
+    Lesser General Public License, Version 2.1, the GNU Affero General
+    Public License, Version 3.0, or any later versions of those
+    licenses.
+
+1.13. "Source Code Form"
+    means the form of the work preferred for making modifications.
+
+1.14. "You" (or "Your")
+    means an individual or a legal entity exercising rights under this
+    License. For legal entities, "You" includes any entity that
+    controls, is controlled by, or is under common control with You. For
+    purposes of this definition, "control" means (a) the power, direct
+    or indirect, to cause the direction or management of such entity,
+    whether by contract or otherwise, or (b) ownership of more than
+    fifty percent (50%) of the outstanding shares or beneficial
+    ownership of such entity.
+
+2. License Grants and Conditions
+--------------------------------
+
+2.1. Grants
+
+Each Contributor hereby grants You a world-wide, royalty-free,
+non-exclusive license:
+
+(a) under intellectual property rights (other than patent or trademark)
+    Licensable by such Contributor to use, reproduce, make available,
+    modify, display, perform, distribute, and otherwise exploit its
+    Contributions, either on an unmodified basis, with Modifications, or
+    as part of a Larger Work; and
+
+(b) under Patent Claims of such Contributor to make, use, sell, offer
+    for sale, have made, import, and otherwise transfer either its
+    Contributions or its Contributor Version.
+
+2.2. Effective Date
+
+The licenses granted in Section 2.1 with respect to any Contribution
+become effective for each Contribution on the date the Contributor first
+distributes such Contribution.
+
+2.3. Limitations on Grant Scope
+
+The licenses granted in this Section 2 are the only rights granted under
+this License. No additional rights or licenses will be implied from the
+distribution or licensing of Covered Software under this License.
+Notwithstanding Section 2.1(b) above, no patent license is granted by a
+Contributor:
+
+(a) for any code that a Contributor has removed from Covered Software;
+    or
+
+(b) for infringements caused by: (i) Your and any other third party's
+    modifications of Covered Software, or (ii) the combination of its
+    Contributions with other software (except as part of its Contributor
+    Version); or
+
+(c) under Patent Claims infringed by Covered Software in the absence of
+    its Contributions.
+
+This License does not grant any rights in the trademarks, service marks,
+or logos of any Contributor (except as may be necessary to comply with
+the notice requirements in Section 3.4).
+
+2.4. Subsequent Licenses
+
+No Contributor makes additional grants as a result of Your choice to
+distribute the Covered Software under a subsequent version of this
+License (see Section 10.2) or under the terms of a Secondary License (if
+permitted under the terms of Section 3.3).
+
+2.5. Representation
+
+Each Contributor represents that the Contributor believes its
+Contributions are its original creation(s) or it has sufficient rights
+to grant the rights to its Contributions conveyed by this License.
+
+2.6. Fair Use
+
+This License is not intended to limit any rights You have under
+applicable copyright doctrines of fair use, fair dealing, or other
+equivalents.
+
+2.7. Conditions
+
+Sections 3.1, 3.2, 3.3, and 3.4 are conditions of the licenses granted
+in Section 2.1.
+
+3. Responsibilities
+-------------------
+
+3.1. Distribution of Source Form
+
+All distribution of Covered Software in Source Code Form, including any
+Modifications that You create or to which You contribute, must be under
+the terms of this License. You must inform recipients that the Source
+Code Form of the Covered Software is governed by the terms of this
+License, and how they can obtain a copy of this License. You may not
+attempt to alter or restrict the recipients' rights in the Source Code
+Form.
+
+3.2. Distribution of Executable Form
+
+If You distribute Covered Software in Executable Form then:
+
+(a) such Covered Software must also be made available in Source Code
+    Form, as described in Section 3.1, and You must inform recipients of
+    the Executable Form how they can obtain a copy of such Source Code
+    Form by reasonable means in a timely manner, at a charge no more
+    than the cost of distribution to the recipient; and
+
+(b) You may distribute such Executable Form under the terms of this
+    License, or sublicense it under different terms, provided that the
+    license for the Executable Form does not attempt to limit or alter
+    the recipients' rights in the Source Code Form under this License.
+
+3.3. Distribution of a Larger Work
+
+You may create and distribute a Larger Work under terms of Your choice,
+provided that You also comply with the requirements of this License for
+the Covered Software. If the Larger Work is a combination of Covered
+Software with a work governed by one or more Secondary Licenses, and the
+Covered Software is not Incompatible With Secondary Licenses, this
+License permits You to additionally distribute such Covered Software
+under the terms of such Secondary License(s), so that the recipient of
+the Larger Work may, at their option, further distribute the Covered
+Software under the terms of either this License or such Secondary
+License(s).
+
+3.4. Notices
+
+You may not remove or alter the substance of any license notices
+(including copyright notices, patent notices, disclaimers of warranty,
+or limitations of liability) contained within the Source Code Form of
+the Covered Software, except that You may alter any license notices to
+the extent required to remedy known factual inaccuracies.
+
+3.5. Application of Additional Terms
+
+You may choose to offer, and to charge a fee for, warranty, support,
+indemnity or liability obligations to one or more recipients of Covered
+Software. However, You may do so only on Your own behalf, and not on
+behalf of any Contributor. You must make it absolutely clear that any
+such warranty, support, indemnity, or liability obligation is offered by
+You alone, and You hereby agree to indemnify every Contributor for any
+liability incurred by such Contributor as a result of warranty, support,
+indemnity or liability terms You offer. You may include additional
+disclaimers of warranty and limitations of liability specific to any
+jurisdiction.
+
+4. Inability to Comply Due to Statute or Regulation
+---------------------------------------------------
+
+If it is impossible for You to comply with any of the terms of this
+License with respect to some or all of the Covered Software due to
+statute, judicial order, or regulation then You must: (a) comply with
+the terms of this License to the maximum extent possible; and (b)
+describe the limitations and the code they affect. Such description must
+be placed in a text file included with all distributions of the Covered
+Software under this License. Except to the extent prohibited by statute
+or regulation, such description must be sufficiently detailed for a
+recipient of ordinary skill to be able to understand it.
+
+5. Termination
+--------------
+
+5.1. The rights granted under this License will terminate automatically
+if You fail to comply with any of its terms. However, if You become
+compliant, then the rights granted under this License from a particular
+Contributor are reinstated (a) provisionally, unless and until such
+Contributor explicitly and finally terminates Your grants, and (b) on an
+ongoing basis, if such Contributor fails to notify You of the
+non-compliance by some reasonable means prior to 60 days after You have
+come back into compliance. Moreover, Your grants from a particular
+Contributor are reinstated on an ongoing basis if such Contributor
+notifies You of the non-compliance by some reasonable means, this is the
+first time You have received notice of non-compliance with this License
+from such Contributor, and You become compliant prior to 30 days after
+Your receipt of the notice.
+
+5.2. If You initiate litigation against any entity by asserting a patent
+infringement claim (excluding declaratory judgment actions,
+counter-claims, and cross-claims) alleging that a Contributor Version
+directly or indirectly infringes any patent, then the rights granted to
+You by any and all Contributors for the Covered Software under Section
+2.1 of this License shall terminate.
+
+5.3. In the event of termination under Sections 5.1 or 5.2 above, all
+end user license agreements (excluding distributors and resellers) which
+have been validly granted by You or Your distributors under this License
+prior to termination shall survive termination.
+
+************************************************************************
+*                                                                      *
+*  6. Disclaimer of Warranty                                           *
+*  -------------------------                                           *
+*                                                                      *
+*  Covered Software is provided under this License on an "as is"       *
+*  basis, without warranty of any kind, either expressed, implied, or  *
+*  statutory, including, without limitation, warranties that the       *
+*  Covered Software is free of defects, merchantable, fit for a        *
+*  particular purpose or non-infringing. The entire risk as to the     *
+*  quality and performance of the Covered Software is with You.        *
+*  Should any Covered Software prove defective in any respect, You     *
+*  (not any Contributor) assume the cost of any necessary servicing,   *
+*  repair, or correction. This disclaimer of warranty constitutes an   *
+*  essential part of this License. No use of any Covered Software is   *
+*  authorized under this License except under this disclaimer.         *
+*                                                                      *
+************************************************************************
+
+************************************************************************
+*                                                                      *
+*  7. Limitation of Liability                                          *
+*  --------------------------                                          *
+*                                                                      *
+*  Under no circumstances and under no legal theory, whether tort      *
+*  (including negligence), contract, or otherwise, shall any           *
+*  Contributor, or anyone who distributes Covered Software as          *
+*  permitted above, be liable to You for any direct, indirect,         *
+*  special, incidental, or consequential damages of any character      *
+*  including, without limitation, damages for lost profits, loss of    *
+*  goodwill, work stoppage, computer failure or malfunction, or any    *
+*  and all other commercial damages or losses, even if such party      *
+*  shall have been informed of the possibility of such damages. This   *
+*  limitation of liability shall not apply to liability for death or   *
+*  personal injury resulting from such party's negligence to the       *
+*  extent applicable law prohibits such limitation. Some               *
+*  jurisdictions do not allow the exclusion or limitation of           *
+*  incidental or consequential damages, so this exclusion and          *
+*  limitation may not apply to You.                                    *
+*                                                                      *
+************************************************************************
+
+8. Litigation
+-------------
+
+Any litigation relating to this License may be brought only in the
+courts of a jurisdiction where the defendant maintains its principal
+place of business and such litigation shall be governed by laws of that
+jurisdiction, without reference to its conflict-of-law provisions.
+Nothing in this Section shall prevent a party's ability to bring
+cross-claims or counter-claims.
+
+9. Miscellaneous
+----------------
+
+This License represents the complete agreement concerning the subject
+matter hereof. If any provision of this License is held to be
+unenforceable, such provision shall be reformed only to the extent
+necessary to make it enforceable. Any law or regulation which provides
+that the language of a contract shall be construed against the drafter
+shall not be used to construe this License against a Contributor.
+
+10. Versions of the License
+---------------------------
+
+10.1. New Versions
+
+Mozilla Foundation is the license steward. Except as provided in Section
+10.3, no one other than the license steward has the right to modify or
+publish new versions of this License. Each version will be given a
+distinguishing version number.
+
+10.2. Effect of New Versions
+
+You may distribute the Covered Software under the terms of the version
+of the License under which You originally received the Covered Software,
+or under the terms of any subsequent version published by the license
+steward.
+
+10.3. Modified Versions
+
+If you create software not governed by this License, and you want to
+create a new license for such software, you may create and use a
+modified version of this License if you rename the license and remove
+any references to the name of the license steward (except to note that
+such modified license differs from this License).
+
+10.4. Distributing Source Code Form that is Incompatible With Secondary
+Licenses
+
+If You choose to distribute Source Code Form that is Incompatible With
+Secondary Licenses under the terms of this version of the License, the
+notice described in Exhibit B of this License must be attached.
+
+Exhibit A - Source Code Form License Notice
+-------------------------------------------
+
+  This Source Code Form is subject to the terms of the Mozilla Public
+  License, v. 2.0. If a copy of the MPL was not distributed with this
+  file, You can obtain one at http://mozilla.org/MPL/2.0/.
+
+If it is not possible or desirable to put the notice in a particular
+file, then You may include the notice in a location (such as a LICENSE
+file in a relevant directory) where a recipient would be likely to look
+for such a notice.
+
+You may add additional accurate notices of copyright ownership.
+
+Exhibit B - "Incompatible With Secondary Licenses" Notice
+---------------------------------------------------------
+
+  This Source Code Form is "Incompatible With Secondary Licenses", as
+  defined by the Mozilla Public License, v. 2.0.
+
+Copyright (c) 2026 Eduardo J. Barrios

context_dedup-0.1.0/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.4
+Name: context-dedup
+Version: 0.1.0
+Summary: Detect and remove redundant context before sending it to an LLM.
+Author-email: "Eduardo J. Barrios" <edujbarrios@outlook.com>
+License-Expression: MPL-2.0
+Project-URL: Homepage, https://github.com/edujbarrios/context-dedup
+Project-URL: Source, https://github.com/edujbarrios/context-dedup
+Project-URL: Issues, https://github.com/edujbarrios/context-dedup/issues
+Keywords: llm,rag,context,deduplication,retrieval,ai,prompt,context-window
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# context-dedup
+
+[![PyPI version](https://img.shields.io/pypi/v/context-dedup.svg?label=PyPI&logo=pypi&cacheSeconds=300)](https://pypi.org/project/context-dedup/)
+[![License: MPL 2.0](https://img.shields.io/badge/license-MPL%202.0-blue.svg)](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)
+
+Detect and remove redundant context before sending it to an LLM.
+
+context-dedup is a small, deterministic Python library for finding exact and near-duplicate chunks in RAG results, agent pipelines, and assembled prompts. It uses lexical word n-gram overlap: no embeddings, no LLM calls, and no runtime dependencies.
+
+## Installation
+
+```bash
+pip install context-dedup
+```
+
+## Usage
+
+```python
+from context_dedup import deduplicate, inspect_context, similarity
+
+chunks = [
+    "The Eiffel Tower is located in Paris.",
+    "The Eiffel Tower is located in Paris. It was completed in 1889.",
+    "Madrid is the capital of Spain.",
+]
+
+print(similarity(chunks[0], chunks[1]))
+print(inspect_context(chunks))
+print(deduplicate(chunks))
+```
+
+## Why context-dedup?
+
+Retrieval and prompt assembly can repeat the same facts across overlapping passages, wasting context-window space and attention. context-dedup provides a transparent heuristic between raw retrieval and an LLM call, without adding a model, service, database, or provider SDK.
+
+It is intentionally focused:
+
+* No embeddings or transformer models
+* No LLM calls or external APIs
+* No runtime dependencies or infrastructure
+
+## Inspect context
+
+`inspect_context` returns an ordinary, serializable dictionary with pair scores, connected duplicate groups, recommended representatives, removable indices, and aggregate redundancy estimates:
+
+```python
+report = inspect_context(chunks)
+
+print(report["redundant_pairs"])
+print(report["groups"])
+print(report["estimated_redundant_words"])
+```
+
+Thresholds are configurable. Their defaults are practical heuristics, not statistically universal values:
+
+```python
+report = inspect_context(
+    chunks,
+    similarity_threshold=0.75,
+    containment_threshold=0.85,
+    n=3,
+)
+```
+
+## Deduplicate context
+
+The default strategy keeps the longest chunk in each duplicate group, preserving more information. Use `first` to keep the earliest chunk instead:
+
+```python
+clean_chunks = deduplicate(chunks)
+first_chunks = deduplicate(chunks, strategy="first")
+```
+
+Objects with metadata are supported through `key`; returned objects are the originals:
+
+```python
+retrieved = [
+    {"text": "Refunds are available within 30 days.", "source": "policy.pdf", "page": 4},
+    {"text": "Refunds are available within 30 days. Contact support to begin.", "source": "faq.pdf", "page": 2},
+]
+
+clean = deduplicate(retrieved, key=lambda item: item["text"])
+```
+
+## Algorithm
+
+Text is lowercased, stripped, and normalized to single spaces. The library builds sets of word trigrams by default, then calculates Jaccard similarity and containment in both directions. A pair is redundant when either configured threshold is reached. Connected components combine transitive pairs into groups, and selection is deterministic.
+
+## Limitations
+
+This is a lexical heuristic, not a semantic-equivalence detector. Paraphrases with different wording may not match, while repeated wording can match despite different meaning. Punctuation remains part of word tokens. Version 0.1.0 compares every pair in `O(n²)` time, which is appropriate for contexts with tens or hundreds of chunks but not large document collections.
+
+## Use cases
+
+* Remove overlap from RAG retrieval results before prompt construction
+* Inspect context redundancy in LLM and agent pipelines
+* Deduplicate passages assembled from multiple document sources
+* Reduce repeated prompt content without provider-specific infrastructure
+
+## Features
+
+* Deterministic word n-gram Jaccard similarity and directional containment
+* Transitive duplicate groups and serializable inspection reports
+* `longest` and `first` representative strategies
+* Metadata-preserving `key` support
+* Configurable thresholds and n-gram size
+* Standard-library implementation with no runtime dependencies
+
+## Issues
+
+Report issues in the [GitHub issue tracker](https://github.com/edujbarrios/context-dedup/issues).
+
+## Author
+
+Eduardo J. Barrios — [edujbarrios@outlook.com](mailto:edujbarrios@outlook.com)
+
+## License
+
+[Mozilla Public License 2.0](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)

context_dedup-0.1.0/README.md

@@ -0,0 +1,119 @@
+# context-dedup
+
+[![PyPI version](https://img.shields.io/pypi/v/context-dedup.svg?label=PyPI&logo=pypi&cacheSeconds=300)](https://pypi.org/project/context-dedup/)
+[![License: MPL 2.0](https://img.shields.io/badge/license-MPL%202.0-blue.svg)](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)
+
+Detect and remove redundant context before sending it to an LLM.
+
+context-dedup is a small, deterministic Python library for finding exact and near-duplicate chunks in RAG results, agent pipelines, and assembled prompts. It uses lexical word n-gram overlap: no embeddings, no LLM calls, and no runtime dependencies.
+
+## Installation
+
+```bash
+pip install context-dedup
+```
+
+## Usage
+
+```python
+from context_dedup import deduplicate, inspect_context, similarity
+
+chunks = [
+    "The Eiffel Tower is located in Paris.",
+    "The Eiffel Tower is located in Paris. It was completed in 1889.",
+    "Madrid is the capital of Spain.",
+]
+
+print(similarity(chunks[0], chunks[1]))
+print(inspect_context(chunks))
+print(deduplicate(chunks))
+```
+
+## Why context-dedup?
+
+Retrieval and prompt assembly can repeat the same facts across overlapping passages, wasting context-window space and attention. context-dedup provides a transparent heuristic between raw retrieval and an LLM call, without adding a model, service, database, or provider SDK.
+
+It is intentionally focused:
+
+* No embeddings or transformer models
+* No LLM calls or external APIs
+* No runtime dependencies or infrastructure
+
+## Inspect context
+
+`inspect_context` returns an ordinary, serializable dictionary with pair scores, connected duplicate groups, recommended representatives, removable indices, and aggregate redundancy estimates:
+
+```python
+report = inspect_context(chunks)
+
+print(report["redundant_pairs"])
+print(report["groups"])
+print(report["estimated_redundant_words"])
+```
+
+Thresholds are configurable. Their defaults are practical heuristics, not statistically universal values:
+
+```python
+report = inspect_context(
+    chunks,
+    similarity_threshold=0.75,
+    containment_threshold=0.85,
+    n=3,
+)
+```
+
+## Deduplicate context
+
+The default strategy keeps the longest chunk in each duplicate group, preserving more information. Use `first` to keep the earliest chunk instead:
+
+```python
+clean_chunks = deduplicate(chunks)
+first_chunks = deduplicate(chunks, strategy="first")
+```
+
+Objects with metadata are supported through `key`; returned objects are the originals:
+
+```python
+retrieved = [
+    {"text": "Refunds are available within 30 days.", "source": "policy.pdf", "page": 4},
+    {"text": "Refunds are available within 30 days. Contact support to begin.", "source": "faq.pdf", "page": 2},
+]
+
+clean = deduplicate(retrieved, key=lambda item: item["text"])
+```
+
+## Algorithm
+
+Text is lowercased, stripped, and normalized to single spaces. The library builds sets of word trigrams by default, then calculates Jaccard similarity and containment in both directions. A pair is redundant when either configured threshold is reached. Connected components combine transitive pairs into groups, and selection is deterministic.
+
+## Limitations
+
+This is a lexical heuristic, not a semantic-equivalence detector. Paraphrases with different wording may not match, while repeated wording can match despite different meaning. Punctuation remains part of word tokens. Version 0.1.0 compares every pair in `O(n²)` time, which is appropriate for contexts with tens or hundreds of chunks but not large document collections.
+
+## Use cases
+
+* Remove overlap from RAG retrieval results before prompt construction
+* Inspect context redundancy in LLM and agent pipelines
+* Deduplicate passages assembled from multiple document sources
+* Reduce repeated prompt content without provider-specific infrastructure
+
+## Features
+
+* Deterministic word n-gram Jaccard similarity and directional containment
+* Transitive duplicate groups and serializable inspection reports
+* `longest` and `first` representative strategies
+* Metadata-preserving `key` support
+* Configurable thresholds and n-gram size
+* Standard-library implementation with no runtime dependencies
+
+## Issues
+
+Report issues in the [GitHub issue tracker](https://github.com/edujbarrios/context-dedup/issues).
+
+## Author
+
+Eduardo J. Barrios — [edujbarrios@outlook.com](mailto:edujbarrios@outlook.com)
+
+## License
+
+[Mozilla Public License 2.0](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)

context_dedup-0.1.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "context-dedup"
+version = "0.1.0"
+description = "Detect and remove redundant context before sending it to an LLM."
+readme = "README.md"
+requires-python = ">=3.10"
+license = "MPL-2.0"
+license-files = ["LICENSE"]
+authors = [{ name = "Eduardo J. Barrios", email = "edujbarrios@outlook.com" }]
+keywords = ["llm", "rag", "context", "deduplication", "retrieval", "ai", "prompt", "context-window"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Operating System :: OS Independent",
+]
+
+[project.optional-dependencies]
+dev = ["pytest", "build", "twine"]
+
+[project.urls]
+Homepage = "https://github.com/edujbarrios/context-dedup"
+Source = "https://github.com/edujbarrios/context-dedup"
+Issues = "https://github.com/edujbarrios/context-dedup/issues"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-ra"
+

context_dedup-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

context_dedup-0.1.0/src/context_dedup/__init__.py

@@ -0,0 +1,5 @@
+"""Small, deterministic tools for removing redundant LLM context."""
+
+from .core import deduplicate, inspect_context, similarity
+
+__all__ = ["similarity", "inspect_context", "deduplicate"]

context_dedup-0.1.0/src/context_dedup/core.py

@@ -0,0 +1,229 @@
+"""Core implementation for context-dedup."""
+
+from __future__ import annotations
+
+import math
+from collections.abc import Callable, Iterable
+from numbers import Real
+from typing import Any
+
+DEFAULT_SIMILARITY_THRESHOLD = 0.8
+DEFAULT_CONTAINMENT_THRESHOLD = 0.65
+
+
+def _normalize(text: str) -> str:
+    if not isinstance(text, str):
+        raise TypeError("text must be a string")
+    return " ".join(text.lower().split())
+
+
+def _validate_n(n: int) -> int:
+    if isinstance(n, bool) or not isinstance(n, int):
+        raise TypeError("n must be an integer")
+    if n < 1:
+        raise ValueError("n must be at least 1")
+    return n
+
+
+def _validate_threshold(value: float, label: str) -> float:
+    if isinstance(value, bool) or not isinstance(value, Real):
+        raise TypeError(f"{label} must be a number between 0.0 and 1.0")
+    result = float(value)
+    if not math.isfinite(result) or not 0.0 <= result <= 1.0:
+        raise ValueError(f"{label} must be between 0.0 and 1.0 and finite")
+    return result
+
+
+def _word_ngrams(text: str, n: int = 3) -> set[tuple[str, ...]]:
+    size = _validate_n(n)
+    words = _normalize(text).split()
+    if not words:
+        return set()
+    if len(words) < size:
+        return {tuple(words)}
+    return {tuple(words[index : index + size]) for index in range(len(words) - size + 1)}
+
+
+def _scores(left: set[tuple[str, ...]], right: set[tuple[str, ...]]) -> tuple[float, float, float]:
+    if not left and not right:
+        return 1.0, 1.0, 1.0
+    shared = len(left & right)
+    union = len(left | right)
+    jaccard = shared / union if union else 1.0
+    left_containment = shared / len(left) if left else 0.0
+    right_containment = shared / len(right) if right else 0.0
+    return jaccard, left_containment, right_containment
+
+
+def similarity(text_a: str, text_b: str, *, n: int = 3) -> float:
+    """Return word n-gram Jaccard similarity from 0.0 to 1.0.
+
+    Text is lowercased and whitespace is collapsed. Trigrams are used by
+    default; a non-empty text shorter than ``n`` is treated as one n-gram.
+    Two empty or whitespace-only strings have similarity 1.0.
+    """
+    return _scores(_word_ngrams(text_a, n), _word_ngrams(text_b, n))[0]
+
+
+def _find_groups(
+    chunks: Iterable[Any],
+    *,
+    key: Callable[[Any], str],
+    similarity_threshold: float,
+    containment_threshold: float,
+    n: int,
+) -> tuple[list[Any], list[str], list[dict[str, Any]], list[list[int]]]:
+    items = list(chunks)
+    texts = [key(item) for item in items]
+    if any(not isinstance(text, str) for text in texts):
+        raise TypeError("key must return a string for every chunk")
+    grams = [_word_ngrams(text, n) for text in texts]
+    parents = list(range(len(items)))
+
+    def find(index: int) -> int:
+        while parents[index] != index:
+            parents[index] = parents[parents[index]]
+            index = parents[index]
+        return index
+
+    def union(left: int, right: int) -> None:
+        left_root, right_root = find(left), find(right)
+        if left_root != right_root:
+            parents[right_root] = left_root
+
+    pairs = []
+    for left in range(len(items)):
+        for right in range(left + 1, len(items)):
+            jaccard, left_containment, right_containment = _scores(grams[left], grams[right])
+            if jaccard >= similarity_threshold or max(left_containment, right_containment) >= containment_threshold:
+                pairs.append({
+                    "indices": [left, right],
+                    "similarity": jaccard,
+                    "containment": [left_containment, right_containment],
+                })
+                union(left, right)
+
+    components: dict[int, list[int]] = {}
+    for index in range(len(items)):
+        components.setdefault(find(index), []).append(index)
+    groups = [indices for indices in components.values() if len(indices) > 1]
+    return items, texts, pairs, groups
+
+
+def _identity(value: Any) -> Any:
+    return value
+
+
+def _prepare(
+    chunks: Iterable[Any],
+    key: Callable[[Any], str] | None,
+    similarity_threshold: float,
+    containment_threshold: float,
+    n: int,
+) -> tuple[list[Any], list[str], list[dict[str, Any]], list[list[int]]]:
+    if isinstance(chunks, (str, bytes)):
+        raise TypeError("chunks must be a non-string iterable")
+    try:
+        iter(chunks)
+    except TypeError as error:
+        raise TypeError("chunks must be a non-string iterable") from error
+    extractor = _identity if key is None else key
+    if not callable(extractor):
+        raise TypeError("key must be callable or None")
+    similarity_limit = _validate_threshold(similarity_threshold, "similarity_threshold")
+    containment_limit = _validate_threshold(containment_threshold, "containment_threshold")
+    items, texts, pairs, groups = _find_groups(
+        chunks,
+        key=extractor,
+        similarity_threshold=similarity_limit,
+        containment_threshold=containment_limit,
+        n=n,
+    )
+    return items, texts, pairs, groups
+
+
+def _representative(indices: list[int], texts: list[str], strategy: str) -> int:
+    if strategy == "first":
+        return indices[0]
+    if strategy == "longest":
+        return max(indices, key=lambda index: len(texts[index]))
+    raise ValueError("strategy must be 'longest' or 'first'")
+
+
+def inspect_context(
+    chunks: Iterable[Any],
+    *,
+    strategy: str = "longest",
+    key: Callable[[Any], str] | None = None,
+    similarity_threshold: float = DEFAULT_SIMILARITY_THRESHOLD,
+    containment_threshold: float = DEFAULT_CONTAINMENT_THRESHOLD,
+    n: int = 3,
+) -> dict[str, Any]:
+    """Inspect chunks and return a serializable lexical-redundancy report.
+
+    A pair is redundant when its Jaccard score reaches ``similarity_threshold``
+    or either directional containment score reaches ``containment_threshold``.
+    The defaults are practical heuristics, not universal statistical cutoffs.
+    """
+    if strategy not in {"longest", "first"}:
+        raise ValueError("strategy must be 'longest' or 'first'")
+    items, texts, pairs, components = _prepare(
+        chunks, key, similarity_threshold, containment_threshold, n
+    )
+    groups = []
+    removable: list[int] = []
+    for indices in components:
+        keep = _representative(indices, texts, strategy)
+        remove = [index for index in indices if index != keep]
+        removable.extend(remove)
+        group_pairs = [
+            pair for pair in pairs if pair["indices"][0] in indices and pair["indices"][1] in indices
+        ]
+        groups.append({
+            "indices": indices,
+            "representative": keep,
+            "remove_indices": remove,
+            "max_similarity": max(pair["similarity"] for pair in group_pairs),
+            "max_containment": max(max(pair["containment"]) for pair in group_pairs),
+        })
+    removable.sort()
+    total = len(items)
+    return {
+        "total_chunks": total,
+        "redundant_pairs": len(pairs),
+        "redundancy_groups": len(groups),
+        "redundant_chunks": len(removable),
+        "redundancy_ratio": len(removable) / total if total else 0.0,
+        "estimated_redundant_characters": sum(len(texts[index]) for index in removable),
+        "estimated_redundant_words": sum(len(texts[index].split()) for index in removable),
+        "removable_indices": removable,
+        "pairs": pairs,
+        "groups": groups,
+    }
+
+
+def deduplicate(
+    chunks: Iterable[Any],
+    *,
+    strategy: str = "longest",
+    key: Callable[[Any], str] | None = None,
+    similarity_threshold: float = DEFAULT_SIMILARITY_THRESHOLD,
+    containment_threshold: float = DEFAULT_CONTAINMENT_THRESHOLD,
+    n: int = 3,
+) -> list[Any]:
+    """Return chunks with lexical duplicates removed, preserving input order.
+
+    ``longest`` keeps the longest text in each group (and the first on ties).
+    ``first`` keeps the earliest chunk. Original objects are returned unchanged.
+    """
+    items = list(chunks) if not isinstance(chunks, (str, bytes)) else chunks
+    report = inspect_context(
+        items,
+        strategy=strategy,
+        key=key,
+        similarity_threshold=similarity_threshold,
+        containment_threshold=containment_threshold,
+        n=n,
+    )
+    remove = set(report["removable_indices"])
+    return [item for index, item in enumerate(items) if index not in remove]

context_dedup-0.1.0/src/context_dedup.egg-info/PKG-INFO

@@ -0,0 +1,146 @@
+Metadata-Version: 2.4
+Name: context-dedup
+Version: 0.1.0
+Summary: Detect and remove redundant context before sending it to an LLM.
+Author-email: "Eduardo J. Barrios" <edujbarrios@outlook.com>
+License-Expression: MPL-2.0
+Project-URL: Homepage, https://github.com/edujbarrios/context-dedup
+Project-URL: Source, https://github.com/edujbarrios/context-dedup
+Project-URL: Issues, https://github.com/edujbarrios/context-dedup/issues
+Keywords: llm,rag,context,deduplication,retrieval,ai,prompt,context-window
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: twine; extra == "dev"
+Dynamic: license-file
+
+# context-dedup
+
+[![PyPI version](https://img.shields.io/pypi/v/context-dedup.svg?label=PyPI&logo=pypi&cacheSeconds=300)](https://pypi.org/project/context-dedup/)
+[![License: MPL 2.0](https://img.shields.io/badge/license-MPL%202.0-blue.svg)](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)
+
+Detect and remove redundant context before sending it to an LLM.
+
+context-dedup is a small, deterministic Python library for finding exact and near-duplicate chunks in RAG results, agent pipelines, and assembled prompts. It uses lexical word n-gram overlap: no embeddings, no LLM calls, and no runtime dependencies.
+
+## Installation
+
+```bash
+pip install context-dedup
+```
+
+## Usage
+
+```python
+from context_dedup import deduplicate, inspect_context, similarity
+
+chunks = [
+    "The Eiffel Tower is located in Paris.",
+    "The Eiffel Tower is located in Paris. It was completed in 1889.",
+    "Madrid is the capital of Spain.",
+]
+
+print(similarity(chunks[0], chunks[1]))
+print(inspect_context(chunks))
+print(deduplicate(chunks))
+```
+
+## Why context-dedup?
+
+Retrieval and prompt assembly can repeat the same facts across overlapping passages, wasting context-window space and attention. context-dedup provides a transparent heuristic between raw retrieval and an LLM call, without adding a model, service, database, or provider SDK.
+
+It is intentionally focused:
+
+* No embeddings or transformer models
+* No LLM calls or external APIs
+* No runtime dependencies or infrastructure
+
+## Inspect context
+
+`inspect_context` returns an ordinary, serializable dictionary with pair scores, connected duplicate groups, recommended representatives, removable indices, and aggregate redundancy estimates:
+
+```python
+report = inspect_context(chunks)
+
+print(report["redundant_pairs"])
+print(report["groups"])
+print(report["estimated_redundant_words"])
+```
+
+Thresholds are configurable. Their defaults are practical heuristics, not statistically universal values:
+
+```python
+report = inspect_context(
+    chunks,
+    similarity_threshold=0.75,
+    containment_threshold=0.85,
+    n=3,
+)
+```
+
+## Deduplicate context
+
+The default strategy keeps the longest chunk in each duplicate group, preserving more information. Use `first` to keep the earliest chunk instead:
+
+```python
+clean_chunks = deduplicate(chunks)
+first_chunks = deduplicate(chunks, strategy="first")
+```
+
+Objects with metadata are supported through `key`; returned objects are the originals:
+
+```python
+retrieved = [
+    {"text": "Refunds are available within 30 days.", "source": "policy.pdf", "page": 4},
+    {"text": "Refunds are available within 30 days. Contact support to begin.", "source": "faq.pdf", "page": 2},
+]
+
+clean = deduplicate(retrieved, key=lambda item: item["text"])
+```
+
+## Algorithm
+
+Text is lowercased, stripped, and normalized to single spaces. The library builds sets of word trigrams by default, then calculates Jaccard similarity and containment in both directions. A pair is redundant when either configured threshold is reached. Connected components combine transitive pairs into groups, and selection is deterministic.
+
+## Limitations
+
+This is a lexical heuristic, not a semantic-equivalence detector. Paraphrases with different wording may not match, while repeated wording can match despite different meaning. Punctuation remains part of word tokens. Version 0.1.0 compares every pair in `O(n²)` time, which is appropriate for contexts with tens or hundreds of chunks but not large document collections.
+
+## Use cases
+
+* Remove overlap from RAG retrieval results before prompt construction
+* Inspect context redundancy in LLM and agent pipelines
+* Deduplicate passages assembled from multiple document sources
+* Reduce repeated prompt content without provider-specific infrastructure
+
+## Features
+
+* Deterministic word n-gram Jaccard similarity and directional containment
+* Transitive duplicate groups and serializable inspection reports
+* `longest` and `first` representative strategies
+* Metadata-preserving `key` support
+* Configurable thresholds and n-gram size
+* Standard-library implementation with no runtime dependencies
+
+## Issues
+
+Report issues in the [GitHub issue tracker](https://github.com/edujbarrios/context-dedup/issues).
+
+## Author
+
+Eduardo J. Barrios — [edujbarrios@outlook.com](mailto:edujbarrios@outlook.com)
+
+## License
+
+[Mozilla Public License 2.0](https://github.com/edujbarrios/context-dedup/blob/main/LICENSE)

context_dedup-0.1.0/src/context_dedup.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+README.md
+pyproject.toml
+src/context_dedup/__init__.py
+src/context_dedup/core.py
+src/context_dedup.egg-info/PKG-INFO
+src/context_dedup.egg-info/SOURCES.txt
+src/context_dedup.egg-info/dependency_links.txt
+src/context_dedup.egg-info/requires.txt
+src/context_dedup.egg-info/top_level.txt
+tests/test_core.py

context_dedup-0.1.0/src/context_dedup.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

context_dedup-0.1.0/src/context_dedup.egg-info/requires.txt

@@ -0,0 +1,5 @@
+
+[dev]
+pytest
+build
+twine

context_dedup-0.1.0/src/context_dedup.egg-info/top_level.txt

@@ -0,0 +1 @@
+context_dedup

context_dedup-0.1.0/tests/test_core.py

@@ -0,0 +1,164 @@
+import math
+
+import pytest
+
+from context_dedup import deduplicate, inspect_context, similarity
+from context_dedup.core import _find_groups
+
+
+def test_exact_capitalization_and_whitespace_duplicates():
+    assert similarity("Python is a language", "Python is a language") == 1.0
+    assert similarity("  PYTHON  is\na language ", "python is a language") == 1.0
+
+
+def test_different_strings_and_symmetry():
+    left = "Madrid is the capital of Spain"
+    right = "Whales live in the ocean"
+    assert similarity(left, right) == 0.0
+    assert similarity(left, right) == similarity(right, left)
+
+
+def test_near_duplicate_has_partial_similarity():
+    score = similarity("one two three four five", "one two three four six")
+    assert 0.0 < score < 1.0
+
+
+def test_empty_and_short_strings():
+    assert similarity("", " \n") == 1.0
+    assert similarity("", "content") == 0.0
+    assert similarity("short text", "SHORT   TEXT") == 1.0
+
+
+def test_containment_and_transitive_groups():
+    chunks = [
+        "alpha beta gamma delta",
+        "alpha beta gamma delta epsilon zeta",
+        "gamma delta epsilon zeta eta",
+    ]
+    _, _, pairs, groups = _find_groups(
+        chunks,
+        key=lambda value: value,
+        similarity_threshold=0.9,
+        containment_threshold=0.65,
+        n=3,
+    )
+    assert [pair["indices"] for pair in pairs] == [[0, 1], [1, 2]]
+    assert groups == [[0, 1, 2]]
+
+
+@pytest.mark.parametrize("n,error", [(0, ValueError), (-1, ValueError), (1.5, TypeError), (True, TypeError)])
+def test_invalid_ngram_size(n, error):
+    with pytest.raises(error):
+        similarity("a", "a", n=n)
+
+
+@pytest.mark.parametrize("value", [None, 1, []])
+def test_non_string_input(value):
+    with pytest.raises(TypeError):
+        similarity(value, "text")
+
+
+def test_report_consistency_and_longest_selection():
+    chunks = [
+        "The Eiffel Tower is located in Paris.",
+        "The Eiffel Tower is located in Paris. It was completed in 1889.",
+        "Madrid is the capital of Spain.",
+    ]
+    report = inspect_context(chunks)
+    assert report["total_chunks"] == 3
+    assert report["redundant_pairs"] == 1
+    assert report["redundancy_groups"] == 1
+    assert report["redundant_chunks"] == 1
+    assert report["removable_indices"] == [0]
+    assert report["groups"][0]["representative"] == 1
+    assert report["redundancy_ratio"] == pytest.approx(1 / 3)
+    assert report["estimated_redundant_characters"] == len(chunks[0])
+    assert report["estimated_redundant_words"] == len(chunks[0].split())
+    assert deduplicate(chunks) == chunks[1:]
+
+
+def test_default_containment_matches_overlapping_rag_chunks():
+    chunks = [
+        "Python is a programming language.",
+        "Python is a programming language used for data science.",
+        "Madrid is the capital of Spain.",
+    ]
+    assert inspect_context(chunks)["redundant_pairs"] == 1
+    assert deduplicate(chunks) == chunks[1:]
+
+
+def test_first_strategy_and_no_mutation():
+    chunks = ["alpha beta gamma", "alpha beta gamma delta"]
+    original = chunks.copy()
+    assert deduplicate(chunks, strategy="first") == [chunks[0]]
+    assert chunks == original
+
+
+def test_configurable_thresholds():
+    chunks = ["one two three four five", "one two three four six"]
+    assert deduplicate(chunks, similarity_threshold=0.49, containment_threshold=1.0) == [chunks[0]]
+    assert deduplicate(chunks, similarity_threshold=0.51, containment_threshold=1.0) == chunks
+
+
+def test_metadata_key_preserves_original_objects():
+    chunks = [
+        {"text": "same useful context", "page": 1},
+        {"text": "same useful context with detail", "page": 2},
+    ]
+    result = deduplicate(chunks, key=lambda item: item["text"])
+    assert result == [chunks[1]]
+    assert result[0] is chunks[1]
+
+
+def test_key_is_called_once_per_item():
+    calls = []
+
+    def extract(item):
+        calls.append(item)
+        return item["text"]
+
+    chunks = [{"text": "one"}, {"text": "two"}]
+    inspect_context(chunks, key=extract)
+    assert calls == chunks
+
+
+def test_transitive_deduplicate_is_deterministic():
+    chunks = [
+        "alpha beta gamma delta",
+        "alpha beta gamma delta epsilon zeta",
+        "gamma delta epsilon zeta eta",
+    ]
+    options = {"similarity_threshold": 0.9, "containment_threshold": 0.65}
+    first = inspect_context(chunks, **options)
+    assert first == inspect_context(chunks, **options)
+    assert first["groups"][0]["indices"] == [0, 1, 2]
+    assert deduplicate(chunks, **options) == [chunks[1]]
+
+
+def test_empty_context_report():
+    report = inspect_context([])
+    assert report["total_chunks"] == 0
+    assert report["redundancy_ratio"] == 0.0
+    assert report["groups"] == []
+    assert deduplicate([]) == []
+
+
+@pytest.mark.parametrize("strategy", ["last", "shortest", ""])
+def test_invalid_strategy(strategy):
+    with pytest.raises(ValueError):
+        deduplicate(["a"], strategy=strategy)
+
+
+@pytest.mark.parametrize("threshold", [-0.1, 1.1, math.nan, math.inf])
+def test_invalid_threshold(threshold):
+    with pytest.raises(ValueError):
+        inspect_context(["a"], similarity_threshold=threshold)
+
+
+def test_invalid_chunks_and_key():
+    with pytest.raises(TypeError):
+        inspect_context("not a chunk collection")
+    with pytest.raises(TypeError):
+        inspect_context([{"text": "x"}], key=None)
+    with pytest.raises(TypeError):
+        inspect_context(["x"], key="text")