ferpa-haystack 0.1.0__py3-none-any.whl

ferpa_haystack-0.1.0.dist-info/METADATA

@@ -0,0 +1,236 @@
+Metadata-Version: 2.4
+Name: ferpa-haystack
+Version: 0.1.0
+Summary: FERPA-compliant document filter for Haystack RAG pipelines — enforces identity-scoped access control before documents reach the LLM
+Project-URL: Homepage, https://github.com/ashutoshrana/ferpa-haystack
+Project-URL: Documentation, https://github.com/ashutoshrana/ferpa-haystack#readme
+Project-URL: Bug Tracker, https://github.com/ashutoshrana/ferpa-haystack/issues
+Project-URL: Source Code, https://github.com/ashutoshrana/ferpa-haystack
+Author-email: Ashutosh Rana <ai.automate101@gmail.com>
+License-Expression: Apache-2.0
+License-File: LICENSE
+Keywords: AI-governance,FERPA,LLM,NLP,RAG,compliance,data-privacy,education,haystack,higher-education
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: haystack-ai>=2.0.0
+Provides-Extra: dev
+Requires-Dist: hatch; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.21; extra == 'dev'
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# ferpa-haystack
+
+[![PyPI](https://img.shields.io/pypi/v/ferpa-haystack.svg)](https://pypi.org/project/ferpa-haystack/)
+[![Python](https://img.shields.io/pypi/pyversions/ferpa-haystack.svg)](https://pypi.org/project/ferpa-haystack/)
+[![Tests](https://github.com/ashutoshrana/ferpa-haystack/actions/workflows/ci.yml/badge.svg)](https://github.com/ashutoshrana/ferpa-haystack/actions/workflows/ci.yml)
+[![License](https://img.shields.io/badge/license-Apache%202.0-blue.svg)](LICENSE)
+[![Downloads](https://img.shields.io/pypi/dm/ferpa-haystack.svg)](https://pypi.org/project/ferpa-haystack/)
+
+**FERPA-compliant document filtering for Haystack RAG pipelines.**
+
+Enforces 34 CFR § 99 identity-scoped access control at the retrieval layer — before any document reaches the LLM context window.
+
+---
+
+## The Problem
+
+Standard Haystack pipelines retrieve documents and pass them directly to the LLM with no enforcement of who is allowed to see what. In higher-education deployments, this creates a structural FERPA compliance gap: a student advising chatbot may return another student's academic record, financial aid details, or disciplinary history in response to a query.
+
+This component closes that gap by adding a two-layer compliance filter between your retriever and your LLM.
+
+---
+
+## Architecture
+
+```
+Haystack Pipeline
+     │
+     ▼
+InMemoryEmbeddingRetriever (or any retriever)
+     │  documents (all retrieved)
+     ▼
+FERPAMetadataFilter
+     │  Layer 1: Identity pre-filter (student_id + institution_id)
+     │  Layer 2: Category authorization (academic_record, financial_aid, ...)
+     │
+     ├── documents ──────────────► LLM (only authorized records)
+     └── disclosure_record ──────► Audit log (34 CFR § 99.32)
+```
+
+**Documents without identity metadata** (course catalogues, policy handbooks) pass through both layers unchanged — shared knowledge-base content is never blocked.
+
+---
+
+## Installation
+
+```bash
+pip install ferpa-haystack
+```
+
+---
+
+## Quick Start
+
+```python
+from haystack import Pipeline
+from haystack.components.generators import OpenAIGenerator
+from haystack.components.retrievers import InMemoryEmbeddingRetriever
+from haystack.document_stores.in_memory import InMemoryDocumentStore
+from haystack_integrations.components.filters.ferpa_filter import FERPAMetadataFilter
+
+doc_store = InMemoryDocumentStore()
+
+ferpa_filter = FERPAMetadataFilter(
+    student_id="stu_001",
+    institution_id="univ_abc",
+    authorized_categories=["academic_record", "financial_aid"],
+    requesting_user_id="advisor_007",
+)
+
+pipeline = Pipeline()
+pipeline.add_component("retriever", InMemoryEmbeddingRetriever(doc_store))
+pipeline.add_component("ferpa_filter", ferpa_filter)
+pipeline.add_component("llm", OpenAIGenerator(model="gpt-4o"))
+
+pipeline.connect("retriever.documents", "ferpa_filter.documents")
+pipeline.connect("ferpa_filter.documents", "llm.documents")
+
+result = pipeline.run({"retriever": {"query_embedding": query_emb}})
+
+# Only stu_001's authorized records reached the LLM
+authorized_docs = result["ferpa_filter"]["documents"]
+
+# 34 CFR § 99.32 audit entry — log this to your compliance system
+audit_record = result["ferpa_filter"]["disclosure_record"]
+print(audit_record.to_log_entry())
+```
+
+---
+
+## Filtering Layers
+
+### Layer 1 — Identity Pre-Filter
+
+Documents are matched against `student_id` and `institution_id` metadata fields.
+
+| Document metadata | Outcome |
+|-------------------|---------|
+| No `student_id` or `institution_id` | **Pass** — treated as shared content |
+| `student_id` matches | **Continue to Layer 2** |
+| `student_id` does not match | **Blocked** |
+
+### Layer 2 — Category Authorization
+
+When `authorized_categories` is non-empty, the document's `category` field must be in the authorized set.
+
+```python
+# Only academic records and financial aid — disciplinary records are blocked
+FERPAMetadataFilter(
+    student_id="stu_001",
+    institution_id="univ_abc",
+    authorized_categories=["academic_record", "financial_aid"],
+    # "disciplinary" is blocked even if identity matches
+)
+```
+
+---
+
+## Audit Record (34 CFR § 99.32)
+
+Every call to `run()` produces a `FERPADisclosureRecord` regardless of how many documents are authorized:
+
+```python
+@dataclass
+class FERPADisclosureRecord:
+    student_id: str
+    institution_id: str
+    requesting_user_id: str
+    disclosed_at: datetime          # UTC timestamp
+    total_retrieved: int            # documents from retriever
+    total_disclosed: int            # documents that passed filtering
+    categories_disclosed: list[str] # record categories in result
+    pipeline_context: str           # pipeline/workflow label
+```
+
+Log it to your compliance database:
+
+```python
+import logging
+compliance_logger = logging.getLogger("ferpa.audit")
+compliance_logger.info(result["ferpa_filter"]["disclosure_record"].to_log_entry())
+```
+
+---
+
+## Configuration
+
+```python
+FERPAMetadataFilter(
+    student_id="stu_001",
+    institution_id="univ_abc",
+    authorized_categories=["academic_record"],   # empty = all categories allowed
+    requesting_user_id="advisor_007",            # recorded in audit log
+    student_id_field="student_id",               # custom meta key
+    institution_id_field="institution_id",       # custom meta key
+    category_field="category",                   # custom meta key
+    pipeline_context="advising_pipeline",        # audit label
+    raise_on_violation=False,                    # True = raise PermissionError
+)
+```
+
+---
+
+## Custom Field Names
+
+If your document store uses different metadata keys:
+
+```python
+FERPAMetadataFilter(
+    student_id="stu_001",
+    institution_id="univ_abc",
+    student_id_field="learner_id",        # your custom key
+    institution_id_field="campus_code",   # your custom key
+    category_field="record_type",         # your custom key
+)
+```
+
+---
+
+## Pipeline Serialization
+
+The component is fully serializable for YAML/JSON pipeline storage:
+
+```python
+pipeline.to_yaml("advising_pipeline.yaml")
+pipeline_restored = Pipeline.from_yaml("advising_pipeline.yaml")
+```
+
+---
+
+## Regulatory Basis
+
+| Regulation | Section | What this component enforces |
+|-----------|---------|------------------------------|
+| FERPA | 34 CFR § 99.31(a)(1) | Legitimate educational interest — only authorized roles access records |
+| FERPA | 34 CFR § 99.32 | Record of disclosures — structured audit entry on every access |
+
+---
+
+## Related Projects
+
+- **[enterprise-rag-patterns](https://github.com/ashutoshrana/enterprise-rag-patterns)** — FERPA, HIPAA, GDPR compliance patterns for RAG across 50+ regulated sectors
+- **[regulated-ai-governance](https://github.com/ashutoshrana/regulated-ai-governance)** — Policy enforcement for AI agents across 25 jurisdictions
+
+---
+
+## License
+
+Apache License 2.0 — see [LICENSE](LICENSE)

ferpa_haystack-0.1.0.dist-info/RECORD

@@ -0,0 +1,10 @@
+haystack_integrations/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+haystack_integrations/components/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+haystack_integrations/components/filters/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+haystack_integrations/components/filters/ferpa_filter/__about__.py,sha256=kUR5RAFc7HCeiqdlX36dZOHkUI5wI6V_43RpEcD8b-0,22
+haystack_integrations/components/filters/ferpa_filter/__init__.py,sha256=CHis063Qne0jEUAUprYiHDyjuThBw0xGiVgdhaMoAHY,204
+haystack_integrations/components/filters/ferpa_filter/ferpa_metadata_filter.py,sha256=yJO9m38cyPOMLx_ndmnvpz3J_Wm5bfdqQsjlxML_U08,10253
+ferpa_haystack-0.1.0.dist-info/METADATA,sha256=2ajhsWCzmMKVtsgfE3CJ7GkMNcTQdO4QtyY-s08zphY,8253
+ferpa_haystack-0.1.0.dist-info/WHEEL,sha256=QccIxa26bgl1E6uMy58deGWi-0aeIkkangHcxk2kWfw,87
+ferpa_haystack-0.1.0.dist-info/licenses/LICENSE,sha256=bNTXqHZyIInyEg4qC0U3lWDgV0XqOrafcmBK-TVMeWs,742
+ferpa_haystack-0.1.0.dist-info/RECORD,,

ferpa_haystack-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.29.0
+Root-Is-Purelib: true
+Tag: py3-none-any

ferpa_haystack-0.1.0.dist-info/licenses/LICENSE

@@ -0,0 +1,17 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Copyright 2026 Ashutosh Rana
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

haystack_integrations/components/filters/ferpa_filter/__about__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

haystack_integrations/components/filters/ferpa_filter/__init__.py

@@ -0,0 +1,6 @@
+from haystack_integrations.components.filters.ferpa_filter.ferpa_metadata_filter import (
+    FERPADisclosureRecord,
+    FERPAMetadataFilter,
+)
+
+__all__ = ["FERPADisclosureRecord", "FERPAMetadataFilter"]

haystack_integrations/components/filters/ferpa_filter/ferpa_metadata_filter.py

@@ -0,0 +1,250 @@
+"""
+FERPAMetadataFilter — FERPA-compliant document filter for Haystack RAG pipelines.
+
+Enforces identity-scoped access control on retriever results before they reach
+the LLM context window. Complies with 34 CFR § 99.31(a)(1) (legitimate educational
+interest) and § 99.32 (record of disclosures).
+
+Two filtering layers applied in sequence:
+
+1. Identity pre-filter — removes documents whose student_id or institution_id
+   metadata does not match the authorized scope.
+2. Category authorization — removes documents whose category is not in the
+   authorized set (e.g., only ACADEMIC_RECORD, not DISCIPLINARY).
+
+Documents with no identity metadata are treated as shared knowledge-base content
+(course catalogues, policy handbooks) and pass through unchanged.
+
+Usage::
+
+    from haystack import Pipeline
+    from haystack.components.retrievers import InMemoryEmbeddingRetriever
+    from haystack_integrations.components.filters.ferpa_filter import FERPAMetadataFilter
+    from haystack.document_stores.in_memory import InMemoryDocumentStore
+
+    ferpa_filter = FERPAMetadataFilter(
+        student_id="stu_001",
+        institution_id="inst_abc",
+        authorized_categories=["academic_record", "financial_aid"],
+        requesting_user_id="advisor_007",
+    )
+
+    pipeline = Pipeline()
+    pipeline.add_component("retriever", InMemoryEmbeddingRetriever(doc_store))
+    pipeline.add_component("ferpa_filter", ferpa_filter)
+    pipeline.connect("retriever.documents", "ferpa_filter.documents")
+
+    result = pipeline.run({"retriever": {"query_embedding": query_emb}})
+    # result["ferpa_filter"]["documents"] — only stu_001's authorized records
+    # result["ferpa_filter"]["disclosure_record"] — 34 CFR § 99.32 audit entry
+
+Regulatory basis:
+    34 CFR § 99.31(a)(1) — legitimate educational interest
+    34 CFR § 99.32       — record of disclosures
+"""
+
+import logging
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+from typing import Any
+
+from haystack import Document, component, default_from_dict, default_to_dict
+
+logger = logging.getLogger(__name__)
+
+_SENTINEL = object()
+
+
+@dataclass
+class FERPADisclosureRecord:
+    """
+    Structured audit record of a FERPA disclosure event (34 CFR § 99.32).
+
+    Attributes:
+        student_id: Identifier of the student whose records were accessed.
+        institution_id: Identifier of the institution.
+        requesting_user_id: User or system that requested access.
+        disclosed_at: UTC timestamp of the disclosure.
+        total_retrieved: Documents returned by the retriever before filtering.
+        total_disclosed: Documents that passed FERPA filtering.
+        categories_disclosed: Record categories included in the result.
+        pipeline_context: Label identifying the pipeline or workflow context.
+    """
+
+    student_id: str
+    institution_id: str
+    requesting_user_id: str
+    disclosed_at: datetime = field(default_factory=lambda: datetime.now(timezone.utc))
+    total_retrieved: int = 0
+    total_disclosed: int = 0
+    categories_disclosed: list[str] = field(default_factory=list)
+    pipeline_context: str = "haystack_pipeline"
+
+    def to_log_entry(self) -> str:
+        return (
+            f"[FERPA_DISCLOSURE] student_id={self.student_id!r} "
+            f"institution_id={self.institution_id!r} "
+            f"requesting_user_id={self.requesting_user_id!r} "
+            f"disclosed_at={self.disclosed_at.isoformat()} "
+            f"total_retrieved={self.total_retrieved} "
+            f"total_disclosed={self.total_disclosed} "
+            f"categories_disclosed={self.categories_disclosed!r} "
+            f"pipeline_context={self.pipeline_context!r}"
+        )
+
+
+@component
+class FERPAMetadataFilter:
+    """
+    Haystack component that enforces FERPA identity-scope filtering on retrieved
+    documents before they enter the LLM context window.
+
+    Connects to any retriever output and emits only the documents that fall within
+    the authorized identity scope. Always emits a FERPADisclosureRecord for
+    downstream compliance logging (34 CFR § 99.32).
+
+    Two enforcement layers:
+
+    1. Identity pre-filter: student_id and institution_id in Document.meta must
+       match the authorized scope. Documents with neither field are shared content
+       and pass through unchanged.
+
+    2. Category authorization: when authorized_categories is non-empty, the
+       document's category field must be in the authorized set.
+
+    Args:
+        student_id: Authorized student identifier.
+        institution_id: Authorized institution identifier.
+        authorized_categories: Permitted record category strings.
+            Empty list means all categories are allowed.
+        requesting_user_id: Identifier of the requesting user (for audit log).
+        student_id_field: Meta key for student identifier. Default: "student_id".
+        institution_id_field: Meta key for institution identifier. Default: "institution_id".
+        category_field: Meta key for record category. Default: "category".
+        pipeline_context: Label for the audit record. Default: "haystack_pipeline".
+        raise_on_violation: When True, raise PermissionError on unauthorized docs.
+            When False (default), silently remove and emit WARNING.
+    """
+
+    def __init__(
+        self,
+        student_id: str,
+        institution_id: str,
+        authorized_categories: list[str] | None = None,
+        requesting_user_id: str = "unknown",
+        student_id_field: str = "student_id",
+        institution_id_field: str = "institution_id",
+        category_field: str = "category",
+        pipeline_context: str = "haystack_pipeline",
+        raise_on_violation: bool = False,
+    ) -> None:
+        self.student_id = student_id
+        self.institution_id = institution_id
+        self.authorized_categories = list(authorized_categories) if authorized_categories else []
+        self.requesting_user_id = requesting_user_id
+        self.student_id_field = student_id_field
+        self.institution_id_field = institution_id_field
+        self.category_field = category_field
+        self.pipeline_context = pipeline_context
+        self.raise_on_violation = raise_on_violation
+
+    def to_dict(self) -> dict[str, Any]:
+        return default_to_dict(
+            self,
+            student_id=self.student_id,
+            institution_id=self.institution_id,
+            authorized_categories=self.authorized_categories,
+            requesting_user_id=self.requesting_user_id,
+            student_id_field=self.student_id_field,
+            institution_id_field=self.institution_id_field,
+            category_field=self.category_field,
+            pipeline_context=self.pipeline_context,
+            raise_on_violation=self.raise_on_violation,
+        )
+
+    @classmethod
+    def from_dict(cls, data: dict[str, Any]) -> "FERPAMetadataFilter":
+        return default_from_dict(cls, data)
+
+    @component.output_types(documents=list[Document], disclosure_record=FERPADisclosureRecord)
+    def run(self, documents: list[Document]) -> dict[str, Any]:
+        """
+        Filter documents to the authorized identity scope.
+
+        Args:
+            documents: Documents from an upstream retriever.
+
+        Returns:
+            documents: Authorized documents only.
+            disclosure_record: FERPADisclosureRecord for compliance logging.
+
+        Raises:
+            PermissionError: Only when raise_on_violation=True and unauthorized
+                documents were detected.
+        """
+        total_retrieved = len(documents)
+        authorized: list[Document] = []
+
+        for doc in documents:
+            if self._is_authorized(doc):
+                authorized.append(doc)
+
+        removed = total_retrieved - len(authorized)
+
+        if removed > 0:
+            if self.raise_on_violation:
+                raise PermissionError(
+                    f"FERPA violation: {removed} unauthorized document(s) blocked for "
+                    f"student={self.student_id!r}, institution={self.institution_id!r}."
+                )
+            logger.warning(
+                "[FERPA_FILTER] Blocked %d unauthorized document(s) student_id=%r institution_id=%r",
+                removed, self.student_id, self.institution_id,
+            )
+
+        record = FERPADisclosureRecord(
+            student_id=self.student_id,
+            institution_id=self.institution_id,
+            requesting_user_id=self.requesting_user_id,
+            total_retrieved=total_retrieved,
+            total_disclosed=len(authorized),
+            categories_disclosed=self._extract_categories(authorized),
+            pipeline_context=self.pipeline_context,
+        )
+        logger.info(record.to_log_entry())
+        return {"documents": authorized, "disclosure_record": record}
+
+    @component.output_types(documents=list[Document], disclosure_record=FERPADisclosureRecord)
+    async def run_async(self, documents: list[Document]) -> dict[str, Any]:
+        """Async variant of run — filtering is CPU-bound, runs synchronously."""
+        return self.run(documents)
+
+    def _is_authorized(self, doc: Document) -> bool:
+        meta = doc.meta or {}
+        doc_student_id = meta.get(self.student_id_field, _SENTINEL)
+        doc_institution_id = meta.get(self.institution_id_field, _SENTINEL)
+
+        # Shared content (no identity metadata) passes through
+        if doc_student_id is _SENTINEL and doc_institution_id is _SENTINEL:
+            return True
+
+        if doc_student_id != self.student_id:
+            return False
+        if doc_institution_id is not _SENTINEL and doc_institution_id != self.institution_id:
+            return False
+
+        if self.authorized_categories:
+            doc_category = meta.get(self.category_field, _SENTINEL)
+            if doc_category is not _SENTINEL and doc_category not in self.authorized_categories:
+                return False
+
+        return True
+
+    def _extract_categories(self, documents: list[Document]) -> list[str]:
+        categories: set[str] = set()
+        for doc in documents:
+            meta = doc.meta or {}
+            cat = meta.get(self.category_field)
+            if cat is not None:
+                categories.add(str(cat))
+        return sorted(categories)