recallfsck 0.2.0rc2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- recallfsck-0.2.0rc2/.github/ISSUE_TEMPLATE/adapter.md +12 -0
- recallfsck-0.2.0rc2/.github/ISSUE_TEMPLATE/bug_report.md +15 -0
- recallfsck-0.2.0rc2/.github/ISSUE_TEMPLATE/feedback.yml +34 -0
- recallfsck-0.2.0rc2/.github/workflows/publish.yml +57 -0
- recallfsck-0.2.0rc2/.github/workflows/release.yml +21 -0
- recallfsck-0.2.0rc2/.github/workflows/tests.yml +78 -0
- recallfsck-0.2.0rc2/CHANGELOG.md +21 -0
- recallfsck-0.2.0rc2/CODE_OF_CONDUCT.md +11 -0
- recallfsck-0.2.0rc2/CONTRIBUTING.md +20 -0
- recallfsck-0.2.0rc2/LICENSE +21 -0
- recallfsck-0.2.0rc2/MANIFEST.in +7 -0
- recallfsck-0.2.0rc2/PKG-INFO +178 -0
- recallfsck-0.2.0rc2/README.md +147 -0
- recallfsck-0.2.0rc2/SECURITY.md +15 -0
- recallfsck-0.2.0rc2/VALIDATION.md +86 -0
- recallfsck-0.2.0rc2/compose.yaml +22 -0
- recallfsck-0.2.0rc2/docs/adapters.md +44 -0
- recallfsck-0.2.0rc2/docs/architecture.md +27 -0
- recallfsck-0.2.0rc2/docs/feedback-study.md +23 -0
- recallfsck-0.2.0rc2/docs/launch.md +31 -0
- recallfsck-0.2.0rc2/docs/release-notes.md +34 -0
- recallfsck-0.2.0rc2/docs/releasing.md +29 -0
- recallfsck-0.2.0rc2/docs/roadmap.md +43 -0
- recallfsck-0.2.0rc2/docs/usage.md +75 -0
- recallfsck-0.2.0rc2/examples/__init__.py +1 -0
- recallfsck-0.2.0rc2/examples/adapter_template.py +34 -0
- recallfsck-0.2.0rc2/examples/http_application.py +5 -0
- recallfsck-0.2.0rc2/examples/init-vector.sql +1 -0
- recallfsck-0.2.0rc2/examples/resurrection.json +225 -0
- recallfsck-0.2.0rc2/pyproject.toml +42 -0
- recallfsck-0.2.0rc2/recallfsck/__init__.py +1 -0
- recallfsck-0.2.0rc2/recallfsck/__main__.py +3 -0
- recallfsck-0.2.0rc2/recallfsck/backends.py +66 -0
- recallfsck-0.2.0rc2/recallfsck/cli.py +109 -0
- recallfsck-0.2.0rc2/recallfsck/demo_http.py +124 -0
- recallfsck-0.2.0rc2/recallfsck/discovery.py +50 -0
- recallfsck-0.2.0rc2/recallfsck/engine.py +112 -0
- recallfsck-0.2.0rc2/recallfsck/integrations/__init__.py +1 -0
- recallfsck-0.2.0rc2/recallfsck/integrations/langchain_chroma.py +58 -0
- recallfsck-0.2.0rc2/recallfsck/product_cli.py +191 -0
- recallfsck-0.2.0rc2/recallfsck/pytest_plugin.py +28 -0
- recallfsck-0.2.0rc2/recallfsck/reporting.py +57 -0
- recallfsck-0.2.0rc2/recallfsck/stores.py +252 -0
- recallfsck-0.2.0rc2/recallfsck/suite.py +353 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/PKG-INFO +178 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/SOURCES.txt +52 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/dependency_links.txt +1 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/entry_points.txt +5 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/requires.txt +24 -0
- recallfsck-0.2.0rc2/recallfsck.egg-info/top_level.txt +1 -0
- recallfsck-0.2.0rc2/recallfsck.yaml +17 -0
- recallfsck-0.2.0rc2/setup.cfg +4 -0
- recallfsck-0.2.0rc2/tests/test_engine.py +111 -0
- recallfsck-0.2.0rc2/tests/test_product.py +355 -0
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Application or backend adapter
|
|
3
|
+
about: Propose a real integration with observable lifecycle semantics
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Application/backend:
|
|
7
|
+
Acknowledgement and consistency guarantees:
|
|
8
|
+
Disposable namespace strategy:
|
|
9
|
+
Retrieval paths:
|
|
10
|
+
Controlled pause/resume hooks:
|
|
11
|
+
Cleanup behavior:
|
|
12
|
+
Reproducible synthetic failure:
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Integrity failure or tool bug
|
|
3
|
+
about: Share reproducible evidence using synthetic data
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
Backend and version:
|
|
7
|
+
Python and RecallFSCK version:
|
|
8
|
+
Application adapter:
|
|
9
|
+
Command and timing settings (remove credentials):
|
|
10
|
+
Expected state:
|
|
11
|
+
Observed state:
|
|
12
|
+
Synthetic reproduction JSON:
|
|
13
|
+
Does replay reproduce it?
|
|
14
|
+
|
|
15
|
+
Do not include private source documents, credentials or security exploit details.
|
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
name: Independent installation feedback
|
|
2
|
+
description: Record your own attempt to install and run RecallFSCK
|
|
3
|
+
title: 'Installation feedback: '
|
|
4
|
+
body:
|
|
5
|
+
- type: input
|
|
6
|
+
id: platform
|
|
7
|
+
attributes:
|
|
8
|
+
label: Operating system and Python version
|
|
9
|
+
validations:
|
|
10
|
+
required: true
|
|
11
|
+
- type: textarea
|
|
12
|
+
id: steps
|
|
13
|
+
attributes:
|
|
14
|
+
label: Commands you actually ran
|
|
15
|
+
description: Remove credentials and private documents.
|
|
16
|
+
validations:
|
|
17
|
+
required: true
|
|
18
|
+
- type: dropdown
|
|
19
|
+
id: outcome
|
|
20
|
+
attributes:
|
|
21
|
+
label: Outcome
|
|
22
|
+
options:
|
|
23
|
+
- Demo reproduced the expected failures and passes
|
|
24
|
+
- Installation failed
|
|
25
|
+
- Demo produced unexpected results
|
|
26
|
+
validations:
|
|
27
|
+
required: true
|
|
28
|
+
- type: textarea
|
|
29
|
+
id: evidence
|
|
30
|
+
attributes:
|
|
31
|
+
label: Observed results and confusing instructions
|
|
32
|
+
description: Include synthetic evidence or error output, not guesses.
|
|
33
|
+
validations:
|
|
34
|
+
required: true
|
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
name: Publish Python package
|
|
2
|
+
on:
|
|
3
|
+
workflow_dispatch:
|
|
4
|
+
inputs:
|
|
5
|
+
destination:
|
|
6
|
+
description: Package index
|
|
7
|
+
required: true
|
|
8
|
+
default: testpypi
|
|
9
|
+
type: choice
|
|
10
|
+
options: [testpypi, pypi]
|
|
11
|
+
permissions:
|
|
12
|
+
contents: read
|
|
13
|
+
jobs:
|
|
14
|
+
checks:
|
|
15
|
+
uses: ./.github/workflows/tests.yml
|
|
16
|
+
build:
|
|
17
|
+
needs: checks
|
|
18
|
+
runs-on: ubuntu-latest
|
|
19
|
+
steps:
|
|
20
|
+
- uses: actions/checkout@v4
|
|
21
|
+
- uses: actions/setup-python@v5
|
|
22
|
+
with:
|
|
23
|
+
python-version: '3.12'
|
|
24
|
+
- name: Verify tagged package version
|
|
25
|
+
run: |
|
|
26
|
+
python - <<'PY'
|
|
27
|
+
import os, tomllib
|
|
28
|
+
version = tomllib.load(open('pyproject.toml', 'rb'))['project']['version']
|
|
29
|
+
if os.environ['GITHUB_REF'] != 'refs/tags/v' + version:
|
|
30
|
+
raise SystemExit('Dispatch this workflow from the matching release tag')
|
|
31
|
+
PY
|
|
32
|
+
- run: python -m pip install build twine
|
|
33
|
+
- run: python -m build
|
|
34
|
+
- run: python -m twine check dist/*
|
|
35
|
+
- uses: actions/upload-artifact@v4
|
|
36
|
+
with:
|
|
37
|
+
name: publish-distributions
|
|
38
|
+
path: dist/
|
|
39
|
+
publish:
|
|
40
|
+
needs: build
|
|
41
|
+
runs-on: ubuntu-latest
|
|
42
|
+
environment: ${{ inputs.destination }}
|
|
43
|
+
permissions:
|
|
44
|
+
id-token: write
|
|
45
|
+
steps:
|
|
46
|
+
- uses: actions/download-artifact@v4
|
|
47
|
+
with:
|
|
48
|
+
name: publish-distributions
|
|
49
|
+
path: dist/
|
|
50
|
+
- name: Upload to TestPyPI
|
|
51
|
+
if: inputs.destination == 'testpypi'
|
|
52
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
53
|
+
with:
|
|
54
|
+
repository-url: https://test.pypi.org/legacy/
|
|
55
|
+
- name: Upload to PyPI
|
|
56
|
+
if: inputs.destination == 'pypi'
|
|
57
|
+
uses: pypa/gh-action-pypi-publish@release/v1
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
name: Build release candidate
|
|
2
|
+
on:
|
|
3
|
+
workflow_dispatch:
|
|
4
|
+
permissions:
|
|
5
|
+
contents: read
|
|
6
|
+
jobs:
|
|
7
|
+
build:
|
|
8
|
+
runs-on: ubuntu-latest
|
|
9
|
+
steps:
|
|
10
|
+
- uses: actions/checkout@v4
|
|
11
|
+
- uses: actions/setup-python@v5
|
|
12
|
+
with:
|
|
13
|
+
python-version: '3.12'
|
|
14
|
+
- run: python -m pip install -e '.[test]'
|
|
15
|
+
- run: python -m pytest -q
|
|
16
|
+
- run: python -m build
|
|
17
|
+
- run: python -m twine check dist/*
|
|
18
|
+
- uses: actions/upload-artifact@v4
|
|
19
|
+
with:
|
|
20
|
+
name: recallfsck-release-candidate
|
|
21
|
+
path: dist/
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
name: tests
|
|
2
|
+
on:
|
|
3
|
+
push:
|
|
4
|
+
pull_request:
|
|
5
|
+
workflow_call:
|
|
6
|
+
permissions:
|
|
7
|
+
contents: read
|
|
8
|
+
jobs:
|
|
9
|
+
unit:
|
|
10
|
+
runs-on: ubuntu-latest
|
|
11
|
+
strategy:
|
|
12
|
+
matrix:
|
|
13
|
+
python: ['3.11', '3.12', '3.13']
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v4
|
|
16
|
+
- uses: actions/setup-python@v5
|
|
17
|
+
with:
|
|
18
|
+
python-version: '${{ matrix.python }}'
|
|
19
|
+
- run: python -m pip install -e '.[test]'
|
|
20
|
+
- run: ruff check .
|
|
21
|
+
- run: ruff format --check recallfsck tests examples
|
|
22
|
+
- run: python -m pytest -q --junitxml=report/tests.xml
|
|
23
|
+
- run: recallfsck demo --output report/sqlite
|
|
24
|
+
- run: recallfsck demo --app recallfsck.demo_http:create_application --output report/http
|
|
25
|
+
- uses: actions/upload-artifact@v4
|
|
26
|
+
if: always()
|
|
27
|
+
with:
|
|
28
|
+
name: unit-evidence-${{ matrix.python }}
|
|
29
|
+
path: report/
|
|
30
|
+
databases:
|
|
31
|
+
runs-on: ubuntu-latest
|
|
32
|
+
services:
|
|
33
|
+
qdrant:
|
|
34
|
+
image: qdrant/qdrant:v1.13.4
|
|
35
|
+
ports: ['6333:6333']
|
|
36
|
+
postgres:
|
|
37
|
+
image: pgvector/pgvector:pg16
|
|
38
|
+
env:
|
|
39
|
+
POSTGRES_PASSWORD: recallfsck-local-only
|
|
40
|
+
POSTGRES_DB: recallfsck
|
|
41
|
+
ports: ['5432:5432']
|
|
42
|
+
options: >-
|
|
43
|
+
--health-cmd "pg_isready -U postgres -d recallfsck"
|
|
44
|
+
--health-interval 2s --health-timeout 2s --health-retries 30
|
|
45
|
+
env:
|
|
46
|
+
RECALLFSCK_QDRANT_URL: http://localhost:6333
|
|
47
|
+
RECALLFSCK_PGVECTOR_URL: postgresql://postgres:recallfsck-local-only@localhost:5432/recallfsck
|
|
48
|
+
steps:
|
|
49
|
+
- uses: actions/checkout@v4
|
|
50
|
+
- uses: actions/setup-python@v5
|
|
51
|
+
with:
|
|
52
|
+
python-version: '3.12'
|
|
53
|
+
- run: python -m pip install -e '.[all,test]'
|
|
54
|
+
- name: Prepare disposable pgvector database and await Qdrant
|
|
55
|
+
run: |
|
|
56
|
+
python - <<'PY'
|
|
57
|
+
import os, time, urllib.request
|
|
58
|
+
import psycopg
|
|
59
|
+
with psycopg.connect(os.environ['RECALLFSCK_PGVECTOR_URL'], autocommit=True) as conn:
|
|
60
|
+
conn.execute('CREATE EXTENSION IF NOT EXISTS vector')
|
|
61
|
+
for attempt in range(30):
|
|
62
|
+
try:
|
|
63
|
+
urllib.request.urlopen(os.environ['RECALLFSCK_QDRANT_URL'] + '/readyz', timeout=2).close()
|
|
64
|
+
break
|
|
65
|
+
except OSError:
|
|
66
|
+
if attempt == 29:
|
|
67
|
+
raise
|
|
68
|
+
time.sleep(1)
|
|
69
|
+
PY
|
|
70
|
+
- run: python -m pytest -q --junitxml=report/tests.xml
|
|
71
|
+
- run: recallfsck demo --backend qdrant --output report/qdrant
|
|
72
|
+
- run: recallfsck demo --backend chroma --output report/chroma
|
|
73
|
+
- run: recallfsck demo --backend pgvector --url "$RECALLFSCK_PGVECTOR_URL" --output report/pgvector
|
|
74
|
+
- uses: actions/upload-artifact@v4
|
|
75
|
+
if: always()
|
|
76
|
+
with:
|
|
77
|
+
name: database-evidence
|
|
78
|
+
path: report/
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## 0.2.0rc2
|
|
4
|
+
|
|
5
|
+
- Fix hosted CI to use the installed HTTP application module.
|
|
6
|
+
- Verify Python 3.11, 3.12 and 3.13 plus actual database integrations on GitHub Actions.
|
|
7
|
+
- Prepare the first PyPI upload through the registered trusted publisher.
|
|
8
|
+
|
|
9
|
+
## 0.2.0rc1
|
|
10
|
+
|
|
11
|
+
- Add Qdrant, Chroma and pgvector document/vector store adapters.
|
|
12
|
+
- Add ten lifecycle scenarios with full-record and filtered retrieval assertions.
|
|
13
|
+
- Add finite convergence deadlines and post-convergence monitoring.
|
|
14
|
+
- Add YAML init/configuration, schema-2 replay and failure minimization.
|
|
15
|
+
- Add Hypothesis search/shrinking, pytest integration and HTTP application demo.
|
|
16
|
+
- Add JSON, HTML and JUnit evidence, service-error reporting and release tooling.
|
|
17
|
+
- Retain schema-1 replay compatibility.
|
|
18
|
+
|
|
19
|
+
## 0.1.0
|
|
20
|
+
|
|
21
|
+
- Initial deterministic create/delete/resurrection red/green experiment.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
# Code of conduct
|
|
2
|
+
|
|
3
|
+
Treat contributors with respect. Technical disagreement is welcome; harassment,
|
|
4
|
+
threats, discrimination, doxxing and repeated unwanted contact are not.
|
|
5
|
+
Keep discussion focused on evidence and the project. Respect personal boundaries
|
|
6
|
+
and avoid sharing private information or credentials.
|
|
7
|
+
|
|
8
|
+
Maintainers may edit or remove harmful content and restrict participation, using
|
|
9
|
+
proportionate action and considering context. Report concerns privately to the
|
|
10
|
+
repository owner. A public contact address must be established before launch;
|
|
11
|
+
do not post sensitive reports publicly while that channel is being set up.
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Use Python 3.11+ in a virtual environment. Install with
|
|
4
|
+
`python -m pip install -e '.[all,test]'`, then run `ruff check .` and
|
|
5
|
+
`python -m pytest -q`. Start the disposable services with
|
|
6
|
+
`docker compose up -d --wait` for integration coverage:
|
|
7
|
+
|
|
8
|
+
```sh
|
|
9
|
+
RECALLFSCK_QDRANT_URL=http://127.0.0.1:16333 \
|
|
10
|
+
RECALLFSCK_PGVECTOR_URL=postgresql://postgres:recallfsck-local-only@127.0.0.1:15432/recallfsck \
|
|
11
|
+
python -m pytest -q
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
Run affected adapter demos and include replayable synthetic failure evidence.
|
|
15
|
+
Keep changes scoped, preserve old artifact replay, and document the exact scope
|
|
16
|
+
of every invariant. Add a deliberately faulty fixture when adding an invariant;
|
|
17
|
+
checking only a passing implementation cannot show the checker detects a defect.
|
|
18
|
+
Do not expose private documents, tokens or connection credentials in reports.
|
|
19
|
+
|
|
20
|
+
See CODE_OF_CONDUCT.md, SECURITY.md and docs/adapters.md before contributing.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 RecallFSCK contributors
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
include README.md LICENSE CHANGELOG.md CONTRIBUTING.md CODE_OF_CONDUCT.md SECURITY.md VALIDATION.md compose.yaml
|
|
2
|
+
include recallfsck.yaml
|
|
3
|
+
recursive-include docs *.md
|
|
4
|
+
recursive-include examples *.py *.sql
|
|
5
|
+
recursive-include examples *.json
|
|
6
|
+
recursive-include tests *.py
|
|
7
|
+
recursive-include .github *.yml *.md
|
|
@@ -0,0 +1,178 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: recallfsck
|
|
3
|
+
Version: 0.2.0rc2
|
|
4
|
+
Summary: Deterministic lifecycle integrity experiments for AI knowledge indexes
|
|
5
|
+
License-Expression: MIT
|
|
6
|
+
Project-URL: Repository, https://github.com/sanketsinghai/recallfsck
|
|
7
|
+
Project-URL: Issues, https://github.com/sanketsinghai/recallfsck/issues
|
|
8
|
+
Requires-Python: >=3.11
|
|
9
|
+
Description-Content-Type: text/markdown
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Requires-Dist: PyYAML<7,>=6
|
|
12
|
+
Provides-Extra: chroma
|
|
13
|
+
Requires-Dist: chromadb<2,>=1; extra == "chroma"
|
|
14
|
+
Provides-Extra: langchain
|
|
15
|
+
Requires-Dist: langchain-chroma<2,>=1; extra == "langchain"
|
|
16
|
+
Provides-Extra: pgvector
|
|
17
|
+
Requires-Dist: psycopg[binary]<4,>=3.2; extra == "pgvector"
|
|
18
|
+
Provides-Extra: test
|
|
19
|
+
Requires-Dist: pytest<10,>=8; extra == "test"
|
|
20
|
+
Requires-Dist: hypothesis<7,>=6; extra == "test"
|
|
21
|
+
Requires-Dist: build<2,>=1; extra == "test"
|
|
22
|
+
Requires-Dist: twine<7,>=6; extra == "test"
|
|
23
|
+
Requires-Dist: ruff<1,>=0.11; extra == "test"
|
|
24
|
+
Provides-Extra: all
|
|
25
|
+
Requires-Dist: chromadb<2,>=1; extra == "all"
|
|
26
|
+
Requires-Dist: langchain-chroma<2,>=1; extra == "all"
|
|
27
|
+
Requires-Dist: psycopg[binary]<4,>=3.2; extra == "all"
|
|
28
|
+
Requires-Dist: pytest<10,>=8; extra == "all"
|
|
29
|
+
Requires-Dist: hypothesis<7,>=6; extra == "all"
|
|
30
|
+
Dynamic: license-file
|
|
31
|
+
|
|
32
|
+
# RecallFSCK
|
|
33
|
+
|
|
34
|
+
A delete returns success. An older ingestion worker finishes afterward and puts
|
|
35
|
+
that document back into search. RecallFSCK runs that schedule, compares actual
|
|
36
|
+
retrieval with an independent reference model, and saves a replayable failure.
|
|
37
|
+
|
|
38
|
+
## One-command demo
|
|
39
|
+
|
|
40
|
+
From this repository, Python 3.11+:
|
|
41
|
+
|
|
42
|
+
```sh
|
|
43
|
+
python -m pip install -e '.[test]'
|
|
44
|
+
recallfsck demo
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
The demo runs ten scenarios against two applications. The deliberately buggy
|
|
48
|
+
worker fails three races: delete/index, replacement/index, and migration/index.
|
|
49
|
+
The corrected worker passes all ten. Demo exits 0 only when that exact pattern is
|
|
50
|
+
observed. The evidence still labels the buggy cases FAIL.
|
|
51
|
+
|
|
52
|
+
Open `report/index.html`; each case also has JSON evidence. `report/junit.xml`
|
|
53
|
+
records failures for CI. These are executed results, not illustrative screenshots.
|
|
54
|
+
|
|
55
|
+
```sh
|
|
56
|
+
recallfsck init --adapter sqlite
|
|
57
|
+
recallfsck test --config recallfsck.yaml
|
|
58
|
+
recallfsck replay report/buggy-delete-during-index.json
|
|
59
|
+
recallfsck minimize report/buggy-delete-during-index.json --output report/minimal
|
|
60
|
+
recallfsck explore --worker buggy --examples 100 --output report/discovery
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Replay, minimize and explore exit 1 when they reproduce an integrity failure.
|
|
64
|
+
Exit 2 means an input, service, dependency or execution error. A missing database
|
|
65
|
+
never becomes PASS. `explore` uses Hypothesis to search actual executions and
|
|
66
|
+
shrink failures; `minimize` produces a 1-minimal valid deletion subsequence.
|
|
67
|
+
|
|
68
|
+
## What is checked
|
|
69
|
+
|
|
70
|
+
| Invariant | Observed evidence |
|
|
71
|
+
| --- | --- |
|
|
72
|
+
| Create | Acknowledged source appears in application queries and index scan |
|
|
73
|
+
| Replacement | Only the expected revision/content remains |
|
|
74
|
+
| Metadata | Stored and retrieved revision matches expected metadata |
|
|
75
|
+
| Delete | Document remains absent throughout the configured monitor window |
|
|
76
|
+
| Idempotency | Repeat upsert/delete does not introduce duplicate records |
|
|
77
|
+
| Isolation | Alpha, beta, and outsider queries expose only authorized records |
|
|
78
|
+
| Concurrency | Paused stale workers cannot overwrite deletion, revision or model |
|
|
79
|
+
| Model migration | All live records move to the synthetic v2 vector configuration and remain retrievable |
|
|
80
|
+
|
|
81
|
+
The generation check and final commit share the deletion lock in the corrected
|
|
82
|
+
fixture. The scheduler coordinates with events rather than hoping sleeps cause a
|
|
83
|
+
race. Passing results cover finite observation windows, not indefinite absence.
|
|
84
|
+
|
|
85
|
+
## Integrations
|
|
86
|
+
|
|
87
|
+
| Adapter | Installation | Runtime |
|
|
88
|
+
| --- | --- | --- |
|
|
89
|
+
| SQLite | Base package | Disposable in-memory SQL; no vector-search claim |
|
|
90
|
+
| Qdrant | Base package | Disposable Qdrant REST collection |
|
|
91
|
+
| Chroma | `pip install -e '.[chroma]'` | Embedded ephemeral Chroma vector collection |
|
|
92
|
+
| pgvector | `pip install -e '.[pgvector]'` | Disposable table on a PostgreSQL server with vector enabled |
|
|
93
|
+
|
|
94
|
+
```sh
|
|
95
|
+
# Start isolated databases (Docker Compose):
|
|
96
|
+
docker compose up -d --wait
|
|
97
|
+
recallfsck demo --backend qdrant --url http://127.0.0.1:16333 --output report/qdrant
|
|
98
|
+
recallfsck demo --backend chroma --output report/chroma
|
|
99
|
+
RECALLFSCK_URL=postgresql://postgres:recallfsck-local-only@127.0.0.1:15432/recallfsck \
|
|
100
|
+
recallfsck demo --backend pgvector --output report/pgvector
|
|
101
|
+
docker compose down -v
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
For source integration, implement the [application adapter contract](docs/adapters.md).
|
|
105
|
+
A runnable [HTTP application example](examples/http_application.py) goes through
|
|
106
|
+
real local HTTP calls while using the runner's disposable index:
|
|
107
|
+
|
|
108
|
+
```sh
|
|
109
|
+
recallfsck demo --app recallfsck.demo_http:create_application --output report/http
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
A real LangChain/Chroma integration is available below. Dify and other applications
|
|
113
|
+
need their own adapters. Backend conformance and bundled fixtures do not prove an
|
|
114
|
+
unrelated production application's correctness.
|
|
115
|
+
|
|
116
|
+
## pytest
|
|
117
|
+
|
|
118
|
+
```python
|
|
119
|
+
def test_delete_integrity(recallfsck_runner):
|
|
120
|
+
result = recallfsck_runner("delete-during-index", worker="fixed")
|
|
121
|
+
assert result["status"] == "PASS", result["first_divergence"]
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Install the `test` extra for pytest and Hypothesis. Importing the plugin does not
|
|
125
|
+
create a database or automatically add tests to your project.
|
|
126
|
+
|
|
127
|
+
## Development and release
|
|
128
|
+
|
|
129
|
+
```sh
|
|
130
|
+
python -m pip install -e '.[all,test]'
|
|
131
|
+
python -m pytest -q
|
|
132
|
+
ruff check .
|
|
133
|
+
python -m build
|
|
134
|
+
python -m twine check dist/*
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
See [usage](docs/usage.md), [architecture](docs/architecture.md),
|
|
138
|
+
[roadmap status](docs/roadmap.md), [validation](VALIDATION.md), and
|
|
139
|
+
[release instructions](docs/releasing.md).
|
|
140
|
+
|
|
141
|
+
## Limits and safety
|
|
142
|
+
|
|
143
|
+
Run mutating tests only on disposable services and synthetic data. Each adapter
|
|
144
|
+
creates and cleans up its own randomly named collection/table. Interrupted runs
|
|
145
|
+
may leave owned objects behind; existing namespaces cannot be selected.
|
|
146
|
+
|
|
147
|
+
The vector tests use known two-dimensional synthetic embeddings. Migration checks
|
|
148
|
+
cover same-dimensional configuration changes, not a real model's semantic quality
|
|
149
|
+
or cross-dimension rebuild. Queries are top-100 with full index scans as a separate
|
|
150
|
+
check; generated schedules are capped at 200 operations and the bundled corpus
|
|
151
|
+
fits below the query limit. Unknown applications need explicit integration and
|
|
152
|
+
scheduling hooks. Network calls have adapter timeouts; consistency settings are
|
|
153
|
+
not a hard wall-clock cancellation limit for blocking drivers.
|
|
154
|
+
|
|
155
|
+
Logical source deletion, retrieval absence, and physical erasure are distinct.
|
|
156
|
+
This tool does not prove disk/backup erasure, untested query paths, distributed
|
|
157
|
+
linearizability, production authorization policy or arbitrary future absence.
|
|
158
|
+
There is no production scanning mode. Saved reports contain test document text;
|
|
159
|
+
use synthetic inputs. No hosted service or API keys are required for the demos.
|
|
160
|
+
|
|
161
|
+
Repository: https://github.com/sanketsinghai/recallfsck . The PyPI name returned
|
|
162
|
+
HTTP 404 when checked on October 26, 2026; this does not reserve the name or
|
|
163
|
+
establish trademark clearance.
|
|
164
|
+
MIT license; optional dependencies keep their respective licenses.
|
|
165
|
+
|
|
166
|
+
## Real LangChain integration
|
|
167
|
+
|
|
168
|
+
```sh
|
|
169
|
+
python -m pip install -e '.[langchain,test]'
|
|
170
|
+
recallfsck demo --backend chroma \
|
|
171
|
+
--app recallfsck.integrations.langchain_chroma:create_application \
|
|
172
|
+
--output report/langchain
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
In this integration, writes, deletion, embedding and retrieval call the actual
|
|
176
|
+
LangChain Chroma vector-store APIs. The controlled ingestion scheduling and the
|
|
177
|
+
injected bugs belong to RecallFSCK's demonstration application. No upstream
|
|
178
|
+
LangChain defect or production application correctness is claimed.
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# RecallFSCK
|
|
2
|
+
|
|
3
|
+
A delete returns success. An older ingestion worker finishes afterward and puts
|
|
4
|
+
that document back into search. RecallFSCK runs that schedule, compares actual
|
|
5
|
+
retrieval with an independent reference model, and saves a replayable failure.
|
|
6
|
+
|
|
7
|
+
## One-command demo
|
|
8
|
+
|
|
9
|
+
From this repository, Python 3.11+:
|
|
10
|
+
|
|
11
|
+
```sh
|
|
12
|
+
python -m pip install -e '.[test]'
|
|
13
|
+
recallfsck demo
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The demo runs ten scenarios against two applications. The deliberately buggy
|
|
17
|
+
worker fails three races: delete/index, replacement/index, and migration/index.
|
|
18
|
+
The corrected worker passes all ten. Demo exits 0 only when that exact pattern is
|
|
19
|
+
observed. The evidence still labels the buggy cases FAIL.
|
|
20
|
+
|
|
21
|
+
Open `report/index.html`; each case also has JSON evidence. `report/junit.xml`
|
|
22
|
+
records failures for CI. These are executed results, not illustrative screenshots.
|
|
23
|
+
|
|
24
|
+
```sh
|
|
25
|
+
recallfsck init --adapter sqlite
|
|
26
|
+
recallfsck test --config recallfsck.yaml
|
|
27
|
+
recallfsck replay report/buggy-delete-during-index.json
|
|
28
|
+
recallfsck minimize report/buggy-delete-during-index.json --output report/minimal
|
|
29
|
+
recallfsck explore --worker buggy --examples 100 --output report/discovery
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
Replay, minimize and explore exit 1 when they reproduce an integrity failure.
|
|
33
|
+
Exit 2 means an input, service, dependency or execution error. A missing database
|
|
34
|
+
never becomes PASS. `explore` uses Hypothesis to search actual executions and
|
|
35
|
+
shrink failures; `minimize` produces a 1-minimal valid deletion subsequence.
|
|
36
|
+
|
|
37
|
+
## What is checked
|
|
38
|
+
|
|
39
|
+
| Invariant | Observed evidence |
|
|
40
|
+
| --- | --- |
|
|
41
|
+
| Create | Acknowledged source appears in application queries and index scan |
|
|
42
|
+
| Replacement | Only the expected revision/content remains |
|
|
43
|
+
| Metadata | Stored and retrieved revision matches expected metadata |
|
|
44
|
+
| Delete | Document remains absent throughout the configured monitor window |
|
|
45
|
+
| Idempotency | Repeat upsert/delete does not introduce duplicate records |
|
|
46
|
+
| Isolation | Alpha, beta, and outsider queries expose only authorized records |
|
|
47
|
+
| Concurrency | Paused stale workers cannot overwrite deletion, revision or model |
|
|
48
|
+
| Model migration | All live records move to the synthetic v2 vector configuration and remain retrievable |
|
|
49
|
+
|
|
50
|
+
The generation check and final commit share the deletion lock in the corrected
|
|
51
|
+
fixture. The scheduler coordinates with events rather than hoping sleeps cause a
|
|
52
|
+
race. Passing results cover finite observation windows, not indefinite absence.
|
|
53
|
+
|
|
54
|
+
## Integrations
|
|
55
|
+
|
|
56
|
+
| Adapter | Installation | Runtime |
|
|
57
|
+
| --- | --- | --- |
|
|
58
|
+
| SQLite | Base package | Disposable in-memory SQL; no vector-search claim |
|
|
59
|
+
| Qdrant | Base package | Disposable Qdrant REST collection |
|
|
60
|
+
| Chroma | `pip install -e '.[chroma]'` | Embedded ephemeral Chroma vector collection |
|
|
61
|
+
| pgvector | `pip install -e '.[pgvector]'` | Disposable table on a PostgreSQL server with vector enabled |
|
|
62
|
+
|
|
63
|
+
```sh
|
|
64
|
+
# Start isolated databases (Docker Compose):
|
|
65
|
+
docker compose up -d --wait
|
|
66
|
+
recallfsck demo --backend qdrant --url http://127.0.0.1:16333 --output report/qdrant
|
|
67
|
+
recallfsck demo --backend chroma --output report/chroma
|
|
68
|
+
RECALLFSCK_URL=postgresql://postgres:recallfsck-local-only@127.0.0.1:15432/recallfsck \
|
|
69
|
+
recallfsck demo --backend pgvector --output report/pgvector
|
|
70
|
+
docker compose down -v
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
For source integration, implement the [application adapter contract](docs/adapters.md).
|
|
74
|
+
A runnable [HTTP application example](examples/http_application.py) goes through
|
|
75
|
+
real local HTTP calls while using the runner's disposable index:
|
|
76
|
+
|
|
77
|
+
```sh
|
|
78
|
+
recallfsck demo --app recallfsck.demo_http:create_application --output report/http
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
A real LangChain/Chroma integration is available below. Dify and other applications
|
|
82
|
+
need their own adapters. Backend conformance and bundled fixtures do not prove an
|
|
83
|
+
unrelated production application's correctness.
|
|
84
|
+
|
|
85
|
+
## pytest
|
|
86
|
+
|
|
87
|
+
```python
|
|
88
|
+
def test_delete_integrity(recallfsck_runner):
|
|
89
|
+
result = recallfsck_runner("delete-during-index", worker="fixed")
|
|
90
|
+
assert result["status"] == "PASS", result["first_divergence"]
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Install the `test` extra for pytest and Hypothesis. Importing the plugin does not
|
|
94
|
+
create a database or automatically add tests to your project.
|
|
95
|
+
|
|
96
|
+
## Development and release
|
|
97
|
+
|
|
98
|
+
```sh
|
|
99
|
+
python -m pip install -e '.[all,test]'
|
|
100
|
+
python -m pytest -q
|
|
101
|
+
ruff check .
|
|
102
|
+
python -m build
|
|
103
|
+
python -m twine check dist/*
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
See [usage](docs/usage.md), [architecture](docs/architecture.md),
|
|
107
|
+
[roadmap status](docs/roadmap.md), [validation](VALIDATION.md), and
|
|
108
|
+
[release instructions](docs/releasing.md).
|
|
109
|
+
|
|
110
|
+
## Limits and safety
|
|
111
|
+
|
|
112
|
+
Run mutating tests only on disposable services and synthetic data. Each adapter
|
|
113
|
+
creates and cleans up its own randomly named collection/table. Interrupted runs
|
|
114
|
+
may leave owned objects behind; existing namespaces cannot be selected.
|
|
115
|
+
|
|
116
|
+
The vector tests use known two-dimensional synthetic embeddings. Migration checks
|
|
117
|
+
cover same-dimensional configuration changes, not a real model's semantic quality
|
|
118
|
+
or cross-dimension rebuild. Queries are top-100 with full index scans as a separate
|
|
119
|
+
check; generated schedules are capped at 200 operations and the bundled corpus
|
|
120
|
+
fits below the query limit. Unknown applications need explicit integration and
|
|
121
|
+
scheduling hooks. Network calls have adapter timeouts; consistency settings are
|
|
122
|
+
not a hard wall-clock cancellation limit for blocking drivers.
|
|
123
|
+
|
|
124
|
+
Logical source deletion, retrieval absence, and physical erasure are distinct.
|
|
125
|
+
This tool does not prove disk/backup erasure, untested query paths, distributed
|
|
126
|
+
linearizability, production authorization policy or arbitrary future absence.
|
|
127
|
+
There is no production scanning mode. Saved reports contain test document text;
|
|
128
|
+
use synthetic inputs. No hosted service or API keys are required for the demos.
|
|
129
|
+
|
|
130
|
+
Repository: https://github.com/sanketsinghai/recallfsck . The PyPI name returned
|
|
131
|
+
HTTP 404 when checked on October 26, 2026; this does not reserve the name or
|
|
132
|
+
establish trademark clearance.
|
|
133
|
+
MIT license; optional dependencies keep their respective licenses.
|
|
134
|
+
|
|
135
|
+
## Real LangChain integration
|
|
136
|
+
|
|
137
|
+
```sh
|
|
138
|
+
python -m pip install -e '.[langchain,test]'
|
|
139
|
+
recallfsck demo --backend chroma \
|
|
140
|
+
--app recallfsck.integrations.langchain_chroma:create_application \
|
|
141
|
+
--output report/langchain
|
|
142
|
+
```
|
|
143
|
+
|
|
144
|
+
In this integration, writes, deletion, embedding and retrieval call the actual
|
|
145
|
+
LangChain Chroma vector-store APIs. The controlled ingestion scheduling and the
|
|
146
|
+
injected bugs belong to RecallFSCK's demonstration application. No upstream
|
|
147
|
+
LangChain defect or production application correctness is claimed.
|
|
@@ -0,0 +1,15 @@
|
|
|
1
|
+
# Security
|
|
2
|
+
|
|
3
|
+
Report vulnerabilities privately through
|
|
4
|
+
https://github.com/sanketsinghai/recallfsck/security/advisories/new .
|
|
5
|
+
Private vulnerability reporting is enabled. Do not disclose credentials, customer
|
|
6
|
+
source documents or exploit details in public issues.
|
|
7
|
+
|
|
8
|
+
This release candidate supports disposable synthetic tests. Never run mutating
|
|
9
|
+
experiments against production servers or real customer data. Reports contain
|
|
10
|
+
synthetic document text and operation histories. Application adapters must avoid
|
|
11
|
+
persisting credentials and private source content in evidence.
|
|
12
|
+
|
|
13
|
+
The latest release candidate is the supported version during initial development.
|
|
14
|
+
Security reports are reviewed by the repository owner; no response-time service
|
|
15
|
+
level is promised.
|