hadith-detector 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (172) hide show
  1. hadith_detector-0.1.0/.gitattributes +10 -0
  2. hadith_detector-0.1.0/.gitignore +72 -0
  3. hadith_detector-0.1.0/.python-version +1 -0
  4. hadith_detector-0.1.0/CHANGELOG.md +96 -0
  5. hadith_detector-0.1.0/CONTRIBUTING.md +127 -0
  6. hadith_detector-0.1.0/LICENSE +21 -0
  7. hadith_detector-0.1.0/MANIFEST.in +24 -0
  8. hadith_detector-0.1.0/PKG-INFO +170 -0
  9. hadith_detector-0.1.0/README.md +146 -0
  10. hadith_detector-0.1.0/data/README.md +58 -0
  11. hadith_detector-0.1.0/docs/data-preparation.md +122 -0
  12. hadith_detector-0.1.0/docs/fuzzy-benchmark.md +117 -0
  13. hadith_detector-0.1.0/docs/fuzzy-runtime.md +78 -0
  14. hadith_detector-0.1.0/docs/html-annotation.md +91 -0
  15. hadith_detector-0.1.0/docs/index-benchmark.md +91 -0
  16. hadith_detector-0.1.0/docs/library-benchmark-results.md +230 -0
  17. hadith_detector-0.1.0/docs/library-benchmark.md +116 -0
  18. hadith_detector-0.1.0/docs/library-review.md +156 -0
  19. hadith_detector-0.1.0/docs/marisa-production.md +123 -0
  20. hadith_detector-0.1.0/docs/marisa-trial.md +164 -0
  21. hadith_detector-0.1.0/docs/phase2-validation.md +51 -0
  22. hadith_detector-0.1.0/docs/phase3-index-benchmark.md +197 -0
  23. hadith_detector-0.1.0/docs/phase3b-validation.md +133 -0
  24. hadith_detector-0.1.0/docs/phase4-fuzzy-benchmark.md +285 -0
  25. hadith_detector-0.1.0/docs/phase4b-validation.md +136 -0
  26. hadith_detector-0.1.0/docs/phase5a-validation.md +77 -0
  27. hadith_detector-0.1.0/docs/phase5b-plan.md +130 -0
  28. hadith_detector-0.1.0/docs/phase5b-validation.md +80 -0
  29. hadith_detector-0.1.0/docs/phase6-evaluation.md +86 -0
  30. hadith_detector-0.1.0/docs/phase6-notebook-validation.md +72 -0
  31. hadith_detector-0.1.0/docs/phase6-pilot-50.md +179 -0
  32. hadith_detector-0.1.0/docs/phase6-pilot-55.md +166 -0
  33. hadith_detector-0.1.0/docs/phase6-pilot-plan-50.md +166 -0
  34. hadith_detector-0.1.0/docs/phase6-pilot-plan.md +51 -0
  35. hadith_detector-0.1.0/docs/phase6-pilot.md +18 -0
  36. hadith_detector-0.1.0/docs/phase6-preparation.md +105 -0
  37. hadith_detector-0.1.0/docs/phase6-release-evaluation.md +11 -0
  38. hadith_detector-0.1.0/docs/phase6-review.md +248 -0
  39. hadith_detector-0.1.0/docs/public-api.md +139 -0
  40. hadith_detector-0.1.0/docs/release-readiness.md +109 -0
  41. hadith_detector-0.1.0/docs/roadmap.md +221 -0
  42. hadith_detector-0.1.0/docs/runtime-index.md +154 -0
  43. hadith_detector-0.1.0/examples/public_api.py +39 -0
  44. hadith_detector-0.1.0/pyproject.toml +72 -0
  45. hadith_detector-0.1.0/setup.cfg +4 -0
  46. hadith_detector-0.1.0/setup.py +66 -0
  47. hadith_detector-0.1.0/src/hadith_detector/__init__.py +33 -0
  48. hadith_detector-0.1.0/src/hadith_detector/_annotation.py +49 -0
  49. hadith_detector-0.1.0/src/hadith_detector/_bundle.json +45 -0
  50. hadith_detector-0.1.0/src/hadith_detector/_bundled.py +16 -0
  51. hadith_detector-0.1.0/src/hadith_detector/_convenience.py +84 -0
  52. hadith_detector-0.1.0/src/hadith_detector/_data/bundle.json +1 -0
  53. hadith_detector-0.1.0/src/hadith_detector/_data/manifest.json +1 -0
  54. hadith_detector-0.1.0/src/hadith_detector/_data/notices/LICENSES.md +17 -0
  55. hadith_detector-0.1.0/src/hadith_detector/_data/notices/PROVENANCE.md +46 -0
  56. hadith_detector-0.1.0/src/hadith_detector/_data/notices/README.md +866 -0
  57. hadith_detector-0.1.0/src/hadith_detector/_data/notices/SOURCES.md +28 -0
  58. hadith_detector-0.1.0/src/hadith_detector/_data/notices/UPSTREAM-UNLICENSE.txt +24 -0
  59. hadith_detector-0.1.0/src/hadith_detector/_data/notices/collections.csv +14 -0
  60. hadith_detector-0.1.0/src/hadith_detector/_data/notices/configs.csv +139 -0
  61. hadith_detector-0.1.0/src/hadith_detector/_data/record-offsets.bin +0 -0
  62. hadith_detector-0.1.0/src/hadith_detector/_data/records.jsonl +14940 -0
  63. hadith_detector-0.1.0/src/hadith_detector/_data/tokens.bin +0 -0
  64. hadith_detector-0.1.0/src/hadith_detector/_data/trigram-offsets.bin +0 -0
  65. hadith_detector-0.1.0/src/hadith_detector/_data/trigram-postings.bin +0 -0
  66. hadith_detector-0.1.0/src/hadith_detector/_data/trigrams.marisa +0 -0
  67. hadith_detector-0.1.0/src/hadith_detector/_data/unigram-offsets.bin +0 -0
  68. hadith_detector-0.1.0/src/hadith_detector/_data/unigram-postings.bin +0 -0
  69. hadith_detector-0.1.0/src/hadith_detector/_data/vocabulary.marisa +0 -0
  70. hadith_detector-0.1.0/src/hadith_detector/_fuzzy.py +272 -0
  71. hadith_detector-0.1.0/src/hadith_detector/_marisa.py +102 -0
  72. hadith_detector-0.1.0/src/hadith_detector/_resources.py +365 -0
  73. hadith_detector-0.1.0/src/hadith_detector/_serialization.py +210 -0
  74. hadith_detector-0.1.0/src/hadith_detector/_text.py +120 -0
  75. hadith_detector-0.1.0/src/hadith_detector/config.py +31 -0
  76. hadith_detector-0.1.0/src/hadith_detector/detector.py +296 -0
  77. hadith_detector-0.1.0/src/hadith_detector/models.py +161 -0
  78. hadith_detector-0.1.0/src/hadith_detector/py.typed +0 -0
  79. hadith_detector-0.1.0/src/hadith_detector.egg-info/PKG-INFO +170 -0
  80. hadith_detector-0.1.0/src/hadith_detector.egg-info/SOURCES.txt +170 -0
  81. hadith_detector-0.1.0/src/hadith_detector.egg-info/dependency_links.txt +1 -0
  82. hadith_detector-0.1.0/src/hadith_detector.egg-info/requires.txt +2 -0
  83. hadith_detector-0.1.0/src/hadith_detector.egg-info/scm_file_list.json +159 -0
  84. hadith_detector-0.1.0/src/hadith_detector.egg-info/scm_version.json +8 -0
  85. hadith_detector-0.1.0/src/hadith_detector.egg-info/top_level.txt +1 -0
  86. hadith_detector-0.1.0/tests/test_annotation.py +294 -0
  87. hadith_detector-0.1.0/tests/test_bundled.py +126 -0
  88. hadith_detector-0.1.0/tests/test_detector.py +302 -0
  89. hadith_detector-0.1.0/tests/test_evaluation_finalization.py +114 -0
  90. hadith_detector-0.1.0/tests/test_evaluation_preparation.py +362 -0
  91. hadith_detector-0.1.0/tests/test_exact_benchmark.py +304 -0
  92. hadith_detector-0.1.0/tests/test_foundation.py +49 -0
  93. hadith_detector-0.1.0/tests/test_fuzzy_benchmark.py +350 -0
  94. hadith_detector-0.1.0/tests/test_fuzzy_detector.py +184 -0
  95. hadith_detector-0.1.0/tests/test_library_benchmark.py +75 -0
  96. hadith_detector-0.1.0/tests/test_marisa_resources.py +146 -0
  97. hadith_detector-0.1.0/tests/test_marisa_trial.py +134 -0
  98. hadith_detector-0.1.0/tests/test_packaging.py +298 -0
  99. hadith_detector-0.1.0/tests/test_prepare_data.py +227 -0
  100. hadith_detector-0.1.0/tests/test_public_api.py +295 -0
  101. hadith_detector-0.1.0/tests/test_release_probe.py +30 -0
  102. hadith_detector-0.1.0/tests/test_review_notebook.py +266 -0
  103. hadith_detector-0.1.0/tests/test_technical_pilot.py +510 -0
  104. hadith_detector-0.1.0/tests/test_text.py +71 -0
  105. hadith_detector-0.1.0/tools/__init__.py +1 -0
  106. hadith_detector-0.1.0/tools/annotation_validation.py +62 -0
  107. hadith_detector-0.1.0/tools/build_index.py +206 -0
  108. hadith_detector-0.1.0/tools/bundle-notices/LICENSES.md +17 -0
  109. hadith_detector-0.1.0/tools/bundle-notices/PROVENANCE.md +46 -0
  110. hadith_detector-0.1.0/tools/bundle-notices/README.md +866 -0
  111. hadith_detector-0.1.0/tools/bundle-notices/SOURCES.md +28 -0
  112. hadith_detector-0.1.0/tools/bundle-notices/UPSTREAM-UNLICENSE.txt +24 -0
  113. hadith_detector-0.1.0/tools/bundle-notices/collections.csv +14 -0
  114. hadith_detector-0.1.0/tools/bundle-notices/configs.csv +139 -0
  115. hadith_detector-0.1.0/tools/check_marisa_install.py +119 -0
  116. hadith_detector-0.1.0/tools/evaluation/__init__.py +1 -0
  117. hadith_detector-0.1.0/tools/evaluation/__main__.py +57 -0
  118. hadith_detector-0.1.0/tools/evaluation/acquire.py +135 -0
  119. hadith_detector-0.1.0/tools/evaluation/dataset.py +203 -0
  120. hadith_detector-0.1.0/tools/evaluation/finalize.py +709 -0
  121. hadith_detector-0.1.0/tools/evaluation/natural-drafts.json +341 -0
  122. hadith_detector-0.1.0/tools/evaluation/natural-source-lock.json +1 -0
  123. hadith_detector-0.1.0/tools/evaluation/pilot-selection-50.json +1 -0
  124. hadith_detector-0.1.0/tools/evaluation/pilot-selection.json +1 -0
  125. hadith_detector-0.1.0/tools/evaluation/pilot.py +564 -0
  126. hadith_detector-0.1.0/tools/evaluation/pilot_migration.py +140 -0
  127. hadith_detector-0.1.0/tools/evaluation/pilot_review.py +587 -0
  128. hadith_detector-0.1.0/tools/evaluation/pilot_widgets.py +518 -0
  129. hadith_detector-0.1.0/tools/evaluation/pipeline.py +320 -0
  130. hadith_detector-0.1.0/tools/evaluation/review.py +274 -0
  131. hadith_detector-0.1.0/tools/evaluation/review_notebook.py +236 -0
  132. hadith_detector-0.1.0/tools/evaluation/review_widgets.py +643 -0
  133. hadith_detector-0.1.0/tools/evaluation/sources.json +22 -0
  134. hadith_detector-0.1.0/tools/exact_benchmark/__init__.py +1 -0
  135. hadith_detector-0.1.0/tools/exact_benchmark/__main__.py +316 -0
  136. hadith_detector-0.1.0/tools/exact_benchmark/corpus.py +128 -0
  137. hadith_detector-0.1.0/tools/exact_benchmark/indexes.py +222 -0
  138. hadith_detector-0.1.0/tools/exact_benchmark/oracle.py +138 -0
  139. hadith_detector-0.1.0/tools/exact_benchmark/queries.py +127 -0
  140. hadith_detector-0.1.0/tools/fuzzy_benchmark/__init__.py +1 -0
  141. hadith_detector-0.1.0/tools/fuzzy_benchmark/__main__.py +317 -0
  142. hadith_detector-0.1.0/tools/fuzzy_benchmark/common.py +173 -0
  143. hadith_detector-0.1.0/tools/fuzzy_benchmark/distance.py +78 -0
  144. hadith_detector-0.1.0/tools/fuzzy_benchmark/oracle.py +147 -0
  145. hadith_detector-0.1.0/tools/fuzzy_benchmark/prototypes.py +200 -0
  146. hadith_detector-0.1.0/tools/fuzzy_benchmark/queries.py +206 -0
  147. hadith_detector-0.1.0/tools/library_benchmark/__init__.py +1 -0
  148. hadith_detector-0.1.0/tools/library_benchmark/__main__.py +386 -0
  149. hadith_detector-0.1.0/tools/library_benchmark/adapters.py +240 -0
  150. hadith_detector-0.1.0/tools/library_benchmark/compatibility.py +74 -0
  151. hadith_detector-0.1.0/tools/library_benchmark/kernels.py +70 -0
  152. hadith_detector-0.1.0/tools/library_benchmark/order_control.py +22 -0
  153. hadith_detector-0.1.0/tools/library_benchmark/requirements.txt +7 -0
  154. hadith_detector-0.1.0/tools/marisa_trial/__init__.py +1 -0
  155. hadith_detector-0.1.0/tools/marisa_trial/benchmark.py +220 -0
  156. hadith_detector-0.1.0/tools/marisa_trial/build.py +209 -0
  157. hadith_detector-0.1.0/tools/marisa_trial/compatibility.py +48 -0
  158. hadith_detector-0.1.0/tools/marisa_trial/lookups.py +95 -0
  159. hadith_detector-0.1.0/tools/marisa_trial/report.py +134 -0
  160. hadith_detector-0.1.0/tools/marisa_trial/resources.py +365 -0
  161. hadith_detector-0.1.0/tools/prepare_data.py +421 -0
  162. hadith_detector-0.1.0/tools/release-cases.json +50 -0
  163. hadith_detector-0.1.0/tools/release_artifacts.py +114 -0
  164. hadith_detector-0.1.0/tools/release_install.py +81 -0
  165. hadith_detector-0.1.0/tools/release_probe.py +125 -0
  166. hadith_detector-0.1.0/tools/release_wheels.py +64 -0
  167. hadith_detector-0.1.0/tools/source-lock.json +49 -0
  168. hadith_detector-0.1.0/tools/stage_bundle.py +67 -0
  169. hadith_detector-0.1.0/tools/validate_fuzzy_runtime.py +211 -0
  170. hadith_detector-0.1.0/tools/validate_marisa_runtime.py +147 -0
  171. hadith_detector-0.1.0/tools/validate_runtime_index.py +178 -0
  172. hadith_detector-0.1.0/uv.lock +2930 -0
@@ -0,0 +1,10 @@
1
+ * text=auto
2
+ *.py text eol=lf
3
+ *.toml text eol=lf
4
+ *.yml text eol=lf
5
+ *.md text eol=lf
6
+ *.csv text eol=lf
7
+ *.json text eol=lf
8
+ *.lock text eol=lf
9
+ # Preserve checksum-pinned upstream notices byte-for-byte on every platform.
10
+ tools/bundle-notices/** -text
@@ -0,0 +1,72 @@
1
+ # === Python ignores ===
2
+ __pycache__/
3
+ *.pyc
4
+ *.pyo
5
+ *.pyd
6
+ *.pkl
7
+ *.db
8
+ *.sqlite3
9
+
10
+ # Virtual environment folders
11
+ venv/
12
+ env/
13
+ .venv/
14
+ .env/
15
+ ENV/
16
+ *.env
17
+ .env
18
+
19
+ # Logs
20
+ *.log
21
+ logs/*
22
+ !logs/.gitkeep
23
+
24
+ # IDE settings
25
+ .idea/
26
+
27
+ # OS files
28
+ .DS_Store
29
+ Thumbs.db
30
+
31
+ # Secret / private keys
32
+ *secret*.json
33
+ *key*.json
34
+
35
+ # Deployment / containers
36
+ *.pid
37
+ *.sock
38
+ *.crt
39
+ *.pem
40
+
41
+ # Coverage / test artifacts
42
+ *.coverage
43
+ htmlcov/
44
+
45
+ # Cache folders
46
+ .cache/
47
+ .ruff_cache/
48
+ .mypy_cache/
49
+ dist/
50
+ build/
51
+
52
+ # Docker-related
53
+ docker-compose.override.yml
54
+
55
+ # Data
56
+ src/hadith_detector/_data/
57
+ data/*
58
+ !data/.gitkeep
59
+ !data/README.md
60
+ .env.*
61
+ !.env.example
62
+ .ipynb_checkpoints/
63
+ .pytest_cache/
64
+ .coverage*
65
+ *.egg-info/
66
+
67
+ # Local research (preserved, not part of the supported package)
68
+ notebooks/*
69
+ !notebooks/phase6_review.ipynb
70
+ !notebooks/phase6_research_review.ipynb
71
+ !notebooks/phase6_review_50.ipynb
72
+ others/*
@@ -0,0 +1 @@
1
+ 3.14
@@ -0,0 +1,96 @@
1
+ # Changelog
2
+
3
+ Release notes are maintained manually. Versions use Git tags and 0.x development
4
+ conventions described in [CONTRIBUTING.md](CONTRIBUTING.md).
5
+
6
+ ## Unreleased
7
+
8
+ ### Fixed
9
+
10
+ - Release-readiness probes execute the current README examples without depending
11
+ on variable names from an earlier draft; documentation links also work on PyPI.
12
+ - Clean-wheel CI checks now install from hash-verified locked dependency wheels,
13
+ without relying on uv registry metadata left in a developer's cache.
14
+
15
+ ### Added
16
+
17
+ - One bundled Bukhari/Muslim corpus with prebuilt MARISA v2 resources, optional
18
+ `index_dir` defaults, offline staging and fail-closed distribution builds (Phase 7).
19
+ - Manual, non-publishing release-readiness workflow for actual bundled candidates
20
+ on Windows/Linux and Python 3.11–3.14, plus a professional README draft.
21
+ - Finalized the 55-input Phase 6 technical pilot with 382 completed human judgments,
22
+ reproducible offline scoring, evidence validation and separate release-description wording.
23
+ - Expanded the technical pilot to 55 frozen inputs, including all 20 commentary
24
+ paragraphs, with four-group reporting and explicitly approved, lossless review migration.
25
+ - Frozen 50-input technical pilot with source-linked comparisons, 62 deduplicated
26
+ passage pairs and separate automatic review progress; full intent research remains optional.
27
+ - Interactive Phase 6 ipywidgets review notebook with exact-offset Arabic previews,
28
+ lazy supporting evidence and atomic, resumable CSV progress saves.
29
+ - Repository-only Phase 6 input freezing, six-configuration evidence capture,
30
+ reversible review CSV, candidate-unwanted-alert inventory and offline replay.
31
+ - Pinned natural-source provenance and input-first draft spans. Original intent labels
32
+ remain provisional; the smaller technical pilot now has approved correspondence review.
33
+
34
+ - Instance and convenience `annotate()` methods returning safely escaped HTML
35
+ fragments with fixed marks, original offsets and best supporting costs (Phase 5B).
36
+ - Text/offset round trips, injection, exact/fuzzy parity and annotation concurrency checks.
37
+ - Carriage-return preservation and explicit annotation-only rejection of NUL/lone surrogates.
38
+
39
+ - Frozen `DetectorConfig`, directory constructor, read-only resolved `index_dir`,
40
+ and convenience `detect()`/`get_hadith()` with a synchronized single-entry cache.
41
+ - Validated result `from_dict()` methods and isolated JSON metadata round trips.
42
+ - Per-instance reentrant locking: concurrent detection and lookup calls execute sequentially.
43
+ - Public API, cache/concurrency and deserialization regression checks (Phase 5A).
44
+
45
+ - Prebuilt MARISA v2 vocabulary/trigram lookups with explicit preserved IDs,
46
+ complete resource validation and deterministic offline construction.
47
+ - Native-resource corruption tests, supported-version and offline installation checks.
48
+
49
+ - Production fuzzy detection with simultaneous character-edit and internal source-word
50
+ omission budgets, complete anchor retrieval and two-pass occurrence recovery.
51
+ - Per-occurrence costs, canonical word alignments and omitted-token models; JSON result
52
+ dictionaries now include these fields, including zero-cost traces on exact results.
53
+
54
+ - Independent exhaustive fuzzy alignment oracle and tiny direct path enumeration.
55
+ - Complete first-word, guaranteed-anchor and trigram-hybrid retrieval experiments,
56
+ with deterministic word traces, internal source omissions and combined budgets.
57
+ - Repository-only RapidFuzz 3.14.6 comparison, isolated timing/memory tooling and
58
+ supported-version backend compatibility checks (Phase 4A).
59
+
60
+ - Installable `hadith_detector` foundation with metadata-derived version and typing marker.
61
+ - Python 3.11–3.14 policy, uv lockfile, Ruff, strict mypy, pytest, and build validation.
62
+ - CI covering Linux and Windows, including clean wheel and sdist installation checks.
63
+ - Project status, phased roadmap, contribution workflow, and data artifact policy.
64
+ - Pinned QuranLab source lock and repository-only fetch/build/verify workflow.
65
+ - Shared versioned Arabic tokenization with original spans and matching barriers.
66
+ - Canonical JSONL artifacts, build manifests, warning reports, and reproducibility checks.
67
+ - Synthetic preprocessing CI and full-corpus acceptance evidence.
68
+ - Repository-only exhaustive exact-match oracle and positional/2-/3-/5-word prototypes.
69
+ - Direct substring-enumeration checks, deterministic query generation and isolated memory/latency benchmarks.
70
+
71
+ - Compact versioned trigram resources with complete unigram fallback, checksums,
72
+ structural validation, and deterministic offline construction.
73
+ - Provisional `HadithDetector.from_directory()`, exact detection and source lookup;
74
+ typed serializable results with all supporting references and occurrences.
75
+ - Runtime/oracle comparison, corruption tests, metadata isolation and clean-install checks.
76
+
77
+ ### Changed
78
+
79
+ - Detection settings now inherit defaults when omitted or `None`; explicit zero overrides.
80
+ - The provisional constructor now takes a directory; `from_directory()` remains supported.
81
+
82
+ - Require `marisa-trie==1.4.1` alongside RapidFuzz. Index format v2 replaces v1;
83
+ rebuild into a new directory from the prepared corpus and preserve v1 for rollback.
84
+ - Accept increased one-time validated initialization for substantially lower memory
85
+ in long-lived instances. Phase 5A now serializes shared-instance calls.
86
+
87
+ - RapidFuzz is the sole required production distance engine (`>=3.14.6,<4`);
88
+ Python distance implementations remain repository-only testing/benchmark tools.
89
+
90
+ - Dependency policy now requires offline operation while allowing evidence-based
91
+ adoption of maintained dependencies.
92
+
93
+ - Replaced the unused application scaffold with a library using no runtime dependencies.
94
+
95
+ Annotation and bundled resources remain planned. Runtime distributions
96
+ contain neither the corpus nor preprocessing dependencies.
@@ -0,0 +1,127 @@
1
+ # Contributing
2
+
3
+ ## Environment and checks
4
+
5
+ Use Python 3.14 locally and uv (CI pins uv 0.11.26). The supported test matrix is
6
+ Python 3.11–3.14 on Linux and 3.14 on Windows. Install the checkout before importing;
7
+ do not set `PYTHONPATH` or add the source directory to test paths.
8
+
9
+ ```sh
10
+ uv sync --locked
11
+ uv run ruff check .
12
+ uv run ruff format --check .
13
+ uv run mypy src/hadith_detector
14
+ uv run pytest
15
+ ```
16
+
17
+ Default resource use and production builds now require offline staging of the approved
18
+ bundle. Packaging tests prepare synthetic bundles in isolated source trees, so normal
19
+ CI does not fetch the corpus. For real wheel/sdist builds, stage verified resources
20
+ first and follow [release readiness](docs/release-readiness.md). Its manual workflow
21
+ tests actual bundled candidates on both Linux and Windows, Python 3.11–3.14.
22
+
23
+ In PowerShell, if wildcard expansion for twine is unavailable, use
24
+ `Get-ChildItem dist -File | ForEach-Object { uv run twine check $_.FullName }`.
25
+ Packaging tests invoke uv, resolve isolated build dependencies, build a wheel from
26
+ an sdist without Git metadata, and install both wheels in fresh environments.
27
+ Preparation downloads compatible runtime dependency wheels listed in `uv.lock`
28
+ and verifies their SHA-256 hashes. Both clean installations then use that local
29
+ wheelhouse with `--offline --no-index --no-cache`, so a warm registry cache cannot
30
+ hide missing dependencies. Installed-library probes also block network access.
31
+ Build/dependency preparation may access the network; offline installation and
32
+ runtime imports do not.
33
+ Use `uv run pytest -m "not packaging"` for the fast foundation checks.
34
+
35
+ Phase 2 preprocessing uses `uv sync --locked --group data`. Run its tests with
36
+ `uv run --frozen --group data pytest`; the default environment skips the synthetic
37
+ pipeline tests when preprocessing dependencies are absent. Shared normalization
38
+ tests run on every supported Python. Full builds require Python 3.14. Ordinary CI
39
+ uses synthetic Parquet data and does not fetch the real corpus. See the
40
+ [data-preparation guide](docs/data-preparation.md) for full-corpus validation.
41
+
42
+ Index experiments use `uv sync --locked --group benchmark`. Run the synthetic
43
+ worker smoke test with `uv run --frozen --group benchmark pytest tests/test_exact_benchmark.py`.
44
+ The exhaustive oracle, retrieval, serialization and randomized reference tests
45
+ need no benchmark dependency; only the subprocess memory-measurement smoke test
46
+ needs psutil. Full-corpus performance experiments are local and never run in CI.
47
+ See [benchmark usage](docs/index-benchmark.md).
48
+
49
+ Phase 3B runtime and synthetic index-builder tests need no optional dependencies
50
+ and run across the supported Python matrix. Full-corpus index construction uses
51
+ Python 3.14 and the verified prepared artifacts; see [runtime resources](docs/runtime-index.md).
52
+ Fresh-process validation measurements use the separate `benchmark` group.
53
+
54
+ Phase 4B requires RapidFuzz at runtime; the benchmark group pins 3.14.6 for reproduction.
55
+ Run the repository's synthetic
56
+ oracle/retrieval/backend checks with `uv run --frozen --group benchmark pytest tests/test_fuzzy_benchmark.py`.
57
+ The normal environment skips RapidFuzz parity and subprocess tests if their optional
58
+ tools are absent. See [fuzzy experiments](docs/fuzzy-benchmark.md) for full-corpus commands.
59
+ Dependencies are assessed on their merits; offline operation is required, while
60
+ zero runtime dependencies is not a permanent project rule. The maintainer approved
61
+ RapidFuzz as the sole production distance engine and authorized Phase 4B. Python
62
+ distance implementations stay in repository tests/benchmarks. Further phase changes
63
+ require review and approval.
64
+
65
+ Ruff handles linting and formatting (100-character lines, Python 3.11 syntax).
66
+ Mypy checks package code in strict mode. Regenerate `uv.lock` with `uv lock` only
67
+ when dependencies change, then rerun checks. No pre-commit hook is required.
68
+
69
+ ## Review and Git workflow
70
+
71
+ Work in independently reviewable phases. Update documentation and the Unreleased
72
+ changelog with behavior changes. Run relevant checks before each commit, inspect
73
+ the explicit staged diff, and run the complete suite before phase acceptance.
74
+ Do not stage ignored research, raw downloads, large artifacts, or unrelated changes.
75
+
76
+ Phase 1 commit boundaries:
77
+
78
+ 1. `chore: establish installable library foundation`
79
+ 2. `test: validate package builds and add CI`
80
+ 3. `docs: document project status and phased roadmap`
81
+
82
+ Phase 2 proposed commit boundaries:
83
+
84
+ 1. `data: pin QuranLab sources and document provenance`
85
+ 2. `feat: add mapped Arabic normalization and tokenization`
86
+ 3. `feat: add reproducible corpus preparation`
87
+ 4. `docs: record full-corpus preparation validation`
88
+
89
+ Phase 3A proposed commit boundaries:
90
+
91
+ 1. `test: establish exhaustive exact-match oracle`
92
+ 2. `perf: benchmark positional and n-gram indexes`
93
+ 3. `docs: record index benchmark and recommendation`
94
+
95
+ Phase 3B proposed commit boundaries:
96
+
97
+ 1. `feat: add compact offline index resources`
98
+ 2. `feat: implement provisional exact detector`
99
+ 3. `docs: record runtime validation and API usage`
100
+
101
+ Commit and push when requested. Preserve existing history and origin; never
102
+ force-push. Remote changes, destructive Git operations, release tags, public
103
+ releases, and PyPI publication require explicit approval. Stop after each phase
104
+ for review before implementing the next.
105
+
106
+ ## Versions and releases
107
+
108
+ Setuptools-scm derives versions from Git; fetch full history and tags in CI.
109
+ Untagged builds are development versions, not a stable `0.1.0`. Import
110
+ `hadith_detector.__version__` to read installed distribution metadata; runtime
111
+ does not need Git or setuptools-scm. Build from a checkout or an official sdist;
112
+ arbitrary source copies without Git or sdist metadata are unsupported.
113
+
114
+ Reserve `v0.1.0` for the first approved usable release. Milestones need no tags.
115
+ During 0.x development, minor versions introduce features or breaking changes;
116
+ patch versions contain compatible fixes. Document incompatibilities and manually
117
+ move Unreleased notes into a dated release section when releasing.
118
+
119
+ Record schemas, normalization, and index formats have independent versions.
120
+ Dataset builds identify pinned upstream revision, preprocessing code fingerprint,
121
+ configuration, and checksums. Corpus rights are separate from MIT; source-specific
122
+ terms and attribution are documented for the approved Arabic configurations.
123
+ No publishing workflow is enabled in Phase 1.
124
+
125
+ Local notebooks and evaluation files are preserved but are not supported package
126
+ code. Use separate environments for their exploratory dependencies; the new
127
+ foundation environment intentionally excludes the former web/research dependencies.
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Mohamed Rayen Fekih
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,24 @@
1
+ include README.md CONTRIBUTING.md CHANGELOG.md LICENSE pyproject.toml
2
+ include setup.py
3
+ include src/hadith_detector/_bundle.json
4
+ recursive-include src/hadith_detector/_data *
5
+ recursive-include src/hadith_detector *.py py.typed
6
+ recursive-include tests *.py
7
+ recursive-include docs *.md
8
+ # This report records final archive checksums; keep it outside the archives it describes.
9
+ exclude docs/phase7-validation.md
10
+ recursive-include tools *.py *.json
11
+ recursive-include tools/bundle-notices *.md *.txt *.csv
12
+ include uv.lock
13
+ include data/README.md
14
+ prune .github
15
+ prune .vscode
16
+ prune .cache
17
+ prune .ruff_cache
18
+ prune .mypy_cache
19
+ prune .pytest_cache
20
+ prune .venv
21
+ prune notebooks
22
+ prune others
23
+ prune logs
24
+ global-exclude *.py[cod] .gitkeep
@@ -0,0 +1,170 @@
1
+ Metadata-Version: 2.4
2
+ Name: hadith-detector
3
+ Version: 0.1.0
4
+ Summary: Deterministic offline Arabic textual matching against Sahih Bukhari and Sahih Muslim
5
+ Author: Mohamed Rayen Fekih
6
+ License-Expression: MIT
7
+ Project-URL: Repository, https://github.com/RayenFekih/hadith-detector
8
+ Project-URL: Issues, https://github.com/RayenFekih/hadith-detector/issues
9
+ Classifier: Development Status :: 3 - Alpha
10
+ Classifier: Intended Audience :: Developers
11
+ Classifier: Operating System :: OS Independent
12
+ Classifier: Programming Language :: Python :: 3
13
+ Classifier: Programming Language :: Python :: 3.11
14
+ Classifier: Programming Language :: Python :: 3.12
15
+ Classifier: Programming Language :: Python :: 3.13
16
+ Classifier: Programming Language :: Python :: 3.14
17
+ Classifier: Typing :: Typed
18
+ Requires-Python: >=3.11
19
+ Description-Content-Type: text/markdown
20
+ License-File: LICENSE
21
+ Requires-Dist: rapidfuzz<4,>=3.14.6
22
+ Requires-Dist: marisa-trie==1.4.1
23
+ Dynamic: license-file
24
+
25
+ # Hadith Detector
26
+
27
+ **Detect Arabic Hadith text and find its sources. Offline, in Python.**
28
+
29
+ [![Python 3.11–3.14](https://img.shields.io/badge/Python-3.11%E2%80%933.14-3776AB?logo=python&logoColor=white)](https://github.com/RayenFekih/hadith-detector/blob/main/pyproject.toml)
30
+ [![CI](https://github.com/RayenFekih/hadith-detector/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/RayenFekih/hadith-detector/actions/workflows/ci.yml)
31
+ [![Code license: MIT](https://img.shields.io/badge/Code_license-MIT-blue.svg)](https://github.com/RayenFekih/hadith-detector/blob/main/LICENSE)
32
+
33
+ `hadith-detector` is a Python library that finds Arabic passages matching Hadith texts and returns their supporting sources. It works with short quotations or longer documents, supports exact and configurable fuzzy matching, and runs locally without an API key or an internet connection after installation.
34
+
35
+ The current bundled corpus covers **Sahih al-Bukhari and Sahih Muslim**. Support for additional Hadith collections is planned.
36
+
37
+ ## Installation
38
+
39
+ ```bash
40
+ pip install hadith-detector
41
+ ```
42
+
43
+ Or with [uv](https://docs.astral.sh/uv/):
44
+
45
+ ```bash
46
+ uv add hadith-detector
47
+ ```
48
+
49
+ Requires **Python 3.11–3.14**. The package includes the Hadith texts and a prebuilt search index, so there is no separate corpus download or index-building step. Detection works offline after installation.
50
+
51
+ ## Quick start
52
+
53
+ Detect a Hadith quotation and highlight it in the original Arabic text:
54
+
55
+ ```python
56
+ from hadith_detector import detect, annotate
57
+
58
+ text = "قال رسول الله ﷺ: إنما الأعمال بالنيات وإنما لكل امرئ ما نوى."
59
+
60
+ matches = detect(text)
61
+ print(matches[0].text)
62
+ print(matches[0].references[0].record.hadith_key)
63
+
64
+ html = annotate(text)
65
+ print(html) # Escaped HTML fragment with <mark> around detected passages
66
+ ```
67
+
68
+ `detect()` returns matched passages, their original character offsets, and supporting Hadith references. A passage may have multiple supporting references. `annotate()` returns an HTML fragment that highlights matches without changing the original Arabic wording.
69
+
70
+ ## Features
71
+
72
+ - **Exact Arabic text matching**, with normalization of supported Arabic letter forms and diacritics.
73
+ - **Optional fuzzy matching** for limited spelling differences and omitted words from the source.
74
+ - **Source references** with complete original records, occurrences, offsets, and alignment details.
75
+ - **Multiple detections** within the same paragraph or document.
76
+ - **HTML highlighting** of detected passages using `annotate()`.
77
+ - **Offline and deterministic** operation with a bundled index; no LLM or external service required.
78
+
79
+ ## Configuration
80
+
81
+ The default settings are intentionally strict:
82
+
83
+ | Parameter | Default | What it controls |
84
+ | --- | ---: | --- |
85
+ | `min_match_words` | `5` | Minimum number of matched input words. Lower values also detect more short, common expressions; higher values can miss short quotations. |
86
+ | `max_errors` | `0` | Total number of character edits allowed across the matched words. |
87
+ | `max_missing_words` | `0` | Number of internal words in the source passage that may be absent from the input. |
88
+
89
+ ```python
90
+ from hadith_detector import DetectorConfig, HadithDetector
91
+
92
+ detector = HadithDetector(
93
+ config=DetectorConfig(
94
+ min_match_words=5,
95
+ max_errors=1,
96
+ max_missing_words=1,
97
+ )
98
+ )
99
+
100
+ matches = detector.detect("إنما الأعمال بالنياث وإنما لكل امرئ ما نوى")
101
+
102
+ # Override the settings for one call without changing the configuration.
103
+ exact_matches = detector.detect(
104
+ "إنما الأعمال بالنيات وإنما لكل امرئ ما نوى", max_errors=0, max_missing_words=0
105
+ )
106
+ ```
107
+
108
+ For long Arabic documents, consider a higher `min_match_words` if short incidental matches create too much noise. Enable fuzzy budgets only when your input may contain spelling variations or source-word omissions. Broader settings can increase both detections and processing time. There is no universally best configuration; test the trade-offs on your own texts.
109
+
110
+ ## More usage examples
111
+
112
+ **Retrieve a complete source record**
113
+
114
+ ```python
115
+ from hadith_detector import get_hadith
116
+
117
+ record = get_hadith("bukhari:1")
118
+ print(record.text)
119
+ ```
120
+
121
+ **Highlight matches in plain text**
122
+
123
+ ```python
124
+ from hadith_detector import annotate
125
+
126
+ html = annotate("ورد في النص: إنما الأعمال بالنيات وإنما لكل امرئ ما نوى")
127
+ ```
128
+
129
+ `annotate()` produces an escaped HTML fragment with matched passages wrapped in `<mark>` elements. It is designed for plain-text input, not for editing an existing HTML document.
130
+
131
+ ## Important notes
132
+
133
+ - **Generic phrases can trigger incidental matches.** For example, a phrase such as `هو عبد الله` may also appear in Hadith records even when it occurs in an unrelated article. Likewise, narration formulas such as `عن النبي صلى الله عليه وسلم قال` may match without being meaningful standalone quotations. Shorter `min_match_words` values increase this risk, and permissive fuzzy budgets can introduce additional matches. **Tune `min_match_words`, `max_errors`, and `max_missing_words` for your text and use case**, especially for long documents; increasing the minimum can reduce short incidental matches but may miss genuine short quotations.
134
+ - **A match means textual support, not quotation intent.** Common expressions, honorifics, Quranic wording embedded in narrations, and narrator-chain fragments can also match. Do not automatically treat every detection as a deliberate Hadith quotation.
135
+ - **Review citations and source context.** Before publishing religious content or relying on the returned citation, inspect the original narration, relevant supporting occurrences, and data provenance. The library does not independently authenticate Hadith or interpret their meaning.
136
+ - **No match does not mean no Hadith exists.** Coverage is currently limited to the bundled Sahihayn collection, and configured matching thresholds may exclude short or modified quotations.
137
+ - **The dataset is preserved as supplied.** Source data can include duplicate narration texts, missing URLs, or damaged characters. Check that it meets your application’s accuracy and provenance requirements.
138
+
139
+ ## Data
140
+
141
+ The package currently includes **14,940 Arabic Hadith records**:
142
+
143
+ | Collection | Records |
144
+ | --- | ---: |
145
+ | Sahih al-Bukhari | 7,580 |
146
+ | Sahih Muslim | 7,360 |
147
+
148
+ The data comes from the [QuranLab Hadith dataset on Hugging Face](https://huggingface.co/datasets/quranlab/hadith), using its `bukhari-ar` and `muslim-ar` configurations. Original Arabic records, source identities, and accompanying metadata are retained. **Additional Hadith books are planned for future versions.**
149
+
150
+ The library's **code is MIT-licensed**; the underlying Hadith data has separate source terms and notices. Consult the [dataset provenance and redistribution notes](https://github.com/RayenFekih/hadith-detector/blob/main/tools/bundle-notices/PROVENANCE.md) before reusing or redistributing the data independently.
151
+
152
+ ## Use cases
153
+
154
+ - **Arabic documents and articles:** Locate matching Hadith passages inside longer text.
155
+ - **Research and reference lookup:** Find supporting collections and inspect the original narration.
156
+ - **Educational and editorial tools:** Help reviewers locate quotations and check source wording.
157
+ - **Web applications:** Highlight text matches and show citations or source context.
158
+ - **AI and LLM applications:** Check generated Hadith wording, retrieve source passages, and provide grounding evidence for citation workflows. Human review remains important.
159
+
160
+ ## Documentation and community
161
+
162
+ - [Public API and result structure](https://github.com/RayenFekih/hadith-detector/blob/main/docs/public-api.md)
163
+ - [Matching and fuzzy behavior](https://github.com/RayenFekih/hadith-detector/blob/main/docs/fuzzy-runtime.md)
164
+ - [HTML annotation](https://github.com/RayenFekih/hadith-detector/blob/main/docs/html-annotation.md)
165
+ - [Evaluation and limitations](https://github.com/RayenFekih/hadith-detector/blob/main/docs/phase6-evaluation.md)
166
+ - [Contributing](https://github.com/RayenFekih/hadith-detector/blob/main/CONTRIBUTING.md)
167
+ - [Changelog](https://github.com/RayenFekih/hadith-detector/blob/main/CHANGELOG.md)
168
+ - [Report a bug or request a feature](https://github.com/RayenFekih/hadith-detector/issues)
169
+
170
+ Contributions, issue reports, reproducible examples, and suggestions for additional Hadith sources are welcome.