khocr-gen 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (52) hide show
  1. khocr_gen-0.1.0/.gitignore +8 -0
  2. khocr_gen-0.1.0/CHANGELOG.md +29 -0
  3. khocr_gen-0.1.0/LICENSE +21 -0
  4. khocr_gen-0.1.0/PKG-INFO +213 -0
  5. khocr_gen-0.1.0/README.md +173 -0
  6. khocr_gen-0.1.0/docs/AUGMENTATION.md +242 -0
  7. khocr_gen-0.1.0/docs/CLI_REFERENCE.md +266 -0
  8. khocr_gen-0.1.0/docs/CONFIG.md +254 -0
  9. khocr_gen-0.1.0/docs/RUST_ACCELERATION.md +125 -0
  10. khocr_gen-0.1.0/pyproject.toml +124 -0
  11. khocr_gen-0.1.0/rust/Cargo.lock +605 -0
  12. khocr_gen-0.1.0/rust/Cargo.toml +30 -0
  13. khocr_gen-0.1.0/rust/pyproject.toml +13 -0
  14. khocr_gen-0.1.0/rust/src/augmentation.rs +1186 -0
  15. khocr_gen-0.1.0/rust/src/fonts.rs +324 -0
  16. khocr_gen-0.1.0/rust/src/lib.rs +376 -0
  17. khocr_gen-0.1.0/rust/src/rendering.rs +524 -0
  18. khocr_gen-0.1.0/rust/src/utils.rs +125 -0
  19. khocr_gen-0.1.0/src/khocr_gen/__init__.py +8 -0
  20. khocr_gen-0.1.0/src/khocr_gen/_pil_compat.py +25 -0
  21. khocr_gen-0.1.0/src/khocr_gen/_rust_accel.py +296 -0
  22. khocr_gen-0.1.0/src/khocr_gen/augmentation.py +753 -0
  23. khocr_gen-0.1.0/src/khocr_gen/cli.py +208 -0
  24. khocr_gen-0.1.0/src/khocr_gen/combine.py +179 -0
  25. khocr_gen-0.1.0/src/khocr_gen/combine_cmd.py +98 -0
  26. khocr_gen-0.1.0/src/khocr_gen/config.py +693 -0
  27. khocr_gen-0.1.0/src/khocr_gen/config_loader.py +149 -0
  28. khocr_gen-0.1.0/src/khocr_gen/corpus.py +232 -0
  29. khocr_gen-0.1.0/src/khocr_gen/data_generator.py +613 -0
  30. khocr_gen-0.1.0/src/khocr_gen/errors.py +33 -0
  31. khocr_gen-0.1.0/src/khocr_gen/fonts.py +181 -0
  32. khocr_gen-0.1.0/src/khocr_gen/generate.py +187 -0
  33. khocr_gen-0.1.0/src/khocr_gen/lmdb_pack.py +111 -0
  34. khocr_gen-0.1.0/src/khocr_gen/logging.py +25 -0
  35. khocr_gen-0.1.0/src/khocr_gen/normalizer.py +217 -0
  36. khocr_gen-0.1.0/src/khocr_gen/parallel.py +206 -0
  37. khocr_gen-0.1.0/src/khocr_gen/rendering.py +539 -0
  38. khocr_gen-0.1.0/src/khocr_gen/verify.py +288 -0
  39. khocr_gen-0.1.0/src/khocr_gen/viewer.py +168 -0
  40. khocr_gen-0.1.0/tests/__init__.py +0 -0
  41. khocr_gen-0.1.0/tests/test_augmentation.py +421 -0
  42. khocr_gen-0.1.0/tests/test_combine.py +46 -0
  43. khocr_gen-0.1.0/tests/test_config.py +253 -0
  44. khocr_gen-0.1.0/tests/test_config_loader.py +168 -0
  45. khocr_gen-0.1.0/tests/test_corpus.py +203 -0
  46. khocr_gen-0.1.0/tests/test_errors.py +75 -0
  47. khocr_gen-0.1.0/tests/test_fonts.py +133 -0
  48. khocr_gen-0.1.0/tests/test_lmdb_pack.py +31 -0
  49. khocr_gen-0.1.0/tests/test_normalizer.py +103 -0
  50. khocr_gen-0.1.0/tests/test_online_aug.py +82 -0
  51. khocr_gen-0.1.0/tests/test_parallel.py +113 -0
  52. khocr_gen-0.1.0/typings/_rust_accel.pyi +56 -0
@@ -0,0 +1,8 @@
1
+ .pytest_cache
2
+ .ruff_cache
3
+ .venv
4
+
5
+ rust/target/
6
+
7
+ __pycache__
8
+ uv.lock
@@ -0,0 +1,29 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [0.1.0] - Unreleased
9
+
10
+ ### Added
11
+ - Initial release: synthetic OCR dataset generation for Khmer/English text
12
+ - `generate` command: render text corpus to training images with augmentations
13
+ - `combine` command: merge multiple datasets into one
14
+ - `verify` command: render augmentation comparison grids for QA
15
+ - `view` command: preview LMDB dataset contents
16
+ - Configurable augmentation pipeline with min/max/prob per method
17
+ - Multi-process parallel rendering
18
+ - LMDB output packing
19
+ - Khmer text normalization via khmernormalizer
20
+ - Rust acceleration extension (`khocr-gen-core`, in `rust/`): native implementations of
21
+ 21/24 augmentation methods, O(1) cmap-based font glyph checking (`FontFace`), and
22
+ script-span splitting, with automatic pure-Python fallback when unavailable. Built and
23
+ installed automatically by `uv sync` via a `[tool.uv.sources]` path dependency. See
24
+ [docs/RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md).
25
+
26
+ ### Fixed
27
+ - `ImageRenderer._render_mixed_font` passed a `(text, script)` tuple to `draw.text()`
28
+ instead of the span's text string, which would have crashed any mixed-font render
29
+ (dead code path prior to this fix, since `mixed_font_prob` defaults to `0.0`).
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Kosal Seng
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,213 @@
1
+ Metadata-Version: 2.4
2
+ Name: khocr-gen
3
+ Version: 0.1.0
4
+ Summary: Synthetic OCR training data generator for mixed Khmer/English text
5
+ Project-URL: Homepage, https://github.com/LazyGreed/khocr-gen
6
+ Project-URL: Repository, https://github.com/LazyGreed/khocr-gen
7
+ Project-URL: Issues, https://github.com/LazyGreed/khocr-gen/issues
8
+ Project-URL: Changelog, https://github.com/LazyGreed/khocr-gen/blob/main/CHANGELOG.md
9
+ Author: Kosal Seng
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Keywords: data-generation,deep-learning,image-generation,khmer,ocr,synthetic-data,text-recognition
13
+ Classifier: Development Status :: 3 - Alpha
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Intended Audience :: Science/Research
16
+ Classifier: License :: OSI Approved :: MIT License
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.12
19
+ Classifier: Programming Language :: Python :: 3.13
20
+ Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
21
+ Classifier: Topic :: Scientific/Engineering :: Image Recognition
22
+ Classifier: Topic :: Text Processing :: Fonts
23
+ Requires-Python: >=3.12
24
+ Requires-Dist: albumentations>=1.3.0
25
+ Requires-Dist: khmernormalizer>=0.0.4
26
+ Requires-Dist: lmdb>=1.4.0
27
+ Requires-Dist: numpy>=1.24.0
28
+ Requires-Dist: opencv-python-headless>=4.8.0
29
+ Requires-Dist: pillow>=10.0.0
30
+ Requires-Dist: pyyaml>=6.0
31
+ Requires-Dist: tqdm>=4.64.0
32
+ Provides-Extra: dev
33
+ Requires-Dist: khocr-gen-core; extra == 'dev'
34
+ Requires-Dist: pytest-cov>=4.0; extra == 'dev'
35
+ Requires-Dist: pytest>=7.0; extra == 'dev'
36
+ Requires-Dist: ruff>=0.1.0; extra == 'dev'
37
+ Provides-Extra: rust
38
+ Requires-Dist: khocr-gen-core; extra == 'rust'
39
+ Description-Content-Type: text/markdown
40
+
41
+ # khocr-gen
42
+
43
+ Synthetic OCR training data generator for mixed Khmer/English text.
44
+
45
+ ## Features
46
+
47
+ - **Mixed-script rendering:** per-span font selection for Khmer + English
48
+ - **24 augmentation methods:** unified registry covering scanner/camera degradations and training-time transforms
49
+ - **Rust acceleration:** 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
50
+ - **Isolated augmentation:** one effect per image, weighted by configurable probabilities
51
+ - **Configurable intensity ranges:** per-method `[min, max]` with linear sampling
52
+ - **Multiprocess generation:** parallel workers for throughput
53
+ - **LMDB packing:** efficient key–value storage for training pipelines
54
+ - **n-way dataset combine:** merge multiple generated datasets into one
55
+ - **Built-in viewer:** preview and extract images from LMDB databases
56
+ - **YAML config files:** reproducible generation recipes
57
+ - **Khmer normalization:** canonical character ordering via `khmernormalizer`
58
+
59
+ ## Installation
60
+
61
+ ```bash
62
+ # Clone and install with uv
63
+ git clone https://github.com/LazyGreed/khocr-gen.git
64
+ cd khocr-gen
65
+ uv sync
66
+
67
+ # Or install with pip
68
+ pip install .
69
+ ```
70
+
71
+ `uv sync` also compiles and installs the optional Rust acceleration extension
72
+ (`khocr-gen-core`, from `rust/`). This requires a Rust toolchain (`cargo`/`rustc`) on
73
+ `PATH` — see [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md). If the extension isn't
74
+ available, khocr-gen falls back to pure-Python implementations automatically.
75
+
76
+ ### Requirements
77
+
78
+ - Python >= 3.12
79
+ - Rust toolchain (`cargo`/`rustc`, edition 2021) — only needed to build the optional
80
+ acceleration extension; the project works without it
81
+
82
+ ## Quick Start
83
+
84
+ ### 1. Prepare fonts
85
+
86
+ ```text
87
+ fonts/
88
+ ├── khmer/ <- .ttf / .otf Khmer fonts
89
+ └── english/ <- .ttf / .otf English fonts
90
+ ```
91
+
92
+ Fonts placed directly in `fonts/` (not in a subdirectory) are added to both pools as fallbacks.
93
+
94
+ ### 2. Prepare corpus
95
+
96
+ A plain UTF-8 text file, one string per line:
97
+
98
+ ```text
99
+ សួស្តី
100
+ Hello World
101
+ ស្វាគមន៍ Welcome
102
+ ```
103
+
104
+ ### 3. Generate a dataset
105
+
106
+ ```bash
107
+ khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb
108
+ ```
109
+
110
+ ### 4. Verify augmentation quality
111
+
112
+ Renders every augmentation method at min and max intensity on clean canvases:
113
+
114
+ ```bash
115
+ khocr-gen verify --fonts fonts --output verify_output
116
+ ```
117
+
118
+ ### 5. View generated images
119
+
120
+ ```bash
121
+ # Summary of an LMDB database
122
+ khocr-gen view --lmdb data/train/lmdb
123
+
124
+ # Extract images
125
+ khocr-gen view --lmdb data/train/lmdb --output-dir extracted
126
+
127
+ # Show labels only
128
+ khocr-gen view --lmdb data/train/lmdb --labels-only
129
+ ```
130
+
131
+ ### 6. Combine multiple datasets
132
+
133
+ ```bash
134
+ khocr-gen combine data_run1 data_run2 data_run3 --output data_merged
135
+ ```
136
+
137
+ ## Configuration
138
+
139
+ All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML).
140
+ See [CONFIG.md](docs/CONFIG.md) for details.
141
+
142
+ ```bash
143
+ khocr-gen generate --config configs/generate.yml
144
+ ```
145
+
146
+ Config loading order (highest priority wins):
147
+ 1. argparse built-in defaults
148
+ 2. YAML config file values
149
+ 3. Explicit CLI flags
150
+
151
+ ## Augmentation
152
+
153
+ 24 methods in a unified registry (21 of them Rust-accelerated).
154
+ Each generated image receives exactly one augmentation, chosen probabilistically by weight.
155
+ See [AUGMENTATION.md](docs/AUGMENTATION.md) for the full catalog and visual examples.
156
+
157
+ ## CLI Reference
158
+
159
+ See [CLI_REFERENCE.md](docs/CLI_REFERENCE.md) for complete command documentation.
160
+
161
+ ## Rust Acceleration
162
+
163
+ See [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md) for what's accelerated, how the
164
+ native extension is built/installed, and how to iterate on the `rust/` crate.
165
+
166
+ ## Project Structure
167
+
168
+ ```text
169
+ src/khocr_gen/
170
+ ├── cli.py # CLI entry point
171
+ ├── config.py # GenerationConfig + AugMethodConfig dataclasses
172
+ ├── config_loader.py # YAML config loading
173
+ ├── corpus.py # Corpus loading / filtering / counting
174
+ ├── fonts.py # FontManager with script-aware font selection
175
+ ├── normalizer.py # Khmer text normalization wrapper
176
+ ├── rendering.py # ImageRenderer: text -> clean canvas -> augmentation
177
+ ├── augmentation.py # 24 augmentation methods (unified registry)
178
+ ├── _rust_accel.py # Native/pure-Python dispatch layer for the Rust extension
179
+ ├── parallel.py # Multiprocess rendering workers
180
+ ├── lmdb_pack.py # LMDB database packing
181
+ ├── combine.py # n-way dataset merge
182
+ ├── data_generator.py # DatasetGenerator orchestrator
183
+ ├── generate.py # generate CLI command
184
+ ├── verify.py # verify CLI command
185
+ ├── viewer.py # view CLI command
186
+ ├── combine_cmd.py # combine CLI command
187
+ └── errors.py # Exception hierarchy
188
+
189
+ rust/ # khocr-gen-core: PyO3/maturin native acceleration extension
190
+ ├── Cargo.toml
191
+ ├── pyproject.toml
192
+ └── src/
193
+ ├── lib.rs # PyO3 bindings + module registration
194
+ ├── augmentation.rs # Accelerated augmentation methods
195
+ ├── fonts.rs # FontManager + FontFace (cmap glyph lookup)
196
+ ├── rendering.rs # Unwired canvas-rendering prototype (not yet bound to Python)
197
+ └── utils.rs
198
+ ```
199
+
200
+ ## Acknowledgments
201
+
202
+ khocr-gen builds on excellent open-source libraries:
203
+
204
+ - [khmernormalizer](https://github.com/seanghay/khmernormalizer) (MIT) - Khmer text normalization
205
+ - [OpenCV](https://opencv.org/) (Apache 2.0) - Image processing and augmentation
206
+ - [Pillow](https://python-pillow.org/) (HPND) - Font rendering and image creation
207
+ - [Albumentations](https://albumentations.ai/) (MIT) - Augmentation primitives
208
+ - [NumPy](https://numpy.org/) (BSD-3-Clause) - Numerical operations
209
+ - [LMDB](https://www.symas.com/lmdb) (OpenLDAP 2.8) - Embedded database for training pipelines
210
+
211
+ ## License
212
+
213
+ MIT
@@ -0,0 +1,173 @@
1
+ # khocr-gen
2
+
3
+ Synthetic OCR training data generator for mixed Khmer/English text.
4
+
5
+ ## Features
6
+
7
+ - **Mixed-script rendering:** per-span font selection for Khmer + English
8
+ - **24 augmentation methods:** unified registry covering scanner/camera degradations and training-time transforms
9
+ - **Rust acceleration:** 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
10
+ - **Isolated augmentation:** one effect per image, weighted by configurable probabilities
11
+ - **Configurable intensity ranges:** per-method `[min, max]` with linear sampling
12
+ - **Multiprocess generation:** parallel workers for throughput
13
+ - **LMDB packing:** efficient key–value storage for training pipelines
14
+ - **n-way dataset combine:** merge multiple generated datasets into one
15
+ - **Built-in viewer:** preview and extract images from LMDB databases
16
+ - **YAML config files:** reproducible generation recipes
17
+ - **Khmer normalization:** canonical character ordering via `khmernormalizer`
18
+
19
+ ## Installation
20
+
21
+ ```bash
22
+ # Clone and install with uv
23
+ git clone https://github.com/LazyGreed/khocr-gen.git
24
+ cd khocr-gen
25
+ uv sync
26
+
27
+ # Or install with pip
28
+ pip install .
29
+ ```
30
+
31
+ `uv sync` also compiles and installs the optional Rust acceleration extension
32
+ (`khocr-gen-core`, from `rust/`). This requires a Rust toolchain (`cargo`/`rustc`) on
33
+ `PATH` — see [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md). If the extension isn't
34
+ available, khocr-gen falls back to pure-Python implementations automatically.
35
+
36
+ ### Requirements
37
+
38
+ - Python >= 3.12
39
+ - Rust toolchain (`cargo`/`rustc`, edition 2021) — only needed to build the optional
40
+ acceleration extension; the project works without it
41
+
42
+ ## Quick Start
43
+
44
+ ### 1. Prepare fonts
45
+
46
+ ```text
47
+ fonts/
48
+ ├── khmer/ <- .ttf / .otf Khmer fonts
49
+ └── english/ <- .ttf / .otf English fonts
50
+ ```
51
+
52
+ Fonts placed directly in `fonts/` (not in a subdirectory) are added to both pools as fallbacks.
53
+
54
+ ### 2. Prepare corpus
55
+
56
+ A plain UTF-8 text file, one string per line:
57
+
58
+ ```text
59
+ សួស្តី
60
+ Hello World
61
+ ស្វាគមន៍ Welcome
62
+ ```
63
+
64
+ ### 3. Generate a dataset
65
+
66
+ ```bash
67
+ khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb
68
+ ```
69
+
70
+ ### 4. Verify augmentation quality
71
+
72
+ Renders every augmentation method at min and max intensity on clean canvases:
73
+
74
+ ```bash
75
+ khocr-gen verify --fonts fonts --output verify_output
76
+ ```
77
+
78
+ ### 5. View generated images
79
+
80
+ ```bash
81
+ # Summary of an LMDB database
82
+ khocr-gen view --lmdb data/train/lmdb
83
+
84
+ # Extract images
85
+ khocr-gen view --lmdb data/train/lmdb --output-dir extracted
86
+
87
+ # Show labels only
88
+ khocr-gen view --lmdb data/train/lmdb --labels-only
89
+ ```
90
+
91
+ ### 6. Combine multiple datasets
92
+
93
+ ```bash
94
+ khocr-gen combine data_run1 data_run2 data_run3 --output data_merged
95
+ ```
96
+
97
+ ## Configuration
98
+
99
+ All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML).
100
+ See [CONFIG.md](docs/CONFIG.md) for details.
101
+
102
+ ```bash
103
+ khocr-gen generate --config configs/generate.yml
104
+ ```
105
+
106
+ Config loading order (highest priority wins):
107
+ 1. argparse built-in defaults
108
+ 2. YAML config file values
109
+ 3. Explicit CLI flags
110
+
111
+ ## Augmentation
112
+
113
+ 24 methods in a unified registry (21 of them Rust-accelerated).
114
+ Each generated image receives exactly one augmentation, chosen probabilistically by weight.
115
+ See [AUGMENTATION.md](docs/AUGMENTATION.md) for the full catalog and visual examples.
116
+
117
+ ## CLI Reference
118
+
119
+ See [CLI_REFERENCE.md](docs/CLI_REFERENCE.md) for complete command documentation.
120
+
121
+ ## Rust Acceleration
122
+
123
+ See [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md) for what's accelerated, how the
124
+ native extension is built/installed, and how to iterate on the `rust/` crate.
125
+
126
+ ## Project Structure
127
+
128
+ ```text
129
+ src/khocr_gen/
130
+ ├── cli.py # CLI entry point
131
+ ├── config.py # GenerationConfig + AugMethodConfig dataclasses
132
+ ├── config_loader.py # YAML config loading
133
+ ├── corpus.py # Corpus loading / filtering / counting
134
+ ├── fonts.py # FontManager with script-aware font selection
135
+ ├── normalizer.py # Khmer text normalization wrapper
136
+ ├── rendering.py # ImageRenderer: text -> clean canvas -> augmentation
137
+ ├── augmentation.py # 24 augmentation methods (unified registry)
138
+ ├── _rust_accel.py # Native/pure-Python dispatch layer for the Rust extension
139
+ ├── parallel.py # Multiprocess rendering workers
140
+ ├── lmdb_pack.py # LMDB database packing
141
+ ├── combine.py # n-way dataset merge
142
+ ├── data_generator.py # DatasetGenerator orchestrator
143
+ ├── generate.py # generate CLI command
144
+ ├── verify.py # verify CLI command
145
+ ├── viewer.py # view CLI command
146
+ ├── combine_cmd.py # combine CLI command
147
+ └── errors.py # Exception hierarchy
148
+
149
+ rust/ # khocr-gen-core: PyO3/maturin native acceleration extension
150
+ ├── Cargo.toml
151
+ ├── pyproject.toml
152
+ └── src/
153
+ ├── lib.rs # PyO3 bindings + module registration
154
+ ├── augmentation.rs # Accelerated augmentation methods
155
+ ├── fonts.rs # FontManager + FontFace (cmap glyph lookup)
156
+ ├── rendering.rs # Unwired canvas-rendering prototype (not yet bound to Python)
157
+ └── utils.rs
158
+ ```
159
+
160
+ ## Acknowledgments
161
+
162
+ khocr-gen builds on excellent open-source libraries:
163
+
164
+ - [khmernormalizer](https://github.com/seanghay/khmernormalizer) (MIT) - Khmer text normalization
165
+ - [OpenCV](https://opencv.org/) (Apache 2.0) - Image processing and augmentation
166
+ - [Pillow](https://python-pillow.org/) (HPND) - Font rendering and image creation
167
+ - [Albumentations](https://albumentations.ai/) (MIT) - Augmentation primitives
168
+ - [NumPy](https://numpy.org/) (BSD-3-Clause) - Numerical operations
169
+ - [LMDB](https://www.symas.com/lmdb) (OpenLDAP 2.8) - Embedded database for training pipelines
170
+
171
+ ## License
172
+
173
+ MIT
@@ -0,0 +1,242 @@
1
+ # Augmentation Reference
2
+
3
+ `khocr-gen` provides 24 augmentation methods in a unified registry.
4
+ Each generated image receives **exactly one** augmentation applied to a clean rendered canvas; effects are never stacked.
5
+
6
+ 21 of the 24 methods run through a native Rust extension when available (falling back to
7
+ pure Python/OpenCV otherwise) — `distortion`, `albu_noise`, and `gradient_illumination`
8
+ are the exceptions. See [RUST_ACCELERATION.md](RUST_ACCELERATION.md) for details.
9
+
10
+ ## Architecture
11
+
12
+ ```text
13
+ Clean canvas -> pick 1 method (weighted by prob) -> sample intensity ∈ [min, max] -> apply -> output
14
+ ```
15
+
16
+ ### Selection
17
+
18
+ From all methods with `prob > 0`, one is chosen with probability proportional to its `prob` weight:
19
+
20
+ ```
21
+ P(method_i) = prob_i / Σ prob_j (for all enabled methods j)
22
+ ```
23
+
24
+ ### Intensity
25
+
26
+ Once a method is chosen, its intensity is sampled uniformly from its configured `[min, max]` range (both in [0, 1]).
27
+ The method maps this to physical units:
28
+
29
+ | Method | intensity -> physical |
30
+ |--------|---------------------|
31
+ | blur | kernel size [3, 15] |
32
+ | rotation | degrees [0.5, 8.0] |
33
+ | salt_pepper | pixel density [0.001, 0.04] |
34
+ | jpeg_compression | quality [95, 25] (inverted) |
35
+ | sauvola | k value [0.05, 0.50] |
36
+ | geo_warp | corner displacement [0.5, 12] px |
37
+ | lowdpi | downsample ratio [0.15, 0.80] |
38
+
39
+ ---
40
+
41
+ ## Augmentation Methods (24 methods)
42
+
43
+ All methods are applied in isolation; one effect per image, chosen probabilistically from the unified registry.
44
+
45
+ ### `sauvola`: Sauvola Local Threshold Degradation
46
+
47
+ Simulates degraded binarization artifacts common in scanned documents.
48
+
49
+ - **Intensity -> k parameter** [0.05, 0.50]
50
+ - Uses adaptive window (25 px, capped by image size)
51
+ - Blends binary result with original via weighted alpha
52
+
53
+ ### `geo_warp`: Geometric Perspective Warp
54
+
55
+ 4-point perspective distortion simulating camera angle or page curl.
56
+
57
+ - **Intensity -> corner displacement** [0.5, 12] px
58
+ - Background color estimated from image border
59
+ - Border filled with estimated background
60
+
61
+ ### `vertical_crop`: Vertical Crop
62
+
63
+ Crops and pads top or bottom edge, simulating misaligned scanning.
64
+
65
+ - **Intensity -> crop amount** [1, 8] px
66
+ - Crops from foreground region edge
67
+ - Pads with estimated background color
68
+
69
+ ### `blur`: Blur (Motion/Median/Gaussian)
70
+
71
+ Uses Albumentations `OneOf` pipeline (falls back to OpenCV GaussianBlur).
72
+
73
+ - **Intensity -> kernel size** [3, 15] (odd)
74
+ - Random choice: MotionBlur, MedianBlur, or GaussianBlur
75
+ - Pure OpenCV fallback if Albumentations unavailable
76
+
77
+ ### `distortion`: Distortion (Optical/Grid/Elastic)
78
+
79
+ Uses Albumentations `OneOf` pipeline (no-op fallback without Albumentations).
80
+
81
+ - **Intensity -> distort_limit** [0.05, 0.30]
82
+ - Random choice: OpticalDistortion, GridDistortion, or ElasticTransform
83
+ - Grid uses 5 steps; Elastic uses 20 sigma
84
+
85
+ ### `albu_noise`: Noise (Gaussian/Multiplicative)
86
+
87
+ Uses Albumentations `OneOf` pipeline (OpenCV fallback available).
88
+
89
+ - **Intensity -> std deviation** [0.005, 0.20] normalized units
90
+ - Random choice: GaussNoise or MultiplicativeNoise
91
+ - Fallback: additive Gaussian noise σ [3, 25]
92
+
93
+ ### `jpeg_compression`: JPEG Compression Artifacts
94
+
95
+ Encodes and decodes the image at configurable quality levels.
96
+
97
+ - **Intensity -> JPEG quality** [95, 25] (inverted: higher intensity = lower quality)
98
+ - Uses cv2.imencode/imdecode round-trip
99
+ - Handles both grayscale and RGB
100
+
101
+ ### `rotation`: Random Rotation
102
+
103
+ Small-angle rotation with border fill.
104
+
105
+ - **Intensity -> max degrees** [0.5, 8.0]
106
+ - Direction: random clockwise or counter-clockwise
107
+ - Border filled with estimated background color
108
+
109
+ ### `salt_pepper`: Salt-and-Pepper Noise
110
+
111
+ Impulse noise: random white (salt) and black (pepper) pixels.
112
+
113
+ - **Intensity -> pixel density** [0.001, 0.04]
114
+ - Equal distribution of salt and pepper
115
+ - Works on both grayscale and RGB
116
+
117
+ ### `background_texture`: Background Texture Overlay
118
+
119
+ Procedural paper-like texture with three modes:
120
+
121
+ 1. **Fine grain**: per-pixel Gaussian noise
122
+ 2. **Coarse blotches**: low-resolution noise upscaled and blurred
123
+ 3. **Streaks**: horizontal line artifacts (scanner streaks)
124
+
125
+ - **Intensity -> blend alpha** [0.05, 0.30]
126
+ - Texture suppressed over dark ink regions (preserves text legibility)
127
+
128
+ ### `lowdpi`: Low-DPI Simulation
129
+
130
+ Downscale-then-upscale using NEAREST for downsample and LANCZOS4 for upsample.
131
+
132
+ - **Intensity -> downsample ratio** [0.15, 0.80] (inverted: higher = smaller = lower DPI)
133
+ - Minimum output dimension: 2 px
134
+
135
+ ### `oversample`: Oversample Rendering
136
+
137
+ Post-hoc sharpening filter (the rendering-mode oversample is handled in
138
+ the renderer, not here).
139
+
140
+ - **Intensity -> sharpen strength** [1.0, 1.5]
141
+ - Uses a 3×3 unsharp mask kernel
142
+
143
+ ---
144
+
145
+ ## Training-time Augmentations
146
+
147
+ These methods simulate augmentations typically applied during OCR training.
148
+ They are disabled by default (`prob: 0.0`) but can be enabled to make generated data more closely match training-time conditions.
149
+
150
+ ### `perspective`: Online Perspective Warp
151
+
152
+ Mild perspective distortion via OpenCV `getPerspectiveTransform`.
153
+
154
+ - **Intensity -> max displacement fraction** [0.02, 0.12] of image size
155
+ - Border filled with gray (128)
156
+
157
+ ### `elastic`: Online Elastic Distortion
158
+
159
+ Displacement-field-based elastic deformation.
160
+
161
+ - **Intensity -> displacement sigma** [0.02×size, 0.12×size]
162
+ - Coarse grid scaled to full resolution
163
+
164
+ ### `random_crop`: Online Random Height Crop
165
+
166
+ Crop top or bottom and resize back to original dimensions.
167
+
168
+ - **Intensity -> crop fraction** [0.01, 0.08] of height
169
+
170
+ ### `online_blur`: Online Blur
171
+
172
+ Gaussian or Motion blur.
173
+
174
+ - **Intensity -> kernel size** [3, 7] (odd)
175
+ - 50% chance of MotionBlur if Albumentations available
176
+
177
+ ### `online_noise`: Online Gaussian Noise
178
+
179
+ Additive Gaussian noise.
180
+
181
+ - **Intensity -> noise sigma** [3, 30]
182
+
183
+ ### `hsv`: HSV Color Jitter
184
+
185
+ Saturation and value adjustments (grayscale images pass through unchanged).
186
+
187
+ - **Intensity -> jitter factor** [0.60, 0.95]
188
+ - Independent random multiplier for S and V channels
189
+
190
+ ### `reverse`: Color Reversal
191
+
192
+ Inverts pixel values (dark <-> light). Common for handling negative/white-on-black text.
193
+
194
+ - **Intensity ignored**: binary on/off effect
195
+ - `255 - pixel` for every pixel
196
+
197
+ ### `brightness_contrast`: Brightness/Contrast Jitter
198
+
199
+ Alpha (contrast) and beta (brightness) adjustments via `cv2.convertScaleAbs`.
200
+
201
+ - **Intensity -> contrast spread** [0.02, 0.25], brightness [0, ±30]
202
+
203
+ ### `pixelation`: Pixelation
204
+
205
+ Downscale then upscale to produce blocky/pixelated artifacts.
206
+
207
+ - **Intensity -> scale fraction** [0.2, 0.9] (inverted: higher = more pixelated)
208
+ - Random interpolation: NEAREST or LINEAR
209
+
210
+ ### `gradient_illumination`: Gradient Illumination
211
+
212
+ Linear gradient overlay simulating uneven lighting.
213
+
214
+ - **Intensity -> gradient strength** [0.1, 0.6]
215
+ - Random direction: horizontal or vertical, left-to-right or right-to-left
216
+
217
+ ### `morphological`: Morphological Operations
218
+
219
+ Erosion or dilation with a small kernel.
220
+
221
+ - **Intensity 0.0–0.5 -> kernel size 2**; **0.5–1.0 -> kernel size 3**
222
+ - Khmer-aware mode: 70% erode, 30% horizontal dilate
223
+ - English mode: 50/50 erode/dilate with square kernel
224
+
225
+ ### `anisotropic_dilation`: Anisotropic Dilation
226
+
227
+ Directional dilation simulating dot-matrix printer spread.
228
+
229
+ - **Intensity 0.0–0.5 -> kernel size 2**; **0.5–1.0 -> kernel size 3 or 4**
230
+ - Random direction: horizontal or vertical kernel
231
+
232
+ ---
233
+
234
+ ## Verify Command
235
+
236
+ To see every method at min and max intensity on a clean canvas:
237
+
238
+ ```bash
239
+ khocr-gen verify --fonts fonts --output verify_output
240
+ ```
241
+
242
+ This produces one PNG per method showing the clean canvas with the method applied at intensity 0.0 (left half) and 1.0 (right half).