khocr-gen 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- khocr_gen-0.1.0/.gitignore +8 -0
- khocr_gen-0.1.0/CHANGELOG.md +29 -0
- khocr_gen-0.1.0/LICENSE +21 -0
- khocr_gen-0.1.0/PKG-INFO +213 -0
- khocr_gen-0.1.0/README.md +173 -0
- khocr_gen-0.1.0/docs/AUGMENTATION.md +242 -0
- khocr_gen-0.1.0/docs/CLI_REFERENCE.md +266 -0
- khocr_gen-0.1.0/docs/CONFIG.md +254 -0
- khocr_gen-0.1.0/docs/RUST_ACCELERATION.md +125 -0
- khocr_gen-0.1.0/pyproject.toml +124 -0
- khocr_gen-0.1.0/rust/Cargo.lock +605 -0
- khocr_gen-0.1.0/rust/Cargo.toml +30 -0
- khocr_gen-0.1.0/rust/pyproject.toml +13 -0
- khocr_gen-0.1.0/rust/src/augmentation.rs +1186 -0
- khocr_gen-0.1.0/rust/src/fonts.rs +324 -0
- khocr_gen-0.1.0/rust/src/lib.rs +376 -0
- khocr_gen-0.1.0/rust/src/rendering.rs +524 -0
- khocr_gen-0.1.0/rust/src/utils.rs +125 -0
- khocr_gen-0.1.0/src/khocr_gen/__init__.py +8 -0
- khocr_gen-0.1.0/src/khocr_gen/_pil_compat.py +25 -0
- khocr_gen-0.1.0/src/khocr_gen/_rust_accel.py +296 -0
- khocr_gen-0.1.0/src/khocr_gen/augmentation.py +753 -0
- khocr_gen-0.1.0/src/khocr_gen/cli.py +208 -0
- khocr_gen-0.1.0/src/khocr_gen/combine.py +179 -0
- khocr_gen-0.1.0/src/khocr_gen/combine_cmd.py +98 -0
- khocr_gen-0.1.0/src/khocr_gen/config.py +693 -0
- khocr_gen-0.1.0/src/khocr_gen/config_loader.py +149 -0
- khocr_gen-0.1.0/src/khocr_gen/corpus.py +232 -0
- khocr_gen-0.1.0/src/khocr_gen/data_generator.py +613 -0
- khocr_gen-0.1.0/src/khocr_gen/errors.py +33 -0
- khocr_gen-0.1.0/src/khocr_gen/fonts.py +181 -0
- khocr_gen-0.1.0/src/khocr_gen/generate.py +187 -0
- khocr_gen-0.1.0/src/khocr_gen/lmdb_pack.py +111 -0
- khocr_gen-0.1.0/src/khocr_gen/logging.py +25 -0
- khocr_gen-0.1.0/src/khocr_gen/normalizer.py +217 -0
- khocr_gen-0.1.0/src/khocr_gen/parallel.py +206 -0
- khocr_gen-0.1.0/src/khocr_gen/rendering.py +539 -0
- khocr_gen-0.1.0/src/khocr_gen/verify.py +288 -0
- khocr_gen-0.1.0/src/khocr_gen/viewer.py +168 -0
- khocr_gen-0.1.0/tests/__init__.py +0 -0
- khocr_gen-0.1.0/tests/test_augmentation.py +421 -0
- khocr_gen-0.1.0/tests/test_combine.py +46 -0
- khocr_gen-0.1.0/tests/test_config.py +253 -0
- khocr_gen-0.1.0/tests/test_config_loader.py +168 -0
- khocr_gen-0.1.0/tests/test_corpus.py +203 -0
- khocr_gen-0.1.0/tests/test_errors.py +75 -0
- khocr_gen-0.1.0/tests/test_fonts.py +133 -0
- khocr_gen-0.1.0/tests/test_lmdb_pack.py +31 -0
- khocr_gen-0.1.0/tests/test_normalizer.py +103 -0
- khocr_gen-0.1.0/tests/test_online_aug.py +82 -0
- khocr_gen-0.1.0/tests/test_parallel.py +113 -0
- khocr_gen-0.1.0/typings/_rust_accel.pyi +56 -0
|
@@ -0,0 +1,29 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [0.1.0] - Unreleased
|
|
9
|
+
|
|
10
|
+
### Added
|
|
11
|
+
- Initial release: synthetic OCR dataset generation for Khmer/English text
|
|
12
|
+
- `generate` command: render text corpus to training images with augmentations
|
|
13
|
+
- `combine` command: merge multiple datasets into one
|
|
14
|
+
- `verify` command: render augmentation comparison grids for QA
|
|
15
|
+
- `view` command: preview LMDB dataset contents
|
|
16
|
+
- Configurable augmentation pipeline with min/max/prob per method
|
|
17
|
+
- Multi-process parallel rendering
|
|
18
|
+
- LMDB output packing
|
|
19
|
+
- Khmer text normalization via khmernormalizer
|
|
20
|
+
- Rust acceleration extension (`khocr-gen-core`, in `rust/`): native implementations of
|
|
21
|
+
21/24 augmentation methods, O(1) cmap-based font glyph checking (`FontFace`), and
|
|
22
|
+
script-span splitting, with automatic pure-Python fallback when unavailable. Built and
|
|
23
|
+
installed automatically by `uv sync` via a `[tool.uv.sources]` path dependency. See
|
|
24
|
+
[docs/RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md).
|
|
25
|
+
|
|
26
|
+
### Fixed
|
|
27
|
+
- `ImageRenderer._render_mixed_font` passed a `(text, script)` tuple to `draw.text()`
|
|
28
|
+
instead of the span's text string, which would have crashed any mixed-font render
|
|
29
|
+
(dead code path prior to this fix, since `mixed_font_prob` defaults to `0.0`).
|
khocr_gen-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Kosal Seng
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
khocr_gen-0.1.0/PKG-INFO
ADDED
|
@@ -0,0 +1,213 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: khocr-gen
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Synthetic OCR training data generator for mixed Khmer/English text
|
|
5
|
+
Project-URL: Homepage, https://github.com/LazyGreed/khocr-gen
|
|
6
|
+
Project-URL: Repository, https://github.com/LazyGreed/khocr-gen
|
|
7
|
+
Project-URL: Issues, https://github.com/LazyGreed/khocr-gen/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/LazyGreed/khocr-gen/blob/main/CHANGELOG.md
|
|
9
|
+
Author: Kosal Seng
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: data-generation,deep-learning,image-generation,khmer,ocr,synthetic-data,text-recognition
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Intended Audience :: Science/Research
|
|
16
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
|
|
21
|
+
Classifier: Topic :: Scientific/Engineering :: Image Recognition
|
|
22
|
+
Classifier: Topic :: Text Processing :: Fonts
|
|
23
|
+
Requires-Python: >=3.12
|
|
24
|
+
Requires-Dist: albumentations>=1.3.0
|
|
25
|
+
Requires-Dist: khmernormalizer>=0.0.4
|
|
26
|
+
Requires-Dist: lmdb>=1.4.0
|
|
27
|
+
Requires-Dist: numpy>=1.24.0
|
|
28
|
+
Requires-Dist: opencv-python-headless>=4.8.0
|
|
29
|
+
Requires-Dist: pillow>=10.0.0
|
|
30
|
+
Requires-Dist: pyyaml>=6.0
|
|
31
|
+
Requires-Dist: tqdm>=4.64.0
|
|
32
|
+
Provides-Extra: dev
|
|
33
|
+
Requires-Dist: khocr-gen-core; extra == 'dev'
|
|
34
|
+
Requires-Dist: pytest-cov>=4.0; extra == 'dev'
|
|
35
|
+
Requires-Dist: pytest>=7.0; extra == 'dev'
|
|
36
|
+
Requires-Dist: ruff>=0.1.0; extra == 'dev'
|
|
37
|
+
Provides-Extra: rust
|
|
38
|
+
Requires-Dist: khocr-gen-core; extra == 'rust'
|
|
39
|
+
Description-Content-Type: text/markdown
|
|
40
|
+
|
|
41
|
+
# khocr-gen
|
|
42
|
+
|
|
43
|
+
Synthetic OCR training data generator for mixed Khmer/English text.
|
|
44
|
+
|
|
45
|
+
## Features
|
|
46
|
+
|
|
47
|
+
- **Mixed-script rendering:** per-span font selection for Khmer + English
|
|
48
|
+
- **24 augmentation methods:** unified registry covering scanner/camera degradations and training-time transforms
|
|
49
|
+
- **Rust acceleration:** 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
|
|
50
|
+
- **Isolated augmentation:** one effect per image, weighted by configurable probabilities
|
|
51
|
+
- **Configurable intensity ranges:** per-method `[min, max]` with linear sampling
|
|
52
|
+
- **Multiprocess generation:** parallel workers for throughput
|
|
53
|
+
- **LMDB packing:** efficient key–value storage for training pipelines
|
|
54
|
+
- **n-way dataset combine:** merge multiple generated datasets into one
|
|
55
|
+
- **Built-in viewer:** preview and extract images from LMDB databases
|
|
56
|
+
- **YAML config files:** reproducible generation recipes
|
|
57
|
+
- **Khmer normalization:** canonical character ordering via `khmernormalizer`
|
|
58
|
+
|
|
59
|
+
## Installation
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
# Clone and install with uv
|
|
63
|
+
git clone https://github.com/LazyGreed/khocr-gen.git
|
|
64
|
+
cd khocr-gen
|
|
65
|
+
uv sync
|
|
66
|
+
|
|
67
|
+
# Or install with pip
|
|
68
|
+
pip install .
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
`uv sync` also compiles and installs the optional Rust acceleration extension
|
|
72
|
+
(`khocr-gen-core`, from `rust/`). This requires a Rust toolchain (`cargo`/`rustc`) on
|
|
73
|
+
`PATH` — see [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md). If the extension isn't
|
|
74
|
+
available, khocr-gen falls back to pure-Python implementations automatically.
|
|
75
|
+
|
|
76
|
+
### Requirements
|
|
77
|
+
|
|
78
|
+
- Python >= 3.12
|
|
79
|
+
- Rust toolchain (`cargo`/`rustc`, edition 2021) — only needed to build the optional
|
|
80
|
+
acceleration extension; the project works without it
|
|
81
|
+
|
|
82
|
+
## Quick Start
|
|
83
|
+
|
|
84
|
+
### 1. Prepare fonts
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
fonts/
|
|
88
|
+
├── khmer/ <- .ttf / .otf Khmer fonts
|
|
89
|
+
└── english/ <- .ttf / .otf English fonts
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Fonts placed directly in `fonts/` (not in a subdirectory) are added to both pools as fallbacks.
|
|
93
|
+
|
|
94
|
+
### 2. Prepare corpus
|
|
95
|
+
|
|
96
|
+
A plain UTF-8 text file, one string per line:
|
|
97
|
+
|
|
98
|
+
```text
|
|
99
|
+
សួស្តី
|
|
100
|
+
Hello World
|
|
101
|
+
ស្វាគមន៍ Welcome
|
|
102
|
+
```
|
|
103
|
+
|
|
104
|
+
### 3. Generate a dataset
|
|
105
|
+
|
|
106
|
+
```bash
|
|
107
|
+
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb
|
|
108
|
+
```
|
|
109
|
+
|
|
110
|
+
### 4. Verify augmentation quality
|
|
111
|
+
|
|
112
|
+
Renders every augmentation method at min and max intensity on clean canvases:
|
|
113
|
+
|
|
114
|
+
```bash
|
|
115
|
+
khocr-gen verify --fonts fonts --output verify_output
|
|
116
|
+
```
|
|
117
|
+
|
|
118
|
+
### 5. View generated images
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
# Summary of an LMDB database
|
|
122
|
+
khocr-gen view --lmdb data/train/lmdb
|
|
123
|
+
|
|
124
|
+
# Extract images
|
|
125
|
+
khocr-gen view --lmdb data/train/lmdb --output-dir extracted
|
|
126
|
+
|
|
127
|
+
# Show labels only
|
|
128
|
+
khocr-gen view --lmdb data/train/lmdb --labels-only
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
### 6. Combine multiple datasets
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
khocr-gen combine data_run1 data_run2 data_run3 --output data_merged
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
## Configuration
|
|
138
|
+
|
|
139
|
+
All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML).
|
|
140
|
+
See [CONFIG.md](docs/CONFIG.md) for details.
|
|
141
|
+
|
|
142
|
+
```bash
|
|
143
|
+
khocr-gen generate --config configs/generate.yml
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
Config loading order (highest priority wins):
|
|
147
|
+
1. argparse built-in defaults
|
|
148
|
+
2. YAML config file values
|
|
149
|
+
3. Explicit CLI flags
|
|
150
|
+
|
|
151
|
+
## Augmentation
|
|
152
|
+
|
|
153
|
+
24 methods in a unified registry (21 of them Rust-accelerated).
|
|
154
|
+
Each generated image receives exactly one augmentation, chosen probabilistically by weight.
|
|
155
|
+
See [AUGMENTATION.md](docs/AUGMENTATION.md) for the full catalog and visual examples.
|
|
156
|
+
|
|
157
|
+
## CLI Reference
|
|
158
|
+
|
|
159
|
+
See [CLI_REFERENCE.md](docs/CLI_REFERENCE.md) for complete command documentation.
|
|
160
|
+
|
|
161
|
+
## Rust Acceleration
|
|
162
|
+
|
|
163
|
+
See [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md) for what's accelerated, how the
|
|
164
|
+
native extension is built/installed, and how to iterate on the `rust/` crate.
|
|
165
|
+
|
|
166
|
+
## Project Structure
|
|
167
|
+
|
|
168
|
+
```text
|
|
169
|
+
src/khocr_gen/
|
|
170
|
+
├── cli.py # CLI entry point
|
|
171
|
+
├── config.py # GenerationConfig + AugMethodConfig dataclasses
|
|
172
|
+
├── config_loader.py # YAML config loading
|
|
173
|
+
├── corpus.py # Corpus loading / filtering / counting
|
|
174
|
+
├── fonts.py # FontManager with script-aware font selection
|
|
175
|
+
├── normalizer.py # Khmer text normalization wrapper
|
|
176
|
+
├── rendering.py # ImageRenderer: text -> clean canvas -> augmentation
|
|
177
|
+
├── augmentation.py # 24 augmentation methods (unified registry)
|
|
178
|
+
├── _rust_accel.py # Native/pure-Python dispatch layer for the Rust extension
|
|
179
|
+
├── parallel.py # Multiprocess rendering workers
|
|
180
|
+
├── lmdb_pack.py # LMDB database packing
|
|
181
|
+
├── combine.py # n-way dataset merge
|
|
182
|
+
├── data_generator.py # DatasetGenerator orchestrator
|
|
183
|
+
├── generate.py # generate CLI command
|
|
184
|
+
├── verify.py # verify CLI command
|
|
185
|
+
├── viewer.py # view CLI command
|
|
186
|
+
├── combine_cmd.py # combine CLI command
|
|
187
|
+
└── errors.py # Exception hierarchy
|
|
188
|
+
|
|
189
|
+
rust/ # khocr-gen-core: PyO3/maturin native acceleration extension
|
|
190
|
+
├── Cargo.toml
|
|
191
|
+
├── pyproject.toml
|
|
192
|
+
└── src/
|
|
193
|
+
├── lib.rs # PyO3 bindings + module registration
|
|
194
|
+
├── augmentation.rs # Accelerated augmentation methods
|
|
195
|
+
├── fonts.rs # FontManager + FontFace (cmap glyph lookup)
|
|
196
|
+
├── rendering.rs # Unwired canvas-rendering prototype (not yet bound to Python)
|
|
197
|
+
└── utils.rs
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
## Acknowledgments
|
|
201
|
+
|
|
202
|
+
khocr-gen builds on excellent open-source libraries:
|
|
203
|
+
|
|
204
|
+
- [khmernormalizer](https://github.com/seanghay/khmernormalizer) (MIT) - Khmer text normalization
|
|
205
|
+
- [OpenCV](https://opencv.org/) (Apache 2.0) - Image processing and augmentation
|
|
206
|
+
- [Pillow](https://python-pillow.org/) (HPND) - Font rendering and image creation
|
|
207
|
+
- [Albumentations](https://albumentations.ai/) (MIT) - Augmentation primitives
|
|
208
|
+
- [NumPy](https://numpy.org/) (BSD-3-Clause) - Numerical operations
|
|
209
|
+
- [LMDB](https://www.symas.com/lmdb) (OpenLDAP 2.8) - Embedded database for training pipelines
|
|
210
|
+
|
|
211
|
+
## License
|
|
212
|
+
|
|
213
|
+
MIT
|
|
@@ -0,0 +1,173 @@
|
|
|
1
|
+
# khocr-gen
|
|
2
|
+
|
|
3
|
+
Synthetic OCR training data generator for mixed Khmer/English text.
|
|
4
|
+
|
|
5
|
+
## Features
|
|
6
|
+
|
|
7
|
+
- **Mixed-script rendering:** per-span font selection for Khmer + English
|
|
8
|
+
- **24 augmentation methods:** unified registry covering scanner/camera degradations and training-time transforms
|
|
9
|
+
- **Rust acceleration:** 21/24 augmentation methods plus font glyph checking run through a native PyO3 extension, with automatic pure-Python fallback
|
|
10
|
+
- **Isolated augmentation:** one effect per image, weighted by configurable probabilities
|
|
11
|
+
- **Configurable intensity ranges:** per-method `[min, max]` with linear sampling
|
|
12
|
+
- **Multiprocess generation:** parallel workers for throughput
|
|
13
|
+
- **LMDB packing:** efficient key–value storage for training pipelines
|
|
14
|
+
- **n-way dataset combine:** merge multiple generated datasets into one
|
|
15
|
+
- **Built-in viewer:** preview and extract images from LMDB databases
|
|
16
|
+
- **YAML config files:** reproducible generation recipes
|
|
17
|
+
- **Khmer normalization:** canonical character ordering via `khmernormalizer`
|
|
18
|
+
|
|
19
|
+
## Installation
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
# Clone and install with uv
|
|
23
|
+
git clone https://github.com/LazyGreed/khocr-gen.git
|
|
24
|
+
cd khocr-gen
|
|
25
|
+
uv sync
|
|
26
|
+
|
|
27
|
+
# Or install with pip
|
|
28
|
+
pip install .
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
`uv sync` also compiles and installs the optional Rust acceleration extension
|
|
32
|
+
(`khocr-gen-core`, from `rust/`). This requires a Rust toolchain (`cargo`/`rustc`) on
|
|
33
|
+
`PATH` — see [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md). If the extension isn't
|
|
34
|
+
available, khocr-gen falls back to pure-Python implementations automatically.
|
|
35
|
+
|
|
36
|
+
### Requirements
|
|
37
|
+
|
|
38
|
+
- Python >= 3.12
|
|
39
|
+
- Rust toolchain (`cargo`/`rustc`, edition 2021) — only needed to build the optional
|
|
40
|
+
acceleration extension; the project works without it
|
|
41
|
+
|
|
42
|
+
## Quick Start
|
|
43
|
+
|
|
44
|
+
### 1. Prepare fonts
|
|
45
|
+
|
|
46
|
+
```text
|
|
47
|
+
fonts/
|
|
48
|
+
├── khmer/ <- .ttf / .otf Khmer fonts
|
|
49
|
+
└── english/ <- .ttf / .otf English fonts
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Fonts placed directly in `fonts/` (not in a subdirectory) are added to both pools as fallbacks.
|
|
53
|
+
|
|
54
|
+
### 2. Prepare corpus
|
|
55
|
+
|
|
56
|
+
A plain UTF-8 text file, one string per line:
|
|
57
|
+
|
|
58
|
+
```text
|
|
59
|
+
សួស្តី
|
|
60
|
+
Hello World
|
|
61
|
+
ស្វាគមន៍ Welcome
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
### 3. Generate a dataset
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
khocr-gen generate --corpus corpus/corpus.txt --fonts fonts --output data --copies 3 --storage lmdb
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
### 4. Verify augmentation quality
|
|
71
|
+
|
|
72
|
+
Renders every augmentation method at min and max intensity on clean canvases:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
khocr-gen verify --fonts fonts --output verify_output
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
### 5. View generated images
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
# Summary of an LMDB database
|
|
82
|
+
khocr-gen view --lmdb data/train/lmdb
|
|
83
|
+
|
|
84
|
+
# Extract images
|
|
85
|
+
khocr-gen view --lmdb data/train/lmdb --output-dir extracted
|
|
86
|
+
|
|
87
|
+
# Show labels only
|
|
88
|
+
khocr-gen view --lmdb data/train/lmdb --labels-only
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
### 6. Combine multiple datasets
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
khocr-gen combine data_run1 data_run2 data_run3 --output data_merged
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Configuration
|
|
98
|
+
|
|
99
|
+
All generation parameters can be specified via CLI flags, a YAML config file, or both (CLI overrides YAML).
|
|
100
|
+
See [CONFIG.md](docs/CONFIG.md) for details.
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
khocr-gen generate --config configs/generate.yml
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
Config loading order (highest priority wins):
|
|
107
|
+
1. argparse built-in defaults
|
|
108
|
+
2. YAML config file values
|
|
109
|
+
3. Explicit CLI flags
|
|
110
|
+
|
|
111
|
+
## Augmentation
|
|
112
|
+
|
|
113
|
+
24 methods in a unified registry (21 of them Rust-accelerated).
|
|
114
|
+
Each generated image receives exactly one augmentation, chosen probabilistically by weight.
|
|
115
|
+
See [AUGMENTATION.md](docs/AUGMENTATION.md) for the full catalog and visual examples.
|
|
116
|
+
|
|
117
|
+
## CLI Reference
|
|
118
|
+
|
|
119
|
+
See [CLI_REFERENCE.md](docs/CLI_REFERENCE.md) for complete command documentation.
|
|
120
|
+
|
|
121
|
+
## Rust Acceleration
|
|
122
|
+
|
|
123
|
+
See [RUST_ACCELERATION.md](docs/RUST_ACCELERATION.md) for what's accelerated, how the
|
|
124
|
+
native extension is built/installed, and how to iterate on the `rust/` crate.
|
|
125
|
+
|
|
126
|
+
## Project Structure
|
|
127
|
+
|
|
128
|
+
```text
|
|
129
|
+
src/khocr_gen/
|
|
130
|
+
├── cli.py # CLI entry point
|
|
131
|
+
├── config.py # GenerationConfig + AugMethodConfig dataclasses
|
|
132
|
+
├── config_loader.py # YAML config loading
|
|
133
|
+
├── corpus.py # Corpus loading / filtering / counting
|
|
134
|
+
├── fonts.py # FontManager with script-aware font selection
|
|
135
|
+
├── normalizer.py # Khmer text normalization wrapper
|
|
136
|
+
├── rendering.py # ImageRenderer: text -> clean canvas -> augmentation
|
|
137
|
+
├── augmentation.py # 24 augmentation methods (unified registry)
|
|
138
|
+
├── _rust_accel.py # Native/pure-Python dispatch layer for the Rust extension
|
|
139
|
+
├── parallel.py # Multiprocess rendering workers
|
|
140
|
+
├── lmdb_pack.py # LMDB database packing
|
|
141
|
+
├── combine.py # n-way dataset merge
|
|
142
|
+
├── data_generator.py # DatasetGenerator orchestrator
|
|
143
|
+
├── generate.py # generate CLI command
|
|
144
|
+
├── verify.py # verify CLI command
|
|
145
|
+
├── viewer.py # view CLI command
|
|
146
|
+
├── combine_cmd.py # combine CLI command
|
|
147
|
+
└── errors.py # Exception hierarchy
|
|
148
|
+
|
|
149
|
+
rust/ # khocr-gen-core: PyO3/maturin native acceleration extension
|
|
150
|
+
├── Cargo.toml
|
|
151
|
+
├── pyproject.toml
|
|
152
|
+
└── src/
|
|
153
|
+
├── lib.rs # PyO3 bindings + module registration
|
|
154
|
+
├── augmentation.rs # Accelerated augmentation methods
|
|
155
|
+
├── fonts.rs # FontManager + FontFace (cmap glyph lookup)
|
|
156
|
+
├── rendering.rs # Unwired canvas-rendering prototype (not yet bound to Python)
|
|
157
|
+
└── utils.rs
|
|
158
|
+
```
|
|
159
|
+
|
|
160
|
+
## Acknowledgments
|
|
161
|
+
|
|
162
|
+
khocr-gen builds on excellent open-source libraries:
|
|
163
|
+
|
|
164
|
+
- [khmernormalizer](https://github.com/seanghay/khmernormalizer) (MIT) - Khmer text normalization
|
|
165
|
+
- [OpenCV](https://opencv.org/) (Apache 2.0) - Image processing and augmentation
|
|
166
|
+
- [Pillow](https://python-pillow.org/) (HPND) - Font rendering and image creation
|
|
167
|
+
- [Albumentations](https://albumentations.ai/) (MIT) - Augmentation primitives
|
|
168
|
+
- [NumPy](https://numpy.org/) (BSD-3-Clause) - Numerical operations
|
|
169
|
+
- [LMDB](https://www.symas.com/lmdb) (OpenLDAP 2.8) - Embedded database for training pipelines
|
|
170
|
+
|
|
171
|
+
## License
|
|
172
|
+
|
|
173
|
+
MIT
|
|
@@ -0,0 +1,242 @@
|
|
|
1
|
+
# Augmentation Reference
|
|
2
|
+
|
|
3
|
+
`khocr-gen` provides 24 augmentation methods in a unified registry.
|
|
4
|
+
Each generated image receives **exactly one** augmentation applied to a clean rendered canvas; effects are never stacked.
|
|
5
|
+
|
|
6
|
+
21 of the 24 methods run through a native Rust extension when available (falling back to
|
|
7
|
+
pure Python/OpenCV otherwise) — `distortion`, `albu_noise`, and `gradient_illumination`
|
|
8
|
+
are the exceptions. See [RUST_ACCELERATION.md](RUST_ACCELERATION.md) for details.
|
|
9
|
+
|
|
10
|
+
## Architecture
|
|
11
|
+
|
|
12
|
+
```text
|
|
13
|
+
Clean canvas -> pick 1 method (weighted by prob) -> sample intensity ∈ [min, max] -> apply -> output
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
### Selection
|
|
17
|
+
|
|
18
|
+
From all methods with `prob > 0`, one is chosen with probability proportional to its `prob` weight:
|
|
19
|
+
|
|
20
|
+
```
|
|
21
|
+
P(method_i) = prob_i / Σ prob_j (for all enabled methods j)
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
### Intensity
|
|
25
|
+
|
|
26
|
+
Once a method is chosen, its intensity is sampled uniformly from its configured `[min, max]` range (both in [0, 1]).
|
|
27
|
+
The method maps this to physical units:
|
|
28
|
+
|
|
29
|
+
| Method | intensity -> physical |
|
|
30
|
+
|--------|---------------------|
|
|
31
|
+
| blur | kernel size [3, 15] |
|
|
32
|
+
| rotation | degrees [0.5, 8.0] |
|
|
33
|
+
| salt_pepper | pixel density [0.001, 0.04] |
|
|
34
|
+
| jpeg_compression | quality [95, 25] (inverted) |
|
|
35
|
+
| sauvola | k value [0.05, 0.50] |
|
|
36
|
+
| geo_warp | corner displacement [0.5, 12] px |
|
|
37
|
+
| lowdpi | downsample ratio [0.15, 0.80] |
|
|
38
|
+
|
|
39
|
+
---
|
|
40
|
+
|
|
41
|
+
## Augmentation Methods (24 methods)
|
|
42
|
+
|
|
43
|
+
All methods are applied in isolation; one effect per image, chosen probabilistically from the unified registry.
|
|
44
|
+
|
|
45
|
+
### `sauvola`: Sauvola Local Threshold Degradation
|
|
46
|
+
|
|
47
|
+
Simulates degraded binarization artifacts common in scanned documents.
|
|
48
|
+
|
|
49
|
+
- **Intensity -> k parameter** [0.05, 0.50]
|
|
50
|
+
- Uses adaptive window (25 px, capped by image size)
|
|
51
|
+
- Blends binary result with original via weighted alpha
|
|
52
|
+
|
|
53
|
+
### `geo_warp`: Geometric Perspective Warp
|
|
54
|
+
|
|
55
|
+
4-point perspective distortion simulating camera angle or page curl.
|
|
56
|
+
|
|
57
|
+
- **Intensity -> corner displacement** [0.5, 12] px
|
|
58
|
+
- Background color estimated from image border
|
|
59
|
+
- Border filled with estimated background
|
|
60
|
+
|
|
61
|
+
### `vertical_crop`: Vertical Crop
|
|
62
|
+
|
|
63
|
+
Crops and pads top or bottom edge, simulating misaligned scanning.
|
|
64
|
+
|
|
65
|
+
- **Intensity -> crop amount** [1, 8] px
|
|
66
|
+
- Crops from foreground region edge
|
|
67
|
+
- Pads with estimated background color
|
|
68
|
+
|
|
69
|
+
### `blur`: Blur (Motion/Median/Gaussian)
|
|
70
|
+
|
|
71
|
+
Uses Albumentations `OneOf` pipeline (falls back to OpenCV GaussianBlur).
|
|
72
|
+
|
|
73
|
+
- **Intensity -> kernel size** [3, 15] (odd)
|
|
74
|
+
- Random choice: MotionBlur, MedianBlur, or GaussianBlur
|
|
75
|
+
- Pure OpenCV fallback if Albumentations unavailable
|
|
76
|
+
|
|
77
|
+
### `distortion`: Distortion (Optical/Grid/Elastic)
|
|
78
|
+
|
|
79
|
+
Uses Albumentations `OneOf` pipeline (no-op fallback without Albumentations).
|
|
80
|
+
|
|
81
|
+
- **Intensity -> distort_limit** [0.05, 0.30]
|
|
82
|
+
- Random choice: OpticalDistortion, GridDistortion, or ElasticTransform
|
|
83
|
+
- Grid uses 5 steps; Elastic uses 20 sigma
|
|
84
|
+
|
|
85
|
+
### `albu_noise`: Noise (Gaussian/Multiplicative)
|
|
86
|
+
|
|
87
|
+
Uses Albumentations `OneOf` pipeline (OpenCV fallback available).
|
|
88
|
+
|
|
89
|
+
- **Intensity -> std deviation** [0.005, 0.20] normalized units
|
|
90
|
+
- Random choice: GaussNoise or MultiplicativeNoise
|
|
91
|
+
- Fallback: additive Gaussian noise σ [3, 25]
|
|
92
|
+
|
|
93
|
+
### `jpeg_compression`: JPEG Compression Artifacts
|
|
94
|
+
|
|
95
|
+
Encodes and decodes the image at configurable quality levels.
|
|
96
|
+
|
|
97
|
+
- **Intensity -> JPEG quality** [95, 25] (inverted: higher intensity = lower quality)
|
|
98
|
+
- Uses cv2.imencode/imdecode round-trip
|
|
99
|
+
- Handles both grayscale and RGB
|
|
100
|
+
|
|
101
|
+
### `rotation`: Random Rotation
|
|
102
|
+
|
|
103
|
+
Small-angle rotation with border fill.
|
|
104
|
+
|
|
105
|
+
- **Intensity -> max degrees** [0.5, 8.0]
|
|
106
|
+
- Direction: random clockwise or counter-clockwise
|
|
107
|
+
- Border filled with estimated background color
|
|
108
|
+
|
|
109
|
+
### `salt_pepper`: Salt-and-Pepper Noise
|
|
110
|
+
|
|
111
|
+
Impulse noise: random white (salt) and black (pepper) pixels.
|
|
112
|
+
|
|
113
|
+
- **Intensity -> pixel density** [0.001, 0.04]
|
|
114
|
+
- Equal distribution of salt and pepper
|
|
115
|
+
- Works on both grayscale and RGB
|
|
116
|
+
|
|
117
|
+
### `background_texture`: Background Texture Overlay
|
|
118
|
+
|
|
119
|
+
Procedural paper-like texture with three modes:
|
|
120
|
+
|
|
121
|
+
1. **Fine grain**: per-pixel Gaussian noise
|
|
122
|
+
2. **Coarse blotches**: low-resolution noise upscaled and blurred
|
|
123
|
+
3. **Streaks**: horizontal line artifacts (scanner streaks)
|
|
124
|
+
|
|
125
|
+
- **Intensity -> blend alpha** [0.05, 0.30]
|
|
126
|
+
- Texture suppressed over dark ink regions (preserves text legibility)
|
|
127
|
+
|
|
128
|
+
### `lowdpi`: Low-DPI Simulation
|
|
129
|
+
|
|
130
|
+
Downscale-then-upscale using NEAREST for downsample and LANCZOS4 for upsample.
|
|
131
|
+
|
|
132
|
+
- **Intensity -> downsample ratio** [0.15, 0.80] (inverted: higher = smaller = lower DPI)
|
|
133
|
+
- Minimum output dimension: 2 px
|
|
134
|
+
|
|
135
|
+
### `oversample`: Oversample Rendering
|
|
136
|
+
|
|
137
|
+
Post-hoc sharpening filter (the rendering-mode oversample is handled in
|
|
138
|
+
the renderer, not here).
|
|
139
|
+
|
|
140
|
+
- **Intensity -> sharpen strength** [1.0, 1.5]
|
|
141
|
+
- Uses a 3×3 unsharp mask kernel
|
|
142
|
+
|
|
143
|
+
---
|
|
144
|
+
|
|
145
|
+
## Training-time Augmentations
|
|
146
|
+
|
|
147
|
+
These methods simulate augmentations typically applied during OCR training.
|
|
148
|
+
They are disabled by default (`prob: 0.0`) but can be enabled to make generated data more closely match training-time conditions.
|
|
149
|
+
|
|
150
|
+
### `perspective`: Online Perspective Warp
|
|
151
|
+
|
|
152
|
+
Mild perspective distortion via OpenCV `getPerspectiveTransform`.
|
|
153
|
+
|
|
154
|
+
- **Intensity -> max displacement fraction** [0.02, 0.12] of image size
|
|
155
|
+
- Border filled with gray (128)
|
|
156
|
+
|
|
157
|
+
### `elastic`: Online Elastic Distortion
|
|
158
|
+
|
|
159
|
+
Displacement-field-based elastic deformation.
|
|
160
|
+
|
|
161
|
+
- **Intensity -> displacement sigma** [0.02×size, 0.12×size]
|
|
162
|
+
- Coarse grid scaled to full resolution
|
|
163
|
+
|
|
164
|
+
### `random_crop`: Online Random Height Crop
|
|
165
|
+
|
|
166
|
+
Crop top or bottom and resize back to original dimensions.
|
|
167
|
+
|
|
168
|
+
- **Intensity -> crop fraction** [0.01, 0.08] of height
|
|
169
|
+
|
|
170
|
+
### `online_blur`: Online Blur
|
|
171
|
+
|
|
172
|
+
Gaussian or Motion blur.
|
|
173
|
+
|
|
174
|
+
- **Intensity -> kernel size** [3, 7] (odd)
|
|
175
|
+
- 50% chance of MotionBlur if Albumentations available
|
|
176
|
+
|
|
177
|
+
### `online_noise`: Online Gaussian Noise
|
|
178
|
+
|
|
179
|
+
Additive Gaussian noise.
|
|
180
|
+
|
|
181
|
+
- **Intensity -> noise sigma** [3, 30]
|
|
182
|
+
|
|
183
|
+
### `hsv`: HSV Color Jitter
|
|
184
|
+
|
|
185
|
+
Saturation and value adjustments (grayscale images pass through unchanged).
|
|
186
|
+
|
|
187
|
+
- **Intensity -> jitter factor** [0.60, 0.95]
|
|
188
|
+
- Independent random multiplier for S and V channels
|
|
189
|
+
|
|
190
|
+
### `reverse`: Color Reversal
|
|
191
|
+
|
|
192
|
+
Inverts pixel values (dark <-> light). Common for handling negative/white-on-black text.
|
|
193
|
+
|
|
194
|
+
- **Intensity ignored**: binary on/off effect
|
|
195
|
+
- `255 - pixel` for every pixel
|
|
196
|
+
|
|
197
|
+
### `brightness_contrast`: Brightness/Contrast Jitter
|
|
198
|
+
|
|
199
|
+
Alpha (contrast) and beta (brightness) adjustments via `cv2.convertScaleAbs`.
|
|
200
|
+
|
|
201
|
+
- **Intensity -> contrast spread** [0.02, 0.25], brightness [0, ±30]
|
|
202
|
+
|
|
203
|
+
### `pixelation`: Pixelation
|
|
204
|
+
|
|
205
|
+
Downscale then upscale to produce blocky/pixelated artifacts.
|
|
206
|
+
|
|
207
|
+
- **Intensity -> scale fraction** [0.2, 0.9] (inverted: higher = more pixelated)
|
|
208
|
+
- Random interpolation: NEAREST or LINEAR
|
|
209
|
+
|
|
210
|
+
### `gradient_illumination`: Gradient Illumination
|
|
211
|
+
|
|
212
|
+
Linear gradient overlay simulating uneven lighting.
|
|
213
|
+
|
|
214
|
+
- **Intensity -> gradient strength** [0.1, 0.6]
|
|
215
|
+
- Random direction: horizontal or vertical, left-to-right or right-to-left
|
|
216
|
+
|
|
217
|
+
### `morphological`: Morphological Operations
|
|
218
|
+
|
|
219
|
+
Erosion or dilation with a small kernel.
|
|
220
|
+
|
|
221
|
+
- **Intensity 0.0–0.5 -> kernel size 2**; **0.5–1.0 -> kernel size 3**
|
|
222
|
+
- Khmer-aware mode: 70% erode, 30% horizontal dilate
|
|
223
|
+
- English mode: 50/50 erode/dilate with square kernel
|
|
224
|
+
|
|
225
|
+
### `anisotropic_dilation`: Anisotropic Dilation
|
|
226
|
+
|
|
227
|
+
Directional dilation simulating dot-matrix printer spread.
|
|
228
|
+
|
|
229
|
+
- **Intensity 0.0–0.5 -> kernel size 2**; **0.5–1.0 -> kernel size 3 or 4**
|
|
230
|
+
- Random direction: horizontal or vertical kernel
|
|
231
|
+
|
|
232
|
+
---
|
|
233
|
+
|
|
234
|
+
## Verify Command
|
|
235
|
+
|
|
236
|
+
To see every method at min and max intensity on a clean canvas:
|
|
237
|
+
|
|
238
|
+
```bash
|
|
239
|
+
khocr-gen verify --fonts fonts --output verify_output
|
|
240
|
+
```
|
|
241
|
+
|
|
242
|
+
This produces one PNG per method showing the clean canvas with the method applied at intensity 0.0 (left half) and 1.0 (right half).
|