ke-data-clean 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. ke_data_clean-0.1.0/.gitignore +18 -0
  2. ke_data_clean-0.1.0/CHANGELOG.md +26 -0
  3. ke_data_clean-0.1.0/LICENSE +21 -0
  4. ke_data_clean-0.1.0/PKG-INFO +127 -0
  5. ke_data_clean-0.1.0/README.md +95 -0
  6. ke_data_clean-0.1.0/docs/complexity-report.md +63 -0
  7. ke_data_clean-0.1.0/docs/explanation.md +65 -0
  8. ke_data_clean-0.1.0/docs/how-to.md +137 -0
  9. ke_data_clean-0.1.0/docs/reference.md +145 -0
  10. ke_data_clean-0.1.0/docs/tutorial.md +124 -0
  11. ke_data_clean-0.1.0/pyproject.toml +67 -0
  12. ke_data_clean-0.1.0/scripts/checkpoint.py +160 -0
  13. ke_data_clean-0.1.0/scripts/complexity_report.py +95 -0
  14. ke_data_clean-0.1.0/src/ke_data_clean/__init__.py +7 -0
  15. ke_data_clean-0.1.0/src/ke_data_clean/cli.py +281 -0
  16. ke_data_clean-0.1.0/src/ke_data_clean/county.py +161 -0
  17. ke_data_clean-0.1.0/src/ke_data_clean/data/__init__.py +3 -0
  18. ke_data_clean-0.1.0/src/ke_data_clean/data/counties.csv +48 -0
  19. ke_data_clean-0.1.0/src/ke_data_clean/kinds.py +48 -0
  20. ke_data_clean-0.1.0/src/ke_data_clean/mask.py +64 -0
  21. ke_data_clean-0.1.0/src/ke_data_clean/money.py +154 -0
  22. ke_data_clean-0.1.0/src/ke_data_clean/mpesa.py +99 -0
  23. ke_data_clean-0.1.0/src/ke_data_clean/pandas_ext.py +75 -0
  24. ke_data_clean-0.1.0/src/ke_data_clean/phone.py +152 -0
  25. ke_data_clean-0.1.0/src/ke_data_clean/pin.py +99 -0
  26. ke_data_clean-0.1.0/src/ke_data_clean/py.typed +0 -0
  27. ke_data_clean-0.1.0/src/ke_data_clean/result.py +30 -0
  28. ke_data_clean-0.1.0/tests/test_cli.py +225 -0
  29. ke_data_clean-0.1.0/tests/test_county.py +88 -0
  30. ke_data_clean-0.1.0/tests/test_mask.py +67 -0
  31. ke_data_clean-0.1.0/tests/test_money.py +76 -0
  32. ke_data_clean-0.1.0/tests/test_mpesa.py +90 -0
  33. ke_data_clean-0.1.0/tests/test_pandas_ext.py +76 -0
  34. ke_data_clean-0.1.0/tests/test_phone.py +104 -0
  35. ke_data_clean-0.1.0/tests/test_phone_crosscheck.py +73 -0
  36. ke_data_clean-0.1.0/tests/test_pin.py +57 -0
  37. ke_data_clean-0.1.0/tests/test_result.py +25 -0
@@ -0,0 +1,18 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ *.egg-info/
4
+ .eggs/
5
+ build/
6
+ dist/
7
+ .venv/
8
+ venv/
9
+ .pytest_cache/
10
+ .ruff_cache/
11
+ .coverage
12
+ htmlcov/
13
+ .ipynb_checkpoints/
14
+ *.csv.tmp
15
+
16
+ # Publishing and implementation notes stay local to the working copy.
17
+ PUBLISHING.md
18
+ ke-data-clean-implementation-guide.md
@@ -0,0 +1,26 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project are documented here. The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/); the project uses [Semantic Versioning](https://semver.org/).
4
+
5
+ ## [0.1.0] - 2026-10-05
6
+
7
+ First release.
8
+
9
+ ### Added
10
+
11
+ - The `Result` contract (`value`, `status`, `reason`) shared by every validator. Nothing is fixed silently: `fixed` changes are mechanical and reported, `invalid` and `ambiguous` refuse with a reason.
12
+ - `phone.normalize`: Kenyan mobile numbers to E.164. The 7-series is accepted as a block; the fragmented 1-series is accepted only on its assigned prefixes; landlines are rejected with a reason.
13
+ - `pin.validate`: KRA PIN format check (letter, 9 digits, letter), default prefixes A and P, configurable.
14
+ - `money.parse_kes`: KES amounts from decorated text to canonical decimal strings. Ragged thousands grouping and negatives are rejected; `1.2k` shorthand is opt-in.
15
+ - `county.normalize`, `county.code_for`, `county.all_counties`: all 47 counties with official names and constitutional codes (001-047), alias and fuzzy matching, and `ambiguous` instead of a guess when two counties fit equally well.
16
+ - `mpesa.parse_details`: M-Pesa statement Details lines into `Parsed(kind, id, name, account)`; unknown shapes return `None`.
17
+ - `mask.phone` and `mask.national_id`: fixed-width masks for sharing extracts under the Data Protection Act, 2019.
18
+ - `kinds.clean_value`: one dispatch table shared by the pandas helper and the CLI.
19
+ - `pandas_ext.clean_column` (pandas extra): one column to `value`/`status`/`reason` with cached repeats.
20
+ - The `ke-clean` command (cli extra): cleans named columns beside the originals, writes an auditable change report, `--strict` for pipelines, idempotent re-runs, refuses to clobber data it did not produce.
21
+
22
+ ### Sources for the rules (checked at release)
23
+
24
+ - Phone numbering plan: Communications Authority of Kenya assignments via libphonenumber metadata for +254, retrieved October 2026.
25
+ - KRA PIN format: KRA publications on kra.go.ke, September 2024 (A0… individuals, P0… companies).
26
+ - Counties: First Schedule of the Constitution of Kenya (2010).
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 ke-data-clean contributors
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,127 @@
1
+ Metadata-Version: 2.5
2
+ Name: ke-data-clean
3
+ Version: 0.1.0
4
+ Summary: Clean Kenyan data: phone numbers, KRA PINs, counties, KES amounts, M-Pesa details.
5
+ Project-URL: Changelog, https://github.com/MghongoDev/ke-data-clean/blob/main/CHANGELOG.md
6
+ Author: ke-data-clean contributors
7
+ License: MIT
8
+ License-File: LICENSE
9
+ Keywords: counties,data-cleaning,e164,kenya,kes,kra-pin,mpesa
10
+ Classifier: Development Status :: 4 - Beta
11
+ Classifier: Intended Audience :: Developers
12
+ Classifier: License :: OSI Approved :: MIT License
13
+ Classifier: Operating System :: OS Independent
14
+ Classifier: Programming Language :: Python :: 3
15
+ Classifier: Programming Language :: Python :: 3.10
16
+ Classifier: Programming Language :: Python :: 3.11
17
+ Classifier: Programming Language :: Python :: 3.12
18
+ Classifier: Programming Language :: Python :: 3.13
19
+ Classifier: Programming Language :: Python :: 3.14
20
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
21
+ Requires-Python: >=3.10
22
+ Provides-Extra: cli
23
+ Requires-Dist: pandas>=1.5; extra == 'cli'
24
+ Provides-Extra: dev
25
+ Requires-Dist: pandas>=1.5; extra == 'dev'
26
+ Requires-Dist: phonenumbers<9.0.42,>=8.13; extra == 'dev'
27
+ Requires-Dist: pytest>=8; extra == 'dev'
28
+ Requires-Dist: ruff>=0.5; extra == 'dev'
29
+ Provides-Extra: pandas
30
+ Requires-Dist: pandas>=1.5; extra == 'pandas'
31
+ Description-Content-Type: text/markdown
32
+
33
+ # ke-data-clean
34
+
35
+ Clean the dirty fields Kenyan datasets keep arriving with. Free, open source (MIT), zero dependencies.
36
+
37
+ - Phone numbers in a dozen formats → E.164 (`+2547XXXXXXXX`, `+2541XXXXXXXX`)
38
+ - KRA PINs → format-checked (shape only; it cannot prove a PIN exists)
39
+ - County names spelled five ways → the official name and code, all 47 counties
40
+ - KES amounts stored as text → `Decimal`
41
+ - M-Pesa "Details" strings → sender kind, id, name and account
42
+
43
+ **Nothing is fixed silently.** Every validator returns a `Result` with a value, a status and a reason. The CLI writes a change report listing every change it made, so you can audit what happened to your data.
44
+
45
+ ## Before and after
46
+
47
+ Ten messy rows in, cleaned values out. This table is the spec the package is tested against.
48
+
49
+ | # | Column | Messy input | Cleaned output | Status | Reason |
50
+ |---|--------|-------------|----------------|--------|--------|
51
+ | 1 | phone | `0712 345 678` | `+254712345678` | fixed | reformatted to E.164 |
52
+ | 2 | phone | `+254-712-345-678` | `+254712345678` | fixed | reformatted to E.164 |
53
+ | 3 | phone | `712345678` | `+254712345678` | fixed | reformatted to E.164 |
54
+ | 4 | phone | `0110 987 654` | `+254110987654` | fixed | reformatted to E.164 |
55
+ | 5 | phone | `07123456789` | *(empty)* | invalid | too long for a Kenyan mobile |
56
+ | 6 | county | `Murang'a` | `Murang'a` (021) | valid | |
57
+ | 7 | county | `muranga` | `Murang'a` (021) | fixed | matched alias, fuzzy |
58
+ | 8 | county | `Nairobi City` | `Nairobi` (047) | fixed | matched alias, exact |
59
+ | 9 | amount | `Ksh 1,200/=` | `1200` | fixed | parsed KES amount |
60
+ | 10 | pin | ` a012345678b` | `A012345678B` | fixed | removed spaces, uppercased |
61
+
62
+ ## Install
63
+
64
+ ```bash
65
+ pip install ke-data-clean
66
+ ```
67
+
68
+ Optional extras:
69
+
70
+ ```bash
71
+ pip install "ke-data-clean[pandas]" # clean_column helper
72
+ pip install "ke-data-clean[cli]" # ke-clean command
73
+ ```
74
+
75
+ ## Quick start
76
+
77
+ ```python
78
+ from ke_data_clean import phone
79
+
80
+ result = phone.normalize("0712 345 678")
81
+ print(result)
82
+ # Result(value='+254712345678', status='fixed', reason='reformatted to E.164')
83
+ ```
84
+
85
+ ### Clean a CSV from the command line
86
+
87
+ ```bash
88
+ ke-clean customers.csv \
89
+ --phone phone_no --pin kra_pin --county county --amount total \
90
+ --out cleaned.csv --report changes.csv
91
+ ```
92
+
93
+ The command writes `<col>_clean` and `<col>_status` columns beside the originals. It never overwrites your data. It prints a summary of counts by status per column. With `--strict`, it exits non-zero when any value stays `invalid`, so a pipeline can fail on it.
94
+
95
+ ## Statuses
96
+
97
+ | Status | Meaning | Value |
98
+ |--------|---------|-------|
99
+ | `valid` | Already clean | unchanged |
100
+ | `fixed` | Cleaned, and the change is recorded | new value |
101
+ | `invalid` | Cannot be cleaned; the reason says why | `None` |
102
+ | `ambiguous` | Two or more equally close matches; the tool refuses to guess | `None` |
103
+
104
+ ## Limitations
105
+
106
+ - KRA PIN validation checks format only. It cannot prove a PIN exists at KRA.
107
+ - County data covers the 47 counties of Kenya (constitution of 2010). Wards and constituencies are not covered yet.
108
+ - KES shorthand like `1.2k` is not parsed unless you opt in. In business data it is ambiguous.
109
+ - Landline numbers are rejected with a reason. Mobile-only for now.
110
+ - Formats follow the Kenyan numbering plan and county lists as understood at release time. Verify against the Communications Authority and KNBS before you rely on this in production. Sources and dates are in the module docstrings.
111
+
112
+ ## Documentation
113
+
114
+ - [Getting started](docs/tutorial.md) — a worked tutorial from raw CSV to cleaned output
115
+ - [How-to guides](docs/how-to.md) — specific tasks: pandas, CLI, strict mode, masking
116
+ - [Reference](docs/reference.md) — every public function and its signatures
117
+ - [Explanation](docs/explanation.md) — why the Result contract exists, and the rules behind each validator
118
+
119
+ ## Contributing
120
+
121
+ Issues and pull requests are welcome. Run `ruff check .` and `pytest` before you submit. Rule changes (phone prefixes, county data) must cite a primary source in the changelog.
122
+
123
+ Two checkpoints guard the rest: `scripts/checkpoint.py` (counts add up on 10,000 synthetic rows, re-runs are byte-identical, 100,000 rows stay under a minute) and `scripts/complexity_report.py` (no function over cyclomatic complexity 10; latest output in [docs/complexity-report.md](docs/complexity-report.md)).
124
+
125
+ ## License
126
+
127
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,95 @@
1
+ # ke-data-clean
2
+
3
+ Clean the dirty fields Kenyan datasets keep arriving with. Free, open source (MIT), zero dependencies.
4
+
5
+ - Phone numbers in a dozen formats → E.164 (`+2547XXXXXXXX`, `+2541XXXXXXXX`)
6
+ - KRA PINs → format-checked (shape only; it cannot prove a PIN exists)
7
+ - County names spelled five ways → the official name and code, all 47 counties
8
+ - KES amounts stored as text → `Decimal`
9
+ - M-Pesa "Details" strings → sender kind, id, name and account
10
+
11
+ **Nothing is fixed silently.** Every validator returns a `Result` with a value, a status and a reason. The CLI writes a change report listing every change it made, so you can audit what happened to your data.
12
+
13
+ ## Before and after
14
+
15
+ Ten messy rows in, cleaned values out. This table is the spec the package is tested against.
16
+
17
+ | # | Column | Messy input | Cleaned output | Status | Reason |
18
+ |---|--------|-------------|----------------|--------|--------|
19
+ | 1 | phone | `0712 345 678` | `+254712345678` | fixed | reformatted to E.164 |
20
+ | 2 | phone | `+254-712-345-678` | `+254712345678` | fixed | reformatted to E.164 |
21
+ | 3 | phone | `712345678` | `+254712345678` | fixed | reformatted to E.164 |
22
+ | 4 | phone | `0110 987 654` | `+254110987654` | fixed | reformatted to E.164 |
23
+ | 5 | phone | `07123456789` | *(empty)* | invalid | too long for a Kenyan mobile |
24
+ | 6 | county | `Murang'a` | `Murang'a` (021) | valid | |
25
+ | 7 | county | `muranga` | `Murang'a` (021) | fixed | matched alias, fuzzy |
26
+ | 8 | county | `Nairobi City` | `Nairobi` (047) | fixed | matched alias, exact |
27
+ | 9 | amount | `Ksh 1,200/=` | `1200` | fixed | parsed KES amount |
28
+ | 10 | pin | ` a012345678b` | `A012345678B` | fixed | removed spaces, uppercased |
29
+
30
+ ## Install
31
+
32
+ ```bash
33
+ pip install ke-data-clean
34
+ ```
35
+
36
+ Optional extras:
37
+
38
+ ```bash
39
+ pip install "ke-data-clean[pandas]" # clean_column helper
40
+ pip install "ke-data-clean[cli]" # ke-clean command
41
+ ```
42
+
43
+ ## Quick start
44
+
45
+ ```python
46
+ from ke_data_clean import phone
47
+
48
+ result = phone.normalize("0712 345 678")
49
+ print(result)
50
+ # Result(value='+254712345678', status='fixed', reason='reformatted to E.164')
51
+ ```
52
+
53
+ ### Clean a CSV from the command line
54
+
55
+ ```bash
56
+ ke-clean customers.csv \
57
+ --phone phone_no --pin kra_pin --county county --amount total \
58
+ --out cleaned.csv --report changes.csv
59
+ ```
60
+
61
+ The command writes `<col>_clean` and `<col>_status` columns beside the originals. It never overwrites your data. It prints a summary of counts by status per column. With `--strict`, it exits non-zero when any value stays `invalid`, so a pipeline can fail on it.
62
+
63
+ ## Statuses
64
+
65
+ | Status | Meaning | Value |
66
+ |--------|---------|-------|
67
+ | `valid` | Already clean | unchanged |
68
+ | `fixed` | Cleaned, and the change is recorded | new value |
69
+ | `invalid` | Cannot be cleaned; the reason says why | `None` |
70
+ | `ambiguous` | Two or more equally close matches; the tool refuses to guess | `None` |
71
+
72
+ ## Limitations
73
+
74
+ - KRA PIN validation checks format only. It cannot prove a PIN exists at KRA.
75
+ - County data covers the 47 counties of Kenya (constitution of 2010). Wards and constituencies are not covered yet.
76
+ - KES shorthand like `1.2k` is not parsed unless you opt in. In business data it is ambiguous.
77
+ - Landline numbers are rejected with a reason. Mobile-only for now.
78
+ - Formats follow the Kenyan numbering plan and county lists as understood at release time. Verify against the Communications Authority and KNBS before you rely on this in production. Sources and dates are in the module docstrings.
79
+
80
+ ## Documentation
81
+
82
+ - [Getting started](docs/tutorial.md) — a worked tutorial from raw CSV to cleaned output
83
+ - [How-to guides](docs/how-to.md) — specific tasks: pandas, CLI, strict mode, masking
84
+ - [Reference](docs/reference.md) — every public function and its signatures
85
+ - [Explanation](docs/explanation.md) — why the Result contract exists, and the rules behind each validator
86
+
87
+ ## Contributing
88
+
89
+ Issues and pull requests are welcome. Run `ruff check .` and `pytest` before you submit. Rule changes (phone prefixes, county data) must cite a primary source in the changelog.
90
+
91
+ Two checkpoints guard the rest: `scripts/checkpoint.py` (counts add up on 10,000 synthetic rows, re-runs are byte-identical, 100,000 rows stay under a minute) and `scripts/complexity_report.py` (no function over cyclomatic complexity 10; latest output in [docs/complexity-report.md](docs/complexity-report.md)).
92
+
93
+ ## License
94
+
95
+ MIT. See [LICENSE](LICENSE).
@@ -0,0 +1,63 @@
1
+ # Code complexity report
2
+
3
+ Per-function cyclomatic complexity for `src/ke_data_clean`, measured with
4
+ `scripts/complexity_report.py` (classic McCabe rules: one point per branch,
5
+ loop, except handler, ternary, assert, match case and comprehension clause,
6
+ plus one per boolean operator beyond the first, plus one for the function
7
+ itself). Nested functions are reported separately.
8
+
9
+ The enforced gate is ruff's C901 at complexity 10 (`pyproject.toml`). This
10
+ report counts more strictly than ruff, so a clean report here implies a
11
+ clean `ruff check --select C901`. Regenerate after changes:
12
+
13
+ ```bash
14
+ python scripts/complexity_report.py
15
+ ```
16
+
17
+ Measured at release 0.1.0: 43 functions, none over the gate.
18
+
19
+ | Complexity | Function | Gate |
20
+ |-----------:|----------|:----:|
21
+ | 9 | `cli.main` | ok |
22
+ | 8 | `phone.normalize` | ok |
23
+ | 8 | `money.parse_kes` | ok |
24
+ | 7 | `phone._judge_national` | ok |
25
+ | 7 | `money._split_groups` | ok |
26
+ | 6 | `money._canonicalize_number` | ok |
27
+ | 6 | `pin._shape_error` | ok |
28
+ | 6 | `mask.national_id` | ok |
29
+ | 6 | `county.normalize` | ok |
30
+ | 5 | `cli._build_report` | ok |
31
+ | 5 | `cli._check_would_not_clobber` | ok |
32
+ | 5 | `county._fuzzy_key` | ok |
33
+ | 5 | `county._fuzzy_lookup` | ok |
34
+ | 5 | `money._last_group_ok` | ok |
35
+ | 5 | `mpesa.parse_details` | ok |
36
+ | 5 | `pin.validate` | ok |
37
+ | 4 | `cli._check_columns_exist` | ok |
38
+ | 4 | `cli._clean_all` | ok |
39
+ | 4 | `phone._to_national` | ok |
40
+ | 4 | `pandas_ext.clean_column` | ok |
41
+ | 3 | `county._alias_map` | ok |
42
+ | 3 | `pandas_ext._cache_key` | ok |
43
+ | 3 | `pin._canonicalize` | ok |
44
+ | 3 | `cli._check_output_paths` | ok |
45
+ | 3 | `county._code_of` | ok |
46
+ | 3 | `money._middle_groups_ok` | ok |
47
+ | 3 | `money._reject_or_expand_shorthand` | ok |
48
+ | 3 | `cli._summarize` | ok |
49
+ | 3 | `cli._targets_from` | ok |
50
+ | 3 | `county.code_for` | ok |
51
+ | 2 | `county._exact_lookup` | ok |
52
+ | 2 | `money._first_group_ok` | ok |
53
+ | 2 | `county._load_rows` | ok |
54
+ | 2 | `county._official_names` | ok |
55
+ | 2 | `kinds.clean_value` | ok |
56
+ | 2 | `mask.phone` | ok |
57
+ | 1 | `cli._existing_column_matches` | ok |
58
+ | 1 | `pandas_ext._missing_result` | ok |
59
+ | 1 | `money._remove_decorations` | ok |
60
+ | 1 | `cli._same_path` | ok |
61
+ | 1 | `phone._strip` | ok |
62
+ | 1 | `county.all_counties` | ok |
63
+ | 1 | `cli.build_parser` | ok |
@@ -0,0 +1,65 @@
1
+ # Explanation
2
+
3
+ Why ke-data-clean works the way it works.
4
+
5
+ ## The contract exists because silent fixes destroy data
6
+
7
+ A phone column that arrives as `0712 345 678`, `+254-712-345-678` and `712345678` is the same number three times. A column with `07123456789` (ten digits) is a different problem: some digit is wrong, and no tool can know which. The failure mode this package guards against is the second case being quietly "fixed" into a plausible number that belongs to somebody else.
8
+
9
+ So every validator returns the same frozen dataclass:
10
+
11
+ ```python
12
+ Result(value=..., status="valid" | "fixed" | "invalid" | "ambiguous", reason=...)
13
+ ```
14
+
15
+ Four statuses, no exceptions:
16
+
17
+ - **valid** means the input already had the canonical shape. The tool touched nothing.
18
+ - **fixed** means the tool made a mechanical change it can defend: removing spaces, uppercasing, expanding a documented shorthand. Every fix lands in the change report with its reason.
19
+ - **invalid** means the tool refused. The reason states the specific failure (`expected 9 digits, got 10`), so a human can correct the source instead of guessing what the tool meant.
20
+ - **ambiguous** means two answers fit equally well. The tool names the candidates and stops. A wrong pick here is silent corruption; an empty cell with a reason is a work item.
21
+
22
+ `value` is a string or `None` so the contract stays identical across validators. Amounts are canonical decimal *strings* (`1200`, `1200.50`); wrap them with `Decimal(result.value)` for arithmetic. Floats were never considered: `0.1 + 0.2` has no place near money.
23
+
24
+ ## Why some plausible-looking inputs are rejected on purpose
25
+
26
+ **Shorthand (`1.2k`) is opt-in.** In business data `1.2k` usually means 1,200, but it can be a typo for 1.20 — a thousandfold error. The caller who knows the column's provenance opts in with `allow_shorthand=True`; everyone else gets an `invalid` with that exact explanation.
27
+
28
+ **Ragged grouping (`1,20,0`) is always rejected.** Real thousands grouping has a first group of 1–3 digits and later groups of exactly 3. `1,20,0` could be 1,200 with mangled separators or three numbers. Collapsing it would be a guess wearing a parse's clothes.
29
+
30
+ **The 1-series needs a prefix list.** Kenyan mobiles live on the 7-series (700–799, assigned as a block) and the 1-series, which the Communications Authority assigned piecemeal from 2019. A rule like "starts with 7 or 1" would accept numbers on unassigned 1-prefixes. The package keeps the assigned prefixes explicitly (`_ONE_SERIES_PREFIXES`, from libphonenumber metadata for +254, retrieved October 2026) and rejects the rest. A randomized test cross-checks this list against the `phonenumbers` package. A first version of this rule over-accepted; the cross-check caught it, which is why the cross-check stays in the suite.
31
+
32
+ **Landlines are deferred, not approximated.** Area codes 2–6 identify landlines unambiguously, so they are rejected with a reason instead of being silently dropped or mis-shaped.
33
+
34
+ ## Rules, and where they come from
35
+
36
+ Rule drift is a real risk: number plans change, and so does the data behind the rest of the rules. Every module docstring names its source and date, and the changelog records rule changes.
37
+
38
+ | Rule | Source | Checked |
39
+ |------|--------|---------|
40
+ | 9-digit NSN, 7-series block, 1-series prefixes | CA numbering plan via libphonenumber +254 metadata | October 2026 |
41
+ | PIN shape: letter + 9 digits + letter; A0… individuals, P0… companies | KRA blog on kra.go.ke | September 2024 |
42
+ | 47 counties, codes 001–047 | First Schedule, Constitution of Kenya (2010) | stable since 2010 |
43
+
44
+ Format checks can never prove existence: a format-valid PIN may not be registered, a valid E.164 number may be disconnected. That is why the README says format only.
45
+
46
+ ## Why argparse, not typer
47
+
48
+ The CLI could have been thirty lines shorter with typer. It would also have pulled in `click`, breaking the zero-dependency promise for the `cli` extra — the people installing this are often in locked-down data environments where one extra transitive dependency is the difference between installing and not. argparse ships with Python, handles the flag surface (repeated flags, `--version`, help text) without ceremony, and its worst rough edges (manual exit codes) are exactly the behaviour this tool needs to control anyway. The command is small; the dependency list stays empty.
49
+
50
+ ## Idempotency, and what the tool will not overwrite
51
+
52
+ `ke-clean` writes `<col>_clean` and `<col>_status` beside the originals and never writes to an original column. That alone would allow one nasty case: re-running on the command's own output would hit the derived columns, and a naive guard would refuse, so pipelines could not safely run twice. A naive non-guard would clobber a column somebody else named `phone_no_clean`.
53
+
54
+ The implemented rule splits the cases by content: a derived column is regenerated only when it already holds exactly the values this run computes. Re-running on ke-clean's own output is therefore a byte-for-byte fixpoint, while a foreign `phone_no_clean` column is refused with `exit 2` rather than destroyed. Similarly, `--out` or `--report` pointing at the input file is refused: writing cleaned data over the source is the one mistake there is no report to recover from.
55
+
56
+ ## Why the masks have fixed widths
57
+
58
+ `mask.phone` writes exactly 5 stars and `mask.national_id` exactly 3, always. A mask that wrote one star per hidden digit would publish the length of every ID in the file — length itself leaks (older ID series differ from newer ones). Fewer than 7 digits is refused because keeping the first 3 and last 3 of a 6-digit number leaves nothing hidden.
59
+
60
+ ## What this package is not
61
+
62
+ - Not a lookup service: nothing here queries KRA, the CA or an HLR.
63
+ - Not a landline formatter (deferred, with a reason when met).
64
+ - Not a ward/constituency gazetteer: counties only.
65
+ - Not a replacement for reviewing the change report. The report is the product; the cleaned CSV is its by-product.
@@ -0,0 +1,137 @@
1
+ # How-to guides
2
+
3
+ Task-by-task recipes. Each one assumes the package is installed; see the [tutorial](tutorial.md) for setup.
4
+
5
+ ## Clean one column of a pandas DataFrame
6
+
7
+ ```python
8
+ import pandas as pd
9
+ from ke_data_clean.pandas_ext import clean_column
10
+
11
+ frame = pd.read_csv("customers.csv", dtype=str, keep_default_na=False, na_values=[""])
12
+ result = clean_column(frame["phone_no"], "phone")
13
+ frame["phone_no_clean"] = result["value"]
14
+ frame["phone_no_status"] = result["status"]
15
+ ```
16
+
17
+ `clean_column` returns a DataFrame with `value`, `status` and `reason` columns and the same index as the input. It returns an object-dtype frame on purpose: missing values stay `None` instead of turning into `NaN`. Missing input becomes an `invalid` result with the reason `no value found`. Repeated raw values are cleaned once and reused from a cache, so a column with many duplicates stays fast.
18
+
19
+ Pass options through to the validator: `clean_column(frame["total"], "amount", allow_shorthand=True)`.
20
+
21
+ ## Fail a pipeline when bad values remain
22
+
23
+ ```bash
24
+ ke-clean customers.csv --phone phone_no --amount total \
25
+ --out cleaned.csv --report changes.csv --strict
26
+ ```
27
+
28
+ Exit codes:
29
+
30
+ | Code | Meaning |
31
+ |------|---------|
32
+ | 0 | Finished. `invalid` values may exist; read the report. |
33
+ | 1 | `--strict` was set and at least one value stayed `invalid`. |
34
+ | 2 | Usage or file problem: unknown column, unreadable file, output path pointing at the input. |
35
+
36
+ Without `--strict` the same run exits 0. Use `--strict` in CI or scheduled jobs where a silent bad column would flow downstream.
37
+
38
+ ## Accept 1.2k-style amounts
39
+
40
+ ```bash
41
+ ke-clean customers.csv --amount total ... --allow-shorthand
42
+ ```
43
+
44
+ ```python
45
+ money.parse_kes("1.2k", allow_shorthand=True)
46
+ # Result(value='1200', status='fixed', reason='expanded shorthand')
47
+ ```
48
+
49
+ The flag exists because `1.2k` is ambiguous in business data: it can mean 1,200 or be a typo for 1.20. Opt in only when you know who typed the column. Ragged thousands grouping such as `1,20,0` stays rejected either way.
50
+
51
+ ## Accept other KRA PIN prefixes
52
+
53
+ KRA documents A for individuals and P for companies, which is the default:
54
+
55
+ ```python
56
+ pin.validate("A012345678B")
57
+ # Result(value='A012345678B', status='valid')
58
+ ```
59
+
60
+ To accept another documented series, name its leading letter:
61
+
62
+ ```python
63
+ pin.validate(raw, allowed_prefixes=("A", "P", "H"))
64
+ ```
65
+
66
+ The format check stays the same: one letter, nine digits, one letter.
67
+
68
+ ## Mask phone numbers and national IDs before sharing
69
+
70
+ ```python
71
+ from ke_data_clean import mask
72
+
73
+ mask.phone("0712 345 678")
74
+ # Result(value='+25471*****78', status='fixed', reason='reformatted to E.164')
75
+
76
+ mask.national_id("12345678")
77
+ # Result(value='123***678', status='fixed', reason='masked for sharing')
78
+ ```
79
+
80
+ The phone mask keeps the first six and last two characters of the E.164 form, so duplicates stay detectable. The ID mask always writes three stars whatever the length, so the mask does not reveal how long the ID is. Fewer than seven digits is refused: a shorter number would stay readable through the mask.
81
+
82
+ ## Get county codes and the full list
83
+
84
+ ```python
85
+ from ke_data_clean import county
86
+
87
+ county.code_for("Nairobi")
88
+ # '047'
89
+
90
+ counties = county.all_counties() # {'001': 'Mombasa', ..., '047': 'Nairobi'}
91
+ ```
92
+
93
+ Codes follow the First Schedule of the Constitution of Kenya (2010). Add `county_clean_code = county.code_for(result.value)` to a pipeline when you want a stable join key next to the name.
94
+
95
+ ## Parse M-Pesa statement Details
96
+
97
+ ```python
98
+ from ke_data_clean import mpesa
99
+
100
+ mpesa.parse_details("Pay Bill to KPLC POSTPAID - 888880 for account 123456789")
101
+ # Parsed(kind='paybill', id='888880', name='KPLC POSTPAID', account='123456789')
102
+
103
+ mpesa.parse_details("Some new statement wording")
104
+ # None
105
+ ```
106
+
107
+ `None` means the line matched no known pattern. Filter those rows out and inspect them; the parser will not invent a shape. `Parsed` carries `kind`, `id`, `name` and `account` (the last only for paybill payments).
108
+
109
+ ## Deal with an ambiguous county
110
+
111
+ The reason names every candidate:
112
+
113
+ ```python
114
+ county.normalize("kiifi")
115
+ # Result(value=None, status='ambiguous', reason='could be any of: Kilifi, Kisii')
116
+ ```
117
+
118
+ Three ways to resolve it:
119
+
120
+ 1. Fix the source data to the official spelling (`Kilifi`).
121
+ 2. Pick per row with extra context (for example the town column) and pass the choice downstream.
122
+ 3. Keep the rows in a review queue. Do not code a "pick the first candidate" shortcut; the ambiguity is real.
123
+
124
+ ## Work with the Result contract directly
125
+
126
+ Every validator returns the same frozen dataclass, so one helper can handle all kinds:
127
+
128
+ ```python
129
+ from ke_data_clean.kinds import clean_value
130
+
131
+ clean_value("0712 345 678", "phone")
132
+ # Result(value='+254712345678', status='fixed', reason='reformatted to E.164')
133
+
134
+ clean_value("x", "nope") # raises ValueError, listing the known kinds
135
+ ```
136
+
137
+ The known kinds are `phone`, `pin`, `county` and `amount`. Use `clean_value` when the kind arrives as data, for example from a CLI flag or a config file.