kloship-utils 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of kloship-utils might be problematic. Click here for more details.

Files changed (45) hide show
  1. kloship_utils-0.1.0/.gitignore +26 -0
  2. kloship_utils-0.1.0/CLAUDE.md +88 -0
  3. kloship_utils-0.1.0/LICENSE +21 -0
  4. kloship_utils-0.1.0/PKG-INFO +252 -0
  5. kloship_utils-0.1.0/README.md +203 -0
  6. kloship_utils-0.1.0/docs/MIGRATION.md +165 -0
  7. kloship_utils-0.1.0/docs/SHARED_FUNCTIONS_ANALYSIS.md +102 -0
  8. kloship_utils-0.1.0/pyproject.toml +121 -0
  9. kloship_utils-0.1.0/src/kloship_utils/__init__.py +26 -0
  10. kloship_utils-0.1.0/src/kloship_utils/_version.py +24 -0
  11. kloship_utils-0.1.0/src/kloship_utils/aws/__init__.py +36 -0
  12. kloship_utils-0.1.0/src/kloship_utils/aws/dynamodb.py +376 -0
  13. kloship_utils-0.1.0/src/kloship_utils/aws/s3.py +216 -0
  14. kloship_utils-0.1.0/src/kloship_utils/cella.py +255 -0
  15. kloship_utils-0.1.0/src/kloship_utils/encoding.py +41 -0
  16. kloship_utils-0.1.0/src/kloship_utils/exceptions.py +45 -0
  17. kloship_utils-0.1.0/src/kloship_utils/handle_units.py +136 -0
  18. kloship_utils-0.1.0/src/kloship_utils/helpers.py +51 -0
  19. kloship_utils-0.1.0/src/kloship_utils/py.typed +0 -0
  20. kloship_utils-0.1.0/src/kloship_utils/templates.py +73 -0
  21. kloship_utils-0.1.0/src/kloship_utils/text.py +57 -0
  22. kloship_utils-0.1.0/src/kloship_utils/transfers/__init__.py +11 -0
  23. kloship_utils-0.1.0/src/kloship_utils/transfers/config.py +177 -0
  24. kloship_utils-0.1.0/src/kloship_utils/transfers/ftp.py +97 -0
  25. kloship_utils-0.1.0/src/kloship_utils/transfers/sftp.py +148 -0
  26. kloship_utils-0.1.0/src/kloship_utils/units.py +255 -0
  27. kloship_utils-0.1.0/src/kloship_utils/xml.py +67 -0
  28. kloship_utils-0.1.0/tests/__init__.py +0 -0
  29. kloship_utils-0.1.0/tests/conftest.py +123 -0
  30. kloship_utils-0.1.0/tests/fixtures/carrier_account_config.json +16 -0
  31. kloship_utils-0.1.0/tests/fixtures/handle_units.json +53 -0
  32. kloship_utils-0.1.0/tests/unit/__init__.py +0 -0
  33. kloship_utils-0.1.0/tests/unit/test_cella.py +180 -0
  34. kloship_utils-0.1.0/tests/unit/test_dynamodb.py +198 -0
  35. kloship_utils-0.1.0/tests/unit/test_encoding.py +37 -0
  36. kloship_utils-0.1.0/tests/unit/test_ftp.py +92 -0
  37. kloship_utils-0.1.0/tests/unit/test_handle_units.py +86 -0
  38. kloship_utils-0.1.0/tests/unit/test_helpers.py +29 -0
  39. kloship_utils-0.1.0/tests/unit/test_s3.py +98 -0
  40. kloship_utils-0.1.0/tests/unit/test_sftp.py +155 -0
  41. kloship_utils-0.1.0/tests/unit/test_templates.py +39 -0
  42. kloship_utils-0.1.0/tests/unit/test_text.py +34 -0
  43. kloship_utils-0.1.0/tests/unit/test_transfers_config.py +95 -0
  44. kloship_utils-0.1.0/tests/unit/test_units.py +164 -0
  45. kloship_utils-0.1.0/tests/unit/test_xml.py +46 -0
@@ -0,0 +1,26 @@
1
+ # Python
2
+ __pycache__/
3
+ *.py[cod]
4
+ *.egg-info/
5
+ .venv/
6
+ env*/
7
+
8
+ # Generated by hatch-vcs at build / install time
9
+ src/kloship_utils/_version.py
10
+
11
+ # Build
12
+ build/
13
+ dist/
14
+
15
+ # Tools
16
+ .mypy_cache/
17
+ .ruff_cache/
18
+ .pytest_cache/
19
+ .coverage
20
+ coverage.xml
21
+ htmlcov/
22
+ .task/
23
+
24
+ # IDE
25
+ .vscode/
26
+ .idea/
@@ -0,0 +1,88 @@
1
+ # CLAUDE.md
2
+
3
+ ## Project
4
+
5
+ `kloship-utils` is the **public** PyPI package (MIT) of utilities shared by the KloShip carrier repositories
6
+ (`Groupe-Klocel/Kloship-<carrier>`): payload encoding, unit / date conversions, text cleaning, Jinja2 templates,
7
+ dict to XML, HandleUnit helpers, S3 archiving, DynamoDB lookups / counters / conversions, CELLA rules, FTP / SFTP uploads.
8
+ Carriers install it in their Lambda layers (`layer/requirements.txt`, exact version pin) and import it instead of
9
+ copy-pasting helpers.
10
+
11
+ Read at the start of a session (versioned in git, unlike the local Claude memory):
12
+
13
+ - [`docs/SHARED_FUNCTIONS_ANALYSIS.md`](docs/SHARED_FUNCTIONS_ANALYSIS.md) — which carrier helpers were moved here,
14
+ their variants, the legacy bugs, what stays in the carriers and the candidates for the next versions.
15
+ - [`docs/MIGRATION.md`](docs/MIGRATION.md) — how a carrier switches to the library, legacy → library mapping and
16
+ behavior differences.
17
+ - [`README.md`](README.md) — usage, development and release process.
18
+
19
+ ## Constraints
20
+
21
+ - **Public package**: never commit secrets, real customer data (names, addresses, account numbers), internal host
22
+ names or client specific configuration — in code, docs or test fixtures. Test data must be synthetic.
23
+ - **Python ≥ 3.11** (AWS Lambda `python3.11` / `python3.12` runtimes, `arm64`).
24
+ - **Light dependencies**: core deps are pure Python (`requests`, `Jinja2`, `Unidecode`). `boto3` is provided by the
25
+ Lambda runtime and is only an optional extra (`aws`); import it lazily (`kloship_utils.aws.boto3_module()`).
26
+ `paramiko` is the optional `sftp` extra, imported lazily too. Do not add heavy dependencies to the core.
27
+ - **Backward compatibility**: carriers pin exact versions, but follow SemVer — a change of signature or behavior of a
28
+ public function is a breaking change. The legacy-compatible names / signatures (`convert_SI`,
29
+ `get_handle_unit_from_shipment_number`, `get_rule_result_from_kloship_front`...) must be kept.
30
+
31
+ ## Layout
32
+
33
+ ```
34
+ src/kloship_utils/
35
+ encoding.py helpers.py units.py text.py templates.py xml.py handle_units.py cella.py exceptions.py
36
+ aws/ s3.py, dynamodb.py (boto3, lazy)
37
+ transfers/ config.py (TransferConfig, TunnelConfig, TransferResult), ftp.py, sftp.py (paramiko, lazy)
38
+ tests/
39
+ conftest.py moto fixtures (S3, DynamoDB), env vars, FakeDynamoDBClient
40
+ fixtures/ synthetic CarrierAccountConfig / HandleUnits
41
+ unit/ one test file per module
42
+ ```
43
+
44
+ ## Conventions
45
+
46
+ - Everything in **English**: code, names, docstrings, comments, log / error messages, docs, commit messages.
47
+ - Google style docstrings with `Args` / `Returns` / `Raises`; full type hints (`mypy --strict` must pass).
48
+ - Use the module logger (`logging.getLogger(__name__)`), never `print`. Never log passwords or tokens.
49
+ - Raise the exceptions of `kloship_utils.exceptions` (subclasses of `KloshipUtilsError`; keep the builtin base class
50
+ the legacy code raised, e.g. `ValueError`, so that carriers' `except` blocks keep working).
51
+ - DynamoDB: table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
52
+ `CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and must stay overridable
53
+ by argument. PartiQL statements are **always parameterized** (`?`), never built with f-strings of values.
54
+ - AWS / HTTP helpers accept an optional `client` / `resource` / `session` argument for reuse and tests.
55
+ - Ruff (lint + format, 120 columns) and mypy configuration live in `pyproject.toml`.
56
+
57
+ ## Commands
58
+
59
+ ```bash
60
+ task install # .venv + pip install -e ".[dev]"
61
+ task lint # ruff check + ruff format --check
62
+ task typecheck # mypy (strict)
63
+ task test # pytest --cov, fails under 90 %
64
+ task check # all of the above
65
+ task build # sdist + wheel + twine check
66
+ ```
67
+
68
+ Without Task: `.venv/bin/pytest --cov`, `.venv/bin/ruff check src tests`, `.venv/bin/mypy`.
69
+
70
+ ## Rules
71
+
72
+ - **Every new or changed function comes with unit tests** (moto for AWS, `responses` for HTTP, mocks for FTP /
73
+ SFTP). Coverage must stay ≥ 90 % and the CI must be green before merging.
74
+ - CI: `.github/workflows/ci.yml` runs lint, mypy, tests (Python 3.11 / 3.12 / 3.13) and the build on every push and
75
+ pull request.
76
+ - When a helper moves from a carrier to the library, update `docs/SHARED_FUNCTIONS_ANALYSIS.md` and
77
+ `docs/MIGRATION.md` (mapping and behavior differences).
78
+
79
+ ## Release
80
+
81
+ The version comes from the git tag (`hatch-vcs`), there is no version number in the code.
82
+
83
+ ```bash
84
+ git tag vX.Y.Z && git push origin vX.Y.Z # pre-release: vX.Y.ZrcN
85
+ ```
86
+
87
+ `.github/workflows/release.yml` then runs the CI, builds, publishes to PyPI (secret `PYPI_TOKEN`) and creates the
88
+ GitHub Release with notes generated from the merged pull requests (labels in `.github/release.yml`).
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Groupe Klocel
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,252 @@
1
+ Metadata-Version: 2.5
2
+ Name: kloship-utils
3
+ Version: 0.1.0
4
+ Summary: Shared utilities for KloShip carrier integrations (encoding, unit conversions, S3, DynamoDB, CELLA, FTP/SFTP transfers).
5
+ Project-URL: Homepage, https://github.com/Groupe-Klocel/Kloship-utils
6
+ Project-URL: Repository, https://github.com/Groupe-Klocel/Kloship-utils
7
+ Project-URL: Issues, https://github.com/Groupe-Klocel/Kloship-utils/issues
8
+ Project-URL: Changelog, https://github.com/Groupe-Klocel/Kloship-utils/releases
9
+ Author-email: Guillaume Candau <guillaume.candau@klocel.com>
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Keywords: aws-lambda,carrier,edi,klocel,kloship,shipping,zpl
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Operating System :: OS Independent
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3 :: Only
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Topic :: Software Development :: Libraries
22
+ Classifier: Typing :: Typed
23
+ Requires-Python: >=3.11
24
+ Requires-Dist: jinja2>=3.1
25
+ Requires-Dist: requests>=2.31
26
+ Requires-Dist: unidecode>=1.3
27
+ Provides-Extra: all
28
+ Requires-Dist: boto3>=1.28; extra == 'all'
29
+ Requires-Dist: paramiko>=3.4; extra == 'all'
30
+ Provides-Extra: aws
31
+ Requires-Dist: boto3>=1.28; extra == 'aws'
32
+ Provides-Extra: dev
33
+ Requires-Dist: boto3-stubs[dynamodb,s3]>=1.28; extra == 'dev'
34
+ Requires-Dist: boto3>=1.28; extra == 'dev'
35
+ Requires-Dist: build>=1.2; extra == 'dev'
36
+ Requires-Dist: moto[dynamodb,s3]>=5.0; extra == 'dev'
37
+ Requires-Dist: mypy>=1.10; extra == 'dev'
38
+ Requires-Dist: paramiko>=3.4; extra == 'dev'
39
+ Requires-Dist: pytest-cov>=5.0; extra == 'dev'
40
+ Requires-Dist: pytest>=8.0; extra == 'dev'
41
+ Requires-Dist: responses>=0.25; extra == 'dev'
42
+ Requires-Dist: ruff>=0.6; extra == 'dev'
43
+ Requires-Dist: twine>=5.1; extra == 'dev'
44
+ Requires-Dist: types-paramiko>=3.4; extra == 'dev'
45
+ Requires-Dist: types-requests>=2.31; extra == 'dev'
46
+ Provides-Extra: sftp
47
+ Requires-Dist: paramiko>=3.4; extra == 'sftp'
48
+ Description-Content-Type: text/markdown
49
+
50
+ # kloship-utils
51
+
52
+ Shared utilities for the [KloShip](https://www.klocel.com) carrier integrations.
53
+
54
+ Every KloShip carrier repository (Chronopost, Colissimo, DHL, UPS, Heppner, XPO, Mondial Relay...) used to copy the
55
+ same helpers into each Lambda: payload encoding, unit conversions, S3 archiving, DynamoDB lookups, CELLA rules,
56
+ FTP/SFTP uploads... `kloship-utils` gathers them in a single, tested and versioned package published on PyPI, so that
57
+ carriers install it in their Lambda layer and import it instead of copy-pasting.
58
+
59
+ See [docs/SHARED_FUNCTIONS_ANALYSIS.md](docs/SHARED_FUNCTIONS_ANALYSIS.md) for the analysis of the duplicated code and
60
+ [docs/MIGRATION.md](docs/MIGRATION.md) to migrate a carrier.
61
+
62
+ ## Installation
63
+
64
+ Requires Python 3.11 or later.
65
+
66
+ ```bash
67
+ pip install kloship-utils # core: encoding, units, text, templates, xml, handle units, CELLA, FTP
68
+ pip install "kloship-utils[sftp]" # + SFTP uploads (paramiko)
69
+ pip install "kloship-utils[aws]" # + boto3, for S3 / DynamoDB helpers outside AWS Lambda
70
+ ```
71
+
72
+ `boto3` is **not** a mandatory dependency: the AWS Lambda Python runtime already provides it, and bundling it would
73
+ add ~80 MB to every layer.
74
+
75
+ ### In a carrier Lambda layer
76
+
77
+ Add the package, **pinned to an exact version**, to the layer requirements of each function
78
+ (`src/functions/<carrier>/<function>/layer/requirements.txt`):
79
+
80
+ ```text
81
+ kloship-utils[sftp]==0.1.0
82
+ ```
83
+
84
+ `task buildlayer` (`sam build --use-container`) then installs it like any other dependency. Every dependency has
85
+ `aarch64` wheels, so it works with the `arm64` Lambdas.
86
+
87
+ ## Modules
88
+
89
+ | Module | Content |
90
+ |---|---|
91
+ | `kloship_utils.encoding` | `to_base64_gzip_json`, `from_base64_gzip_json`: payloads exchanged between Step Functions and Lambdas |
92
+ | `kloship_utils.helpers` | `to_list`, `empty_if_none`, `chunks`, `get_path` |
93
+ | `kloship_utils.units` | `convert_SI` (length, volume, weight, liquid, dates), `convert_date`, `convert_values` |
94
+ | `kloship_utils.text` | `strip_diacritics`, `strip_diacritics_in`, `replace_special_chars` |
95
+ | `kloship_utils.templates` | `render_template`, `render_zpl_label`, `split_zpl` (Jinja2) |
96
+ | `kloship_utils.xml` | `dict_to_xml`, `to_xml_string` (attributes, lists, CDATA) |
97
+ | `kloship_utils.handle_units` | `Status`, `global_group_id`, `global_group_id2`, `group_by_delivery`, `next_index_in_order`, `delete_attachments_from_result`, `clear_fields` |
98
+ | `kloship_utils.aws.s3` | `build_key`, `put_file`, `get_file`, `save_label`, `save_shipment_file`, file name helpers |
99
+ | `kloship_utils.aws.dynamodb` | HandleUnit lookups, `increment_counter`, `get_parameter`, `get_carrier_account_config`, `get_conversion`, `get_packaging_conversion`, `execute_statement_all`, `batch_execute` |
100
+ | `kloship_utils.cella` | `CellaClient`, `get_rule_result`, `get_rule_result_from_kloship_front`, `pdf_to_zpl` |
101
+ | `kloship_utils.transfers` | `TransferConfig`, `TunnelConfig`, `upload_ftp`, `upload_sftp` |
102
+ | `kloship_utils.exceptions` | `KloshipUtilsError` and its subclasses |
103
+
104
+ ### Encoding
105
+
106
+ ```python
107
+ from kloship_utils.encoding import from_base64_gzip_json, to_base64_gzip_json
108
+
109
+ def lambda_handler(event, context):
110
+ handle_units = from_base64_gzip_json(event["compressed_handle_units"])
111
+ ...
112
+ return {"compressed_handle_units": to_base64_gzip_json(enriched_handle_units)}
113
+ ```
114
+
115
+ Pass `default=str` to `to_base64_gzip_json` to serialize the `Decimal` values returned by DynamoDB.
116
+
117
+ ### Units and dates
118
+
119
+ ```python
120
+ from kloship_utils.units import convert_SI, convert_date
121
+
122
+ convert_SI(hu["WEIGHT"]["#text"], "weight", hu["WEIGHT"]["@format"], "kg") # "10.05" (2 decimals)
123
+ convert_SI("1234", "length", "mm", "cm", number_decimal=0) # "123"
124
+ convert_SI("1234", "length", "mm", "m", number_decimal=None) # "1.234" (no rounding)
125
+ convert_SI("2024/01/31", "date", "YYYY/MM/DD", "%d.%m.%Y") # "31.01.2024"
126
+ convert_date("2024-01-31 08:05:00", "YYYY-MM-DD HH:MI:SS", "CCYYMMDDHHMM") # "202401310805"
127
+ ```
128
+
129
+ Date formats accept KloShip tokens (`yyyy`, `ccyy`, `yy`, `mm`, `dd`, `hh`, `mi`/`ii`, `ss`; `mm` after `hh` means
130
+ minutes) or `strftime` patterns.
131
+
132
+ ### Labels
133
+
134
+ ```python
135
+ from kloship_utils.templates import render_zpl_label, split_zpl
136
+ from kloship_utils.aws.s3 import save_label
137
+
138
+ zpl = render_zpl_label(handle_unit, config, "layout_label.zpl") # templates/ directory of the Lambda
139
+ handle_unit["label"] = split_zpl(zpl) # {"header", "body", "footer"}
140
+ handle_unit["KLOSHIP_LABEL_FILENAME"] = save_label(
141
+ handle_unit, account_id, config, zpl, extension="zpl", content_type="x-application/zpl", carrier="MYCARRIER"
142
+ )
143
+ ```
144
+
145
+ ### S3
146
+
147
+ ```python
148
+ from kloship_utils.aws.s3 import save_shipment_file, shipment_file_name
149
+
150
+ file_name = shipment_file_name(config, "edi") # 20240131_080509_42.edi
151
+ save_shipment_file(
152
+ edi, account_id, config, file_name,
153
+ content_type="application/EDI-EDIFACT", carrier="MYCARRIER",
154
+ encoding="ISO-8859-1", content_encoding="ISO-8859-1",
155
+ )
156
+ ```
157
+
158
+ Keys follow the KloShip layout `{account}/{@senderCodeTDM}/{@serviceTagTDM}/{YYYY}/{MM}/{DD}/{file}`, in the
159
+ `STANDARD_IA` storage class. Buckets default to the `LABELS_BUCKET` / `SHIPMENT_BUCKET` environment variables.
160
+
161
+ ### DynamoDB
162
+
163
+ ```python
164
+ from kloship_utils.aws import dynamodb
165
+
166
+ handle_units = dynamodb.get_handle_units_by_global_group_id(account_id, config) # PRE-SHIPPED / SHIPPED HUs
167
+ number = dynamodb.increment_counter(config, parameter_name=config.get("ParameterNameHUCounter"))
168
+ packaging = dynamodb.get_packaging_conversion(hu["PACKAGING"], config, default=config.get("DefaultPackaging"))
169
+ ```
170
+
171
+ Table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
172
+ `CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and can be given explicitly.
173
+ All PartiQL statements are parameterized.
174
+
175
+ ### CELLA rules
176
+
177
+ ```python
178
+ from kloship_utils.cella import get_rule_result
179
+
180
+ values = get_rule_result(
181
+ config["CreateLabel"]["FrontendConversion"], # Url, WarehouseWorkerUsername, WarehouseWorkerPwd, WarehouseId, StockOwnerId
182
+ rule_config, # Name + Exit [{name, type, defaultValue}]
183
+ {"packagingDescription": hu["PACKAGING"]},
184
+ )
185
+ ```
186
+
187
+ ### FTP / SFTP
188
+
189
+ ```python
190
+ from kloship_utils.transfers import TransferConfig, TunnelConfig, upload_sftp
191
+
192
+ sftp = TransferConfig.from_carrier_config(config["CARRIER"]["Shipment"]["Sftp"])
193
+ tunnel = TunnelConfig.from_carrier_config(config["CARRIER"]["Shipment"]["TunnelSSH"], password_base64=True)
194
+ upload_sftp(sftp, file_name, edi, tunnel=tunnel) # raises TransferError (with .code 401/402/404) on failure
195
+ ```
196
+
197
+ `TransferConfig.from_carrier_config` reads the usual carrier keys whatever their case (`Host`, `Port`, `Login`,
198
+ `Password`, `RemotePath`, `EDIFileEncoding`, `UseCnOptsHostkeys`, `ParamikoKeyData`, `ParamikoSignVerifySSHData`).
199
+
200
+ ## Development
201
+
202
+ The tasks are defined in [Taskfile.yml](Taskfile.yml) ([Task](https://taskfile.dev)):
203
+
204
+ ```bash
205
+ task install # virtualenv + editable install with every extra and the dev tools
206
+ task lint # ruff (lint + format check)
207
+ task typecheck # mypy --strict
208
+ task test # pytest with coverage (fails under 90 %)
209
+ task check # lint + typecheck + test
210
+ task build # sdist + wheel in dist/, checked with twine
211
+ ```
212
+
213
+ Without Task: `python -m venv .venv && .venv/bin/pip install -e ".[dev]" && .venv/bin/pytest --cov`.
214
+
215
+ Every function has unit tests: AWS is mocked with [moto](https://github.com/getmoto/moto), HTTP with
216
+ [responses](https://github.com/getsentry/responses), FTP / SFTP with mocks. The tests use synthetic data only.
217
+
218
+ ### Continuous integration
219
+
220
+ - [`ci.yml`](.github/workflows/ci.yml) runs on every push and pull request: ruff, mypy, the tests on Python 3.11,
221
+ 3.12 and 3.13 (coverage ≥ 90 %), then builds and smoke-tests the wheel.
222
+ - [`release.yml`](.github/workflows/release.yml) runs on `vX.Y.Z` tags: CI, build, publication on PyPI, GitHub
223
+ Release with generated notes.
224
+
225
+ ## Versioning and releases
226
+
227
+ The package follows [Semantic Versioning](https://semver.org): breaking changes bump the major version (minor while
228
+ in `0.x`). The version is computed from the git tag by `hatch-vcs`: there is no version to edit in the code.
229
+
230
+ To publish a release:
231
+
232
+ ```bash
233
+ git checkout main && git pull
234
+ git tag v0.1.0
235
+ git push origin v0.1.0
236
+ ```
237
+
238
+ Pre-releases use PEP 440 suffixes (`v0.2.0rc1`) and are flagged as pre-releases on GitHub. Release notes are
239
+ generated from the merged pull requests, grouped by label (see [.github/release.yml](.github/release.yml)).
240
+
241
+ ### First publication (one-time setup)
242
+
243
+ 1. Create an API token on [pypi.org](https://pypi.org/manage/account/token/). The project does not exist yet, so the
244
+ first token must have the *entire account* scope.
245
+ 2. In the GitHub repository settings, add it as the `PYPI_TOKEN` secret (repository secret, or secret of the `pypi`
246
+ environment used by the publish job).
247
+ 3. Push the first tag (`v0.1.0`).
248
+ 4. Once the project exists on PyPI, replace the token with one scoped to the `kloship-utils` project.
249
+
250
+ ## License
251
+
252
+ [MIT](LICENSE) © Groupe Klocel
@@ -0,0 +1,203 @@
1
+ # kloship-utils
2
+
3
+ Shared utilities for the [KloShip](https://www.klocel.com) carrier integrations.
4
+
5
+ Every KloShip carrier repository (Chronopost, Colissimo, DHL, UPS, Heppner, XPO, Mondial Relay...) used to copy the
6
+ same helpers into each Lambda: payload encoding, unit conversions, S3 archiving, DynamoDB lookups, CELLA rules,
7
+ FTP/SFTP uploads... `kloship-utils` gathers them in a single, tested and versioned package published on PyPI, so that
8
+ carriers install it in their Lambda layer and import it instead of copy-pasting.
9
+
10
+ See [docs/SHARED_FUNCTIONS_ANALYSIS.md](docs/SHARED_FUNCTIONS_ANALYSIS.md) for the analysis of the duplicated code and
11
+ [docs/MIGRATION.md](docs/MIGRATION.md) to migrate a carrier.
12
+
13
+ ## Installation
14
+
15
+ Requires Python 3.11 or later.
16
+
17
+ ```bash
18
+ pip install kloship-utils # core: encoding, units, text, templates, xml, handle units, CELLA, FTP
19
+ pip install "kloship-utils[sftp]" # + SFTP uploads (paramiko)
20
+ pip install "kloship-utils[aws]" # + boto3, for S3 / DynamoDB helpers outside AWS Lambda
21
+ ```
22
+
23
+ `boto3` is **not** a mandatory dependency: the AWS Lambda Python runtime already provides it, and bundling it would
24
+ add ~80 MB to every layer.
25
+
26
+ ### In a carrier Lambda layer
27
+
28
+ Add the package, **pinned to an exact version**, to the layer requirements of each function
29
+ (`src/functions/<carrier>/<function>/layer/requirements.txt`):
30
+
31
+ ```text
32
+ kloship-utils[sftp]==0.1.0
33
+ ```
34
+
35
+ `task buildlayer` (`sam build --use-container`) then installs it like any other dependency. Every dependency has
36
+ `aarch64` wheels, so it works with the `arm64` Lambdas.
37
+
38
+ ## Modules
39
+
40
+ | Module | Content |
41
+ |---|---|
42
+ | `kloship_utils.encoding` | `to_base64_gzip_json`, `from_base64_gzip_json`: payloads exchanged between Step Functions and Lambdas |
43
+ | `kloship_utils.helpers` | `to_list`, `empty_if_none`, `chunks`, `get_path` |
44
+ | `kloship_utils.units` | `convert_SI` (length, volume, weight, liquid, dates), `convert_date`, `convert_values` |
45
+ | `kloship_utils.text` | `strip_diacritics`, `strip_diacritics_in`, `replace_special_chars` |
46
+ | `kloship_utils.templates` | `render_template`, `render_zpl_label`, `split_zpl` (Jinja2) |
47
+ | `kloship_utils.xml` | `dict_to_xml`, `to_xml_string` (attributes, lists, CDATA) |
48
+ | `kloship_utils.handle_units` | `Status`, `global_group_id`, `global_group_id2`, `group_by_delivery`, `next_index_in_order`, `delete_attachments_from_result`, `clear_fields` |
49
+ | `kloship_utils.aws.s3` | `build_key`, `put_file`, `get_file`, `save_label`, `save_shipment_file`, file name helpers |
50
+ | `kloship_utils.aws.dynamodb` | HandleUnit lookups, `increment_counter`, `get_parameter`, `get_carrier_account_config`, `get_conversion`, `get_packaging_conversion`, `execute_statement_all`, `batch_execute` |
51
+ | `kloship_utils.cella` | `CellaClient`, `get_rule_result`, `get_rule_result_from_kloship_front`, `pdf_to_zpl` |
52
+ | `kloship_utils.transfers` | `TransferConfig`, `TunnelConfig`, `upload_ftp`, `upload_sftp` |
53
+ | `kloship_utils.exceptions` | `KloshipUtilsError` and its subclasses |
54
+
55
+ ### Encoding
56
+
57
+ ```python
58
+ from kloship_utils.encoding import from_base64_gzip_json, to_base64_gzip_json
59
+
60
+ def lambda_handler(event, context):
61
+ handle_units = from_base64_gzip_json(event["compressed_handle_units"])
62
+ ...
63
+ return {"compressed_handle_units": to_base64_gzip_json(enriched_handle_units)}
64
+ ```
65
+
66
+ Pass `default=str` to `to_base64_gzip_json` to serialize the `Decimal` values returned by DynamoDB.
67
+
68
+ ### Units and dates
69
+
70
+ ```python
71
+ from kloship_utils.units import convert_SI, convert_date
72
+
73
+ convert_SI(hu["WEIGHT"]["#text"], "weight", hu["WEIGHT"]["@format"], "kg") # "10.05" (2 decimals)
74
+ convert_SI("1234", "length", "mm", "cm", number_decimal=0) # "123"
75
+ convert_SI("1234", "length", "mm", "m", number_decimal=None) # "1.234" (no rounding)
76
+ convert_SI("2024/01/31", "date", "YYYY/MM/DD", "%d.%m.%Y") # "31.01.2024"
77
+ convert_date("2024-01-31 08:05:00", "YYYY-MM-DD HH:MI:SS", "CCYYMMDDHHMM") # "202401310805"
78
+ ```
79
+
80
+ Date formats accept KloShip tokens (`yyyy`, `ccyy`, `yy`, `mm`, `dd`, `hh`, `mi`/`ii`, `ss`; `mm` after `hh` means
81
+ minutes) or `strftime` patterns.
82
+
83
+ ### Labels
84
+
85
+ ```python
86
+ from kloship_utils.templates import render_zpl_label, split_zpl
87
+ from kloship_utils.aws.s3 import save_label
88
+
89
+ zpl = render_zpl_label(handle_unit, config, "layout_label.zpl") # templates/ directory of the Lambda
90
+ handle_unit["label"] = split_zpl(zpl) # {"header", "body", "footer"}
91
+ handle_unit["KLOSHIP_LABEL_FILENAME"] = save_label(
92
+ handle_unit, account_id, config, zpl, extension="zpl", content_type="x-application/zpl", carrier="MYCARRIER"
93
+ )
94
+ ```
95
+
96
+ ### S3
97
+
98
+ ```python
99
+ from kloship_utils.aws.s3 import save_shipment_file, shipment_file_name
100
+
101
+ file_name = shipment_file_name(config, "edi") # 20240131_080509_42.edi
102
+ save_shipment_file(
103
+ edi, account_id, config, file_name,
104
+ content_type="application/EDI-EDIFACT", carrier="MYCARRIER",
105
+ encoding="ISO-8859-1", content_encoding="ISO-8859-1",
106
+ )
107
+ ```
108
+
109
+ Keys follow the KloShip layout `{account}/{@senderCodeTDM}/{@serviceTagTDM}/{YYYY}/{MM}/{DD}/{file}`, in the
110
+ `STANDARD_IA` storage class. Buckets default to the `LABELS_BUCKET` / `SHIPMENT_BUCKET` environment variables.
111
+
112
+ ### DynamoDB
113
+
114
+ ```python
115
+ from kloship_utils.aws import dynamodb
116
+
117
+ handle_units = dynamodb.get_handle_units_by_global_group_id(account_id, config) # PRE-SHIPPED / SHIPPED HUs
118
+ number = dynamodb.increment_counter(config, parameter_name=config.get("ParameterNameHUCounter"))
119
+ packaging = dynamodb.get_packaging_conversion(hu["PACKAGING"], config, default=config.get("DefaultPackaging"))
120
+ ```
121
+
122
+ Table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
123
+ `CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and can be given explicitly.
124
+ All PartiQL statements are parameterized.
125
+
126
+ ### CELLA rules
127
+
128
+ ```python
129
+ from kloship_utils.cella import get_rule_result
130
+
131
+ values = get_rule_result(
132
+ config["CreateLabel"]["FrontendConversion"], # Url, WarehouseWorkerUsername, WarehouseWorkerPwd, WarehouseId, StockOwnerId
133
+ rule_config, # Name + Exit [{name, type, defaultValue}]
134
+ {"packagingDescription": hu["PACKAGING"]},
135
+ )
136
+ ```
137
+
138
+ ### FTP / SFTP
139
+
140
+ ```python
141
+ from kloship_utils.transfers import TransferConfig, TunnelConfig, upload_sftp
142
+
143
+ sftp = TransferConfig.from_carrier_config(config["CARRIER"]["Shipment"]["Sftp"])
144
+ tunnel = TunnelConfig.from_carrier_config(config["CARRIER"]["Shipment"]["TunnelSSH"], password_base64=True)
145
+ upload_sftp(sftp, file_name, edi, tunnel=tunnel) # raises TransferError (with .code 401/402/404) on failure
146
+ ```
147
+
148
+ `TransferConfig.from_carrier_config` reads the usual carrier keys whatever their case (`Host`, `Port`, `Login`,
149
+ `Password`, `RemotePath`, `EDIFileEncoding`, `UseCnOptsHostkeys`, `ParamikoKeyData`, `ParamikoSignVerifySSHData`).
150
+
151
+ ## Development
152
+
153
+ The tasks are defined in [Taskfile.yml](Taskfile.yml) ([Task](https://taskfile.dev)):
154
+
155
+ ```bash
156
+ task install # virtualenv + editable install with every extra and the dev tools
157
+ task lint # ruff (lint + format check)
158
+ task typecheck # mypy --strict
159
+ task test # pytest with coverage (fails under 90 %)
160
+ task check # lint + typecheck + test
161
+ task build # sdist + wheel in dist/, checked with twine
162
+ ```
163
+
164
+ Without Task: `python -m venv .venv && .venv/bin/pip install -e ".[dev]" && .venv/bin/pytest --cov`.
165
+
166
+ Every function has unit tests: AWS is mocked with [moto](https://github.com/getmoto/moto), HTTP with
167
+ [responses](https://github.com/getsentry/responses), FTP / SFTP with mocks. The tests use synthetic data only.
168
+
169
+ ### Continuous integration
170
+
171
+ - [`ci.yml`](.github/workflows/ci.yml) runs on every push and pull request: ruff, mypy, the tests on Python 3.11,
172
+ 3.12 and 3.13 (coverage ≥ 90 %), then builds and smoke-tests the wheel.
173
+ - [`release.yml`](.github/workflows/release.yml) runs on `vX.Y.Z` tags: CI, build, publication on PyPI, GitHub
174
+ Release with generated notes.
175
+
176
+ ## Versioning and releases
177
+
178
+ The package follows [Semantic Versioning](https://semver.org): breaking changes bump the major version (minor while
179
+ in `0.x`). The version is computed from the git tag by `hatch-vcs`: there is no version to edit in the code.
180
+
181
+ To publish a release:
182
+
183
+ ```bash
184
+ git checkout main && git pull
185
+ git tag v0.1.0
186
+ git push origin v0.1.0
187
+ ```
188
+
189
+ Pre-releases use PEP 440 suffixes (`v0.2.0rc1`) and are flagged as pre-releases on GitHub. Release notes are
190
+ generated from the merged pull requests, grouped by label (see [.github/release.yml](.github/release.yml)).
191
+
192
+ ### First publication (one-time setup)
193
+
194
+ 1. Create an API token on [pypi.org](https://pypi.org/manage/account/token/). The project does not exist yet, so the
195
+ first token must have the *entire account* scope.
196
+ 2. In the GitHub repository settings, add it as the `PYPI_TOKEN` secret (repository secret, or secret of the `pypi`
197
+ environment used by the publish job).
198
+ 3. Push the first tag (`v0.1.0`).
199
+ 4. Once the project exists on PyPI, replace the token with one scoped to the `kloship-utils` project.
200
+
201
+ ## License
202
+
203
+ [MIT](LICENSE) © Groupe Klocel