kloship-utils 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of kloship-utils might be problematic. Click here for more details.
- kloship_utils-0.1.0/.gitignore +26 -0
- kloship_utils-0.1.0/CLAUDE.md +88 -0
- kloship_utils-0.1.0/LICENSE +21 -0
- kloship_utils-0.1.0/PKG-INFO +252 -0
- kloship_utils-0.1.0/README.md +203 -0
- kloship_utils-0.1.0/docs/MIGRATION.md +165 -0
- kloship_utils-0.1.0/docs/SHARED_FUNCTIONS_ANALYSIS.md +102 -0
- kloship_utils-0.1.0/pyproject.toml +121 -0
- kloship_utils-0.1.0/src/kloship_utils/__init__.py +26 -0
- kloship_utils-0.1.0/src/kloship_utils/_version.py +24 -0
- kloship_utils-0.1.0/src/kloship_utils/aws/__init__.py +36 -0
- kloship_utils-0.1.0/src/kloship_utils/aws/dynamodb.py +376 -0
- kloship_utils-0.1.0/src/kloship_utils/aws/s3.py +216 -0
- kloship_utils-0.1.0/src/kloship_utils/cella.py +255 -0
- kloship_utils-0.1.0/src/kloship_utils/encoding.py +41 -0
- kloship_utils-0.1.0/src/kloship_utils/exceptions.py +45 -0
- kloship_utils-0.1.0/src/kloship_utils/handle_units.py +136 -0
- kloship_utils-0.1.0/src/kloship_utils/helpers.py +51 -0
- kloship_utils-0.1.0/src/kloship_utils/py.typed +0 -0
- kloship_utils-0.1.0/src/kloship_utils/templates.py +73 -0
- kloship_utils-0.1.0/src/kloship_utils/text.py +57 -0
- kloship_utils-0.1.0/src/kloship_utils/transfers/__init__.py +11 -0
- kloship_utils-0.1.0/src/kloship_utils/transfers/config.py +177 -0
- kloship_utils-0.1.0/src/kloship_utils/transfers/ftp.py +97 -0
- kloship_utils-0.1.0/src/kloship_utils/transfers/sftp.py +148 -0
- kloship_utils-0.1.0/src/kloship_utils/units.py +255 -0
- kloship_utils-0.1.0/src/kloship_utils/xml.py +67 -0
- kloship_utils-0.1.0/tests/__init__.py +0 -0
- kloship_utils-0.1.0/tests/conftest.py +123 -0
- kloship_utils-0.1.0/tests/fixtures/carrier_account_config.json +16 -0
- kloship_utils-0.1.0/tests/fixtures/handle_units.json +53 -0
- kloship_utils-0.1.0/tests/unit/__init__.py +0 -0
- kloship_utils-0.1.0/tests/unit/test_cella.py +180 -0
- kloship_utils-0.1.0/tests/unit/test_dynamodb.py +198 -0
- kloship_utils-0.1.0/tests/unit/test_encoding.py +37 -0
- kloship_utils-0.1.0/tests/unit/test_ftp.py +92 -0
- kloship_utils-0.1.0/tests/unit/test_handle_units.py +86 -0
- kloship_utils-0.1.0/tests/unit/test_helpers.py +29 -0
- kloship_utils-0.1.0/tests/unit/test_s3.py +98 -0
- kloship_utils-0.1.0/tests/unit/test_sftp.py +155 -0
- kloship_utils-0.1.0/tests/unit/test_templates.py +39 -0
- kloship_utils-0.1.0/tests/unit/test_text.py +34 -0
- kloship_utils-0.1.0/tests/unit/test_transfers_config.py +95 -0
- kloship_utils-0.1.0/tests/unit/test_units.py +164 -0
- kloship_utils-0.1.0/tests/unit/test_xml.py +46 -0
|
@@ -0,0 +1,26 @@
|
|
|
1
|
+
# Python
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
.venv/
|
|
6
|
+
env*/
|
|
7
|
+
|
|
8
|
+
# Generated by hatch-vcs at build / install time
|
|
9
|
+
src/kloship_utils/_version.py
|
|
10
|
+
|
|
11
|
+
# Build
|
|
12
|
+
build/
|
|
13
|
+
dist/
|
|
14
|
+
|
|
15
|
+
# Tools
|
|
16
|
+
.mypy_cache/
|
|
17
|
+
.ruff_cache/
|
|
18
|
+
.pytest_cache/
|
|
19
|
+
.coverage
|
|
20
|
+
coverage.xml
|
|
21
|
+
htmlcov/
|
|
22
|
+
.task/
|
|
23
|
+
|
|
24
|
+
# IDE
|
|
25
|
+
.vscode/
|
|
26
|
+
.idea/
|
|
@@ -0,0 +1,88 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
## Project
|
|
4
|
+
|
|
5
|
+
`kloship-utils` is the **public** PyPI package (MIT) of utilities shared by the KloShip carrier repositories
|
|
6
|
+
(`Groupe-Klocel/Kloship-<carrier>`): payload encoding, unit / date conversions, text cleaning, Jinja2 templates,
|
|
7
|
+
dict to XML, HandleUnit helpers, S3 archiving, DynamoDB lookups / counters / conversions, CELLA rules, FTP / SFTP uploads.
|
|
8
|
+
Carriers install it in their Lambda layers (`layer/requirements.txt`, exact version pin) and import it instead of
|
|
9
|
+
copy-pasting helpers.
|
|
10
|
+
|
|
11
|
+
Read at the start of a session (versioned in git, unlike the local Claude memory):
|
|
12
|
+
|
|
13
|
+
- [`docs/SHARED_FUNCTIONS_ANALYSIS.md`](docs/SHARED_FUNCTIONS_ANALYSIS.md) — which carrier helpers were moved here,
|
|
14
|
+
their variants, the legacy bugs, what stays in the carriers and the candidates for the next versions.
|
|
15
|
+
- [`docs/MIGRATION.md`](docs/MIGRATION.md) — how a carrier switches to the library, legacy → library mapping and
|
|
16
|
+
behavior differences.
|
|
17
|
+
- [`README.md`](README.md) — usage, development and release process.
|
|
18
|
+
|
|
19
|
+
## Constraints
|
|
20
|
+
|
|
21
|
+
- **Public package**: never commit secrets, real customer data (names, addresses, account numbers), internal host
|
|
22
|
+
names or client specific configuration — in code, docs or test fixtures. Test data must be synthetic.
|
|
23
|
+
- **Python ≥ 3.11** (AWS Lambda `python3.11` / `python3.12` runtimes, `arm64`).
|
|
24
|
+
- **Light dependencies**: core deps are pure Python (`requests`, `Jinja2`, `Unidecode`). `boto3` is provided by the
|
|
25
|
+
Lambda runtime and is only an optional extra (`aws`); import it lazily (`kloship_utils.aws.boto3_module()`).
|
|
26
|
+
`paramiko` is the optional `sftp` extra, imported lazily too. Do not add heavy dependencies to the core.
|
|
27
|
+
- **Backward compatibility**: carriers pin exact versions, but follow SemVer — a change of signature or behavior of a
|
|
28
|
+
public function is a breaking change. The legacy-compatible names / signatures (`convert_SI`,
|
|
29
|
+
`get_handle_unit_from_shipment_number`, `get_rule_result_from_kloship_front`...) must be kept.
|
|
30
|
+
|
|
31
|
+
## Layout
|
|
32
|
+
|
|
33
|
+
```
|
|
34
|
+
src/kloship_utils/
|
|
35
|
+
encoding.py helpers.py units.py text.py templates.py xml.py handle_units.py cella.py exceptions.py
|
|
36
|
+
aws/ s3.py, dynamodb.py (boto3, lazy)
|
|
37
|
+
transfers/ config.py (TransferConfig, TunnelConfig, TransferResult), ftp.py, sftp.py (paramiko, lazy)
|
|
38
|
+
tests/
|
|
39
|
+
conftest.py moto fixtures (S3, DynamoDB), env vars, FakeDynamoDBClient
|
|
40
|
+
fixtures/ synthetic CarrierAccountConfig / HandleUnits
|
|
41
|
+
unit/ one test file per module
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
## Conventions
|
|
45
|
+
|
|
46
|
+
- Everything in **English**: code, names, docstrings, comments, log / error messages, docs, commit messages.
|
|
47
|
+
- Google style docstrings with `Args` / `Returns` / `Raises`; full type hints (`mypy --strict` must pass).
|
|
48
|
+
- Use the module logger (`logging.getLogger(__name__)`), never `print`. Never log passwords or tokens.
|
|
49
|
+
- Raise the exceptions of `kloship_utils.exceptions` (subclasses of `KloshipUtilsError`; keep the builtin base class
|
|
50
|
+
the legacy code raised, e.g. `ValueError`, so that carriers' `except` blocks keep working).
|
|
51
|
+
- DynamoDB: table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
|
|
52
|
+
`CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and must stay overridable
|
|
53
|
+
by argument. PartiQL statements are **always parameterized** (`?`), never built with f-strings of values.
|
|
54
|
+
- AWS / HTTP helpers accept an optional `client` / `resource` / `session` argument for reuse and tests.
|
|
55
|
+
- Ruff (lint + format, 120 columns) and mypy configuration live in `pyproject.toml`.
|
|
56
|
+
|
|
57
|
+
## Commands
|
|
58
|
+
|
|
59
|
+
```bash
|
|
60
|
+
task install # .venv + pip install -e ".[dev]"
|
|
61
|
+
task lint # ruff check + ruff format --check
|
|
62
|
+
task typecheck # mypy (strict)
|
|
63
|
+
task test # pytest --cov, fails under 90 %
|
|
64
|
+
task check # all of the above
|
|
65
|
+
task build # sdist + wheel + twine check
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Without Task: `.venv/bin/pytest --cov`, `.venv/bin/ruff check src tests`, `.venv/bin/mypy`.
|
|
69
|
+
|
|
70
|
+
## Rules
|
|
71
|
+
|
|
72
|
+
- **Every new or changed function comes with unit tests** (moto for AWS, `responses` for HTTP, mocks for FTP /
|
|
73
|
+
SFTP). Coverage must stay ≥ 90 % and the CI must be green before merging.
|
|
74
|
+
- CI: `.github/workflows/ci.yml` runs lint, mypy, tests (Python 3.11 / 3.12 / 3.13) and the build on every push and
|
|
75
|
+
pull request.
|
|
76
|
+
- When a helper moves from a carrier to the library, update `docs/SHARED_FUNCTIONS_ANALYSIS.md` and
|
|
77
|
+
`docs/MIGRATION.md` (mapping and behavior differences).
|
|
78
|
+
|
|
79
|
+
## Release
|
|
80
|
+
|
|
81
|
+
The version comes from the git tag (`hatch-vcs`), there is no version number in the code.
|
|
82
|
+
|
|
83
|
+
```bash
|
|
84
|
+
git tag vX.Y.Z && git push origin vX.Y.Z # pre-release: vX.Y.ZrcN
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`.github/workflows/release.yml` then runs the CI, builds, publishes to PyPI (secret `PYPI_TOKEN`) and creates the
|
|
88
|
+
GitHub Release with notes generated from the merged pull requests (labels in `.github/release.yml`).
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Groupe Klocel
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,252 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: kloship-utils
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Shared utilities for KloShip carrier integrations (encoding, unit conversions, S3, DynamoDB, CELLA, FTP/SFTP transfers).
|
|
5
|
+
Project-URL: Homepage, https://github.com/Groupe-Klocel/Kloship-utils
|
|
6
|
+
Project-URL: Repository, https://github.com/Groupe-Klocel/Kloship-utils
|
|
7
|
+
Project-URL: Issues, https://github.com/Groupe-Klocel/Kloship-utils/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/Groupe-Klocel/Kloship-utils/releases
|
|
9
|
+
Author-email: Guillaume Candau <guillaume.candau@klocel.com>
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: aws-lambda,carrier,edi,klocel,kloship,shipping,zpl
|
|
13
|
+
Classifier: Development Status :: 4 - Beta
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Topic :: Software Development :: Libraries
|
|
22
|
+
Classifier: Typing :: Typed
|
|
23
|
+
Requires-Python: >=3.11
|
|
24
|
+
Requires-Dist: jinja2>=3.1
|
|
25
|
+
Requires-Dist: requests>=2.31
|
|
26
|
+
Requires-Dist: unidecode>=1.3
|
|
27
|
+
Provides-Extra: all
|
|
28
|
+
Requires-Dist: boto3>=1.28; extra == 'all'
|
|
29
|
+
Requires-Dist: paramiko>=3.4; extra == 'all'
|
|
30
|
+
Provides-Extra: aws
|
|
31
|
+
Requires-Dist: boto3>=1.28; extra == 'aws'
|
|
32
|
+
Provides-Extra: dev
|
|
33
|
+
Requires-Dist: boto3-stubs[dynamodb,s3]>=1.28; extra == 'dev'
|
|
34
|
+
Requires-Dist: boto3>=1.28; extra == 'dev'
|
|
35
|
+
Requires-Dist: build>=1.2; extra == 'dev'
|
|
36
|
+
Requires-Dist: moto[dynamodb,s3]>=5.0; extra == 'dev'
|
|
37
|
+
Requires-Dist: mypy>=1.10; extra == 'dev'
|
|
38
|
+
Requires-Dist: paramiko>=3.4; extra == 'dev'
|
|
39
|
+
Requires-Dist: pytest-cov>=5.0; extra == 'dev'
|
|
40
|
+
Requires-Dist: pytest>=8.0; extra == 'dev'
|
|
41
|
+
Requires-Dist: responses>=0.25; extra == 'dev'
|
|
42
|
+
Requires-Dist: ruff>=0.6; extra == 'dev'
|
|
43
|
+
Requires-Dist: twine>=5.1; extra == 'dev'
|
|
44
|
+
Requires-Dist: types-paramiko>=3.4; extra == 'dev'
|
|
45
|
+
Requires-Dist: types-requests>=2.31; extra == 'dev'
|
|
46
|
+
Provides-Extra: sftp
|
|
47
|
+
Requires-Dist: paramiko>=3.4; extra == 'sftp'
|
|
48
|
+
Description-Content-Type: text/markdown
|
|
49
|
+
|
|
50
|
+
# kloship-utils
|
|
51
|
+
|
|
52
|
+
Shared utilities for the [KloShip](https://www.klocel.com) carrier integrations.
|
|
53
|
+
|
|
54
|
+
Every KloShip carrier repository (Chronopost, Colissimo, DHL, UPS, Heppner, XPO, Mondial Relay...) used to copy the
|
|
55
|
+
same helpers into each Lambda: payload encoding, unit conversions, S3 archiving, DynamoDB lookups, CELLA rules,
|
|
56
|
+
FTP/SFTP uploads... `kloship-utils` gathers them in a single, tested and versioned package published on PyPI, so that
|
|
57
|
+
carriers install it in their Lambda layer and import it instead of copy-pasting.
|
|
58
|
+
|
|
59
|
+
See [docs/SHARED_FUNCTIONS_ANALYSIS.md](docs/SHARED_FUNCTIONS_ANALYSIS.md) for the analysis of the duplicated code and
|
|
60
|
+
[docs/MIGRATION.md](docs/MIGRATION.md) to migrate a carrier.
|
|
61
|
+
|
|
62
|
+
## Installation
|
|
63
|
+
|
|
64
|
+
Requires Python 3.11 or later.
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
pip install kloship-utils # core: encoding, units, text, templates, xml, handle units, CELLA, FTP
|
|
68
|
+
pip install "kloship-utils[sftp]" # + SFTP uploads (paramiko)
|
|
69
|
+
pip install "kloship-utils[aws]" # + boto3, for S3 / DynamoDB helpers outside AWS Lambda
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
`boto3` is **not** a mandatory dependency: the AWS Lambda Python runtime already provides it, and bundling it would
|
|
73
|
+
add ~80 MB to every layer.
|
|
74
|
+
|
|
75
|
+
### In a carrier Lambda layer
|
|
76
|
+
|
|
77
|
+
Add the package, **pinned to an exact version**, to the layer requirements of each function
|
|
78
|
+
(`src/functions/<carrier>/<function>/layer/requirements.txt`):
|
|
79
|
+
|
|
80
|
+
```text
|
|
81
|
+
kloship-utils[sftp]==0.1.0
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
`task buildlayer` (`sam build --use-container`) then installs it like any other dependency. Every dependency has
|
|
85
|
+
`aarch64` wheels, so it works with the `arm64` Lambdas.
|
|
86
|
+
|
|
87
|
+
## Modules
|
|
88
|
+
|
|
89
|
+
| Module | Content |
|
|
90
|
+
|---|---|
|
|
91
|
+
| `kloship_utils.encoding` | `to_base64_gzip_json`, `from_base64_gzip_json`: payloads exchanged between Step Functions and Lambdas |
|
|
92
|
+
| `kloship_utils.helpers` | `to_list`, `empty_if_none`, `chunks`, `get_path` |
|
|
93
|
+
| `kloship_utils.units` | `convert_SI` (length, volume, weight, liquid, dates), `convert_date`, `convert_values` |
|
|
94
|
+
| `kloship_utils.text` | `strip_diacritics`, `strip_diacritics_in`, `replace_special_chars` |
|
|
95
|
+
| `kloship_utils.templates` | `render_template`, `render_zpl_label`, `split_zpl` (Jinja2) |
|
|
96
|
+
| `kloship_utils.xml` | `dict_to_xml`, `to_xml_string` (attributes, lists, CDATA) |
|
|
97
|
+
| `kloship_utils.handle_units` | `Status`, `global_group_id`, `global_group_id2`, `group_by_delivery`, `next_index_in_order`, `delete_attachments_from_result`, `clear_fields` |
|
|
98
|
+
| `kloship_utils.aws.s3` | `build_key`, `put_file`, `get_file`, `save_label`, `save_shipment_file`, file name helpers |
|
|
99
|
+
| `kloship_utils.aws.dynamodb` | HandleUnit lookups, `increment_counter`, `get_parameter`, `get_carrier_account_config`, `get_conversion`, `get_packaging_conversion`, `execute_statement_all`, `batch_execute` |
|
|
100
|
+
| `kloship_utils.cella` | `CellaClient`, `get_rule_result`, `get_rule_result_from_kloship_front`, `pdf_to_zpl` |
|
|
101
|
+
| `kloship_utils.transfers` | `TransferConfig`, `TunnelConfig`, `upload_ftp`, `upload_sftp` |
|
|
102
|
+
| `kloship_utils.exceptions` | `KloshipUtilsError` and its subclasses |
|
|
103
|
+
|
|
104
|
+
### Encoding
|
|
105
|
+
|
|
106
|
+
```python
|
|
107
|
+
from kloship_utils.encoding import from_base64_gzip_json, to_base64_gzip_json
|
|
108
|
+
|
|
109
|
+
def lambda_handler(event, context):
|
|
110
|
+
handle_units = from_base64_gzip_json(event["compressed_handle_units"])
|
|
111
|
+
...
|
|
112
|
+
return {"compressed_handle_units": to_base64_gzip_json(enriched_handle_units)}
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
Pass `default=str` to `to_base64_gzip_json` to serialize the `Decimal` values returned by DynamoDB.
|
|
116
|
+
|
|
117
|
+
### Units and dates
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
from kloship_utils.units import convert_SI, convert_date
|
|
121
|
+
|
|
122
|
+
convert_SI(hu["WEIGHT"]["#text"], "weight", hu["WEIGHT"]["@format"], "kg") # "10.05" (2 decimals)
|
|
123
|
+
convert_SI("1234", "length", "mm", "cm", number_decimal=0) # "123"
|
|
124
|
+
convert_SI("1234", "length", "mm", "m", number_decimal=None) # "1.234" (no rounding)
|
|
125
|
+
convert_SI("2024/01/31", "date", "YYYY/MM/DD", "%d.%m.%Y") # "31.01.2024"
|
|
126
|
+
convert_date("2024-01-31 08:05:00", "YYYY-MM-DD HH:MI:SS", "CCYYMMDDHHMM") # "202401310805"
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
Date formats accept KloShip tokens (`yyyy`, `ccyy`, `yy`, `mm`, `dd`, `hh`, `mi`/`ii`, `ss`; `mm` after `hh` means
|
|
130
|
+
minutes) or `strftime` patterns.
|
|
131
|
+
|
|
132
|
+
### Labels
|
|
133
|
+
|
|
134
|
+
```python
|
|
135
|
+
from kloship_utils.templates import render_zpl_label, split_zpl
|
|
136
|
+
from kloship_utils.aws.s3 import save_label
|
|
137
|
+
|
|
138
|
+
zpl = render_zpl_label(handle_unit, config, "layout_label.zpl") # templates/ directory of the Lambda
|
|
139
|
+
handle_unit["label"] = split_zpl(zpl) # {"header", "body", "footer"}
|
|
140
|
+
handle_unit["KLOSHIP_LABEL_FILENAME"] = save_label(
|
|
141
|
+
handle_unit, account_id, config, zpl, extension="zpl", content_type="x-application/zpl", carrier="MYCARRIER"
|
|
142
|
+
)
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
### S3
|
|
146
|
+
|
|
147
|
+
```python
|
|
148
|
+
from kloship_utils.aws.s3 import save_shipment_file, shipment_file_name
|
|
149
|
+
|
|
150
|
+
file_name = shipment_file_name(config, "edi") # 20240131_080509_42.edi
|
|
151
|
+
save_shipment_file(
|
|
152
|
+
edi, account_id, config, file_name,
|
|
153
|
+
content_type="application/EDI-EDIFACT", carrier="MYCARRIER",
|
|
154
|
+
encoding="ISO-8859-1", content_encoding="ISO-8859-1",
|
|
155
|
+
)
|
|
156
|
+
```
|
|
157
|
+
|
|
158
|
+
Keys follow the KloShip layout `{account}/{@senderCodeTDM}/{@serviceTagTDM}/{YYYY}/{MM}/{DD}/{file}`, in the
|
|
159
|
+
`STANDARD_IA` storage class. Buckets default to the `LABELS_BUCKET` / `SHIPMENT_BUCKET` environment variables.
|
|
160
|
+
|
|
161
|
+
### DynamoDB
|
|
162
|
+
|
|
163
|
+
```python
|
|
164
|
+
from kloship_utils.aws import dynamodb
|
|
165
|
+
|
|
166
|
+
handle_units = dynamodb.get_handle_units_by_global_group_id(account_id, config) # PRE-SHIPPED / SHIPPED HUs
|
|
167
|
+
number = dynamodb.increment_counter(config, parameter_name=config.get("ParameterNameHUCounter"))
|
|
168
|
+
packaging = dynamodb.get_packaging_conversion(hu["PACKAGING"], config, default=config.get("DefaultPackaging"))
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
Table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
|
|
172
|
+
`CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and can be given explicitly.
|
|
173
|
+
All PartiQL statements are parameterized.
|
|
174
|
+
|
|
175
|
+
### CELLA rules
|
|
176
|
+
|
|
177
|
+
```python
|
|
178
|
+
from kloship_utils.cella import get_rule_result
|
|
179
|
+
|
|
180
|
+
values = get_rule_result(
|
|
181
|
+
config["CreateLabel"]["FrontendConversion"], # Url, WarehouseWorkerUsername, WarehouseWorkerPwd, WarehouseId, StockOwnerId
|
|
182
|
+
rule_config, # Name + Exit [{name, type, defaultValue}]
|
|
183
|
+
{"packagingDescription": hu["PACKAGING"]},
|
|
184
|
+
)
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
### FTP / SFTP
|
|
188
|
+
|
|
189
|
+
```python
|
|
190
|
+
from kloship_utils.transfers import TransferConfig, TunnelConfig, upload_sftp
|
|
191
|
+
|
|
192
|
+
sftp = TransferConfig.from_carrier_config(config["CARRIER"]["Shipment"]["Sftp"])
|
|
193
|
+
tunnel = TunnelConfig.from_carrier_config(config["CARRIER"]["Shipment"]["TunnelSSH"], password_base64=True)
|
|
194
|
+
upload_sftp(sftp, file_name, edi, tunnel=tunnel) # raises TransferError (with .code 401/402/404) on failure
|
|
195
|
+
```
|
|
196
|
+
|
|
197
|
+
`TransferConfig.from_carrier_config` reads the usual carrier keys whatever their case (`Host`, `Port`, `Login`,
|
|
198
|
+
`Password`, `RemotePath`, `EDIFileEncoding`, `UseCnOptsHostkeys`, `ParamikoKeyData`, `ParamikoSignVerifySSHData`).
|
|
199
|
+
|
|
200
|
+
## Development
|
|
201
|
+
|
|
202
|
+
The tasks are defined in [Taskfile.yml](Taskfile.yml) ([Task](https://taskfile.dev)):
|
|
203
|
+
|
|
204
|
+
```bash
|
|
205
|
+
task install # virtualenv + editable install with every extra and the dev tools
|
|
206
|
+
task lint # ruff (lint + format check)
|
|
207
|
+
task typecheck # mypy --strict
|
|
208
|
+
task test # pytest with coverage (fails under 90 %)
|
|
209
|
+
task check # lint + typecheck + test
|
|
210
|
+
task build # sdist + wheel in dist/, checked with twine
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Without Task: `python -m venv .venv && .venv/bin/pip install -e ".[dev]" && .venv/bin/pytest --cov`.
|
|
214
|
+
|
|
215
|
+
Every function has unit tests: AWS is mocked with [moto](https://github.com/getmoto/moto), HTTP with
|
|
216
|
+
[responses](https://github.com/getsentry/responses), FTP / SFTP with mocks. The tests use synthetic data only.
|
|
217
|
+
|
|
218
|
+
### Continuous integration
|
|
219
|
+
|
|
220
|
+
- [`ci.yml`](.github/workflows/ci.yml) runs on every push and pull request: ruff, mypy, the tests on Python 3.11,
|
|
221
|
+
3.12 and 3.13 (coverage ≥ 90 %), then builds and smoke-tests the wheel.
|
|
222
|
+
- [`release.yml`](.github/workflows/release.yml) runs on `vX.Y.Z` tags: CI, build, publication on PyPI, GitHub
|
|
223
|
+
Release with generated notes.
|
|
224
|
+
|
|
225
|
+
## Versioning and releases
|
|
226
|
+
|
|
227
|
+
The package follows [Semantic Versioning](https://semver.org): breaking changes bump the major version (minor while
|
|
228
|
+
in `0.x`). The version is computed from the git tag by `hatch-vcs`: there is no version to edit in the code.
|
|
229
|
+
|
|
230
|
+
To publish a release:
|
|
231
|
+
|
|
232
|
+
```bash
|
|
233
|
+
git checkout main && git pull
|
|
234
|
+
git tag v0.1.0
|
|
235
|
+
git push origin v0.1.0
|
|
236
|
+
```
|
|
237
|
+
|
|
238
|
+
Pre-releases use PEP 440 suffixes (`v0.2.0rc1`) and are flagged as pre-releases on GitHub. Release notes are
|
|
239
|
+
generated from the merged pull requests, grouped by label (see [.github/release.yml](.github/release.yml)).
|
|
240
|
+
|
|
241
|
+
### First publication (one-time setup)
|
|
242
|
+
|
|
243
|
+
1. Create an API token on [pypi.org](https://pypi.org/manage/account/token/). The project does not exist yet, so the
|
|
244
|
+
first token must have the *entire account* scope.
|
|
245
|
+
2. In the GitHub repository settings, add it as the `PYPI_TOKEN` secret (repository secret, or secret of the `pypi`
|
|
246
|
+
environment used by the publish job).
|
|
247
|
+
3. Push the first tag (`v0.1.0`).
|
|
248
|
+
4. Once the project exists on PyPI, replace the token with one scoped to the `kloship-utils` project.
|
|
249
|
+
|
|
250
|
+
## License
|
|
251
|
+
|
|
252
|
+
[MIT](LICENSE) © Groupe Klocel
|
|
@@ -0,0 +1,203 @@
|
|
|
1
|
+
# kloship-utils
|
|
2
|
+
|
|
3
|
+
Shared utilities for the [KloShip](https://www.klocel.com) carrier integrations.
|
|
4
|
+
|
|
5
|
+
Every KloShip carrier repository (Chronopost, Colissimo, DHL, UPS, Heppner, XPO, Mondial Relay...) used to copy the
|
|
6
|
+
same helpers into each Lambda: payload encoding, unit conversions, S3 archiving, DynamoDB lookups, CELLA rules,
|
|
7
|
+
FTP/SFTP uploads... `kloship-utils` gathers them in a single, tested and versioned package published on PyPI, so that
|
|
8
|
+
carriers install it in their Lambda layer and import it instead of copy-pasting.
|
|
9
|
+
|
|
10
|
+
See [docs/SHARED_FUNCTIONS_ANALYSIS.md](docs/SHARED_FUNCTIONS_ANALYSIS.md) for the analysis of the duplicated code and
|
|
11
|
+
[docs/MIGRATION.md](docs/MIGRATION.md) to migrate a carrier.
|
|
12
|
+
|
|
13
|
+
## Installation
|
|
14
|
+
|
|
15
|
+
Requires Python 3.11 or later.
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
pip install kloship-utils # core: encoding, units, text, templates, xml, handle units, CELLA, FTP
|
|
19
|
+
pip install "kloship-utils[sftp]" # + SFTP uploads (paramiko)
|
|
20
|
+
pip install "kloship-utils[aws]" # + boto3, for S3 / DynamoDB helpers outside AWS Lambda
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
`boto3` is **not** a mandatory dependency: the AWS Lambda Python runtime already provides it, and bundling it would
|
|
24
|
+
add ~80 MB to every layer.
|
|
25
|
+
|
|
26
|
+
### In a carrier Lambda layer
|
|
27
|
+
|
|
28
|
+
Add the package, **pinned to an exact version**, to the layer requirements of each function
|
|
29
|
+
(`src/functions/<carrier>/<function>/layer/requirements.txt`):
|
|
30
|
+
|
|
31
|
+
```text
|
|
32
|
+
kloship-utils[sftp]==0.1.0
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
`task buildlayer` (`sam build --use-container`) then installs it like any other dependency. Every dependency has
|
|
36
|
+
`aarch64` wheels, so it works with the `arm64` Lambdas.
|
|
37
|
+
|
|
38
|
+
## Modules
|
|
39
|
+
|
|
40
|
+
| Module | Content |
|
|
41
|
+
|---|---|
|
|
42
|
+
| `kloship_utils.encoding` | `to_base64_gzip_json`, `from_base64_gzip_json`: payloads exchanged between Step Functions and Lambdas |
|
|
43
|
+
| `kloship_utils.helpers` | `to_list`, `empty_if_none`, `chunks`, `get_path` |
|
|
44
|
+
| `kloship_utils.units` | `convert_SI` (length, volume, weight, liquid, dates), `convert_date`, `convert_values` |
|
|
45
|
+
| `kloship_utils.text` | `strip_diacritics`, `strip_diacritics_in`, `replace_special_chars` |
|
|
46
|
+
| `kloship_utils.templates` | `render_template`, `render_zpl_label`, `split_zpl` (Jinja2) |
|
|
47
|
+
| `kloship_utils.xml` | `dict_to_xml`, `to_xml_string` (attributes, lists, CDATA) |
|
|
48
|
+
| `kloship_utils.handle_units` | `Status`, `global_group_id`, `global_group_id2`, `group_by_delivery`, `next_index_in_order`, `delete_attachments_from_result`, `clear_fields` |
|
|
49
|
+
| `kloship_utils.aws.s3` | `build_key`, `put_file`, `get_file`, `save_label`, `save_shipment_file`, file name helpers |
|
|
50
|
+
| `kloship_utils.aws.dynamodb` | HandleUnit lookups, `increment_counter`, `get_parameter`, `get_carrier_account_config`, `get_conversion`, `get_packaging_conversion`, `execute_statement_all`, `batch_execute` |
|
|
51
|
+
| `kloship_utils.cella` | `CellaClient`, `get_rule_result`, `get_rule_result_from_kloship_front`, `pdf_to_zpl` |
|
|
52
|
+
| `kloship_utils.transfers` | `TransferConfig`, `TunnelConfig`, `upload_ftp`, `upload_sftp` |
|
|
53
|
+
| `kloship_utils.exceptions` | `KloshipUtilsError` and its subclasses |
|
|
54
|
+
|
|
55
|
+
### Encoding
|
|
56
|
+
|
|
57
|
+
```python
|
|
58
|
+
from kloship_utils.encoding import from_base64_gzip_json, to_base64_gzip_json
|
|
59
|
+
|
|
60
|
+
def lambda_handler(event, context):
|
|
61
|
+
handle_units = from_base64_gzip_json(event["compressed_handle_units"])
|
|
62
|
+
...
|
|
63
|
+
return {"compressed_handle_units": to_base64_gzip_json(enriched_handle_units)}
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Pass `default=str` to `to_base64_gzip_json` to serialize the `Decimal` values returned by DynamoDB.
|
|
67
|
+
|
|
68
|
+
### Units and dates
|
|
69
|
+
|
|
70
|
+
```python
|
|
71
|
+
from kloship_utils.units import convert_SI, convert_date
|
|
72
|
+
|
|
73
|
+
convert_SI(hu["WEIGHT"]["#text"], "weight", hu["WEIGHT"]["@format"], "kg") # "10.05" (2 decimals)
|
|
74
|
+
convert_SI("1234", "length", "mm", "cm", number_decimal=0) # "123"
|
|
75
|
+
convert_SI("1234", "length", "mm", "m", number_decimal=None) # "1.234" (no rounding)
|
|
76
|
+
convert_SI("2024/01/31", "date", "YYYY/MM/DD", "%d.%m.%Y") # "31.01.2024"
|
|
77
|
+
convert_date("2024-01-31 08:05:00", "YYYY-MM-DD HH:MI:SS", "CCYYMMDDHHMM") # "202401310805"
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Date formats accept KloShip tokens (`yyyy`, `ccyy`, `yy`, `mm`, `dd`, `hh`, `mi`/`ii`, `ss`; `mm` after `hh` means
|
|
81
|
+
minutes) or `strftime` patterns.
|
|
82
|
+
|
|
83
|
+
### Labels
|
|
84
|
+
|
|
85
|
+
```python
|
|
86
|
+
from kloship_utils.templates import render_zpl_label, split_zpl
|
|
87
|
+
from kloship_utils.aws.s3 import save_label
|
|
88
|
+
|
|
89
|
+
zpl = render_zpl_label(handle_unit, config, "layout_label.zpl") # templates/ directory of the Lambda
|
|
90
|
+
handle_unit["label"] = split_zpl(zpl) # {"header", "body", "footer"}
|
|
91
|
+
handle_unit["KLOSHIP_LABEL_FILENAME"] = save_label(
|
|
92
|
+
handle_unit, account_id, config, zpl, extension="zpl", content_type="x-application/zpl", carrier="MYCARRIER"
|
|
93
|
+
)
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
### S3
|
|
97
|
+
|
|
98
|
+
```python
|
|
99
|
+
from kloship_utils.aws.s3 import save_shipment_file, shipment_file_name
|
|
100
|
+
|
|
101
|
+
file_name = shipment_file_name(config, "edi") # 20240131_080509_42.edi
|
|
102
|
+
save_shipment_file(
|
|
103
|
+
edi, account_id, config, file_name,
|
|
104
|
+
content_type="application/EDI-EDIFACT", carrier="MYCARRIER",
|
|
105
|
+
encoding="ISO-8859-1", content_encoding="ISO-8859-1",
|
|
106
|
+
)
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
Keys follow the KloShip layout `{account}/{@senderCodeTDM}/{@serviceTagTDM}/{YYYY}/{MM}/{DD}/{file}`, in the
|
|
110
|
+
`STANDARD_IA` storage class. Buckets default to the `LABELS_BUCKET` / `SHIPMENT_BUCKET` environment variables.
|
|
111
|
+
|
|
112
|
+
### DynamoDB
|
|
113
|
+
|
|
114
|
+
```python
|
|
115
|
+
from kloship_utils.aws import dynamodb
|
|
116
|
+
|
|
117
|
+
handle_units = dynamodb.get_handle_units_by_global_group_id(account_id, config) # PRE-SHIPPED / SHIPPED HUs
|
|
118
|
+
number = dynamodb.increment_counter(config, parameter_name=config.get("ParameterNameHUCounter"))
|
|
119
|
+
packaging = dynamodb.get_packaging_conversion(hu["PACKAGING"], config, default=config.get("DefaultPackaging"))
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
Table names default to the standard environment variables (`HANDLE_UNIT_TABLE_NAME`,
|
|
123
|
+
`CARRIER_ACCOUNT_CONFIG_TABLE_NAME`, `PARAMETERS_TABLE_NAME`, `CONVERSIONS_TABLE_NAME`) and can be given explicitly.
|
|
124
|
+
All PartiQL statements are parameterized.
|
|
125
|
+
|
|
126
|
+
### CELLA rules
|
|
127
|
+
|
|
128
|
+
```python
|
|
129
|
+
from kloship_utils.cella import get_rule_result
|
|
130
|
+
|
|
131
|
+
values = get_rule_result(
|
|
132
|
+
config["CreateLabel"]["FrontendConversion"], # Url, WarehouseWorkerUsername, WarehouseWorkerPwd, WarehouseId, StockOwnerId
|
|
133
|
+
rule_config, # Name + Exit [{name, type, defaultValue}]
|
|
134
|
+
{"packagingDescription": hu["PACKAGING"]},
|
|
135
|
+
)
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
### FTP / SFTP
|
|
139
|
+
|
|
140
|
+
```python
|
|
141
|
+
from kloship_utils.transfers import TransferConfig, TunnelConfig, upload_sftp
|
|
142
|
+
|
|
143
|
+
sftp = TransferConfig.from_carrier_config(config["CARRIER"]["Shipment"]["Sftp"])
|
|
144
|
+
tunnel = TunnelConfig.from_carrier_config(config["CARRIER"]["Shipment"]["TunnelSSH"], password_base64=True)
|
|
145
|
+
upload_sftp(sftp, file_name, edi, tunnel=tunnel) # raises TransferError (with .code 401/402/404) on failure
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
`TransferConfig.from_carrier_config` reads the usual carrier keys whatever their case (`Host`, `Port`, `Login`,
|
|
149
|
+
`Password`, `RemotePath`, `EDIFileEncoding`, `UseCnOptsHostkeys`, `ParamikoKeyData`, `ParamikoSignVerifySSHData`).
|
|
150
|
+
|
|
151
|
+
## Development
|
|
152
|
+
|
|
153
|
+
The tasks are defined in [Taskfile.yml](Taskfile.yml) ([Task](https://taskfile.dev)):
|
|
154
|
+
|
|
155
|
+
```bash
|
|
156
|
+
task install # virtualenv + editable install with every extra and the dev tools
|
|
157
|
+
task lint # ruff (lint + format check)
|
|
158
|
+
task typecheck # mypy --strict
|
|
159
|
+
task test # pytest with coverage (fails under 90 %)
|
|
160
|
+
task check # lint + typecheck + test
|
|
161
|
+
task build # sdist + wheel in dist/, checked with twine
|
|
162
|
+
```
|
|
163
|
+
|
|
164
|
+
Without Task: `python -m venv .venv && .venv/bin/pip install -e ".[dev]" && .venv/bin/pytest --cov`.
|
|
165
|
+
|
|
166
|
+
Every function has unit tests: AWS is mocked with [moto](https://github.com/getmoto/moto), HTTP with
|
|
167
|
+
[responses](https://github.com/getsentry/responses), FTP / SFTP with mocks. The tests use synthetic data only.
|
|
168
|
+
|
|
169
|
+
### Continuous integration
|
|
170
|
+
|
|
171
|
+
- [`ci.yml`](.github/workflows/ci.yml) runs on every push and pull request: ruff, mypy, the tests on Python 3.11,
|
|
172
|
+
3.12 and 3.13 (coverage ≥ 90 %), then builds and smoke-tests the wheel.
|
|
173
|
+
- [`release.yml`](.github/workflows/release.yml) runs on `vX.Y.Z` tags: CI, build, publication on PyPI, GitHub
|
|
174
|
+
Release with generated notes.
|
|
175
|
+
|
|
176
|
+
## Versioning and releases
|
|
177
|
+
|
|
178
|
+
The package follows [Semantic Versioning](https://semver.org): breaking changes bump the major version (minor while
|
|
179
|
+
in `0.x`). The version is computed from the git tag by `hatch-vcs`: there is no version to edit in the code.
|
|
180
|
+
|
|
181
|
+
To publish a release:
|
|
182
|
+
|
|
183
|
+
```bash
|
|
184
|
+
git checkout main && git pull
|
|
185
|
+
git tag v0.1.0
|
|
186
|
+
git push origin v0.1.0
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Pre-releases use PEP 440 suffixes (`v0.2.0rc1`) and are flagged as pre-releases on GitHub. Release notes are
|
|
190
|
+
generated from the merged pull requests, grouped by label (see [.github/release.yml](.github/release.yml)).
|
|
191
|
+
|
|
192
|
+
### First publication (one-time setup)
|
|
193
|
+
|
|
194
|
+
1. Create an API token on [pypi.org](https://pypi.org/manage/account/token/). The project does not exist yet, so the
|
|
195
|
+
first token must have the *entire account* scope.
|
|
196
|
+
2. In the GitHub repository settings, add it as the `PYPI_TOKEN` secret (repository secret, or secret of the `pypi`
|
|
197
|
+
environment used by the publish job).
|
|
198
|
+
3. Push the first tag (`v0.1.0`).
|
|
199
|
+
4. Once the project exists on PyPI, replace the token with one scoped to the `kloship-utils` project.
|
|
200
|
+
|
|
201
|
+
## License
|
|
202
|
+
|
|
203
|
+
[MIT](LICENSE) © Groupe Klocel
|