data-manipulation 0.50__tar.gz → 0.54__tar.gz

{data_manipulation-0.50 → data_manipulation-0.54}/.gitignore

@@ -51,15 +51,15 @@ coverage.xml
 # Django stuff:
 *.log
 
-# Sphinx documentation
-# docs/_build/
-!docs/build
+# MkDocs documentation
+/site/
 
 # PyBuilder
 target/
 
 # DotEnv configuration
 .env
+.pypirc
 
 # Database
 *.db
@@ -93,7 +93,8 @@ target/
 .mypy_cache/
 
 # custom
+.opencode
+.specstory
 build.sh
-install_packages.sh
 docs/README*
-.opencode
+install_packages.sh

data_manipulation-0.54/CHANGELOG.md

@@ -0,0 +1,92 @@
+# Changelog
+
+## [Unreleased]
+
+### Removed
+
+- **`django_` module** - removed entirely (breaking change). The Django helpers
+  (`init_django`, `get_django_countries_dict`, `django_validate_email`,
+  `django_validate_url`, `django_validate_phone`) now live in the
+  [`django-util`](https://github.com/shawnngtq/django-util) package:
+  `django_util.utility` (`init_django`, `get_django_countries_dict`) and
+  `django_util.validators` (the `django_validate_*` functions). `data_manipulation`
+  no longer depends on Django. This warrants a major version bump on release.
+
+- **`pandas_.clean_none()`** - function has been removed.
+  Migrate to pandas built-ins:
+
+  ```python
+  df = df.replace(r"^\s*$", np.nan, regex=True)
+  df = df.where(pd.notnull(df), None)
+  ```
+
+- **`pandas_.config_pandas_display()`** - function has been removed.
+  Set display options directly:
+
+  ```python
+  pd.set_option("display.max_columns", 500)
+  pd.set_option("display.max_colwidth", 500)
+  pd.set_option("display.expand_frame_repr", True)
+  ```
+
+- **`prometheus_` module** - was an empty placeholder; removed from
+  the public API and from `__init__` imports.
+
+### Changed
+
+- **`base.get_none_variation()`** - now returns a `frozenset` of 7
+  lowercase canonical strings instead of a 72-item list of case
+  permutations. Use `.casefold()` for membership checks:
+
+  ```python
+  # before
+  if value in get_none_variation():
+      ...
+  # after
+  if isinstance(value, str) and value.casefold() in get_none_variation():
+      ...
+  ```
+
+- **`pyspark_.group_count()`** - the `percent` column is now computed
+  as a native Spark column expression instead of a Python UDF.
+  Behaviour is identical; performance improves on large datasets.
+
+- **`pyspark_.column_into_list()`** - raises `ValueError` (with the
+  list of available columns) when the requested column does not exist.
+  Previously returned `None` silently.
+
+- **`pandas_.print_dataframe_overview()`** - now returns a
+  `Dict[str, Dict[str, Any]]` of per-column stats (`unique`,
+  `null_count`, `value_counts`). Still prints to stdout as before;
+  existing call sites are unaffected.
+
+- **`pandas_.compare_dataframes()`** - now returns a comparison
+  summary dict (`same_length`, `df1_length`, `df2_length`, `columns`).
+  Still prints to stdout as before; existing call sites are unaffected.
+
+- **`pyspark_` module** - all `print()` calls replaced with
+  `logger.info()` / `logger.debug()`. Output is now routed through
+  the standard logging pipeline and can be silenced or redirected.
+
+- **`sqlalchemy_.create_sqlalchemy_engine()`** - duplicate
+  `if/elif` timeout branches (both setting the same value) consolidated
+  into a single condition.
+
+- **`django_.django_validate_phone()`** - now has an explicit
+  `return None` on failure instead of a bare `return`.
+
+### Fixed (previous pass - reference)
+
+- `pyspark_.columns_statistics()` - always-true condition in empty-column
+  detection (third clause was truthy for any non-empty string).
+- `pandas_.dtypes_dictionary()` - `.iteritems()` removed in pandas 2.0;
+  replaced with `.items()`.
+- Bare `except:` replaced with `except Exception:` in `pandas_` and
+  `django_`.
+- `base.string_boolean_to_int()` - removed (used `distutils`, dropped
+  in Python 3.12).
+- `flask_.py` - deleted (was entirely commented-out dead code).
+- `beautifulsoup_.preprocess()` - fixed mixed tabs/spaces and invalid
+  `\s` escape sequences.
+- All modules now guard third-party imports with `try/except`; the
+  package imports cleanly even when optional dependencies are missing.

{data_manipulation-0.50 → data_manipulation-0.54}/CODE_OF_CONDUCT.md

@@ -23,13 +23,16 @@ include:
 Examples of unacceptable behavior by participants include:
 
 * The use of sexualized language or imagery and unwelcome sexual attention or
- advances
+
+  advances
 * Trolling, insulting/derogatory comments, and personal or political attacks
 * Public or private harassment
 * Publishing others' private information, such as a physical or electronic
- address, without explicit permission
+
+  address, without explicit permission
 * Other conduct which could reasonably be considered inappropriate in a
- professional setting
+
+  professional setting
 
 ## Our Responsibilities
 
@@ -55,7 +58,7 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at Shawn.coding.acc@gmail.com. All
+reported by contacting the project team at <Shawn.coding.acc@gmail.com>. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.
@@ -68,9 +71,9 @@ members of the project's leadership.
 ## Attribution
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>
 
 [homepage]: https://www.contributor-covenant.org
 
 For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq
+<https://www.contributor-covenant.org/faq>

{data_manipulation-0.50 → data_manipulation-0.54}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data_manipulation
-Version: 0.50
+Version: 0.54
 Summary: Powerful data manipulation
 Author-email: Shawn Ng <shawn.coding.acc@gmail.com>
 License-Expression: BSD-3-Clause
@@ -10,21 +10,16 @@ Classifier: Environment :: Console
 Classifier: Operating System :: OS Independent
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Cython
 Classifier: Topic :: Scientific/Engineering
-Requires-Python: >=3.9
+Requires-Python: >=3.14
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: beautifulsoup4
 Requires-Dist: boto3
 Requires-Dist: cryptography
-Requires-Dist: django
 Requires-Dist: flask
-Requires-Dist: geopandas
 Requires-Dist: ipython
 Requires-Dist: loguru
 Requires-Dist: mysql-connector-python
@@ -34,6 +29,9 @@ Requires-Dist: psycopg
 Requires-Dist: pyspark
 Requires-Dist: python-dotenv
 Requires-Dist: sqlalchemy
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-mock; extra == "test"
 Dynamic: license-file
 
 # Data Manipulation
@@ -43,6 +41,7 @@ Dynamic: license-file
   - [Where to get it](#where-to-get-it)
   - [Dependencies](#dependencies)
   - [License](#license)
+  - [Maintainer Release](#maintainer-release)
   - [Getting Help](#getting-help)
   - [Discussion and Development](#discussion-and-development)
   - [Contributing to data manipulation](#contributing-to-data-manipulation)
@@ -55,7 +54,7 @@ Data Manipulation is a Python package providing powerful utility functions. It c
 
 ## Where to get it
 
-The source code is currently hosted on GitHub at: https://github.com/shawnngtq/data-manipulation
+The source code is currently hosted on GitHub at: <https://github.com/shawnngtq/data-manipulation>
 
 ```bash
 pip install data-manipulation
@@ -69,6 +68,10 @@ The dependencies will be installed automatically along with this package, using
 
 [BSD 3](LICENSE)
 
+## Maintainer Release
+
+Release and deployment steps are documented in [RELEASE.md](RELEASE.md).
+
 ## Getting Help
 
 For usage questions, the best place to go to is StackOverflow.

{data_manipulation-0.50 → data_manipulation-0.54}/README.md

@@ -5,6 +5,7 @@
   - [Where to get it](#where-to-get-it)
   - [Dependencies](#dependencies)
   - [License](#license)
+  - [Maintainer Release](#maintainer-release)
   - [Getting Help](#getting-help)
   - [Discussion and Development](#discussion-and-development)
   - [Contributing to data manipulation](#contributing-to-data-manipulation)
@@ -17,7 +18,7 @@ Data Manipulation is a Python package providing powerful utility functions. It c
 
 ## Where to get it
 
-The source code is currently hosted on GitHub at: https://github.com/shawnngtq/data-manipulation
+The source code is currently hosted on GitHub at: <https://github.com/shawnngtq/data-manipulation>
 
 ```bash
 pip install data-manipulation
@@ -31,6 +32,10 @@ The dependencies will be installed automatically along with this package, using
 
 [BSD 3](LICENSE)
 
+## Maintainer Release
+
+Release and deployment steps are documented in [RELEASE.md](RELEASE.md).
+
 ## Getting Help
 
 For usage questions, the best place to go to is StackOverflow.

data_manipulation-0.54/RELEASE.md

@@ -0,0 +1,101 @@
+# Release Guide
+
+This guide is for maintainers publishing documentation to GitHub Pages and
+package artifacts to PyPI.
+
+## Preconditions
+
+- Work from a clean `master` branch.
+- Release versions are derived from git tags by `setuptools-scm`.
+- Never build or upload release artifacts from a dirty worktree.
+- Keep PyPI credentials out of git.
+
+Check the release state:
+
+```bash
+git checkout master
+git pull --ff-only origin master
+git status
+```
+
+`git status` should report no changes before continuing.
+
+## Tag The Release
+
+Use the next release version, for example `0.50`:
+
+```bash
+git tag -a 0.50 -m "Release 0.50"
+git push origin master
+git push origin 0.50
+```
+
+The tag must point at the exact commit being released.
+
+## Publish Documentation
+
+Both commands require an environment that provides `mkdocs`, `mkdocs-material`, and
+`mkdocstrings-python` (for example a virtualenv, conda env, or pixi shell). Activate that
+environment before running them.
+
+Build the MkDocs documentation (local preview only):
+
+```bash
+./setup.sh create_update_docs
+```
+
+Deploy to GitHub Pages:
+
+```bash
+./setup.sh deploy_docs
+```
+
+This publishes the contents of `site/` to the `gh-pages` branch.
+
+**Note - documentation and PyPI releases are independent.**
+
+- **Tags are only for PyPI releases.** `setuptools-scm` derives the package
+  version from git tags. Create a tag only when the shipped `data_manipulation/`
+  package changes; tooling- or docs-only changes need no tag.
+- **Docs deploy without a tag.** `deploy_docs` publishes current `master` to the
+  separate `gh-pages` branch, independent of any tag or PyPI release.
+- **Commit and push doc/source changes to `master` before building.** The site
+  renders the working tree, and `create_update_docs` writes only to the
+  gitignored `site/` directory - no tracked files change.
+- **When cutting a release, build docs after tagging** so the published site
+  matches the released commit.
+
+## Publish To PyPI
+
+Build and validate the distribution artifacts:
+
+```bash
+./setup.sh cleanup
+```
+
+Upload the artifacts:
+
+```bash
+./setup.sh upload
+```
+
+When prompted by `twine`, enter the PyPI API token. The token should start with
+`pypi-`. It will not be displayed while typing or pasting.
+
+## Verify The Release
+
+Install the published package in a fresh environment and confirm the version:
+
+```bash
+python3 -m pip install --upgrade data-manipulation
+python3 -c "import data_manipulation; print(data_manipulation.__version__)"
+```
+
+## Security Notes
+
+- Do not commit PyPI API tokens, passwords, or `.pypirc` files with credentials.
+- Do not add tokens to shell history as command-line arguments.
+- Prefer entering the PyPI token at the `twine` prompt or using a local secret
+  store.
+- If a token is accidentally committed or exposed, revoke it in PyPI immediately
+  and create a new one.

data_manipulation-0.54/conftest.py

@@ -0,0 +1,24 @@
+"""Repo-root pytest configuration.
+
+``--doctest-modules`` (see ``[tool.pytest.ini_options]``) collects doctests from
+every ``data_manipulation/*.py``. We only want to *execute* examples that are
+robustly runnable, so ignore the rest here. Their ``Examples:`` blocks still
+render in the mkdocs API reference — they are just not auto-verified.
+
+Auto-verified modules (not ignored): base.py, beautifulsoup_.py, sqlalchemy_.py.
+"""
+
+collect_ignore = [
+    "data_manipulation/__init__.py",
+    "data_manipulation/_version.py",
+    "data_manipulation/pandas_.py",
+    "data_manipulation/pyspark_.py",
+    "data_manipulation/boto3_.py",
+    "data_manipulation/kerberos_.py",
+    "data_manipulation/openldap_.py",
+    "data_manipulation/cryptography_.py",
+    "data_manipulation/mysql_connector_python_.py",
+    "data_manipulation/postgres_.py",
+    "data_manipulation/smtplib_.py",
+    "data_manipulation/polars_.py",
+]

{data_manipulation-0.50 → data_manipulation-0.54}/data_manipulation/__init__.py

@@ -3,15 +3,11 @@ from data_manipulation import (
     beautifulsoup_,
     boto3_,
     cryptography_,
-    django_,
-    flask_,
-    geopandas_,
     kerberos_,
     mysql_connector_python_,
     openldap_,
     pandas_,
     postgres_,
-    prometheus_,
     pyspark_,
     smtplib_,
     sqlalchemy_,

{data_manipulation-0.50 → data_manipulation-0.54}/data_manipulation/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.50'
-__version_tuple__ = version_tuple = (0, 50)
+__version__ = version = '0.54'
+__version_tuple__ = version_tuple = (0, 54)
 
-__commit_id__ = commit_id = 'g3a0e189a0'
+__commit_id__ = commit_id = 'g7826f58d8'