data-manipulation 0.53__tar.gz → 0.54__tar.gz

{data_manipulation-0.53 → data_manipulation-0.54}/CHANGELOG.md

@@ -4,29 +4,40 @@
 
 ### Removed
 
-- **`pandas_.clean_none()`** — function has been removed.
+- **`django_` module** - removed entirely (breaking change). The Django helpers
+  (`init_django`, `get_django_countries_dict`, `django_validate_email`,
+  `django_validate_url`, `django_validate_phone`) now live in the
+  [`django-util`](https://github.com/shawnngtq/django-util) package:
+  `django_util.utility` (`init_django`, `get_django_countries_dict`) and
+  `django_util.validators` (the `django_validate_*` functions). `data_manipulation`
+  no longer depends on Django. This warrants a major version bump on release.
+
+- **`pandas_.clean_none()`** - function has been removed.
   Migrate to pandas built-ins:
+
   ```python
   df = df.replace(r"^\s*$", np.nan, regex=True)
   df = df.where(pd.notnull(df), None)
   ```
 
-- **`pandas_.config_pandas_display()`** — function has been removed.
+- **`pandas_.config_pandas_display()`** - function has been removed.
   Set display options directly:
+
   ```python
   pd.set_option("display.max_columns", 500)
   pd.set_option("display.max_colwidth", 500)
   pd.set_option("display.expand_frame_repr", True)
   ```
 
-- **`prometheus_` module** — was an empty placeholder; removed from
+- **`prometheus_` module** - was an empty placeholder; removed from
   the public API and from `__init__` imports.
 
 ### Changed
 
-- **`base.get_none_variation()`** — now returns a `frozenset` of 7
+- **`base.get_none_variation()`** - now returns a `frozenset` of 7
   lowercase canonical strings instead of a 72-item list of case
   permutations. Use `.casefold()` for membership checks:
+
   ```python
   # before
   if value in get_none_variation():
@@ -36,46 +47,46 @@
       ...
   ```
 
-- **`pyspark_.group_count()`** — the `percent` column is now computed
+- **`pyspark_.group_count()`** - the `percent` column is now computed
   as a native Spark column expression instead of a Python UDF.
   Behaviour is identical; performance improves on large datasets.
 
-- **`pyspark_.column_into_list()`** — raises `ValueError` (with the
+- **`pyspark_.column_into_list()`** - raises `ValueError` (with the
   list of available columns) when the requested column does not exist.
   Previously returned `None` silently.
 
-- **`pandas_.print_dataframe_overview()`** — now returns a
+- **`pandas_.print_dataframe_overview()`** - now returns a
   `Dict[str, Dict[str, Any]]` of per-column stats (`unique`,
   `null_count`, `value_counts`). Still prints to stdout as before;
   existing call sites are unaffected.
 
-- **`pandas_.compare_dataframes()`** — now returns a comparison
+- **`pandas_.compare_dataframes()`** - now returns a comparison
   summary dict (`same_length`, `df1_length`, `df2_length`, `columns`).
   Still prints to stdout as before; existing call sites are unaffected.
 
-- **`pyspark_` module** — all `print()` calls replaced with
+- **`pyspark_` module** - all `print()` calls replaced with
   `logger.info()` / `logger.debug()`. Output is now routed through
   the standard logging pipeline and can be silenced or redirected.
 
-- **`sqlalchemy_.create_sqlalchemy_engine()`** — duplicate
+- **`sqlalchemy_.create_sqlalchemy_engine()`** - duplicate
   `if/elif` timeout branches (both setting the same value) consolidated
   into a single condition.
 
-- **`django_.django_validate_phone()`** — now has an explicit
+- **`django_.django_validate_phone()`** - now has an explicit
   `return None` on failure instead of a bare `return`.
 
-### Fixed (previous pass — reference)
+### Fixed (previous pass - reference)
 
-- `pyspark_.columns_statistics()` — always-true condition in empty-column
+- `pyspark_.columns_statistics()` - always-true condition in empty-column
   detection (third clause was truthy for any non-empty string).
-- `pandas_.dtypes_dictionary()` — `.iteritems()` removed in pandas 2.0;
+- `pandas_.dtypes_dictionary()` - `.iteritems()` removed in pandas 2.0;
   replaced with `.items()`.
 - Bare `except:` replaced with `except Exception:` in `pandas_` and
   `django_`.
-- `base.string_boolean_to_int()` — removed (used `distutils`, dropped
+- `base.string_boolean_to_int()` - removed (used `distutils`, dropped
   in Python 3.12).
-- `flask_.py` — deleted (was entirely commented-out dead code).
-- `beautifulsoup_.preprocess()` — fixed mixed tabs/spaces and invalid
+- `flask_.py` - deleted (was entirely commented-out dead code).
+- `beautifulsoup_.preprocess()` - fixed mixed tabs/spaces and invalid
   `\s` escape sequences.
 - All modules now guard third-party imports with `try/except`; the
   package imports cleanly even when optional dependencies are missing.

{data_manipulation-0.53 → data_manipulation-0.54}/CODE_OF_CONDUCT.md

@@ -23,13 +23,16 @@ include:
 Examples of unacceptable behavior by participants include:
 
 * The use of sexualized language or imagery and unwelcome sexual attention or
- advances
+
+  advances
 * Trolling, insulting/derogatory comments, and personal or political attacks
 * Public or private harassment
 * Publishing others' private information, such as a physical or electronic
- address, without explicit permission
+
+  address, without explicit permission
 * Other conduct which could reasonably be considered inappropriate in a
- professional setting
+
+  professional setting
 
 ## Our Responsibilities
 
@@ -55,7 +58,7 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at Shawn.coding.acc@gmail.com. All
+reported by contacting the project team at <Shawn.coding.acc@gmail.com>. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.
@@ -68,9 +71,9 @@ members of the project's leadership.
 ## Attribution
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+available at <https://www.contributor-covenant.org/version/1/4/code-of-conduct.html>
 
 [homepage]: https://www.contributor-covenant.org
 
 For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq
+<https://www.contributor-covenant.org/faq>

{data_manipulation-0.53 → data_manipulation-0.54}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: data_manipulation
-Version: 0.53
+Version: 0.54
 Summary: Powerful data manipulation
 Author-email: Shawn Ng <shawn.coding.acc@gmail.com>
 License-Expression: BSD-3-Clause
@@ -10,21 +10,16 @@ Classifier: Environment :: Console
 Classifier: Operating System :: OS Independent
 Classifier: Intended Audience :: Science/Research
 Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.14
 Classifier: Programming Language :: Cython
 Classifier: Topic :: Scientific/Engineering
-Requires-Python: >=3.9
+Requires-Python: >=3.14
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: beautifulsoup4
 Requires-Dist: boto3
 Requires-Dist: cryptography
-Requires-Dist: django
 Requires-Dist: flask
-Requires-Dist: geopandas
 Requires-Dist: ipython
 Requires-Dist: loguru
 Requires-Dist: mysql-connector-python
@@ -34,6 +29,9 @@ Requires-Dist: psycopg
 Requires-Dist: pyspark
 Requires-Dist: python-dotenv
 Requires-Dist: sqlalchemy
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Requires-Dist: pytest-mock; extra == "test"
 Dynamic: license-file
 
 # Data Manipulation
@@ -56,7 +54,7 @@ Data Manipulation is a Python package providing powerful utility functions. It c
 
 ## Where to get it
 
-The source code is currently hosted on GitHub at: https://github.com/shawnngtq/data-manipulation
+The source code is currently hosted on GitHub at: <https://github.com/shawnngtq/data-manipulation>
 
 ```bash
 pip install data-manipulation

{data_manipulation-0.53 → data_manipulation-0.54}/README.md

@@ -18,7 +18,7 @@ Data Manipulation is a Python package providing powerful utility functions. It c
 
 ## Where to get it
 
-The source code is currently hosted on GitHub at: https://github.com/shawnngtq/data-manipulation
+The source code is currently hosted on GitHub at: <https://github.com/shawnngtq/data-manipulation>
 
 ```bash
 pip install data-manipulation

{data_manipulation-0.53 → data_manipulation-0.54}/RELEASE.md

@@ -34,20 +34,37 @@ The tag must point at the exact commit being released.
 
 ## Publish Documentation
 
-Build the MkDocs documentation:
+Both commands require an environment that provides `mkdocs`, `mkdocs-material`, and
+`mkdocstrings-python` (for example a virtualenv, conda env, or pixi shell). Activate that
+environment before running them.
+
+Build the MkDocs documentation (local preview only):
 
 ```bash
-mkdocs build
+./setup.sh create_update_docs
 ```
 
-Publish the generated HTML to GitHub Pages:
+Deploy to GitHub Pages:
 
 ```bash
-mkdocs gh-deploy
+./setup.sh deploy_docs
 ```
 
 This publishes the contents of `site/` to the `gh-pages` branch.
 
+**Note - documentation and PyPI releases are independent.**
+
+- **Tags are only for PyPI releases.** `setuptools-scm` derives the package
+  version from git tags. Create a tag only when the shipped `data_manipulation/`
+  package changes; tooling- or docs-only changes need no tag.
+- **Docs deploy without a tag.** `deploy_docs` publishes current `master` to the
+  separate `gh-pages` branch, independent of any tag or PyPI release.
+- **Commit and push doc/source changes to `master` before building.** The site
+  renders the working tree, and `create_update_docs` writes only to the
+  gitignored `site/` directory - no tracked files change.
+- **When cutting a release, build docs after tagging** so the published site
+  matches the released commit.
+
 ## Publish To PyPI
 
 Build and validate the distribution artifacts:

data_manipulation-0.54/conftest.py

@@ -0,0 +1,24 @@
+"""Repo-root pytest configuration.
+
+``--doctest-modules`` (see ``[tool.pytest.ini_options]``) collects doctests from
+every ``data_manipulation/*.py``. We only want to *execute* examples that are
+robustly runnable, so ignore the rest here. Their ``Examples:`` blocks still
+render in the mkdocs API reference — they are just not auto-verified.
+
+Auto-verified modules (not ignored): base.py, beautifulsoup_.py, sqlalchemy_.py.
+"""
+
+collect_ignore = [
+    "data_manipulation/__init__.py",
+    "data_manipulation/_version.py",
+    "data_manipulation/pandas_.py",
+    "data_manipulation/pyspark_.py",
+    "data_manipulation/boto3_.py",
+    "data_manipulation/kerberos_.py",
+    "data_manipulation/openldap_.py",
+    "data_manipulation/cryptography_.py",
+    "data_manipulation/mysql_connector_python_.py",
+    "data_manipulation/postgres_.py",
+    "data_manipulation/smtplib_.py",
+    "data_manipulation/polars_.py",
+]

{data_manipulation-0.53 → data_manipulation-0.54}/data_manipulation/__init__.py

@@ -3,8 +3,6 @@ from data_manipulation import (
     beautifulsoup_,
     boto3_,
     cryptography_,
-    django_,
-    geopandas_,
     kerberos_,
     mysql_connector_python_,
     openldap_,

{data_manipulation-0.53 → data_manipulation-0.54}/data_manipulation/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.53'
-__version_tuple__ = version_tuple = (0, 53)
+__version__ = version = '0.54'
+__version_tuple__ = version_tuple = (0, 54)
 
-__commit_id__ = commit_id = 'g2446c213a'
+__commit_id__ = commit_id = 'g7826f58d8'

{data_manipulation-0.53 → data_manipulation-0.54}/data_manipulation/base.py

@@ -4,13 +4,14 @@ import re
 import subprocess
 from functools import wraps
 from pathlib import Path
-from typing import Any, Dict, List, Optional, Tuple, Union
+from typing import Any
 from urllib.parse import urlencode, urlparse, urlunparse
 
 try:
     from loguru import logger
 except ImportError:
     import logging
+
     logger = logging.getLogger(__name__)
 
 
@@ -67,14 +68,14 @@ def clean_string(
     return " ".join(string.strip().upper().split())
 
 
-def get_string_case_combination(str_: str) -> List[str]:
+def get_string_case_combination(str_: str) -> list[str]:
     """Generates all possible case combinations of a string.
 
     Args:
         str_ (str): Input string to generate combinations for.
 
     Returns:
-        List[str]: List of all possible case combinations.
+        list[str]: List of all possible case combinations.
 
     Examples:
         >>> get_string_case_combination("abc")
@@ -83,7 +84,12 @@ def get_string_case_combination(str_: str) -> List[str]:
     if not str_:
         raise ValueError("Input string cannot be empty or None")
 
-    return list(map("".join, itertools.product(*zip(str_.upper(), str_.lower()))))
+    return list(
+        map(
+            "".join,
+            itertools.product(*zip(str_.upper(), str_.lower(), strict=True)),
+        )
+    )
 
 
 def get_none_variation() -> frozenset:
@@ -150,14 +156,16 @@ def get_country_code_variation() -> dict:
     return variations
 
 
-def list_tuple_without_none(list_tuple: Union[List, Tuple]) -> Union[List, Tuple]:
+def list_tuple_without_none(
+    list_tuple: list | tuple,
+) -> list | tuple:
     """Removes None variations from a list or tuple.
 
     Args:
         list_tuple: Input list or tuple to clean.
 
     Returns:
-        Union[List, Tuple]: Cleaned list or tuple.
+        list | tuple: Cleaned list or tuple.
 
     Raises:
         TypeError: If input is not a list or tuple.
@@ -185,55 +193,9 @@ def list_tuple_without_none(list_tuple: Union[List, Tuple]) -> Union[List, Tuple
     return tuple(item for item in list_tuple if _keep(item))
 
 
-
-def string_dlt_to_dlt(dlt_str_rep: str) -> Union[Dict, List, Tuple]:
-    """Converts string representation of data structures to actual Python objects.
-
-    Args:
-        dlt_str_rep (str): String representation of dictionary/list/tuple.
-
-    Returns:
-        Union[Dict, List, Tuple]: Converted Python data structure.
-
-    Examples:
-        >>> string_dlt_to_dlt("[1, 2, 3]")
-        [1, 2, 3]
-        >>> string_dlt_to_dlt("{'a': 1, 'b': 2}")
-        {'a': 1, 'b': 2}
-        >>> string_dlt_to_dlt("('1', '2', '3')")
-        ('1', '2', '3')
-    """
-    try:
-        from ast import literal_eval
-
-        return literal_eval(dlt_str_rep)
-    except (ValueError, SyntaxError) as e:
-        raise ValueError(f"Invalid data structure string: {dlt_str_rep}") from e
-
-
-def string_str_to_str(string_str_rep: str) -> str:
-    """Converts string representation to a clean string by removing quotes.
-
-    Args:
-        string_str_rep (str): String representation to clean.
-
-    Returns:
-        str: Cleaned string with outer quotes removed.
-
-    Examples:
-        >>> string_str_to_str("'test'")
-        'test'
-        >>> string_str_to_str('"test"')
-        'test'
-    """
-    if not string_str_rep:
-        raise ValueError("Input string cannot be empty or None")
-    return string_str_rep.strip("'\"")
-
-
 def delete_list_indices(
     list_: list,
-    indices: List[int],
+    indices: list[int],
 ) -> None:
     """Deletes multiple indices from a list in-place.
 
@@ -253,9 +215,9 @@ def delete_list_indices(
 
 # FILESYSTEM
 def get_path_files(
-    path: Union[str, Path],
-    keywords: List[str],
-) -> List[str]:
+    path: str | Path,
+    keywords: list[str],
+) -> list[str]:
     """Returns sorted list of files from given path that contain specified keywords.
 
     Args:
@@ -263,10 +225,10 @@ def get_path_files(
         keywords (list): List of keywords to match in filenames.
 
     Returns:
-        List[str]: Sorted list of matching filenames.
+        list[str]: Sorted list of matching filenames.
 
     Examples:
-        >>> get_path_files("test_base_folder", ["py"])
+        >>> get_path_files("test_base_folder", ["py"])  # doctest: +SKIP
         ['test1.py', 'test2.py', 'test3.py', 'test4.py', 'test5.py']
     """
     path = Path(path)
@@ -281,7 +243,7 @@ def get_path_files(
 
 
 def remove_path_file(
-    path: Union[str, Path],
+    path: str | Path,
     keyword: str,
     n: int = 2,
 ) -> None:
@@ -293,7 +255,7 @@ def remove_path_file(
         n (int, optional): Number of newest files to keep. Defaults to 2.
 
     Examples:
-        >>> remove_path_file("test_base_folder", ".py")
+        >>> remove_path_file("test_base_folder", ".py")  # doctest: +SKIP
     """
     if n < 0:
         raise ValueError("n must be non-negative")
@@ -311,8 +273,8 @@ def remove_path_file(
 
 
 def list_to_file(
-    filepath: Union[str, Path],
-    list_: List,
+    filepath: str | Path,
+    list_: list,
     newline: bool = True,
 ) -> None:
     """Writes list contents to a file.
@@ -323,7 +285,7 @@ def list_to_file(
         newline (bool, optional): Whether to add newline after each item. Defaults to True.
 
     Examples:
-        >>> list_to_file("test.txt", [1, 2, 3])
+        >>> list_to_file("test.txt", [1, 2, 3])  # doctest: +SKIP
     """
     filepath = Path(filepath)
     try:
@@ -333,7 +295,7 @@ def list_to_file(
                 if newline:
                     f.write("\n")
         logger.info(f"Successfully wrote to file: {filepath}")
-    except IOError as e:
+    except OSError as e:
         logger.error(f"Failed to write to file {filepath}: {e}")
         raise
 
@@ -341,7 +303,7 @@ def list_to_file(
 # URLLIB
 def create_encode_url(
     url: str,
-    query_params: Optional[Dict[str, Any]] = None,
+    query_params: dict[str, Any] | None = None,
 ) -> str:
     """Creates an encoded URL with query parameters.
 
@@ -378,14 +340,14 @@ def create_encode_url(
 
 
 # SYSTEM
-def parse_ps_aux(ps_aux_commands: str) -> List[List[str]]:
+def parse_ps_aux(ps_aux_commands: str) -> list[list[str]]:
     """Parses Linux ps aux command output into a list of records.
 
     Args:
         ps_aux_commands (str): Linux ps aux command string.
 
     Returns:
-        List[List[str]]: List of process records.
+        list[list[str]]: List of process records.
 
     Examples:
         >>> # parse_ps_aux("ps aux | egrep -i '%cpu|anaconda3' | head")
@@ -413,26 +375,3 @@ def parse_ps_aux(ps_aux_commands: str) -> List[List[str]]:
     except subprocess.SubprocessError as e:
         logger.error(f"Failed to execute command: {e}")
         raise
-
-
-if __name__ == "__main__":
-    import doctest
-
-    # Setup test environment
-    test_folder = Path("test_base_folder")
-    try:
-        test_folder.mkdir(exist_ok=True)
-        for i in range(1, 6):
-            (test_folder / f"test{i}.py").touch()
-
-        doctest.testmod()
-    finally:
-        # Cleanup
-        if test_folder.exists():
-            for file in test_folder.iterdir():
-                file.unlink()
-            test_folder.rmdir()
-
-        test_file = Path("test.txt")
-        if test_file.exists():
-            test_file.unlink()

{data_manipulation-0.53 → data_manipulation-0.54}/data_manipulation/beautifulsoup_.py

@@ -1,10 +1,13 @@
+from __future__ import annotations
+
 import re
-from typing import TYPE_CHECKING, Optional
+from typing import TYPE_CHECKING
 
 try:
     from loguru import logger
 except ImportError:
     import logging
+
     logger = logging.getLogger(__name__)
 
 if TYPE_CHECKING:
@@ -17,14 +20,14 @@ DEFAULT_HEADERS = {
 }
 
 
-def preprocess(html: str) -> Optional[str]:
+def preprocess(html: str) -> str | None:
     """Removes whitespaces and newline characters from HTML string.
 
     Args:
         html (str): HTML string to be cleaned.
 
     Returns:
-        Optional[str]: Cleaned HTML string with normalized whitespace.
+        str | None: Cleaned HTML string with normalized whitespace.
 
     Examples:
         >>> a = "<html>   <p> Something </p>    </html> "
@@ -48,8 +51,8 @@ def build_soup(
     features: str = "lxml",
     to_preprocess: bool = True,
     timeout: int = DEFAULT_TIMEOUT,
-    headers: Optional[dict] = None,
-) -> "Optional[BeautifulSoup]":
+    headers: dict | None = None,
+) -> BeautifulSoup | None:
     """Creates a BeautifulSoup object from a given URL.
 
     Args:
@@ -57,14 +60,14 @@ def build_soup(
         features (str, optional): Parser to use. Defaults to "lxml".
         to_preprocess (bool, optional): Whether to preprocess the HTML. Defaults to True.
         timeout (int, optional): Request timeout in seconds. Defaults to 10.
-        headers (Optional[dict], optional): Custom headers for the request. Defaults to None.
+        headers (dict | None, optional): Custom headers for the request. Defaults to None.
 
     Returns:
-        Optional[BeautifulSoup]: Parsed BeautifulSoup object, or None if request fails.
+        BeautifulSoup | None: Parsed BeautifulSoup object, or None if request fails.
 
     Examples:
-        >>> a = build_soup("https://google.com")
-        >>> type(a)
+        >>> a = build_soup("https://google.com")  # doctest: +SKIP
+        >>> type(a)  # doctest: +SKIP
         <class 'bs4.BeautifulSoup'>
 
     Note:
@@ -83,18 +86,16 @@ def build_soup(
 
     try:
         with requests.Session() as session:
-            response = session.get(url, headers=request_headers, timeout=timeout)
+            response = session.get(
+                url, headers=request_headers, timeout=timeout
+            )
             response.raise_for_status()
 
-            html_content = preprocess(response.text) if to_preprocess else response.text
+            html_content = (
+                preprocess(response.text) if to_preprocess else response.text
+            )
             return BeautifulSoup(html_content, features=features)
 
     except requests.RequestException as e:
         logger.error(f"Failed to fetch URL {url}: {str(e)}")
         return None
-
-
-if __name__ == "__main__":
-    import doctest
-
-    doctest.testmod()