data-manipulation 0.49__tar.gz → 0.53__tar.gz

data_manipulation-0.53/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,38 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

data_manipulation-0.53/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,20 @@
+---
+name: Feature request
+about: Suggest an idea for this project
+title: ''
+labels: ''
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

data_manipulation-0.53/.gitignore

@@ -0,0 +1,100 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+data_manipulation/_version.py
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# MkDocs documentation
+/site/
+
+# PyBuilder
+target/
+
+# DotEnv configuration
+.env
+.pypirc
+
+# Database
+*.db
+*.rdb
+
+# Pycharm
+.idea
+
+# VS Code
+.vscode/
+
+# Spyder
+.spyproject/
+
+# Jupyter NB Checkpoints
+.ipynb_checkpoints/
+
+# exclude data from source control by default
+/data/
+*.csv
+*.xlsx
+
+# Mac OS-specific storage files
+.DS_Store
+
+# vim
+*.swp
+*.swo
+
+# Mypy cache
+.mypy_cache/
+
+# custom
+.opencode
+.specstory
+build.sh
+docs/README*
+install_packages.sh

data_manipulation-0.53/CHANGELOG.md

@@ -0,0 +1,81 @@
+# Changelog
+
+## [Unreleased]
+
+### Removed
+
+- **`pandas_.clean_none()`** — function has been removed.
+  Migrate to pandas built-ins:
+  ```python
+  df = df.replace(r"^\s*$", np.nan, regex=True)
+  df = df.where(pd.notnull(df), None)
+  ```
+
+- **`pandas_.config_pandas_display()`** — function has been removed.
+  Set display options directly:
+  ```python
+  pd.set_option("display.max_columns", 500)
+  pd.set_option("display.max_colwidth", 500)
+  pd.set_option("display.expand_frame_repr", True)
+  ```
+
+- **`prometheus_` module** — was an empty placeholder; removed from
+  the public API and from `__init__` imports.
+
+### Changed
+
+- **`base.get_none_variation()`** — now returns a `frozenset` of 7
+  lowercase canonical strings instead of a 72-item list of case
+  permutations. Use `.casefold()` for membership checks:
+  ```python
+  # before
+  if value in get_none_variation():
+      ...
+  # after
+  if isinstance(value, str) and value.casefold() in get_none_variation():
+      ...
+  ```
+
+- **`pyspark_.group_count()`** — the `percent` column is now computed
+  as a native Spark column expression instead of a Python UDF.
+  Behaviour is identical; performance improves on large datasets.
+
+- **`pyspark_.column_into_list()`** — raises `ValueError` (with the
+  list of available columns) when the requested column does not exist.
+  Previously returned `None` silently.
+
+- **`pandas_.print_dataframe_overview()`** — now returns a
+  `Dict[str, Dict[str, Any]]` of per-column stats (`unique`,
+  `null_count`, `value_counts`). Still prints to stdout as before;
+  existing call sites are unaffected.
+
+- **`pandas_.compare_dataframes()`** — now returns a comparison
+  summary dict (`same_length`, `df1_length`, `df2_length`, `columns`).
+  Still prints to stdout as before; existing call sites are unaffected.
+
+- **`pyspark_` module** — all `print()` calls replaced with
+  `logger.info()` / `logger.debug()`. Output is now routed through
+  the standard logging pipeline and can be silenced or redirected.
+
+- **`sqlalchemy_.create_sqlalchemy_engine()`** — duplicate
+  `if/elif` timeout branches (both setting the same value) consolidated
+  into a single condition.
+
+- **`django_.django_validate_phone()`** — now has an explicit
+  `return None` on failure instead of a bare `return`.
+
+### Fixed (previous pass — reference)
+
+- `pyspark_.columns_statistics()` — always-true condition in empty-column
+  detection (third clause was truthy for any non-empty string).
+- `pandas_.dtypes_dictionary()` — `.iteritems()` removed in pandas 2.0;
+  replaced with `.items()`.
+- Bare `except:` replaced with `except Exception:` in `pandas_` and
+  `django_`.
+- `base.string_boolean_to_int()` — removed (used `distutils`, dropped
+  in Python 3.12).
+- `flask_.py` — deleted (was entirely commented-out dead code).
+- `beautifulsoup_.preprocess()` — fixed mixed tabs/spaces and invalid
+  `\s` escape sequences.
+- All modules now guard third-party imports with `try/except`; the
+  package imports cleanly even when optional dependencies are missing.

data_manipulation-0.53/CODE_OF_CONDUCT.md

@@ -0,0 +1,76 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to making participation in our project and
+our community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, sex characteristics, gender identity and expression,
+level of experience, education, socio-economic status, nationality, personal
+appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or
+ advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic
+ address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a
+ professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable
+behavior and are expected to take appropriate and fair corrective action in
+response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or
+reject comments, commits, code, wiki edits, issues, and other contributions
+that are not aligned to this Code of Conduct, or to ban temporarily or
+permanently any contributor for other behaviors that they deem inappropriate,
+threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces
+when an individual is representing the project or its community. Examples of
+representing a project or community include using an official project e-mail
+address, posting via an official social media account, or acting as an appointed
+representative at an online or offline event. Representation of a project may be
+further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported by contacting the project team at Shawn.coding.acc@gmail.com. All
+complaints will be reviewed and investigated and will result in a response that
+is deemed necessary and appropriate to the circumstances. The project team is
+obligated to maintain confidentiality with regard to the reporter of an incident.
+Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good
+faith may face temporary or permanent repercussions as determined by other
+members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
+available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+
+[homepage]: https://www.contributor-covenant.org
+
+For answers to common questions about this code of conduct, see
+https://www.contributor-covenant.org/faq

data_manipulation-0.53/CONTRIBUTING.md

@@ -0,0 +1,19 @@
+# Contributing to data manipulation
+
+Whether you are a novice or experienced software developer, all contributions and suggestions are welcome!
+
+## Getting Started
+
+If you are looking to contribute to the *data manipulation* codebase, the best place to start is the [GitHub "issues" tab](https://github.com/shawnngtq/data-manipulation/issues). This is also a great place for filing bug reports and making suggestions for ways in which we can improve the code and documentation.
+
+## Filing Issues
+
+If you notice a bug in the code or documentation, or have suggestions for how we can improve either, feel free to create an issue on the [GitHub "issues" tab](https://github.com/shawnngtq/data-manipulation/issues) using [GitHub's "issue" form](https://github.com/shawnngtq/data-manipulation/issues/new). The form contains some questions that will help us best address your issue.
+
+## Contributing to the Codebase
+
+The code is hosted on [GitHub](https://github.com/shawnngtq/data-manipulation), so you will need to use [Git](https://git-scm.com/) to clone the project and make changes to the codebase. Once you have obtained a copy of the code, you should create a development environment that is separate from your existing Python environment so that you can make and test changes without compromising your own work environment.
+
+Before submitting your changes for review, make sure to check that your changes do not break any tests.
+
+Once your changes are ready to be submitted, make sure to push your changes to GitHub before creating a pull request. We will review your changes, and you will most likely be asked to make additional changes before it is finally ready to merge. However, once it's ready, we will merge it, and you will have successfully contributed to the codebase!

data_manipulation-0.53/Makefile

@@ -0,0 +1,144 @@
+.PHONY: clean data lint requirements sync_data_to_s3 sync_data_from_s3
+
+#################################################################################
+# GLOBALS                                                                       #
+#################################################################################
+
+PROJECT_DIR := $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))
+BUCKET = [OPTIONAL] your-bucket-for-syncing-data (do not include 's3://')
+PROFILE = default
+PROJECT_NAME = data_manipulation
+PYTHON_INTERPRETER = python3
+
+ifeq (,$(shell which conda))
+HAS_CONDA=False
+else
+HAS_CONDA=True
+endif
+
+#################################################################################
+# COMMANDS                                                                      #
+#################################################################################
+
+## Install Python Dependencies
+requirements: test_environment
+	$(PYTHON_INTERPRETER) -m pip install -U pip setuptools wheel
+	$(PYTHON_INTERPRETER) -m pip install -r requirements.txt
+
+## Make Dataset
+data: requirements
+	$(PYTHON_INTERPRETER) data_manipulation/data/make_dataset.py data/raw data/processed
+
+## Delete all compiled Python files
+clean:
+	find . -type f -name "*.py[co]" -delete
+	find . -type d -name "__pycache__" -delete
+
+## Lint using flake8
+lint:
+	flake8 data_manipulation
+
+## Upload Data to S3
+sync_data_to_s3:
+ifeq (default,$(PROFILE))
+	aws s3 sync data/ s3://$(BUCKET)/data/
+else
+	aws s3 sync data/ s3://$(BUCKET)/data/ --profile $(PROFILE)
+endif
+
+## Download Data from S3
+sync_data_from_s3:
+ifeq (default,$(PROFILE))
+	aws s3 sync s3://$(BUCKET)/data/ data/
+else
+	aws s3 sync s3://$(BUCKET)/data/ data/ --profile $(PROFILE)
+endif
+
+## Set up python interpreter environment
+create_environment:
+ifeq (True,$(HAS_CONDA))
+		@echo ">>> Detected conda, creating conda environment."
+ifeq (3,$(findstring 3,$(PYTHON_INTERPRETER)))
+	conda create --name $(PROJECT_NAME) python=3
+else
+	conda create --name $(PROJECT_NAME) python=2.7
+endif
+		@echo ">>> New conda env created. Activate with:\nsource activate $(PROJECT_NAME)"
+else
+	$(PYTHON_INTERPRETER) -m pip install -q virtualenv virtualenvwrapper
+	@echo ">>> Installing virtualenvwrapper if not already installed.\nMake sure the following lines are in shell startup file\n\
+	export WORKON_HOME=$$HOME/.virtualenvs\nexport PROJECT_HOME=$$HOME/Devel\nsource /usr/local/bin/virtualenvwrapper.sh\n"
+	@bash -c "source `which virtualenvwrapper.sh`;mkvirtualenv $(PROJECT_NAME) --python=$(PYTHON_INTERPRETER)"
+	@echo ">>> New virtualenv created. Activate with:\nworkon $(PROJECT_NAME)"
+endif
+
+## Test python environment is setup correctly
+test_environment:
+	$(PYTHON_INTERPRETER) test_environment.py
+
+#################################################################################
+# PROJECT RULES                                                                 #
+#################################################################################
+
+
+
+#################################################################################
+# Self Documenting Commands                                                     #
+#################################################################################
+
+.DEFAULT_GOAL := help
+
+# Inspired by <http://marmelab.com/blog/2016/02/29/auto-documented-makefile.html>
+# sed script explained:
+# /^##/:
+# 	* save line in hold space
+# 	* purge line
+# 	* Loop:
+# 		* append newline + line to hold space
+# 		* go to next line
+# 		* if line starts with doc comment, strip comment character off and loop
+# 	* remove target prerequisites
+# 	* append hold space (+ newline) to line
+# 	* replace newline plus comments by `---`
+# 	* print line
+# Separate expressions are necessary because labels cannot be delimited by
+# semicolon; see <http://stackoverflow.com/a/11799865/1968>
+.PHONY: help
+help:
+	@echo "$$(tput bold)Available rules:$$(tput sgr0)"
+	@echo
+	@sed -n -e "/^## / { \
+		h; \
+		s/.*//; \
+		:doc" \
+		-e "H; \
+		n; \
+		s/^## //; \
+		t doc" \
+		-e "s/:.*//; \
+		G; \
+		s/\\n## /---/; \
+		s/\\n/ /g; \
+		p; \
+	}" ${MAKEFILE_LIST} \
+	| LC_ALL='C' sort --ignore-case \
+	| awk -F '---' \
+		-v ncol=$$(tput cols) \
+		-v indent=19 \
+		-v col_on="$$(tput setaf 6)" \
+		-v col_off="$$(tput sgr0)" \
+	'{ \
+		printf "%s%*s%s ", col_on, -indent, $$1, col_off; \
+		n = split($$2, words, " "); \
+		line_length = ncol - indent; \
+		for (i = 1; i <= n; i++) { \
+			line_length -= length(words[i]) + 1; \
+			if (line_length <= 0) { \
+				line_length = ncol - indent - length(words[i]) - 1; \
+				printf "\n%*s ", -indent, " "; \
+			} \
+			printf "%s ", words[i]; \
+		} \
+		printf "\n"; \
+	}' \
+	| more $(shell test $(shell uname) = Darwin && echo '--no-init --raw-control-chars')

data_manipulation-0.53/PKG-INFO

@@ -0,0 +1,89 @@
+Metadata-Version: 2.4
+Name: data_manipulation
+Version: 0.53
+Summary: Powerful data manipulation
+Author-email: Shawn Ng <shawn.coding.acc@gmail.com>
+License-Expression: BSD-3-Clause
+Project-URL: Homepage, https://github.com/shawnngtq/data-manipulation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Operating System :: OS Independent
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Cython
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: beautifulsoup4
+Requires-Dist: boto3
+Requires-Dist: cryptography
+Requires-Dist: django
+Requires-Dist: flask
+Requires-Dist: geopandas
+Requires-Dist: ipython
+Requires-Dist: loguru
+Requires-Dist: mysql-connector-python
+Requires-Dist: numpy
+Requires-Dist: pandas
+Requires-Dist: psycopg
+Requires-Dist: pyspark
+Requires-Dist: python-dotenv
+Requires-Dist: sqlalchemy
+Dynamic: license-file
+
+# Data Manipulation
+
+- [Data Manipulation](#data-manipulation)
+  - [What is it?](#what-is-it)
+  - [Where to get it](#where-to-get-it)
+  - [Dependencies](#dependencies)
+  - [License](#license)
+  - [Maintainer Release](#maintainer-release)
+  - [Getting Help](#getting-help)
+  - [Discussion and Development](#discussion-and-development)
+  - [Contributing to data manipulation](#contributing-to-data-manipulation)
+
+My data manipulation library includes functions build on top of popular Python libraries such as Pandas, PySpark and more.
+
+## What is it?
+
+Data Manipulation is a Python package providing powerful utility functions. It contains many subpackages with utility functions built for popular packages such as Pandas, PySpark and many more.
+
+## Where to get it
+
+The source code is currently hosted on GitHub at: https://github.com/shawnngtq/data-manipulation
+
+```bash
+pip install data-manipulation
+```
+
+## Dependencies
+
+The dependencies will be installed automatically along with this package, using metadata in `pyproject.toml`.
+
+## License
+
+[BSD 3](LICENSE)
+
+## Maintainer Release
+
+Release and deployment steps are documented in [RELEASE.md](RELEASE.md).
+
+## Getting Help
+
+For usage questions, the best place to go to is StackOverflow.
+
+## Discussion and Development
+
+Most development discussion is taking place on github in this repo.
+
+## Contributing to data manipulation
+
+All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.
+
+As contributors and maintainers to this project, you are expected to abide by our code of conduct. More information can be found at: [Contributor Code of Conduct](https://github.com/shawnngtq/data-manipulation/blob/master/CODE_OF_CONDUCT.md)

{data_manipulation-0.49 → data_manipulation-0.53}/README.md

@@ -5,6 +5,7 @@
   - [Where to get it](#where-to-get-it)
   - [Dependencies](#dependencies)
   - [License](#license)
+  - [Maintainer Release](#maintainer-release)
   - [Getting Help](#getting-help)
   - [Discussion and Development](#discussion-and-development)
   - [Contributing to data manipulation](#contributing-to-data-manipulation)
@@ -25,12 +26,16 @@ pip install data-manipulation
 
 ## Dependencies
 
-The dependencies will be installed automatically along with this package, in `setup.py`
+The dependencies will be installed automatically along with this package, using metadata in `pyproject.toml`.
 
 ## License
 
 [BSD 3](LICENSE)
 
+## Maintainer Release
+
+Release and deployment steps are documented in [RELEASE.md](RELEASE.md).
+
 ## Getting Help
 
 For usage questions, the best place to go to is StackOverflow.

data_manipulation-0.53/RELEASE.md

@@ -0,0 +1,84 @@
+# Release Guide
+
+This guide is for maintainers publishing documentation to GitHub Pages and
+package artifacts to PyPI.
+
+## Preconditions
+
+- Work from a clean `master` branch.
+- Release versions are derived from git tags by `setuptools-scm`.
+- Never build or upload release artifacts from a dirty worktree.
+- Keep PyPI credentials out of git.
+
+Check the release state:
+
+```bash
+git checkout master
+git pull --ff-only origin master
+git status
+```
+
+`git status` should report no changes before continuing.
+
+## Tag The Release
+
+Use the next release version, for example `0.50`:
+
+```bash
+git tag -a 0.50 -m "Release 0.50"
+git push origin master
+git push origin 0.50
+```
+
+The tag must point at the exact commit being released.
+
+## Publish Documentation
+
+Build the MkDocs documentation:
+
+```bash
+mkdocs build
+```
+
+Publish the generated HTML to GitHub Pages:
+
+```bash
+mkdocs gh-deploy
+```
+
+This publishes the contents of `site/` to the `gh-pages` branch.
+
+## Publish To PyPI
+
+Build and validate the distribution artifacts:
+
+```bash
+./setup.sh cleanup
+```
+
+Upload the artifacts:
+
+```bash
+./setup.sh upload
+```
+
+When prompted by `twine`, enter the PyPI API token. The token should start with
+`pypi-`. It will not be displayed while typing or pasting.
+
+## Verify The Release
+
+Install the published package in a fresh environment and confirm the version:
+
+```bash
+python3 -m pip install --upgrade data-manipulation
+python3 -c "import data_manipulation; print(data_manipulation.__version__)"
+```
+
+## Security Notes
+
+- Do not commit PyPI API tokens, passwords, or `.pypirc` files with credentials.
+- Do not add tokens to shell history as command-line arguments.
+- Prefer entering the PyPI token at the `twine` prompt or using a local secret
+  store.
+- If a token is accidentally committed or exposed, revoke it in PyPI immediately
+  and create a new one.