unstructured-ingest 0.7.1__tar.gz → 1.0.1__tar.gz

unstructured_ingest-1.0.1/.gitignore

@@ -0,0 +1,212 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+figures/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+pip-wheel-metadata/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Pycharm
+.idea/
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+local_settings.py
+db.sqlite3
+db.sqlite3-journal
+
+# Flask stuff:
+instance/
+.webassets-cache
+
+# Scrapy stuff:
+.scrapy
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+# Jupyter Notebook
+.ipynb_checkpoints
+nbs/
+
+# IPython
+profile_default/
+ipython_config.py
+
+# pyenv
+.python-version
+
+# pipenv
+#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
+#   However, in case of collaboration, if having platform-specific dependencies or dependencies
+#   having no cross-platform support, pipenv may install dependencies that don't work, or not
+#   install all needed dependencies.
+#Pipfile.lock
+
+# PEP 582; used by e.g. github.com/David-OConnor/pyflow
+__pypackages__/
+
+# Celery stuff
+celerybeat-schedule
+celerybeat.pid
+
+# SageMath parsed files
+*.sage.py
+
+# Environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Spyder project settings
+.spyderproject
+.spyproject
+
+# Rope project settings
+.ropeproject
+
+# mkdocs documentation
+/site
+
+# mypy
+.mypy_cache/
+.dmypy.json
+dmypy.json
+
+# Pyre type checker
+.pyre/
+
+# pyright (Python LSP/type-checker in VSCode) config
+/pyrightconfig.json
+
+# ingest outputs
+/structured-output
+test_unstructured_ingest/workdir/
+test_unstructured_ingest/delta-table-dest/
+test_unstructured_ingest/skipped-files.txt
+test_unstructured_ingest/chroma-dest/
+
+# suggested ingest mirror directory
+/mirror
+
+## https://github.com/github/gitignore/blob/main/Global/Emacs.gitignore (partial)
+
+*~
+\#*\#
+/.emacs.desktop
+/.emacs.desktop.lock
+*.elc
+auto-save-list
+tramp
+.\#*
+
+## https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore
+.vscode/*
+!.vscode/tasks.json
+!.vscode/launch.json
+!.vscode/extensions.json
+!.vscode/*.code-snippets
+
+# Local History for Visual Studio Code
+.history/
+
+# Built Visual Studio Code Extensions
+*.vsix
+
+## https://github.com/github/gitignore/blob/main/Global/Vim.gitignore
+# Swap
+[._]*.s[a-v][a-z]
+!*.svg  # comment out if you don't need vector files
+[._]*.sw[a-p]
+[._]s[a-rt-v][a-z]
+[._]ss[a-gi-z]
+[._]sw[a-p]
+
+# Session
+Session.vim
+Sessionx.vim
+
+# Temporary
+.netrwhist
+# Auto-generated tag files
+tags
+# Persistent undo
+[._]*.un~
+
+.DS_Store
+
+# Ruff cache
+.ruff_cache/
+
+.ppm
+.vs
+
+example-docs/*_images
+examples/**/output/
+
+outputdiff.txt
+metricsdiff.txt
+
+# analysis
+annotated/
+
+tmp_ingest/
+.vs
+.report.json

unstructured_ingest-1.0.1/PKG-INFO

@@ -0,0 +1,226 @@
+Metadata-Version: 2.4
+Name: unstructured_ingest
+Version: 1.0.1
+Summary: Local ETL data pipeline to get data RAG ready
+Author-email: Unstructured Technologies <devops@unstructuredai.io>
+License-Expression: Apache-2.0
+License-File: LICENSE.md
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: <3.13,>=3.9
+Requires-Dist: click
+Requires-Dist: dataclasses-json
+Requires-Dist: opentelemetry-sdk
+Requires-Dist: pydantic>=2.7
+Requires-Dist: python-dateutil
+Requires-Dist: tqdm
+Provides-Extra: airtable
+Requires-Dist: pandas; extra == 'airtable'
+Requires-Dist: pyairtable; extra == 'airtable'
+Provides-Extra: astradb
+Requires-Dist: astrapy; extra == 'astradb'
+Provides-Extra: azure
+Requires-Dist: adlfs; extra == 'azure'
+Requires-Dist: fsspec; extra == 'azure'
+Provides-Extra: azure-ai-search
+Requires-Dist: azure-search-documents; extra == 'azure-ai-search'
+Provides-Extra: bedrock
+Requires-Dist: aioboto3; extra == 'bedrock'
+Requires-Dist: boto3; extra == 'bedrock'
+Provides-Extra: biomed
+Requires-Dist: bs4; extra == 'biomed'
+Requires-Dist: requests; extra == 'biomed'
+Provides-Extra: box
+Requires-Dist: boxfs; extra == 'box'
+Requires-Dist: fsspec; extra == 'box'
+Provides-Extra: chroma
+Requires-Dist: chromadb; extra == 'chroma'
+Provides-Extra: clarifai
+Requires-Dist: clarifai; extra == 'clarifai'
+Provides-Extra: confluence
+Requires-Dist: atlassian-python-api; extra == 'confluence'
+Requires-Dist: requests; extra == 'confluence'
+Provides-Extra: couchbase
+Requires-Dist: couchbase; extra == 'couchbase'
+Provides-Extra: databricks-delta-tables
+Requires-Dist: databricks-sql-connector; extra == 'databricks-delta-tables'
+Requires-Dist: pandas; extra == 'databricks-delta-tables'
+Provides-Extra: databricks-volumes
+Requires-Dist: databricks-sdk; extra == 'databricks-volumes'
+Provides-Extra: delta-table
+Requires-Dist: boto3; extra == 'delta-table'
+Requires-Dist: deltalake; extra == 'delta-table'
+Requires-Dist: pandas; extra == 'delta-table'
+Provides-Extra: discord
+Requires-Dist: discord-py; extra == 'discord'
+Provides-Extra: doc
+Requires-Dist: unstructured[doc]; extra == 'doc'
+Provides-Extra: docx
+Requires-Dist: unstructured[docx]; extra == 'docx'
+Provides-Extra: dropbox
+Requires-Dist: dropboxdrivefs; extra == 'dropbox'
+Requires-Dist: fsspec; extra == 'dropbox'
+Provides-Extra: duckdb
+Requires-Dist: duckdb; extra == 'duckdb'
+Requires-Dist: pandas; extra == 'duckdb'
+Provides-Extra: elasticsearch
+Requires-Dist: elasticsearch[async]; extra == 'elasticsearch'
+Provides-Extra: epub
+Requires-Dist: unstructured[epub]; extra == 'epub'
+Provides-Extra: gcs
+Requires-Dist: bs4; extra == 'gcs'
+Requires-Dist: fsspec; extra == 'gcs'
+Requires-Dist: gcsfs; extra == 'gcs'
+Provides-Extra: github
+Requires-Dist: pygithub>1.58.0; extra == 'github'
+Requires-Dist: requests; extra == 'github'
+Provides-Extra: gitlab
+Requires-Dist: python-gitlab; extra == 'gitlab'
+Provides-Extra: google-drive
+Requires-Dist: google-api-python-client; extra == 'google-drive'
+Provides-Extra: hubspot
+Requires-Dist: hubspot-api-client; extra == 'hubspot'
+Requires-Dist: urllib3; extra == 'hubspot'
+Provides-Extra: huggingface
+Requires-Dist: sentence-transformers; extra == 'huggingface'
+Provides-Extra: ibm-watsonx-s3
+Requires-Dist: httpx; extra == 'ibm-watsonx-s3'
+Requires-Dist: pandas; extra == 'ibm-watsonx-s3'
+Requires-Dist: pyarrow; extra == 'ibm-watsonx-s3'
+Requires-Dist: pyiceberg; extra == 'ibm-watsonx-s3'
+Requires-Dist: tenacity; extra == 'ibm-watsonx-s3'
+Provides-Extra: image
+Requires-Dist: unstructured[image]; extra == 'image'
+Provides-Extra: jira
+Requires-Dist: atlassian-python-api; extra == 'jira'
+Provides-Extra: kafka
+Requires-Dist: confluent-kafka; extra == 'kafka'
+Provides-Extra: kdbai
+Requires-Dist: kdbai-client>=1.4.0; extra == 'kdbai'
+Requires-Dist: pandas; extra == 'kdbai'
+Provides-Extra: lancedb
+Requires-Dist: lancedb; extra == 'lancedb'
+Provides-Extra: md
+Requires-Dist: unstructured[md]; extra == 'md'
+Provides-Extra: milvus
+Requires-Dist: pymilvus; extra == 'milvus'
+Provides-Extra: mixedbreadai
+Requires-Dist: mixedbread-ai; extra == 'mixedbreadai'
+Provides-Extra: mongodb
+Requires-Dist: pymongo; extra == 'mongodb'
+Provides-Extra: msg
+Requires-Dist: unstructured[msg]; extra == 'msg'
+Provides-Extra: neo4j
+Requires-Dist: cymple; extra == 'neo4j'
+Requires-Dist: neo4j-rust-ext; extra == 'neo4j'
+Requires-Dist: networkx; extra == 'neo4j'
+Provides-Extra: notion
+Requires-Dist: backoff; extra == 'notion'
+Requires-Dist: htmlbuilder; extra == 'notion'
+Requires-Dist: httpx; extra == 'notion'
+Requires-Dist: notion-client; extra == 'notion'
+Provides-Extra: octoai
+Requires-Dist: openai; extra == 'octoai'
+Requires-Dist: tiktoken; extra == 'octoai'
+Provides-Extra: odt
+Requires-Dist: unstructured[odt]; extra == 'odt'
+Provides-Extra: onedrive
+Requires-Dist: msal; extra == 'onedrive'
+Requires-Dist: office365-rest-python-client; extra == 'onedrive'
+Requires-Dist: requests; extra == 'onedrive'
+Provides-Extra: openai
+Requires-Dist: openai; extra == 'openai'
+Requires-Dist: tiktoken; extra == 'openai'
+Provides-Extra: opensearch
+Requires-Dist: opensearch-py; extra == 'opensearch'
+Provides-Extra: org
+Requires-Dist: unstructured[org]; extra == 'org'
+Provides-Extra: outlook
+Requires-Dist: msal; extra == 'outlook'
+Requires-Dist: office365-rest-python-client; extra == 'outlook'
+Provides-Extra: pdf
+Requires-Dist: unstructured[pdf]; extra == 'pdf'
+Provides-Extra: pinecone
+Requires-Dist: pinecone; extra == 'pinecone'
+Provides-Extra: postgres
+Requires-Dist: pandas; extra == 'postgres'
+Requires-Dist: psycopg2-binary; extra == 'postgres'
+Provides-Extra: ppt
+Requires-Dist: unstructured[ppt]; extra == 'ppt'
+Provides-Extra: pptx
+Requires-Dist: unstructured[pptx]; extra == 'pptx'
+Provides-Extra: qdrant
+Requires-Dist: qdrant-client; extra == 'qdrant'
+Provides-Extra: reddit
+Requires-Dist: praw; extra == 'reddit'
+Provides-Extra: redis
+Requires-Dist: redis; extra == 'redis'
+Provides-Extra: remote
+Requires-Dist: unstructured-client>=0.30.0; extra == 'remote'
+Provides-Extra: rst
+Requires-Dist: unstructured[rst]; extra == 'rst'
+Provides-Extra: rtf
+Requires-Dist: unstructured[rtf]; extra == 'rtf'
+Provides-Extra: s3
+Requires-Dist: fsspec; extra == 's3'
+Requires-Dist: s3fs; extra == 's3'
+Provides-Extra: salesforce
+Requires-Dist: simple-salesforce; extra == 'salesforce'
+Provides-Extra: sftp
+Requires-Dist: fsspec; extra == 'sftp'
+Requires-Dist: paramiko; extra == 'sftp'
+Provides-Extra: sharepoint
+Requires-Dist: msal; extra == 'sharepoint'
+Requires-Dist: office365-rest-python-client; extra == 'sharepoint'
+Requires-Dist: requests; extra == 'sharepoint'
+Provides-Extra: singlestore
+Requires-Dist: pandas; extra == 'singlestore'
+Requires-Dist: singlestoredb; extra == 'singlestore'
+Provides-Extra: slack
+Requires-Dist: slack-sdk[optional]; extra == 'slack'
+Provides-Extra: snowflake
+Requires-Dist: pandas; extra == 'snowflake'
+Requires-Dist: psycopg2-binary; extra == 'snowflake'
+Requires-Dist: snowflake-connector-python; extra == 'snowflake'
+Provides-Extra: togetherai
+Requires-Dist: together; extra == 'togetherai'
+Provides-Extra: tsv
+Requires-Dist: unstructured[tsv]; extra == 'tsv'
+Provides-Extra: vastdb
+Requires-Dist: ibis; extra == 'vastdb'
+Requires-Dist: pandas; extra == 'vastdb'
+Requires-Dist: pyarrow; extra == 'vastdb'
+Requires-Dist: vastdb; extra == 'vastdb'
+Provides-Extra: vectara
+Requires-Dist: aiofiles; extra == 'vectara'
+Requires-Dist: httpx; extra == 'vectara'
+Requires-Dist: requests; extra == 'vectara'
+Provides-Extra: vertexai
+Requires-Dist: vertexai; extra == 'vertexai'
+Provides-Extra: voyageai
+Requires-Dist: voyageai; extra == 'voyageai'
+Provides-Extra: weaviate
+Requires-Dist: weaviate-client; extra == 'weaviate'
+Provides-Extra: wikipedia
+Requires-Dist: wikipedia; extra == 'wikipedia'
+Provides-Extra: xlsx
+Requires-Dist: unstructured[xlsx]; extra == 'xlsx'
+Provides-Extra: zendesk
+Requires-Dist: aiofiles; extra == 'zendesk'
+Requires-Dist: bs4; extra == 'zendesk'
+Requires-Dist: httpx; extra == 'zendesk'
+Description-Content-Type: text/markdown
+
+# Unstructured Ingest  
+
+For details, see the [Unstructured Ingest overview](https://docs.unstructured.io/ingestion/overview) in the Unstructured documentation.

unstructured_ingest-1.0.1/pyproject.toml

@@ -0,0 +1,212 @@
+[project]
+name = "unstructured_ingest"
+description = "Local ETL data pipeline to get data RAG ready"
+requires-python = ">=3.9, <3.13"
+authors = [{name = "Unstructured Technologies", email = "devops@unstructuredai.io"}]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Education",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: Apache Software License",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering :: Artificial Intelligence",
+]
+readme = "README.md"
+license = "Apache-2.0"
+dynamic = ["version", "dependencies", "optional-dependencies"]
+
+[tool.hatch.metadata.hooks.requirements_txt]
+files = ["requirements/base.txt"]
+
+[tool.hatch.metadata.hooks.requirements_txt.optional-dependencies]
+# Connectors
+airtable = ["requirements/connectors/airtable.txt"]
+astradb = ["requirements/connectors/astradb.txt"]
+azure-ai-search = ["requirements/connectors/azure-ai-search.txt"]
+azure = ["requirements/connectors/azure.txt"]
+biomed = ["requirements/connectors/biomed.txt"]
+box = ["requirements/connectors/box.txt"]
+chroma = ["requirements/connectors/chroma.txt"]
+clarifai = ["requirements/connectors/clarifai.txt"]
+confluence = ["requirements/connectors/confluence.txt"]
+couchbase = ["requirements/connectors/couchbase.txt"]
+databricks-delta-tables = ["requirements/connectors/databricks-delta-tables.txt"]
+databricks-volumes = ["requirements/connectors/databricks-volumes.txt"]
+delta-table = ["requirements/connectors/delta-table.txt"]
+discord = ["requirements/connectors/discord.txt"]
+dropbox = ["requirements/connectors/dropbox.txt"]
+duckdb = ["requirements/connectors/duckdb.txt"]
+elasticsearch = ["requirements/connectors/elasticsearch.txt"]
+gcs = ["requirements/connectors/gcs.txt"]
+github = ["requirements/connectors/github.txt"]
+gitlab = ["requirements/connectors/gitlab.txt"]
+google-drive = ["requirements/connectors/google-drive.txt"]
+hubspot = ["requirements/connectors/hubspot.txt"]
+ibm-watsonx-s3 = ["requirements/connectors/ibm-watsonx-s3.txt"]
+jira = ["requirements/connectors/jira.txt"]
+kafka = ["requirements/connectors/kafka.txt"]
+kdbai = ["requirements/connectors/kdbai.txt"]
+lancedb = ["requirements/connectors/lancedb.txt"]
+milvus = ["requirements/connectors/milvus.txt"]
+mongodb = ["requirements/connectors/mongodb.txt"]
+neo4j = ["requirements/connectors/neo4j.txt"]
+notion = ["requirements/connectors/notion.txt"]
+onedrive = ["requirements/connectors/onedrive.txt"]
+opensearch = ["requirements/connectors/opensearch.txt"]
+outlook = ["requirements/connectors/outlook.txt"]
+pinecone = ["requirements/connectors/pinecone.txt"]
+postgres = ["requirements/connectors/postgres.txt"]
+qdrant = ["requirements/connectors/qdrant.txt"]
+reddit = ["requirements/connectors/reddit.txt"]
+redis = ["requirements/connectors/redis.txt"]
+s3 = ["requirements/connectors/s3.txt"]
+salesforce = ["requirements/connectors/salesforce.txt"]
+sftp = ["requirements/connectors/sftp.txt"]
+sharepoint = ["requirements/connectors/sharepoint.txt"]
+singlestore = ["requirements/connectors/singlestore.txt"]
+slack = ["requirements/connectors/slack.txt"]
+snowflake = ["requirements/connectors/snowflake.txt"]
+vastdb = ["requirements/connectors/vastdb.txt"]
+vectara = ["requirements/connectors/vectara.txt"]
+weaviate = ["requirements/connectors/weaviate.txt"]
+wikipedia = ["requirements/connectors/wikipedia.txt"]
+zendesk = ["requirements/connectors/zendesk.txt"]
+
+# Embedders
+bedrock = ["requirements/embed/bedrock.txt"]
+huggingface = ["requirements/embed/huggingface.txt"]
+mixedbreadai = ["requirements/embed/mixedbreadai.txt"]
+octoai = ["requirements/embed/octoai.txt"]
+openai = ["requirements/embed/openai.txt"]
+togetherai = ["requirements/embed/togetherai.txt"]
+vertexai = ["requirements/embed/vertexai.txt"]
+voyageai = ["requirements/embed/voyageai.txt"]
+
+# remote
+remote = ["requirements/remote/client.txt"]
+
+# local partition
+doc = ["requirements/local_partition/doc.txt"]
+docx = ["requirements/local_partition/docx.txt"]
+epub = ["requirements/local_partition/epub.txt"]
+image = ["requirements/local_partition/image.txt"]
+md = ["requirements/local_partition/md.txt"]
+msg = ["requirements/local_partition/msg.txt"]
+odt = ["requirements/local_partition/odt.txt"]
+org = ["requirements/local_partition/org.txt"]
+pdf = ["requirements/local_partition/pdf.txt"]
+ppt = ["requirements/local_partition/ppt.txt"]
+pptx = ["requirements/local_partition/pptx.txt"]
+rst = ["requirements/local_partition/rst.txt"]
+rtf = ["requirements/local_partition/rtf.txt"]
+tsv = ["requirements/local_partition/tsv.txt"]
+xlsx = ["requirements/local_partition/xlsx.txt"]
+
+
+[tool.hatch.version]
+path = "unstructured_ingest/__version__.py"
+
+[dependency-groups]
+release = [
+    "twine",
+    "wheel",
+    "build"
+]
+lint = [
+    "ruff",
+]
+test = [
+    "pytest",
+    "pytest-cov",
+    "pytest-mock",
+    "pytest-check",
+    "pytest-asyncio",
+    "pytest_tagging",
+    "pytest-json-report",
+    "pytest-timeout",
+    "faker",
+    "docker",
+    "universal_pathlib",
+    "deepdiff",
+    "bs4",
+    "pandas",
+
+    # Connector specific deps
+    "cryptography",
+    "fsspec",
+    "vertexai",
+    "pyiceberg",
+    "pyarrow",
+]
+# Add constraints needed for CI
+ci = [
+    # consistency with local-inference-pin
+    "protobuf<4.24",
+    "grpcio>=1.65.5",
+    # TODO: Pinned in transformers package, remove when that gets updated
+    "tokenizers>=0.19,<0.20",
+    # TODO: Constaint due to boto, with python before 3.10 not requiring openssl 1.1.1, remove when that gets
+    # updated or we drop support for 3.9
+    "urllib3<1.27",
+    # TODO: Constriant due to aiobotocore, remove when that gets updates:
+    "botocore<1.34.132",
+    # TODO: Constriant due to both 8.5.0 and 8.4.0 being installed during pip-compile
+    "importlib-metadata>=8.5.0",
+    # TODO: Constraint due to boto, with python before 3.10 not requiring openssl 1.1.1, remove when that gets
+    # updated or we drop support for 3.9
+    "urllib3<1.27",
+    "unstructured-client>= 0.25.8",
+    "fsspec==2024.5.0",
+    # python 3.12 support
+    "wrapt>=1.14.0",
+    "numpy<2",
+    # deltalake >=0.23.0 currently has a bug with the versio of pyarrow it installs
+    "deltalake<=0.22.0",
+    # TODO: investigate breaking changed introduced in lancedb>0.15.0
+    "lancedb<=0.15.0",
+    # TODO: versions higher than this are missing the macos wheel
+    "pykx==2.5.3",
+    "astrapy<2.0.0"
+]
+
+[project.scripts]
+unstructured-ingest = "unstructured_ingest.main:main"
+
+
+[build-system]
+requires = ["hatchling", "hatch-requirements-txt"]
+build-backend = "hatchling.build"
+
+[tool.ruff]
+line-length = 100
+
+[tool.ruff.lint]
+select = [
+    # pycodestyle
+    "E",
+    # Pyflakes
+    "F",
+    # flake8-simplify
+    "SIM",
+    # isort
+    "I",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+
+[tool.coverage.report]
+# TODO: Update as this improves
+fail_under = 0
+
+[tool.hatch.build.targets.wheel]
+    packages = ["/unstructured_ingest"]
+
+[tool.hatch.build.targets.sdist]
+    packages = ["/unstructured_ingest"]

unstructured_ingest-1.0.1/unstructured_ingest/__version__.py

@@ -0,0 +1 @@
+__version__ = "1.0.1"  # pragma: no cover

unstructured_ingest-1.0.1/unstructured_ingest/cli/README.md

@@ -0,0 +1,28 @@
+# Ingest CLI
+This package helps map user input via a cli to the underlying ingest code to run a small ETL pipeline.
+
+## Design Reference
+[cli.py](cli.py) is the main entrypoint to run the cli itself. The key points for this is the interaction between all
+source and destination connectors.
+
+To manually run the cli:
+```shell
+PYTHONPATH=. python unstructured_ingest/v2/main.py --help
+```
+
+The `main.py` file simply wraps the generated Click command created in `cli.py`.
+
+### Source Commands
+All source commands are added as sub commands to the parent ingest Click group. This allows each command to map to
+different connectors with shared and unique parameters.
+
+### Destination Commands
+All destination commands are added as sub commands to each parent source command. This allows each invocation of the source
+sub command to display all possible destination subcommands. The code un [utils.py](./utils.py) helps structure the
+generated text from the Click library to be more intuitive on this approach (i.e. list sub commands as  `Destinations`).
+
+### Configs
+The configs in [configs/](./configs) and connector specific ones in [cmds/](./cmds) help surface all user parameters that
+are needed to marshall the input dictionary from Click into all the respective configs needed to create a full pipeline run.
+Because click returns a flat dictionary of user inputs, the `extract_config` method in `utils.py` helps deserialize this dictionary
+into dataclasses that have nested fields (such as access configs).

{unstructured_ingest-0.7.1 → unstructured_ingest-1.0.1}/unstructured_ingest/embed/mixedbreadai.py

@@ -114,7 +114,6 @@ class MixedbreadAIEmbeddingEncoder(BaseEmbeddingEncoder):
 
 @dataclass
 class AsyncMixedbreadAIEmbeddingEncoder(AsyncBaseEmbeddingEncoder):
-
     config: MixedbreadAIEmbeddingConfig
 
     async def get_exemplary_embedding(self) -> list[float]: