tinybird-toolset 1.0.0__tar.gz

tinybird-toolset-1.0.0/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2019, Raul Ochoa
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

tinybird-toolset-1.0.0/MANIFEST.in

@@ -0,0 +1,2 @@
+global-exclude *.h
+global-exclude *.cpp

tinybird-toolset-1.0.0/PKG-INFO

@@ -0,0 +1,12 @@
+Metadata-Version: 2.1
+Name: tinybird-toolset
+Version: 1.0.0
+Summary: UNKNOWN
+Home-page: https://gitlab.com/tinybird/clickhouse-toolset
+Author: Tinybird.co
+Author-email: support@tinybird.co
+License: UNKNOWN
+Description: UNKNOWN
+Platform: UNKNOWN
+Requires-Python: >=3.8, <3.14
+Provides-Extra: test

tinybird-toolset-1.0.0/README.md

@@ -0,0 +1,188 @@
+# ClickHouse query tools
+
+Exposes ClickHouse internals to parse and manipulate ClickHouse queries.
+
+Currently made of 1 module, clickhouse-toolset, which includes functionality both for the server and the CLI.
+
+## Installing prebuilts
+
+The module is available in pypi:
+
+```bash
+pip install tinybird-toolset
+```
+You need to have access to the API Token, it may be shared from LastPass, if you don't have access request it. 
+
+If we don't have prebuilts for your platform the installation will fail.
+
+## No prebuilts available
+
+To simplify things, the main module source distribution includes only the python code so that installing it is possible,
+but it will throw when trying to use it:
+
+```python
+>>> from chtoolset import query as chquery
+Traceback (most recent call last):
+  File "<stdin>", line 1, in <module>
+  File "/home/raul/.local/lib/python3.9/site-packages/chtoolset/__init__.py", line 1, in <module>
+    from . import query
+  File "/home/raul/.local/lib/python3.9/site-packages/chtoolset/query.py", line 1, in <module>
+    from chtoolset._query import replace_tables, format, tables, table_if_is_simple_query
+ModuleNotFoundError: No module named 'chtoolset._query'
+```
+
+If you see this in the analytics server that means that your platform isn't supported and needs a prebuilt. If you see
+this in the CLI that means that we are not handling the exception as it should (using a remote server).
+
+## Development
+
+### Install pre-requisites (Ubuntu)
+
+```bash
+sudo apt-get update
+sudo apt-get install git cmake ccache python3 python3-pip ninja-build nasm yasm gawk lsb-release wget software-properties-common gnupg
+
+sudo bash -c "$(wget -O - https://apt.llvm.org/llvm.sh)"
+
+sudo add-apt-repository ppa:deadsnakes/ppa
+sudo apt update
+sudo apt install python3.8-dev python3.9-dev python3.10-dev python3.11-dev python3.12-dev python3.13-dev
+
+pip3 install virtualenv
+```
+
+### Run tests
+
+First, you need to clone the repo and **its submodules**.
+
+```bash
+git clone --recursive git@gitlab.com:tinybird/clickhouse-toolset.git
+```
+
+Then, you will compile the dependencies and the module itself. You need a modern compiler (Clang 17) to build it, both under Linux and MacOS (AppleClang is not supported).
+
+The best option is to use the Makefile targets which will use virtualenv to install dependencies, build the packages, install them too and run tests:
+
+```bash
+make test-3.9
+```
+
+### Generate pre-built packages
+
+You need to install all the necessary python releases so they are available via virtualenv.
+
+#### If you are In MacOS, prepare your environment:
+
+You need to be able to compile ClickHouse for MacOS so we follow [their guide](https://github.com/ClickHouse/ClickHouse/blob/master/docs/en/development/build-osx.md) to install the necessary packages:
+
+* Install Homebrew
+* Install Xcode and Command Line Tools
+* Install the necessary tools (cmake ninja libtool gettext llvm@16 gcc ccache findutils grep)
+* Make sure your local clang is pointing to the llvm installation and to the default from OSX / Xcode.  You can check it by running these commands. You should see a similar output
+```bash
+➜  clickhouse-toolset git:(master) ✗ clang --version
+Homebrew clang version 16.0.6
+Target: x86_64-apple-darwin22.5.0
+Thread model: posix
+InstalledDir: /usr/local/opt/llvm/bin
+➜  clickhouse-toolset git:(master) ✗ llvm-as --version
+Homebrew LLVM version 16.0.6
+  Optimized build.
+* Follow the normal build (`make build`)
+```
+
+#### Clean environment if you already did some compilation before
+To clean your environment you need to make sure that the repository is clean and updated with the expected values. To do that you can use:
+
+```bash
+make distclean
+git clean -fdx
+```
+
+#### Make sure you have the expeted version from the submodules
+```bash
+git submodule sync && git submodule update --init --recursive
+
+# If it fails because of changes inside the ClickHouse folder related to the patchs we apply, just reset all the changes in that directory so these patchs can be reaplied in the next step.
+cd clickhouse
+git reset --hard
+```
+
+#### Compile and generate the new .whl files
+Use:
+
+```bash
+make build
+```
+
+```
+Disclaimer: The MacOS package for 3.8 will be generated but it will not work and tests fill fail. We have not been using that version since 0.27.dev0 and no-one has requested it so we'll deprecate the generation for this version soon. You can continue with the process, just don't push that version to Pypi.
+```
+
+Note that to reduce the version of the dependencies in the binary wheel, it is better if generated on an old Linux distribution, and it's best to use the CI.
+
+#### Tip: re-compiling and debugging
+
+A couple of environment variables can be defined with a non-empty value to help retrying the compilation and debugging:
+
+* `OMIT_PATCHES` prevents the application of the ClickHouse patches: this is useful if you've already applied them to avoid having to restore the original source code, since some patches may not re-apply cleanly.
+* `DEBUG_SYMBOLS` triggers the generation of debug symbols for the python extension code (query.cpp and the C++ functions, but not for the ClickHouse code).
+
+#### Finish preparation of Linux packages
+
+In order to improve compatibility for Linux packages you need to use auditwheel to "repair" them  before the upload to pypi:
+
+```bash
+for i in $(ls /tmp/artifacts/*whl); do auditwheel repair --plat manylinux2014_x86_64 $i; done
+```
+
+This will check and rename them to `manylinux2014` or `manylinux_2_17` (provided they have been compiled correctly). If auditwheel fails, or the result is still `linux_x86_64`, then **don't upload them** as they won't be compatible with older Linux releases.
+
+#### Finish preparation of MacOS packages
+
+In the case of MacOS we need to check the dependencies of the generated wheel using [delocate](https://github.com/matthew-brett/delocate). Use `delocate-listdeps` to check that there aren't any external dependencies and `delocate-wheel` if there are.
+
+You can use it by running:
+```bash
+# Install it
+pip install delocate
+
+# Execute it for each .whl generated
+delocate-listdeps ./dist/clickhouse_toolset-0.27.dev0-cp311-cp311-macosx_12_0_x86_64.whl
+```
+
+In addition, to increase compatibility of the generated packages we need to rename them to the oldest release with binary compatibility (based on python tags), which we decided on 11.0:
+
+If you're executing in from Linux, run:
+```bash
+find . -type f -name \*macosx_*_*_x86\* | perl -pe 'print $_; s/macosx_.._.+_x86/macosx_11_0_x86/' | xargs -d "\n" -n2 mv
+find . -type f -name \*macosx_*_*_arm64\* | perl -pe 'print $_; s/macosx_.._.+_arm64/macosx_11_0_arm64/' | xargs -d "\n" -n2 mv
+```
+
+If you're executing in from MacOS, run:
+```bash
+find -E . -type f -regex '.*macosx_.._._x86.*' | perl -pe 'print $_; s/macosx_.._.+_x86/macosx_11_0_x86/' | xargs -n2 mv
+find -E . -type f -regex '.*macosx_.._._arm64.*' | perl -pe 'print $_; s/macosx_.._.+_arm64/macosx_11_0_arm64/' | xargs -n2 mv
+```
+
+## Examples
+
+Check tests directory
+
+## Publish
+
+1. Update VERSION in `setup.py`
+
+2. Publish the source package for the version you want to use to the **test repository** (if you don't have permissions to do that, upload the generated `.whl` packages to [the compiled package's GCP bucket](https://console.cloud.google.com/storage/browser/tinybird-bdist_wheels?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&prefix=&) and ask for help):
+
+```
+twine upload --repository-url https://test.pypi.org/legacy/ dist/tinybird-toolset-1.0.0.tar.gz
+```
+
+3. Publish the whl packages (wheelhouse/ is generated by auditwheel):
+
+```
+twine upload --repository-url https://test.pypi.org/legacy/ wheelhouse/*
+```
+
+4. Once tested, repeat for the production repository (no repository url)

tinybird-toolset-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tinybird-toolset-1.0.0/setup.py

@@ -0,0 +1,60 @@
+from setuptools import setup, Extension
+
+NAME = 'tinybird-toolset'
+VERSION = '1.0.0'
+
+try:
+    from conf import *
+    chquery = Extension(
+        'chtoolset._query',
+        sources=['src/query.cpp',],
+        depends=['conf.py',
+                 'functions/AccessControl.h',
+                 'functions/Aggregation.h',
+                 'functions/CheckCompatibleTypes.h',
+                 'functions/CheckValidWriteQuery.h',
+                 'functions/ReplaceTables.h',
+                 'functions/Tables.h',
+                 'functions/TBQueryParser.h',
+                 'functions/Validation.h',
+                 'functions/simdjsonHelpers.h',
+                 'functions/JSONPathQuery.h',
+                 'functions/JSONPathTree.h',
+                 'functions/DateTimeParser.h',
+                 'functions/RowBinaryEncoder.h',
+                 'src/PythonThreadHandler.h',
+                 'ts_build/libCHToolset.a'],
+    )
+    setup(
+        name=NAME,
+        version=VERSION,
+        url='https://gitlab.com/tinybird/clickhouse-toolset',
+        author='Tinybird.co',
+        author_email='support@tinybird.co',
+        packages=['chtoolset'],
+        package_dir={'': 'src'},
+        python_requires='>=3.8, <3.14',
+        install_requires=[],
+        extras_require={
+            'test': requirements_from_file('requirements-test.txt')
+        },
+        cmdclass={
+            'clickhouse': ClickHouseBuildExt,
+            'toolset': ToolsetBuildWithFromCH,
+            'build_ext': CustomBuildWithFromCH,
+        },
+        ext_modules=[chquery]
+    )
+
+except ModuleNotFoundError:
+    setup(
+        name=NAME,
+        version=VERSION,
+        url='https://gitlab.com/tinybird/clickhouse-toolset',
+        author='Tinybird.co',
+        author_email='support@tinybird.co',
+        packages=['chtoolset'],
+        package_dir={'': 'src'},
+        python_requires='>=3.8, <3.14',
+        install_requires=[],
+    )

tinybird-toolset-1.0.0/src/chtoolset/__init__.py

@@ -0,0 +1 @@
+from . import query

tinybird-toolset-1.0.0/src/chtoolset/query.py

@@ -0,0 +1,59 @@
+from chtoolset._query import replace_tables, \
+    format, \
+    tables, \
+    table_if_is_simple_query, \
+    query_get_type, \
+    check_compatible_types, \
+    check_valid_write_query, \
+    get_left_table, \
+    rewrite_aggregation_states, \
+    parser_cache_info, \
+    parser_cache_reset, \
+    explain_ast, \
+    create_row_binary_encoder, \
+    apply_row_binary_encoder, \
+    delete_row_binary_encoder
+
+
+class RowBinaryEncoderError(Exception):
+    """Custom exception for RowBinaryEncoder errors"""
+    pass
+
+
+class RowBinaryEncoder():
+    def __init__(self, schema: str, legacy_conversion_mode: bool = True):
+        if not isinstance(schema, str):
+            raise TypeError("Schema must be a string")
+
+        try:
+            self._encoder_ptr = create_row_binary_encoder(schema, legacy_conversion_mode)
+            if not self._encoder_ptr:
+                raise RowBinaryEncoderError("Failed to create encoder")
+        except Exception as e:
+            raise RowBinaryEncoderError(f"Error initializing encoder: {str(e)}") from e
+
+    def __enter__(self):
+        if not self._encoder_ptr:
+            raise RowBinaryEncoderError("Encoder was already closed")
+        return self
+
+    def __exit__(self, exception_type, exception_value, exception_traceback):
+        self.close()
+
+    def encode(self, block: str) -> bytes:
+        if not isinstance(block, str):
+            raise TypeError("Block must be a string")
+
+        if not self._encoder_ptr:
+            raise RowBinaryEncoderError("Encoder was already closed")
+
+        try:
+            result = apply_row_binary_encoder(self._encoder_ptr, block)
+            return result
+        except Exception as e:
+            raise RowBinaryEncoderError(f"Error encoding block: {str(e)}") from e
+
+    def close(self):
+        if self._encoder_ptr:
+            delete_row_binary_encoder(self._encoder_ptr)
+            self._encoder_ptr = None

tinybird-toolset-1.0.0/src/tinybird_toolset.egg-info/PKG-INFO

@@ -0,0 +1,12 @@
+Metadata-Version: 2.1
+Name: tinybird-toolset
+Version: 1.0.0
+Summary: UNKNOWN
+Home-page: https://gitlab.com/tinybird/clickhouse-toolset
+Author: Tinybird.co
+Author-email: support@tinybird.co
+License: UNKNOWN
+Description: UNKNOWN
+Platform: UNKNOWN
+Requires-Python: >=3.8, <3.14
+Provides-Extra: test

tinybird-toolset-1.0.0/src/tinybird_toolset.egg-info/SOURCES.txt

@@ -0,0 +1,11 @@
+LICENSE
+MANIFEST.in
+README.md
+setup.py
+src/chtoolset/__init__.py
+src/chtoolset/query.py
+src/tinybird_toolset.egg-info/PKG-INFO
+src/tinybird_toolset.egg-info/SOURCES.txt
+src/tinybird_toolset.egg-info/dependency_links.txt
+src/tinybird_toolset.egg-info/requires.txt
+src/tinybird_toolset.egg-info/top_level.txt

tinybird-toolset-1.0.0/src/tinybird_toolset.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

tinybird-toolset-1.0.0/src/tinybird_toolset.egg-info/requires.txt

@@ -0,0 +1,3 @@
+
+[test]
+pytest

tinybird-toolset-1.0.0/src/tinybird_toolset.egg-info/top_level.txt

@@ -0,0 +1 @@
+chtoolset