gst-python-ml 0.3.0__tar.gz → 1.0.1__tar.gz

{gst_python_ml-0.3.0 → gst_python_ml-1.0.1}/MANIFEST.in

@@ -1,4 +1,3 @@
-include requirements.txt
 include README.md
 include COPYING
 recursive-include plugins/python *

{gst_python_ml-0.3.0/plugins/python/gst_python_ml.egg-info → gst_python_ml-1.0.1}/PKG-INFO

@@ -1,63 +1,47 @@
 Metadata-Version: 2.4
 Name: gst-python-ml
-Version: 0.3.0
+Version: 1.0.1
 Summary: An ML package for GStreamer
-Home-page: https://github.com/collabora/gst-python-ml
-Author: Aaron Boxer
-Author-email: aaron.boxer@collabora.com
-License: LGPL-3.0
+Author-email: Aaron Boxer <aaron.boxer@collabora.com>
+Project-URL: Homepage, https://github.com/collabora/gst-python-ml
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: GNU Lesser General Public License v3 (LGPLv3)
 Classifier: Operating System :: OS Independent
-Requires-Python: >=3.6
+Requires-Python: >=3.13
 Description-Content-Type: text/markdown
 License-File: COPYING
 Requires-Dist: pygobject
-Requires-Dist: torch
-Requires-Dist: torchvision
-Requires-Dist: torchaudio
-Requires-Dist: transformers
+Requires-Dist: torch==2.9.1
+Requires-Dist: torchvision==0.24.1
+Requires-Dist: torchaudio==2.9.1
+Requires-Dist: transformers==4.56.2
+Requires-Dist: qwen-vl-utils[decord]==0.0.8
+Requires-Dist: autoawq==0.2.9
+Requires-Dist: accelerate==1.12.0
+Requires-Dist: bitsandbytes==0.46.1
 Requires-Dist: numpy
 Requires-Dist: huggingface-hub
 Requires-Dist: lap
-Requires-Dist: ultralytics
 Requires-Dist: pycairo
-Requires-Dist: pytest>=7.0
-Provides-Extra: kafka
-Requires-Dist: confluent-kafka; extra == "kafka"
-Provides-Extra: captioning
-Requires-Dist: ninja; extra == "captioning"
-Requires-Dist: flash_attn; extra == "captioning"
+Requires-Dist: ultralytics
+Requires-Dist: confluent_kafka
+Requires-Dist: diffusers
+Requires-Dist: sentencepiece
+Requires-Dist: protobuf
+Requires-Dist: pysilero
+Requires-Dist: faster_whisper
+Requires-Dist: pysilero-vad
+Requires-Dist: whisperspeech
+Requires-Dist: webdataset
+Requires-Dist: easydict
+Requires-Dist: pyflann-py3
+Requires-Dist: speechbrain
 Provides-Extra: language
-Requires-Dist: nvidia-cudnn-cu11==8.9.6.50; extra == "language"
-Requires-Dist: sentencepiece; extra == "language"
-Requires-Dist: protobuf; extra == "language"
-Requires-Dist: accelerate; extra == "language"
-Requires-Dist: pysilero; extra == "language"
-Requires-Dist: faster_whisper; extra == "language"
-Requires-Dist: pysilero-vad; extra == "language"
-Requires-Dist: diffusers; extra == "language"
-Requires-Dist: whisperspeech; extra == "language"
-Requires-Dist: webdataset; extra == "language"
-Requires-Dist: sacremoses; extra == "language"
-Requires-Dist: coqui-tts; extra == "language"
-Dynamic: author
-Dynamic: author-email
-Dynamic: classifier
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
 Dynamic: license-file
-Dynamic: provides-extra
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
 
 # GStreamer Python ML
 
-This project provides Python base classes and GStreamer elements supporting a broad range
-of ML features. 
+This project provides a pure Python ML framework for upstream GStreamer, supporting a broad range of ML vision and language features. 
 
 Supported functionality includes:
 
@@ -72,24 +56,21 @@ Supported functionality includes:
 1. LLMs
 1. serializing model metadata to Kafka server
 
-ML toolkits are supported via the `MLEngine` abstraction - we have nominal support for
+Different ML toolkits are supported via the `MLEngine` abstraction - we have nominal support for
 TensorFlow, LiteRT and OpenVINO, but all testing thus far has been done with PyTorch.
 
-These elements will work with your distribution's GStreamer packages. They have been tested on Ubuntu 24 with GStreamer 1.24.
-
-## Python Version
-
-All elements have been tested with Python 3.12, the installed version of Python on Ubuntu 24
+These elements will work with your distribution's GStreamer packages as long as the GStreamer version
+is >= 1.24.
 
 ## Install
 
-There are two installation options described below: installing on your host machine,
-or installing with a Docker container:
+There are two installation options described below: on host machine or on Docker container:
 
-### Host Install (Ubuntu 24)
+### Host Install
 
-#### Install packages
+#### Install distribution packages
 
+##### Ubuntu
 ```
 sudo apt update && sudo apt -y upgrade
 sudo apt install -y python3-pip  python3-venv \
@@ -99,39 +80,52 @@ sudo apt install -y python3-pip  python3-venv \
     libcairo2 libcairo2-dev git
 ```
 
-#### Install venv
+##### Fedora
+```
+sudo dnf upgrade -y
+sudo dnf install -y python3-pip \
+    gstreamer1-plugins-base gstreamer1-plugins-base-tools \
+    gstreamer1-plugins-good gstreamer1-plugins-bad-free \
+    gstreamer1-plugins-bad-free-devel python3-gstreamer1 \
+    cairo cairo-devel git
+```
 
-`python3 -m venv --system-site-packages ~/venv`
 
 
-#### Clone repo (host)
+#### Manage Python packages with uv
 
-`git clone https://github.com/collabora/gst-python-ml.git`
+##### install
+curl -LsSf https://astral.sh/uv/install.sh | sh
 
-#### Update .bashrc
+##### set up uv venv
 
 ```
-export VIRTUAL_ENV=$HOME/venv
-export PATH=$VIRTUAL_ENV/bin:$PATH
-export GST_PLUGIN_PATH=$HOME/src/gst-python-ml/plugins
+uv venv --system-site-packages
+source .venv/bin/activate
+uv sync
 ```
 
-and then
+Now manually install flash-attn wheel (must match your version of python, torch and cuda)
+For example:
+
+`uv pip install ./flash_attn-2.8.3+cu128torch2.9-cp313-cp313-linux_x86_64.whl`
+
+Pe-built wheels can be found here:
+https://github.com/mjun0812/flash-attention-prebuild-wheels/releases
 
-`source ~/.bashrc`
 
-#### Activate venv and install basic pip packages
+#### Clone repo
 
 ```
-source $VIRTUAL_ENV/bin/activate
-pip install --upgrade pip
+cd $HOME/src
+git clone https://github.com/collabora/gst-python-ml.git
 ```
 
-#### Install pip requirements
+#### Update .bashrc
 
 ```
-cd ~/src/gst-python-ml
-pip install -r requirements.txt
+echo 'export GST_PLUGIN_PATH=$HOME/src/gst-python-ml/plugins:$GST_PLUGIN_PATH' >> ~/.bashrc
+source ~/.bashrc
 ```
 
 ### Docker Install
@@ -141,7 +135,7 @@ pip install -r requirements.txt
 Important Note:
 
 This Dockerfile maps a local `gst-python-ml` repository to the container,
-and expects this repository to be located in `~/src` i.e.  `~/src/gst-python-ml`.
+and expects this repository to be located in `$HOME/src` i.e.  `$HOME/src/gst-python-ml`.
 
 
 #### Enable Docker GPU Support on Host
@@ -149,43 +143,56 @@ and expects this repository to be located in `~/src` i.e.  `~/src/gst-python-ml`
 To use the host GPU in a docker container, you will need to install the nvidia container toolkit. If running on CPU, these steps can be skipped.
 
 
-Add nvidia repository (Ubuntu)
-
+##### Ubuntu
 ```
 curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
   && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
     sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
     sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
-```
 
-Then
-
-```
 sudo apt update
 sudo apt install -y nvidia-container-toolkit
 sudo systemctl restart docker
 ```
 
+##### Fedora
+```
+curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/
+nvidia-container-toolkit.repo
+
+sudo dnf install nvidia-container-toolkit
+sudo nvidia-ctk runtime configure --runtime=docker
+sudo nvidia-ctk runtime configure --runtime=containerd
+sudo systemctl restart containerd
+sudo systemctl restart docker
+```
+
+##### Testing GPU Support
+
+```
+docker pull nvidia/cuda:12.0.0-base-ubuntu22.04
+docker run --rm --runtime=nvidia --gpus all nvidia/cuda:12.0.0-base-ubuntu22.04 nvidia-smi
+
+```
+
+
 #### Build Ubuntu 24.04 Container
+
 `docker build -f ./Dockerfile -t ubuntu24:latest .`
 
 #### Run Docker Container
 
 a) If running on CPU, just remove `--gpus all` from command below
+
 b) This command assumes you have set up a Kafka network as described below
 
 `docker run -v ~/src/gst-python-ml/:/root/gst-python-ml -it --rm --gpus all --name ubuntu24 ubuntu24:latest /bin/bash`
 
-In the container shell, run
+In the container shell, install `uv` following steps above
 
-`pip install -r requirements.txt`
+`cd gst-python-ml` to run the pipelines below.
 
-to install base requirements, and then
-
-`cd gst-python-ml` to run the pipelines below. After installing requirements,
-it is recommended to open another terminal on host and run
-
-`docker ps` to get the container id, and then run
+To persist the container, run `docker ps` to get the container id, and then run
 
 `docker commit $CONTAINER_ID` to commit the changes, where `$CONTAINER_ID`
 is the id for your docker instance.
@@ -199,15 +206,16 @@ docker container prune -f
 docker image prune -a -f
 ```
 
-## IMPORTANT NOTE
+## IMPORTANT NOTES
+
+### Birdseye
+
+To use `pyml_birdseye`, additional pip requirements must be installed from the `plugins/python/birdseye` folder.
 
-To use the language elements included in this project, the `nvidia-cuda-toolkit`
-ubuntu package must be installed, and additional pip requirements must be installed from
-`requirements/language_requrements.txt`
 
 ## Post Install
 
-Run `gst-inspect-1.0 python` to see all of the pyml elements listed.
+Run `gst-inspect-1.0 python` to list pyml elements.
 
 # Building PyPI Package
 
@@ -237,79 +245,6 @@ Run `gst-inspect-1.0 python` to see all of the pyml elements listed.
 
 Below are some sample pipelines for the various elements in this project.
 
-### kafkasink
-
-#### Setting up kafka network
-
-`docker network create kafka-network`
-
-and list networks
-
-`docker network ls`
-
-#### docker launch
-
-To launch a docker instance with the kafka network, add ` --network kafka-network  `
-to the docker launch command above.
-
-#### Set up kafka and zookeeper
-
-Note: setup below assumes you are running your pipeline in a docker container. 
-If running pipeline from host, then the port changes from `9092` to `29092`,
-and the broker changes from `kafka` to `localhost`.
-
-```
-docker stop kafka zookeeper
-docker rm kafka zookeeper
-docker run -d --name zookeeper --network kafka-network -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest
-docker run -d --name kafka --network kafka-network \
-  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
-  -e KAFKA_ADVERTISED_LISTENERS=INSIDE://kafka:9092,OUTSIDE://localhost:29092 \
-  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT \
-  -e KAFKA_LISTENERS=INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092 \
-  -e KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE \
-  -e KAFKA_BROKER_ID=1 \
-  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
-  -p 9092:9092 \
-  -p 29092:29092 \
-  confluentinc/cp-kafka:latest
-```
-
-#### Create test topic
-```
-docker exec kafka kafka-topics --create --topic test-kafkasink-topic --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1
-```
-
-#### list topics
-
-`docker exec -it kafka kafka-topics --list --bootstrap-server kafka:9092`
-
-
-#### delete topic
-
-`docker exec -it kafka kafka-topics --delete --topic test-topic --bootstrap-server kafka:9092`
-
-
-#### consume topic
-
-`docker exec -it kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test-kafkasink-topic --from-beginning`
-
-
-### non ML
-
-`GST_DEBUG=4 gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720 ! pyml_overlay meta-path=data/sample_metadata.json tracking=true ! videoconvert ! autovideosink`
-
-Note: make sure to set the following in `.bashrc` file :
-
-`export GST_PLUGIN_PATH=/home/$USER/src/gst-python-ml/plugins:$GST_PLUGIN_PATH`
-
-
-### Bird's Eye View
-
-`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videoconvert ! pyml_birdseye ! videoconvert ! autovideosink`
-
-`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videorate ! video/x-raw,framerate=30/1 ! videoconvert ! pyml_birdseye ! videoconvert ! openh264enc ! h264parse ! matroskamux ! filesink location=output.mkv`
-
 ### Classification
 
 ```
@@ -366,17 +301,10 @@ GST_DEBUG=4 gst-launch-1.0   filesrc location=data/soccer_tracking.mp4 ! decodeb
 
 ```
 
-#### yolo with overlay
-
 ```
- GST_DEBUG=4 gst-launch-1.0   filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True !  pyml_overlay ! videoconvert !  autovideosink
+GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvertscale ! video/x-raw,width=640,height=480 ! demo_soccer model-name=yolo11m device=cuda:0 ! pyml_overlay ! videoconvert ! autovideosink
 ```
 
-### streammux/streamdemux pipeline
-
-```
- GST_DEBUG=4 gst-launch-1.0   videotestsrc pattern=ball ! video/x-raw, width=320, height=240 ! queue ! pyml_streammux name=mux   videotestsrc pattern=smpte ! video/x-raw, width=320, height=240 ! queue ! mux.sink_1   videotestsrc pattern=smpte ! video/x-raw, width=320, height=240 ! queue ! mux.sink_2   mux.src ! queue ! pyml_streamdemux name=demux   demux.src_0 ! queue ! glimagesink  demux.src_1 ! queue ! glimagesink   demux.src_2 ! queue  ! glimagesink
-```
 
 ### Transcription
 
@@ -392,6 +320,14 @@ GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english
 GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! pyml_whispertranscribe device=cuda language=ko translate=yes ! fakesink
 ```
 
+#### demucs audio separation
+
+
+```
+GST_DEBUG=4 gst-launch-1.0 filesrc location=data/air_traffic_korean_with_english.wav ! decodebin ! audioconvert ! audioresample ! pyml_demucs device=cuda ! wavenc ! filesink location=separated_vocals.wav
+```
+
+
 #### coquitts
 
 ```
@@ -436,11 +372,87 @@ https://huggingface.co/models?sort=trending&search=Helsinki
 
 #### Caption
 
-#### caption + yolo
+#### caption qwen with history
+
+(should also work with "microsoft/Phi-3.5-vision-instruct" model)
+
+```
+GST_DEBUG=3 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videoconvertscale ! video/x-raw,width=640,height=480 ! tee name=t t. ! queue ! textoverlay name=overlay wait-text=false ! videoconvert ! autovideosink t. ! queue leaky=2 max-size-buffers=1 ! videoconvertscale ! video/x-raw,width=240,height=180 ! pyml_caption_qwen device=cuda:0 prompt="In one sentence, describe what you see?" model-name="Qwen/Qwen2.5-VL-3B-Instruct-AWQ" name=cap cap.src ! fakesink async=0 sync=0 cap.text_src ! queue ! coalescehistory history-length=10 ! pyml_llm model-name="Qwen/Qwen3-0.6B" device=cuda system-prompt="You receive the history of what happened in recent times, summarize it nicely with excitement but NEVER mention the specific times. Focus on the most recent events." ! queue ! overlay.text_sink
+```
+
+
+### Bird's Eye View
+
+`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videoconvert ! pyml_birdseye ! videoconvert ! autovideosink`
+
+`GST_DEBUG=4 gst-launch-1.0 filesrc location=data/soccer_single_camera.mp4 ! decodebin ! videorate ! video/x-raw,framerate=30/1 ! videoconvert ! pyml_birdseye ! videoconvert ! openh264enc ! h264parse ! matroskamux ! filesink location=output.mkv`
+
+
+### kafkasink
+
+#### Setting up kafka network
+
+`docker network create kafka-network`
 
-`GST_DEBUG=4 gst-launch-1.0   filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! videoscale ! video/x-raw,width=640,height=480 ! pyml_yolo model-name=yolo11m device=cuda:0 track=True ! pyml_caption device=cuda:0 ! textoverlay !  pyml_overlay ! videoconvert !  autovideosink`
+and list networks
+
+`docker network ls`
+
+#### docker launch
+
+To launch a docker instance with the kafka network, add ` --network kafka-network  `
+to the docker launch command above.
 
+#### Set up kafka and zookeeper
 
-#### caption
+Note: setup below assumes you are running your pipeline in a docker container. 
+If running pipeline from host, then the port changes from `9092` to `29092`,
+and the broker changes from `kafka` to `localhost`.
 
-`GST_DEBUG=4 gst-launch-1.0   filesrc location=data/soccer_tracking.mp4 ! decodebin ! videoconvert ! pyml_caption device=cuda:0 downsampled_width=320 downsampled_height=240 prompt="What is the name of the game being played?" ! textoverlay !  autovideosink`
+```
+docker stop kafka zookeeper
+docker rm kafka zookeeper
+docker run -d --name zookeeper --network kafka-network -e ZOOKEEPER_CLIENT_PORT=2181 confluentinc/cp-zookeeper:latest
+docker run -d --name kafka --network kafka-network \
+  -e KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181 \
+  -e KAFKA_ADVERTISED_LISTENERS=INSIDE://kafka:9092,OUTSIDE://localhost:29092 \
+  -e KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT \
+  -e KAFKA_LISTENERS=INSIDE://0.0.0.0:9092,OUTSIDE://0.0.0.0:29092 \
+  -e KAFKA_INTER_BROKER_LISTENER_NAME=INSIDE \
+  -e KAFKA_BROKER_ID=1 \
+  -e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
+  -p 9092:9092 \
+  -p 29092:29092 \
+  confluentinc/cp-kafka:latest
+```
+
+#### Create test topic
+```
+docker exec kafka kafka-topics --create --topic test-kafkasink-topic --bootstrap-server kafka:9092 --partitions 1 --replication-factor 1
+```
+
+#### list topics
+
+`docker exec -it kafka kafka-topics --list --bootstrap-server kafka:9092`
+
+
+#### delete topic
+
+`docker exec -it kafka kafka-topics --delete --topic test-topic --bootstrap-server kafka:9092`
+
+
+#### consume topic
+
+`docker exec -it kafka kafka-console-consumer --bootstrap-server kafka:9092 --topic test-kafkasink-topic --from-beginning`
+
+
+### non ML
+
+`GST_DEBUG=4 gst-launch-1.0 videotestsrc ! video/x-raw,width=1280,height=720 ! pyml_overlay meta-path=data/sample_metadata.json tracking=true ! videoconvert ! autovideosink`
+
+
+### streammux/streamdemux pipeline
+
+```
+ GST_DEBUG=4 gst-launch-1.0   videotestsrc pattern=ball ! video/x-raw, width=320, height=240 ! queue ! pyml_streammux name=mux   videotestsrc pattern=smpte ! video/x-raw, width=320, height=240 ! queue ! mux.sink_1   videotestsrc pattern=smpte ! video/x-raw, width=320, height=240 ! queue ! mux.sink_2   mux.src ! queue ! pyml_streamdemux name=demux   demux.src_0 ! queue ! glimagesink  demux.src_1 ! queue ! glimagesink   demux.src_2 ! queue  ! glimagesink
+```