video-representations-extractor 1.0.0__tar.gz

video-representations-extractor-1.0.0/LICENSE.TXT

@@ -0,0 +1,13 @@
+        DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
+                    Version 2, December 2004 
+
+ Copyright (C) 2004 Sam Hocevar <sam@hocevar.net> 
+
+ Everyone is permitted to copy and distribute verbatim or modified 
+ copies of this license document, and changing it is allowed as long 
+ as the name is changed. 
+
+            DO WHAT THE FUCK YOU WANT TO PUBLIC LICENSE 
+   TERMS AND CONDITIONS FOR COPYING, DISTRIBUTION AND MODIFICATION 
+
+  0. You just DO WHAT THE FUCK YOU WANT TO.

video-representations-extractor-1.0.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.1
+Name: video-representations-extractor
+Version: 1.0.0
+Summary: Video Representations Extractor (VRE) for computing algorithmic or neural representations of each frame.
+Home-page: https://gitlab.com/meehai/video-representations-extractor
+License: WTFPL
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE.TXT
+Requires-Dist: numpy>=1.21.6
+Requires-Dist: PyYAML==6.0
+Requires-Dist: argparse==1.1
+Requires-Dist: pims==0.6.1
+Requires-Dist: tqdm==4.66.1
+Requires-Dist: scikit-image==0.19.3
+Requires-Dist: natsort==8.2.0
+Requires-Dist: gitpython==3.1.31
+Requires-Dist: gdown==4.6.0
+Requires-Dist: torch==2.1.0
+Requires-Dist: torchvision==0.16.0
+Requires-Dist: timm==0.6.13
+Requires-Dist: transforms3d==0.4.1
+Requires-Dist: pyproj>=3.2.0
+Requires-Dist: overrides==7.3.1
+Requires-Dist: pandas==2.1.3
+Requires-Dist: matplotlib==3.7.1
+Requires-Dist: flow_vis==0.1
+Requires-Dist: colorama==0.4.6
+Requires-Dist: omegaconf==2.3.0
+Requires-Dist: lovely_tensors==0.1.15
+Requires-Dist: ultralytics==8.0.120
+Requires-Dist: fvcore==0.1.5.post20221221
+Requires-Dist: pycocotools==2.0.7
+Requires-Dist: cloudpickle==3.0.0
+
+# Video Representation Extractor
+
+![logo](logo.png)
+
+## 1. Description
+
+The purpose of this repository is to export various representations starting from RGB videos only. Representations are
+defined as ways of 'looking at the world'. One can watch at various levels of information:
+- low level: colors, edges
+- mid level: depth, orientation of planes (normals)
+- high level: semantics and actions
+
+For GitHub users: this is a mirror of the
+[gitlab repository](https://gitlab.com/meehai/video-representations-extractor).
+
+<u>Supported representations</u>
+
+- See [here](vre/representations/build_representations.py) for a comprehensive list, since it updates faster
+than this README.
+
+Weights repository for supported pretrained neural-network based representations is
+[here](https://drive.google.com/drive/folders/1bWKEAiTXDpgaY2YOAFBvMqqyOGSafoIm?usp=sharing).
+
+## 2. Usage
+
+Add `bin/` directory to your `PATH` env variable to be able to access the `vre` tool directly from cli.
+
+```bash
+vre <path/to/video.mp4> --cfg_path <path/to/cfg> -o <path/to/export_dir>
+```
+
+The magic happens inside the config file, where we define *what* representations to extract and *what* parameters are
+used to instantiate said representations.
+
+### Single image usage
+
+You can get the representations for a single image (or a directory of images) by placing your image in a standalone
+directory.
+
+```bash
+vre <path/to/dir_of_images> --cfg_path <path/to/cfg> -o <path/to/export_dir>
+```
+
+Note: use `--cfg_path resources/cfgs/testCfg_ootb.yaml` for 'out of the box' working representations.
+Note2: Use `VRE_DEVICE=cuda vre...` to use cuda. For some representations, this speeds up the process by a lot.
+
+## 3. CFG file
+
+The config file will have the hyperparameters required to instantiate each supported method as well as global
+hyperparameters for the output. This means that if a depth method is pre-traied for 0-300m, this information will be
+encoded in the CFG file and passed to the constructor of that particular depth method. There are also export level
+parameters, such as the output resolution of the representations.
+
+High level format:
+
+```
+name of representation:
+  type: some high level type (such as depth, semantic, edges, etc.)
+  name: the implemented method's name (i.e. dexined, dpt, odoflow etc.)
+  dependencies: [a list of dependencies given by their names]
+  parameters: (as defined in the constructor of the implementation)
+    param1: value1
+    param2: value2
+
+name of representation 2:
+  type: some other type
+  name: some other method
+  dependencies: [name of representation]
+  parameters: []
+```
+
+Example cfg file: See [out of the box supported representations](cfgs/testCfg_ootb.yaml) and the CFG defined in
+the [CI process](.gitlab-ci.yml) for an actual export that is done at every commit on a real video.
+
+Note: If the topological sort fails (because cycle dependencies), an error will be thrown.
+
+Note2: dependencies are provided by names and apply only to the case where one representation (say odo flow) depends
+on a generic secondary representation. In this case *any* optical flow would work as long as we have a motion field
+vector for each frame returned by the required dependency. In cases where dependencies can be infered automatically,
+this is done behind the scenes. All representations require RGB, for example, but this is expected, so we don't need
+to specify it.
+
+## 4. Output format
+
+All the outputs are going to be stored as [0-1] float32 npz files, one for each frame in a directory specified by
+`--output_dir/-o`. A subdirectory will be created for each representation.
+
+For the above CFG file, 2 subdirectories will be created:
+
+```
+/path/to/output_dir/
+  name of representation/
+    1.npz, ..., N.npz + cfg.yaml
+  name of representation 2/
+    1.npz, ..., N.npz + cfg.yaml
+```
+
+The `cfg.yaml` file for each representation is created so that we know what parameters were used for that
+representation.
+
+### 4.1 Exporting PNG images
+
+We can also export images as a grid collage of all exported representations by adding the CLI argument
+`--export_collage` to the `vre` tool.
+
+This yields a new directory with PNGs:
+
+```
+/path/to/output_dir/
+  npy/
+    1.npz, ..., N.npz
+  png/
+    1.png, ..., N.png
+```
+
+## 5. Bonus
+
+### 5.1: Exporting video from PNGs
+
+After exporting pngs, use this command (requires `ffmpeg`)
+
+```bash
+old_path=`pwd`
+cd /path/to/output_dir/collage
+ffmpeg -start_number 1 -framerate 30 -i %d.png -c:v libx264 -pix_fmt yuv420p $oldPath/collage.mp4;
+cd -;
+```
+
+### 5.2 Run in docker
+- use `meehai/vre:latest` from docker hub.
+
+```
+mkdir example
+# move the cfg and the video in some local dir
+gdown https://drive.google.com/uc?id=158U-W-Gal6eXxYtS1ca1DAAxHvknqwAk -O example/vid.mp4
+wget https://gitlab.com/meehai/video-representations-extractor/-/raw/df15af177edf5c101bbb241428c43faac333cea4/test/end_to_end/imgur/cfg.yaml -O example/cfg.yaml
+docker run \
+  -v `pwd`/example:/app/resources \
+  meehai/vre \
+  /app/resources/vid.mp4 --cfg_path /app/resources/cfg.yaml -o /app/resources/result --start_frame 5 --end_frame 6
+```
+
+### 5.3 Batched experiment
+
+Last updated at: 2023.11.11
+
+See [implementation](examples/batched_experiment.py) for cfg and code.
+
+|                        |    batch=1 |    batch=3 |   ratio 1/3 |    batch=5 |   ratio 1/5 |
+|:-----------------------|-----------:|-----------:|------------:|-----------:|------------:|
+| rgb                    | 0.00287311 | 0.00285926 |    1.00485  | 0.00355646 |    0.807857 |
+| hsv                    | 0.227086   | 0.23222    |    0.977893 | 0.234075   |    0.970141 |
+| normals svd (dpth)     | 3.05688    | 3.04678    |    1.00331  | 3.096      |    0.987362 |
+| halftone               | 3.1492     | 3.14151    |    1.00245  | 3.13804    |    1.00356  |
+| softseg kmeans         | 0.821758   | 0.818581   |    1.00388  | 0.816683   |    1.00621  |
+| canny                  | 0.0149038  | 0.0143591  |    1.03793  | 0.0142797  |    1.0437   |
+| softseg gb             | 0.256591   | 0.244486   |    1.04952  | 0.236025   |    1.08714  |
+| dexined                | 0.121858   | 0.10822    |    1.12602  | 0.105016   |    1.16038  |
+| depth dpt              | 0.134931   | 0.121348   |    1.11193  | 0.113324   |    1.19066  |
+| depth odoflow (raft)   | 1.39505    | 1.17315    |    1.18915  | 1.15518    |    1.20764  |
+| fastsam (x)            | 0.0565283  | 0.0414641  |    1.36331  | 0.0392864  |    1.43888  |
+| opticalflow rife       | 0.0435189  | 0.0308007  |    1.41292  | 0.0288509  |    1.50841  |
+| opticalflow raft       | 0.999548   | 0.742625   |    1.34597  | 0.650921   |    1.53559  |
+| fastsam (s)            | 0.0348993  | 0.0215417  |    1.62008  | 0.019354   |    1.80321  |
+| semantic safeuav torch | 0.0251322  | 0.012268   |    2.04859  | 0.00914755 |    2.74742  |
+
+We can easily observe that some representations are not batched yet (all with ratios close to 1). We can also observe
+that some of them are super slow (svd, halftone, odoflow).

video-representations-extractor-1.0.0/README.md

@@ -0,0 +1,168 @@
+# Video Representation Extractor
+
+![logo](logo.png)
+
+## 1. Description
+
+The purpose of this repository is to export various representations starting from RGB videos only. Representations are
+defined as ways of 'looking at the world'. One can watch at various levels of information:
+- low level: colors, edges
+- mid level: depth, orientation of planes (normals)
+- high level: semantics and actions
+
+For GitHub users: this is a mirror of the
+[gitlab repository](https://gitlab.com/meehai/video-representations-extractor).
+
+<u>Supported representations</u>
+
+- See [here](vre/representations/build_representations.py) for a comprehensive list, since it updates faster
+than this README.
+
+Weights repository for supported pretrained neural-network based representations is
+[here](https://drive.google.com/drive/folders/1bWKEAiTXDpgaY2YOAFBvMqqyOGSafoIm?usp=sharing).
+
+## 2. Usage
+
+Add `bin/` directory to your `PATH` env variable to be able to access the `vre` tool directly from cli.
+
+```bash
+vre <path/to/video.mp4> --cfg_path <path/to/cfg> -o <path/to/export_dir>
+```
+
+The magic happens inside the config file, where we define *what* representations to extract and *what* parameters are
+used to instantiate said representations.
+
+### Single image usage
+
+You can get the representations for a single image (or a directory of images) by placing your image in a standalone
+directory.
+
+```bash
+vre <path/to/dir_of_images> --cfg_path <path/to/cfg> -o <path/to/export_dir>
+```
+
+Note: use `--cfg_path resources/cfgs/testCfg_ootb.yaml` for 'out of the box' working representations.
+Note2: Use `VRE_DEVICE=cuda vre...` to use cuda. For some representations, this speeds up the process by a lot.
+
+## 3. CFG file
+
+The config file will have the hyperparameters required to instantiate each supported method as well as global
+hyperparameters for the output. This means that if a depth method is pre-traied for 0-300m, this information will be
+encoded in the CFG file and passed to the constructor of that particular depth method. There are also export level
+parameters, such as the output resolution of the representations.
+
+High level format:
+
+```
+name of representation:
+  type: some high level type (such as depth, semantic, edges, etc.)
+  name: the implemented method's name (i.e. dexined, dpt, odoflow etc.)
+  dependencies: [a list of dependencies given by their names]
+  parameters: (as defined in the constructor of the implementation)
+    param1: value1
+    param2: value2
+
+name of representation 2:
+  type: some other type
+  name: some other method
+  dependencies: [name of representation]
+  parameters: []
+```
+
+Example cfg file: See [out of the box supported representations](cfgs/testCfg_ootb.yaml) and the CFG defined in
+the [CI process](.gitlab-ci.yml) for an actual export that is done at every commit on a real video.
+
+Note: If the topological sort fails (because cycle dependencies), an error will be thrown.
+
+Note2: dependencies are provided by names and apply only to the case where one representation (say odo flow) depends
+on a generic secondary representation. In this case *any* optical flow would work as long as we have a motion field
+vector for each frame returned by the required dependency. In cases where dependencies can be infered automatically,
+this is done behind the scenes. All representations require RGB, for example, but this is expected, so we don't need
+to specify it.
+
+## 4. Output format
+
+All the outputs are going to be stored as [0-1] float32 npz files, one for each frame in a directory specified by
+`--output_dir/-o`. A subdirectory will be created for each representation.
+
+For the above CFG file, 2 subdirectories will be created:
+
+```
+/path/to/output_dir/
+  name of representation/
+    1.npz, ..., N.npz + cfg.yaml
+  name of representation 2/
+    1.npz, ..., N.npz + cfg.yaml
+```
+
+The `cfg.yaml` file for each representation is created so that we know what parameters were used for that
+representation.
+
+### 4.1 Exporting PNG images
+
+We can also export images as a grid collage of all exported representations by adding the CLI argument
+`--export_collage` to the `vre` tool.
+
+This yields a new directory with PNGs:
+
+```
+/path/to/output_dir/
+  npy/
+    1.npz, ..., N.npz
+  png/
+    1.png, ..., N.png
+```
+
+## 5. Bonus
+
+### 5.1: Exporting video from PNGs
+
+After exporting pngs, use this command (requires `ffmpeg`)
+
+```bash
+old_path=`pwd`
+cd /path/to/output_dir/collage
+ffmpeg -start_number 1 -framerate 30 -i %d.png -c:v libx264 -pix_fmt yuv420p $oldPath/collage.mp4;
+cd -;
+```
+
+### 5.2 Run in docker
+- use `meehai/vre:latest` from docker hub.
+
+```
+mkdir example
+# move the cfg and the video in some local dir
+gdown https://drive.google.com/uc?id=158U-W-Gal6eXxYtS1ca1DAAxHvknqwAk -O example/vid.mp4
+wget https://gitlab.com/meehai/video-representations-extractor/-/raw/df15af177edf5c101bbb241428c43faac333cea4/test/end_to_end/imgur/cfg.yaml -O example/cfg.yaml
+docker run \
+  -v `pwd`/example:/app/resources \
+  meehai/vre \
+  /app/resources/vid.mp4 --cfg_path /app/resources/cfg.yaml -o /app/resources/result --start_frame 5 --end_frame 6
+```
+
+### 5.3 Batched experiment
+
+Last updated at: 2023.11.11
+
+See [implementation](examples/batched_experiment.py) for cfg and code.
+
+|                        |    batch=1 |    batch=3 |   ratio 1/3 |    batch=5 |   ratio 1/5 |
+|:-----------------------|-----------:|-----------:|------------:|-----------:|------------:|
+| rgb                    | 0.00287311 | 0.00285926 |    1.00485  | 0.00355646 |    0.807857 |
+| hsv                    | 0.227086   | 0.23222    |    0.977893 | 0.234075   |    0.970141 |
+| normals svd (dpth)     | 3.05688    | 3.04678    |    1.00331  | 3.096      |    0.987362 |
+| halftone               | 3.1492     | 3.14151    |    1.00245  | 3.13804    |    1.00356  |
+| softseg kmeans         | 0.821758   | 0.818581   |    1.00388  | 0.816683   |    1.00621  |
+| canny                  | 0.0149038  | 0.0143591  |    1.03793  | 0.0142797  |    1.0437   |
+| softseg gb             | 0.256591   | 0.244486   |    1.04952  | 0.236025   |    1.08714  |
+| dexined                | 0.121858   | 0.10822    |    1.12602  | 0.105016   |    1.16038  |
+| depth dpt              | 0.134931   | 0.121348   |    1.11193  | 0.113324   |    1.19066  |
+| depth odoflow (raft)   | 1.39505    | 1.17315    |    1.18915  | 1.15518    |    1.20764  |
+| fastsam (x)            | 0.0565283  | 0.0414641  |    1.36331  | 0.0392864  |    1.43888  |
+| opticalflow rife       | 0.0435189  | 0.0308007  |    1.41292  | 0.0288509  |    1.50841  |
+| opticalflow raft       | 0.999548   | 0.742625   |    1.34597  | 0.650921   |    1.53559  |
+| fastsam (s)            | 0.0348993  | 0.0215417  |    1.62008  | 0.019354   |    1.80321  |
+| semantic safeuav torch | 0.0251322  | 0.012268   |    2.04859  | 0.00914755 |    2.74742  |
+
+We can easily observe that some representations are not batched yet (all with ratios close to 1). We can also observe
+that some of them are super slow (svd, halftone, odoflow).

video-representations-extractor-1.0.0/bin/vre

@@ -0,0 +1,85 @@
+#!/usr/bin/env python3
+from argparse import ArgumentParser
+from typing import List
+from pathlib import Path
+from datetime import datetime
+import os
+from natsort import natsorted
+import numpy as np
+import pandas as pd
+import pims
+from vre import VRE
+from vre.utils import FakeVideo, image_read, get_project_root
+from vre.representations import build_representations_from_cfg
+from vre.logger import logger
+from omegaconf import OmegaConf
+
+def _print_and_store_run_stats(run_stats: pd.DataFrame):
+    print(run_stats if len(run_stats) > 1 else run_stats.T)
+    (logs_dir := Path(os.getenv("VRE_LOGS_DIR", get_project_root() / "logs"))).mkdir(exist_ok=True, parents=True)
+    run_stats_out_file = f"{logs_dir}/run_stats_{datetime.now().replace(microsecond=0).isoformat()}.csv"
+    logger.info(f"Stored vre run log file at '{run_stats_out_file}")
+    run_stats.to_csv(run_stats_out_file)
+
+def get_args():
+    """cli args"""
+    parser = ArgumentParser()
+    parser.add_argument("video_path", help="Path to the scene video we are processing")
+    parser.add_argument("--output_path", "-o", required=True,
+                        help="Path to the output directory where representations are stored")
+    parser.add_argument("--cfg_path", required=True, help="Path to global YAML cfg file")
+    parser.add_argument("--start_frame", type=int, default=0, help="The first frame. If not set, defaults to first.")
+    parser.add_argument("--end_frame", type=int, help="End frame. If not set, defaults to length of video")
+    parser.add_argument("--output_dir_exist_mode", choices=["overwrite", "skip_computed", "raise"])
+    parser.add_argument("--frame_rate", type=int, help="For a directory of images. Required for optical flow")
+    parser.add_argument("--batch_size", type=int)
+    parser.add_argument("--output_size", nargs="+", default=["video_shape"])
+    parser.add_argument("--representations", nargs="+", help="Subset of representations to run VRE for")
+    parser.add_argument("--n_threads_data_storer", type=int, default=0)
+
+    args = parser.parse_args()
+    args.video_path = Path(args.video_path).absolute()
+    args.output_path = Path(args.output_path).absolute()
+    assert 0 < len(args.output_size) < 3, args.output_size
+    args.output_size = args.output_size[0] if len(args.output_size) == 1 else tuple(map(int, args.output_size))
+    args.cfg_path = Path(args.cfg_path).absolute() if args.cfg_path is not None else args.cfg_path
+    return args
+
+def get_cfg():
+    """cfg and cli args"""
+    args = get_args()
+    cfg = OmegaConf.load(open(args.cfg_path, "r"))
+
+    def _update_cfg_from_args(cfg, args, items: List[str]):
+        for item in items:
+            if getattr(args, item) is None:
+                continue
+            cfg[item] = getattr(args, item)
+            logger.warning(f"--{item} is set to '{cfg[item]}'. Will update the cfg value")
+
+        return cfg
+
+    cfg.vre = _update_cfg_from_args(cfg.vre, args, ["start_frame", "end_frame", "output_dir_exist_mode",
+                                                    "batch_size", "output_size", "n_threads_data_storer"])
+    if args.representations is not None:
+        assert (d := set(args.representations).difference(r := cfg.representations.keys())) == set(), f"{d} not in {r}"
+        logger.info(f"Keeping only {args.representations} from the config")
+        cfg.representations = {k: v for k, v in cfg.representations.items() if k in args.representations}
+    return args, cfg
+
+def main():
+    args, cfg = get_cfg()
+    if args.video_path.is_dir():
+        raw_data = [image_read(f) for f in natsorted(args.video_path.glob("*.png"), key=lambda p: p.name)]
+        assert all(x.shape == raw_data[0].shape for x in raw_data), f"Images shape differ in '{args.video_path}'"
+        logger.info(f"--video_path is a directory. Assuming a directory of images. Found {len(raw_data)} images.")
+        video = FakeVideo(np.array(raw_data, dtype=np.uint8), frame_rate=args.frame_rate)
+    else:
+        video = pims.Video(args.video_path)
+    representations = build_representations_from_cfg(cfg.representations)
+    vre = VRE(video, representations)
+    run_stats = vre(args.output_path, **cfg.vre)
+    _print_and_store_run_stats(run_stats)
+
+if __name__ == "__main__":
+    main()

video-representations-extractor-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

video-representations-extractor-1.0.0/setup.py

@@ -0,0 +1,52 @@
+from setuptools import setup, find_packages
+from os import path
+
+
+def get_reqs(requirements):
+    # Do not add to required lines pointing to Git repositories
+    required, dependency_links = [], []
+    EGG_MARK = "#egg="
+    for line in requirements:
+        if line.startswith("-e git:") or line.startswith("-e git+") or line.startswith("git:") \
+                or line.startswith("git+"):
+            line = line.lstrip("-e ")  # in case that is using "-e"
+            if EGG_MARK in line:
+                package_name = line[line.find(EGG_MARK) + len(EGG_MARK) :]
+                repository = line[: line.find(EGG_MARK)]
+                required.append(f"{package_name} @ {repository}")
+                dependency_links.append(line)
+            else:
+                print("Dependency to a git repository should have the format:")
+                print("git+ssh://git@github.com/xxxxx/xxxxxx#egg=package_name")
+        else:
+            required.append(line)
+    return required, dependency_links
+
+
+name = "video-representations-extractor"
+version = "1.0.0"
+description = "Video Representations Extractor (VRE) for computing algorithmic or neural representations of each frame."
+url = "https://gitlab.com/meehai/video-representations-extractor"
+
+loc = path.abspath(path.dirname(__file__))
+with open(f"{loc}/README.md", "r", encoding="utf-8") as fh:
+    long_description = fh.read()
+
+with open(f"{loc}/requirements.txt") as f:
+    requirements = f.read().splitlines()
+required, dependency_links = get_reqs(requirements)
+
+setup(
+    name=name,
+    version=version,
+    description=description,
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    url=url,
+    packages=find_packages(),
+    install_requires=required,
+    dependency_links=dependency_links,
+    license="WTFPL",
+    python_requires=">=3.9",
+    scripts=["bin/vre"],
+)