mapillary-tools 0.14.0a2__tar.gz → 0.14.0b1__tar.gz

{mapillary_tools-0.14.0a2 → mapillary_tools-0.14.0b1}/PKG-INFO

@@ -1,32 +1,37 @@
 Metadata-Version: 2.4
 Name: mapillary_tools
-Version: 0.14.0a2
+Version: 0.14.0b1
 Summary: Mapillary Image/Video Import Pipeline
-Home-page: https://github.com/mapillary/mapillary_tools
-Author: Mapillary
+Author-email: Mapillary <support@mapillary.com>
 License: BSD
-Requires-Python: >=3.8
+Project-URL: Homepage, https://github.com/mapillary/mapillary_tools
+Project-URL: Repository, https://github.com/mapillary/mapillary_tools
+Project-URL: Issues, https://github.com/mapillary/mapillary_tools/issues
+Keywords: mapillary,gis,computer vision,street view
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: End Users/Desktop
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: appdirs<2.0.0,>=1.4.4
 Requires-Dist: construct<3.0.0,>=2.10.0
 Requires-Dist: exifread==2.3.2
-Requires-Dist: piexif==1.1.3
 Requires-Dist: gpxpy<1.6.0,>=1.5.0
+Requires-Dist: jsonschema~=4.17.3
+Requires-Dist: piexif==1.1.3
 Requires-Dist: pynmea2<2.0.0,>=1.12.0
 Requires-Dist: requests[socks]<3.0.0,>=2.20.0
 Requires-Dist: tqdm<5.0,>=4.0
-Requires-Dist: typing_extensions
-Requires-Dist: jsonschema~=4.17.3
-Dynamic: author
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
+Requires-Dist: typing-extensions>=4.12.2
 Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
 
 <p align="center">
   <a href="https://github.com/mapillary/mapillary_tools/">
@@ -58,18 +63,40 @@ mapillary_tools --help
 <!--ts-->
 
 - [Supported File Formats](#supported-file-formats)
+  - [Image Formats](#image-formats)
+  - [Video Formats](#video-formats)
 - [Installation](#installation)
+  - [Standalone Executable](#standalone-executable)
+  - [Installing via pip](#installing-via-pip)
+    - [Installing on Android Devices](#installing-on-android-devices)
 - [Usage](#usage)
   - [Process and Upload](#process-and-upload)
   - [Process](#process)
   - [Upload](#upload)
 - [Advanced Usage](#advanced-usage)
   - [Local Video Processing](#local-video-processing)
+    - [Install FFmpeg](#install-ffmpeg)
+    - [Video Processing](#video-processing)
   - [Geotagging with GPX](#geotagging-with-gpx)
+  - [New video geotagging features (experimental)](#new-video-geotagging-features-experimental)
+    - [Usage](#usage-1)
+    - [Examples](#examples)
+      - [Generic supported videos](#generic-supported-videos)
+      - [External GPX](#external-gpx)
+      - [Insta360 stitched videos](#insta360-stitched-videos)
+    - [Limitations of `--video_geotag_source`](#limitations-of---video_geotag_source)
   - [Authenticate](#authenticate)
+    - [Examples](#examples-1)
   - [Image Description](#image-description)
   - [Zip Images](#zip-images)
 - [Development](#development)
+  - [Setup](#setup)
+    - [Option 1: Using uv (Recommended)](#option-1-using-uv-recommended)
+    - [Option 2: Using pip with virtual environment](#option-2-using-pip-with-virtual-environment)
+  - [Running the code](#running-the-code)
+  - [Tests](#tests)
+  - [Code Quality](#code-quality)
+  - [Release and Build](#release-and-build)
 
 <!--te-->
 
@@ -495,29 +522,49 @@ git clone git@github.com:mapillary/mapillary_tools.git
 cd mapillary_tools
 ```
 
-Set up the virtual environment. It is optional but recommended:
+### Option 1: Using uv (Recommended)
+
+Use [uv](https://docs.astral.sh/uv/) - a fast Python package manager.
+
+Install the project in development mode with all dependencies:
 
 ```sh
-pip install pipenv
+# Install the project and development dependencies
+uv sync --group dev
+
+# Activate the virtual environment
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 ```
 
-Install dependencies:
+### Option 2: Using pip with virtual environment
+
+Set up a virtual environment (recommended):
 
 ```sh
-pipenv install -r requirements.txt
-pipenv install -r requirements-dev.txt
+python -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 ```
 
-Enter the virtualenv shell:
+Install the project in development mode:
 
 ```sh
-pipenv shell
+# Install the project and all dependencies in editable mode
+pip install -e .
+
+# Install development dependencies
+pip install --group dev
 ```
 
+## Running the code
+
 Run the code from the repository:
 
 ```sh
-python3 -m mapillary_tools.commands --version
+# If you have mapillary_tools installed in editable mode
+mapillary_tools --version
+
+# Alternatively
+python -m mapillary_tools.commands --version
 ```
 
 ## Tests
@@ -525,19 +572,28 @@ python3 -m mapillary_tools.commands --version
 Run tests:
 
 ```sh
-# test all cases
-python3 -m pytest -s -vv tests
-# or test a single case specifically
-python3 -m pytest -s -vv tests/unit/test_camm_parser.py::test_build_and_parse
+# Test all cases
+pytest -s -vv tests
+# Or test a single case specifically
+pytest -s -vv tests/unit/test_camm_parser.py::test_build_and_parse
 ```
 
-Run linting:
+## Code Quality
+
+Run code formatting and linting:
 
 ```sh
-# format code
-black mapillary_tools tests
-# sort imports
+# Format code with ruff
+ruff format mapillary_tools tests
+
+# Lint code with ruff
+ruff check mapillary_tools tests
+
+# Sort imports with usort
 usort format mapillary_tools tests
+
+# Type checking with mypy
+mypy mapillary_tools
 ```
 
 ## Release and Build

mapillary_tools-0.14.0a2/mapillary_tools.egg-info/PKG-INFO → mapillary_tools-0.14.0b1/README.md

@@ -1,33 +1,3 @@
-Metadata-Version: 2.4
-Name: mapillary_tools
-Version: 0.14.0a2
-Summary: Mapillary Image/Video Import Pipeline
-Home-page: https://github.com/mapillary/mapillary_tools
-Author: Mapillary
-License: BSD
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: appdirs<2.0.0,>=1.4.4
-Requires-Dist: construct<3.0.0,>=2.10.0
-Requires-Dist: exifread==2.3.2
-Requires-Dist: piexif==1.1.3
-Requires-Dist: gpxpy<1.6.0,>=1.5.0
-Requires-Dist: pynmea2<2.0.0,>=1.12.0
-Requires-Dist: requests[socks]<3.0.0,>=2.20.0
-Requires-Dist: tqdm<5.0,>=4.0
-Requires-Dist: typing_extensions
-Requires-Dist: jsonschema~=4.17.3
-Dynamic: author
-Dynamic: description
-Dynamic: description-content-type
-Dynamic: home-page
-Dynamic: license
-Dynamic: license-file
-Dynamic: requires-dist
-Dynamic: requires-python
-Dynamic: summary
-
 <p align="center">
   <a href="https://github.com/mapillary/mapillary_tools/">
     <img src="https://raw.githubusercontent.com/mapillary/mapillary_tools/main/docs/images/logo.png">
@@ -58,18 +28,40 @@ mapillary_tools --help
 <!--ts-->
 
 - [Supported File Formats](#supported-file-formats)
+  - [Image Formats](#image-formats)
+  - [Video Formats](#video-formats)
 - [Installation](#installation)
+  - [Standalone Executable](#standalone-executable)
+  - [Installing via pip](#installing-via-pip)
+    - [Installing on Android Devices](#installing-on-android-devices)
 - [Usage](#usage)
   - [Process and Upload](#process-and-upload)
   - [Process](#process)
   - [Upload](#upload)
 - [Advanced Usage](#advanced-usage)
   - [Local Video Processing](#local-video-processing)
+    - [Install FFmpeg](#install-ffmpeg)
+    - [Video Processing](#video-processing)
   - [Geotagging with GPX](#geotagging-with-gpx)
+  - [New video geotagging features (experimental)](#new-video-geotagging-features-experimental)
+    - [Usage](#usage-1)
+    - [Examples](#examples)
+      - [Generic supported videos](#generic-supported-videos)
+      - [External GPX](#external-gpx)
+      - [Insta360 stitched videos](#insta360-stitched-videos)
+    - [Limitations of `--video_geotag_source`](#limitations-of---video_geotag_source)
   - [Authenticate](#authenticate)
+    - [Examples](#examples-1)
   - [Image Description](#image-description)
   - [Zip Images](#zip-images)
 - [Development](#development)
+  - [Setup](#setup)
+    - [Option 1: Using uv (Recommended)](#option-1-using-uv-recommended)
+    - [Option 2: Using pip with virtual environment](#option-2-using-pip-with-virtual-environment)
+  - [Running the code](#running-the-code)
+  - [Tests](#tests)
+  - [Code Quality](#code-quality)
+  - [Release and Build](#release-and-build)
 
 <!--te-->
 
@@ -495,29 +487,49 @@ git clone git@github.com:mapillary/mapillary_tools.git
 cd mapillary_tools
 ```
 
-Set up the virtual environment. It is optional but recommended:
+### Option 1: Using uv (Recommended)
+
+Use [uv](https://docs.astral.sh/uv/) - a fast Python package manager.
+
+Install the project in development mode with all dependencies:
 
 ```sh
-pip install pipenv
+# Install the project and development dependencies
+uv sync --group dev
+
+# Activate the virtual environment
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 ```
 
-Install dependencies:
+### Option 2: Using pip with virtual environment
+
+Set up a virtual environment (recommended):
 
 ```sh
-pipenv install -r requirements.txt
-pipenv install -r requirements-dev.txt
+python -m venv .venv
+source .venv/bin/activate  # On Windows: .venv\Scripts\activate
 ```
 
-Enter the virtualenv shell:
+Install the project in development mode:
 
 ```sh
-pipenv shell
+# Install the project and all dependencies in editable mode
+pip install -e .
+
+# Install development dependencies
+pip install --group dev
 ```
 
+## Running the code
+
 Run the code from the repository:
 
 ```sh
-python3 -m mapillary_tools.commands --version
+# If you have mapillary_tools installed in editable mode
+mapillary_tools --version
+
+# Alternatively
+python -m mapillary_tools.commands --version
 ```
 
 ## Tests
@@ -525,19 +537,28 @@ python3 -m mapillary_tools.commands --version
 Run tests:
 
 ```sh
-# test all cases
-python3 -m pytest -s -vv tests
-# or test a single case specifically
-python3 -m pytest -s -vv tests/unit/test_camm_parser.py::test_build_and_parse
+# Test all cases
+pytest -s -vv tests
+# Or test a single case specifically
+pytest -s -vv tests/unit/test_camm_parser.py::test_build_and_parse
 ```
 
-Run linting:
+## Code Quality
+
+Run code formatting and linting:
 
 ```sh
-# format code
-black mapillary_tools tests
-# sort imports
+# Format code with ruff
+ruff format mapillary_tools tests
+
+# Lint code with ruff
+ruff check mapillary_tools tests
+
+# Sort imports with usort
 usort format mapillary_tools tests
+
+# Type checking with mypy
+mypy mapillary_tools
 ```
 
 ## Release and Build

mapillary_tools-0.14.0b1/mapillary_tools/__init__.py

@@ -0,0 +1 @@
+VERSION = "0.14.0b1"

{mapillary_tools-0.14.0a2 → mapillary_tools-0.14.0b1}/mapillary_tools/api_v4.py

@@ -25,6 +25,7 @@ class ClusterFileType(enum.Enum):
     ZIP = "zip"
     BLACKVUE = "mly_blackvue_video"
     CAMM = "mly_camm_video"
+    MLY_BUNDLE_MANIFEST = "mly_bundle_manifest"
 
 
 class HTTPSystemCertsAdapter(HTTPAdapter):

{mapillary_tools-0.14.0a2 → mapillary_tools-0.14.0b1}/mapillary_tools/authenticate.py

@@ -11,7 +11,7 @@ import jsonschema
 
 import requests
 
-from . import api_v4, config, constants, exceptions, types
+from . import api_v4, config, constants, exceptions
 
 
 LOG = logging.getLogger(__name__)
@@ -64,7 +64,7 @@ def authenticate(
             LOG.info('Creating new profile: "%s"', profile_name)
 
         if jwt:
-            user_items: types.UserItem = {"user_upload_token": jwt}
+            user_items: config.UserItem = {"user_upload_token": jwt}
             user_items = _verify_user_auth(_validate_profile(user_items))
         else:
             user_items = _prompt_login(
@@ -89,7 +89,7 @@ def authenticate(
 def fetch_user_items(
     user_name: str | None = None,
     organization_key: str | None = None,
-) -> types.UserItem:
+) -> config.UserItem:
     """
     Read user information from the config file,
     or prompt the user to authenticate if the specified profile does not exist
@@ -155,9 +155,9 @@ def _prompt(message: str) -> str:
     return input()
 
 
-def _validate_profile(user_items: types.UserItem) -> types.UserItem:
+def _validate_profile(user_items: config.UserItem) -> config.UserItem:
     try:
-        jsonschema.validate(user_items, types.UserItemSchema)
+        jsonschema.validate(user_items, config.UserItemSchema)
     except jsonschema.ValidationError as ex:
         raise exceptions.MapillaryBadParameterError(
             f"Invalid profile format: {ex.message}"
@@ -165,7 +165,7 @@ def _validate_profile(user_items: types.UserItem) -> types.UserItem:
     return user_items
 
 
-def _verify_user_auth(user_items: types.UserItem) -> types.UserItem:
+def _verify_user_auth(user_items: config.UserItem) -> config.UserItem:
     """
     Verify that the user access token is valid
     """
@@ -205,7 +205,7 @@ def _validate_profile_name(profile_name: str):
         )
 
 
-def _list_all_profiles(profiles: dict[str, types.UserItem]) -> None:
+def _list_all_profiles(profiles: dict[str, config.UserItem]) -> None:
     _echo("Existing Mapillary profiles:")
 
     # Header
@@ -256,7 +256,7 @@ def _is_login_retryable(ex: requests.HTTPError) -> bool:
 def _prompt_login(
     user_email: str | None = None,
     user_password: str | None = None,
-) -> types.UserItem:
+) -> config.UserItem:
     _enabled = _prompt_enabled()
 
     if user_email is None:
@@ -288,7 +288,7 @@ def _prompt_login(
 
     data = resp.json()
 
-    user_items: types.UserItem = {
+    user_items: config.UserItem = {
         "user_upload_token": str(data["access_token"]),
         "MAPSettingsUserKey": str(data["user_id"]),
     }

mapillary_tools-0.14.0b1/mapillary_tools/blackvue_parser.py

@@ -0,0 +1,195 @@
+from __future__ import annotations
+
+import dataclasses
+
+import json
+import logging
+import re
+import typing as T
+
+import pynmea2
+
+from . import geo
+from .mp4 import simple_mp4_parser as sparser
+
+
+LOG = logging.getLogger(__name__)
+NMEA_LINE_REGEX = re.compile(
+    rb"""
+    ^\s*
+    \[(\d+)\] # Timestamp
+    \s*
+    (\$\w{5}.*) # NMEA message
+    \s*
+    (\[\d+\])? # Strange timestamp
+    \s*$
+    """,
+    re.X,
+)
+
+
+@dataclasses.dataclass
+class BlackVueInfo:
+    # None and [] are equivalent here. Use None as default because:
+    # ValueError: mutable default <class 'list'> for field gps is not allowed: use default_factory
+    gps: list[geo.Point] | None = None
+    make: str = "BlackVue"
+    model: str = ""
+
+
+def extract_blackvue_info(fp: T.BinaryIO) -> BlackVueInfo | None:
+    try:
+        gps_data = sparser.parse_mp4_data_first(fp, [b"free", b"gps "])
+    except sparser.ParsingError:
+        gps_data = None
+
+    if gps_data is None:
+        return None
+
+    points = _parse_gps_box(gps_data)
+    points.sort(key=lambda p: p.time)
+
+    if points:
+        first_point_time = points[0].time
+        for p in points:
+            p.time = (p.time - first_point_time) / 1000
+
+    # Camera model
+    try:
+        cprt_bytes = sparser.parse_mp4_data_first(fp, [b"free", b"cprt"])
+    except sparser.ParsingError:
+        cprt_bytes = None
+        model = ""
+
+    if cprt_bytes is None:
+        model = ""
+    else:
+        model = _extract_camera_model_from_cprt(cprt_bytes)
+
+    return BlackVueInfo(model=model, gps=points)
+
+
+def extract_camera_model(fp: T.BinaryIO) -> str:
+    try:
+        cprt_bytes = sparser.parse_mp4_data_first(fp, [b"free", b"cprt"])
+    except sparser.ParsingError:
+        return ""
+
+    if cprt_bytes is None:
+        return ""
+
+    return _extract_camera_model_from_cprt(cprt_bytes)
+
+
+def _extract_camera_model_from_cprt(cprt_bytes: bytes) -> str:
+    """
+    >>> _extract_camera_model_from_cprt(b' {"model":"DR900X Plus","ver":0.918,"lang":"English","direct":1,"psn":"","temp":34,"GPS":1}')
+    'DR900X Plus'
+    >>> _extract_camera_model_from_cprt(b' Pittasoft Co., Ltd.;DR900S-1CH;1.008;English;1;D90SS1HAE00661;T69;')
+    'DR900S-1CH'
+    """
+    cprt_bytes = cprt_bytes.strip().strip(b"\x00")
+
+    try:
+        cprt_str = cprt_bytes.decode("utf8")
+    except UnicodeDecodeError:
+        return ""
+
+    try:
+        cprt_json = json.loads(cprt_str)
+    except json.JSONDecodeError:
+        cprt_json = None
+
+    if cprt_json is not None:
+        return str(cprt_json.get("model", "")).strip()
+
+    fields = cprt_str.split(";")
+    if 2 <= len(fields):
+        model = fields[1]
+        if model:
+            return model.strip()
+        else:
+            return ""
+    else:
+        return ""
+
+
+def _parse_gps_box(gps_data: bytes) -> list[geo.Point]:
+    """
+    >>> list(_parse_gps_box(b"[1623057074211]$GPGGA,202530.00,5109.0262,N,11401.8407,W,5,40,0.5,1097.36,M,-17.00,M,18,TSTR*61"))
+    [Point(time=1623057074211, lat=51.150436666666664, lon=-114.03067833333333, alt=1097.36, angle=None)]
+
+    >>> list(_parse_gps_box(b"[1629874404069]$GNGGA,175322.00,3244.53126,N,11710.97811,W,1,12,0.84,17.4,M,-34.0,M,,*45"))
+    [Point(time=1629874404069, lat=32.742187666666666, lon=-117.1829685, alt=17.4, angle=None)]
+
+    >>> list(_parse_gps_box(b"[1629874404069]$GNGLL,4404.14012,N,12118.85993,W,001037.00,A,A*67"))
+    [Point(time=1629874404069, lat=44.069002, lon=-121.31433216666667, alt=None, angle=None)]
+
+    >>> list(_parse_gps_box(b"[1629874404069]$GNRMC,001031.00,A,4404.13993,N,12118.86023,W,0.146,,100117,,,A*7B"))
+    [Point(time=1629874404069, lat=44.06899883333333, lon=-121.31433716666666, alt=None, angle=None)]
+
+    >>> list(_parse_gps_box(b"[1623057074211]$GPVTG,,T,,M,0.078,N,0.144,K,D*28[1623057075215]"))
+    []
+    """
+    points_by_sentence_type: dict[str, list[geo.Point]] = {}
+
+    for line_bytes in gps_data.splitlines():
+        match = NMEA_LINE_REGEX.match(line_bytes)
+        if match is None:
+            continue
+        nmea_line_bytes = match.group(2)
+
+        if not nmea_line_bytes:
+            continue
+
+        try:
+            nmea_line = nmea_line_bytes.decode("utf8")
+        except UnicodeDecodeError:
+            continue
+
+        if not nmea_line:
+            continue
+
+        try:
+            message = pynmea2.parse(nmea_line)
+        except pynmea2.nmea.ParseError:
+            continue
+
+        epoch_ms = int(match.group(1))
+
+        # https://tavotech.com/gps-nmea-sentence-structure/
+        if message.sentence_type in ["GGA"]:
+            if not message.is_valid:
+                continue
+            point = geo.Point(
+                time=epoch_ms,
+                lat=message.latitude,
+                lon=message.longitude,
+                alt=message.altitude,
+                angle=None,
+            )
+            points_by_sentence_type.setdefault(message.sentence_type, []).append(point)
+
+        elif message.sentence_type in ["RMC", "GLL"]:
+            if not message.is_valid:
+                continue
+            point = geo.Point(
+                time=epoch_ms,
+                lat=message.latitude,
+                lon=message.longitude,
+                alt=None,
+                angle=None,
+            )
+            points_by_sentence_type.setdefault(message.sentence_type, []).append(point)
+
+    # This is the extraction order in exiftool
+    if "RMC" in points_by_sentence_type:
+        return points_by_sentence_type["RMC"]
+
+    if "GGA" in points_by_sentence_type:
+        return points_by_sentence_type["GGA"]
+
+    if "GLL" in points_by_sentence_type:
+        return points_by_sentence_type["GLL"]
+
+    return []