mapillary-tools 0.0.1__tar.gz → 0.0.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1 @@
1
+ include README.rst
@@ -0,0 +1,108 @@
1
+ Metadata-Version: 2.1
2
+ Name: mapillary_tools
3
+ Version: 0.0.2
4
+ Summary: Mapillary Commandline Image Uploader
5
+ Home-page: https://github.com/MarcelloPerathoner/mapillary-tools
6
+ Author: Marcello Perathoner
7
+ Author-email: marcello@perathoner.de
8
+ License: GPL3
9
+ Description: ======================================
10
+ Mapillary Commandline Image Uploader
11
+ ======================================
12
+
13
+
14
+ Commandline tools to upload discrete images to Mapillary.
15
+ Does not support videos.
16
+
17
+
18
+ Rewrite
19
+ =======
20
+
21
+ This is a rewrite of a portion of the official Mapillary tools found at:
22
+ https://github.com/mapillary/mapillary_tools
23
+
24
+ The official Mapillary tools are broken because Python 2 is dead. Major
25
+ distributions have already removed Python 2 from their standard installation.
26
+ An official Python 3 port has been requested in May 2018 but has not
27
+ materialized so far.
28
+
29
+ A quick look at the official code shows that it is not worth porting anyway.
30
+ That code was obviously written by lots of different people with varying
31
+ understanding of Python and then duct-taped together to make it look like one
32
+ application. It also pulls in way too many dependencies, like a full image
33
+ manipulation library, 2 different Exif libraries (one of them custom patched)
34
+ and an advanced keystore.
35
+
36
+ The rewritten code uses Python 3 and only a few external libraries.
37
+
38
+ This code does not change your image files, neither does it copy them. All data
39
+ is put into small sidecar files. This keeps your disk lean and your backups
40
+ small.
41
+
42
+
43
+ Usage
44
+ =====
45
+
46
+ 1. Authorize: This will prompt for your Mapillary password:
47
+
48
+ .. code-block:: shell
49
+
50
+ mapillary_auth.py --user_name <your_username> --user_email <your_email>
51
+
52
+ This step is needed only once. Your credentials are now stored in the file
53
+ :code:`~/.config/mapillary/configs/<CLIENT_ID>`. Keep this file secret.
54
+ Mapillary credentials do not expire.
55
+
56
+
57
+ 2. Pre-process the images:
58
+
59
+ .. code-block:: shell
60
+
61
+ mapillary_process.py ~/Pictures/Mapillary/*.jpg
62
+
63
+ This step extracts GPS data from your images and stores it in sidecar files.
64
+
65
+
66
+ 3. Upload the images:
67
+
68
+ .. code-block:: shell
69
+
70
+ mapillary_upload.py ~/Pictures/Mapillary/*.jpg
71
+
72
+ The script remembers which images were successfully uploaded. If you run the
73
+ upload script on the same images again, the ones already uploaded will not be
74
+ uploaded again.
75
+
76
+ 4. Cleanup:
77
+
78
+ .. code-block:: shell
79
+
80
+ mapillary_process.py --clean ~/Pictures/Mapillary/*.jpg
81
+
82
+ Delete all sidecar files. Caution: This will erase all memory about which
83
+ files where already uploaded.
84
+
85
+ Run the scripts with '-h' to see more options.
86
+
87
+
88
+ GPS Tracks
89
+ ==========
90
+
91
+ This code has no support for GPS tracks.
92
+
93
+ To use GPS information stored in a GPS track use exiftool to insert GPS
94
+ information into your image files, then proces them.
95
+
96
+ https://exiftool.org/geotag.html
97
+
98
+ Keywords: mapillary commandline console image upload
99
+ Platform: UNKNOWN
100
+ Classifier: Development Status :: 3 - Alpha
101
+ Classifier: Environment :: Console
102
+ Classifier: Intended Audience :: End Users/Desktop
103
+ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
104
+ Classifier: Operating System :: POSIX :: Linux
105
+ Classifier: Programming Language :: Python :: 3 :: Only
106
+ Classifier: Topic :: Utilities
107
+ Requires-Python: >=3.5.0
108
+ Description-Content-Type: text/x-rst
@@ -1,20 +1,22 @@
1
- ==============================
2
- Mapillary Tools for Python 3
3
- ==============================
1
+ ======================================
2
+ Mapillary Commandline Image Uploader
3
+ ======================================
4
4
 
5
5
 
6
- Tool to upload discrete images to Mapillary. Does not support videos.
6
+ Commandline tools to upload discrete images to Mapillary.
7
+ Does not support videos.
7
8
 
8
9
 
9
10
  Rewrite
10
11
  =======
11
12
 
12
- This is a complete rewrite of a portion of the official Mapillary tools found at:
13
+ This is a rewrite of a portion of the official Mapillary tools found at:
13
14
  https://github.com/mapillary/mapillary_tools
14
15
 
15
16
  The official Mapillary tools are broken because Python 2 is dead. Major
16
- distributions have already removed Python 2 from their standard installation. A
17
- Python 3 port has been requested a long time ago but nothing happened.
17
+ distributions have already removed Python 2 from their standard installation.
18
+ An official Python 3 port has been requested in May 2018 but has not
19
+ materialized so far.
18
20
 
19
21
  A quick look at the official code shows that it is not worth porting anyway.
20
22
  That code was obviously written by lots of different people with varying
@@ -40,7 +42,7 @@ Usage
40
42
  mapillary_auth.py --user_name <your_username> --user_email <your_email>
41
43
 
42
44
  This step is needed only once. Your credentials are now stored in the file
43
- :file:`~/.config/mapillary/configs/<CLIENT_ID>`. Keep this file secret.
45
+ :code:`~/.config/mapillary/configs/<CLIENT_ID>`. Keep this file secret.
44
46
  Mapillary credentials do not expire.
45
47
 
46
48
 
@@ -63,6 +65,15 @@ Usage
63
65
  upload script on the same images again, the ones already uploaded will not be
64
66
  uploaded again.
65
67
 
68
+ 4. Cleanup:
69
+
70
+ .. code-block:: shell
71
+
72
+ mapillary_process.py --clean ~/Pictures/Mapillary/*.jpg
73
+
74
+ Delete all sidecar files. Caution: This will erase all memory about which
75
+ files where already uploaded.
76
+
66
77
  Run the scripts with '-h' to see more options.
67
78
 
68
79
 
@@ -1,4 +1,4 @@
1
- #!/usr/bin/python3
1
+ #!/usr/bin/env python
2
2
 
3
3
  '''Get Mapillary authentication tokens and store them.
4
4
 
@@ -10,24 +10,17 @@ only once.
10
10
 
11
11
  '''
12
12
 
13
- import argparse
14
13
  import getpass
14
+ import logging
15
15
 
16
- from mapillary_tools import get_auth_tokens, write_config_file
16
+ import mapillary_tools as mt
17
17
 
18
18
 
19
19
  def build_parser ():
20
20
  ''' Build the commandline parser. '''
21
21
 
22
- parser = argparse.ArgumentParser (
23
- description = __doc__,
24
- formatter_class = argparse.RawDescriptionHelpFormatter # don't wrap my description
25
- )
22
+ parser = mt.build_parser (__doc__)
26
23
 
27
- parser.add_argument (
28
- '-v', '--verbose', dest='verbose', action='count',
29
- help='increase output verbosity', default=0
30
- )
31
24
  parser.add_argument (
32
25
  '-u', '--user_name', required = True,
33
26
  help='your username on Mapillary'
@@ -36,12 +29,18 @@ def build_parser ():
36
29
  '-e', '--user_email', required = True,
37
30
  help='your email on Mapillary'
38
31
  )
32
+
39
33
  return parser
40
34
 
41
35
 
42
36
  if __name__ == '__main__':
43
37
  args = build_parser ().parse_args ()
38
+ mt.init_logging (args.verbose)
44
39
 
45
40
  password = getpass.getpass ('Please enter your Mapillary user password : ')
46
41
 
47
- write_config_file (get_auth_tokens (args.user_name, args.user_email, password))
42
+ try:
43
+ mt.write_config_file (mt.get_auth_tokens (args.user_name, args.user_email, password))
44
+
45
+ except mt.MapillaryError as e:
46
+ logging.exception (e)
@@ -0,0 +1,75 @@
1
+ #!/usr/bin/env python
2
+
3
+ '''Extract EXIF info from discrete image files.
4
+
5
+ Extract Latitude, Longitude, Altitude, DateTime and Heading from the image files
6
+ and put the information into sidecar files (same filename with an added extension
7
+ of .mapillary).
8
+
9
+ Process images from a directory:
10
+
11
+ mapillary_process.py ~/Pictures/Mapillary/*.jpg
12
+
13
+ Process images from a list of images (one complete filepath per line):
14
+
15
+ mapillary_process.py @picture_list.txt
16
+ find ~/Pictures/Mapillary -name "*.jpg" | mapillary_process.py @/dev/stdin
17
+
18
+ '''
19
+
20
+ import logging
21
+
22
+ import mapillary_tools as mt
23
+
24
+
25
+ def build_parser ():
26
+ ''' Build the commandline parser. '''
27
+
28
+ parser = mt.build_parser (__doc__)
29
+
30
+ parser.add_argument (
31
+ 'images', metavar='FILENAME', type=str, nargs='+',
32
+ help='the images to process'
33
+ )
34
+
35
+ parser.add_argument (
36
+ '--clean', action='store_true',
37
+ help='remove sidecar files for selected images'
38
+ )
39
+ return parser
40
+
41
+
42
+ def main ():
43
+ ''' The main function. '''
44
+
45
+ args = build_parser ().parse_args ()
46
+ mt.init_logging (args.verbose)
47
+
48
+ if args.clean:
49
+ for image in args.images:
50
+ mt.delete_sidecar_file (image)
51
+ logging.info (image)
52
+ return
53
+
54
+ for image in args.images:
55
+ logging.info (image)
56
+
57
+ geotags = mt.read_sidecar_file (image)
58
+ mt.set_uuid (geotags)
59
+
60
+ try:
61
+ geotags.update (mt.get_image_geotags (image))
62
+ mt.write_sidecar_file (image, geotags)
63
+
64
+ except mt.MapillaryError as e:
65
+ logging.exception (e)
66
+
67
+ if 'MAPCaptureTime' not in geotags:
68
+ logging.warning ('%s has no timestamp', image)
69
+
70
+ if 'MAPLatitude' not in geotags:
71
+ logging.warning ('%s has no GPS data', image)
72
+
73
+
74
+ if __name__ == '__main__':
75
+ main ()
@@ -1,4 +1,4 @@
1
- #!/usr/bin/python3
1
+ #!/usr/bin/env python
2
2
 
3
3
  ''' Upload images to Mapillary.
4
4
 
@@ -13,22 +13,19 @@ Upload images from a directory:
13
13
  Upload images from a list of images (one complete filepath per line):
14
14
 
15
15
  mapillary_upload.py @picture_list.txt
16
- list_builder | mapillary_upload.py @/dev/stdin
16
+ find ~/Pictures/Mapillary/ -name "*.jpg" | mapillary_upload.py @/dev/stdin
17
17
 
18
18
  '''
19
19
 
20
- import argparse
21
20
  import itertools
22
- import json
21
+ import logging
23
22
  from multiprocessing.dummy import Pool
24
23
  import operator
25
- import sys
26
24
 
27
- import requests
28
25
  from tqdm import tqdm
29
26
 
30
- from mapillary_tools import read_config_file, read_sidecar_file, \
31
- cut_sequences, managed_session, upload_image
27
+ import mapillary_tools as mt
28
+
32
29
 
33
30
  SEQUENCE_THREADS = 2 # upload this many sequences in parallel
34
31
  UPLOAD_THREADS = 2 # upload this many images in parallel per sequence
@@ -37,24 +34,17 @@ DEFAULT_MAX_TIME = 5 * 60.0 # in seconds. max time between images in same sequen
37
34
  DEFAULT_MAX_DIST = 100.0 # in meters. max distance between images in same seq.
38
35
  DEFAULT_MAX_DOP = 20.0 # discard images with GPS DOP greater than this
39
36
 
37
+
40
38
  def build_parser ():
41
39
  ''' Build the commandline parser. '''
42
40
 
43
- parser = argparse.ArgumentParser (
44
- description = __doc__,
45
- formatter_class = argparse.RawDescriptionHelpFormatter, # don't wrap my description
46
- fromfile_prefix_chars = '@'
47
- )
41
+ parser = mt.build_parser (__doc__)
48
42
 
49
43
  parser.add_argument (
50
44
  'images', metavar='FILENAME', type=str, nargs='+',
51
45
  help='the images to upload'
52
46
  )
53
47
 
54
- parser.add_argument (
55
- '-v', '--verbose', dest='verbose', action='count',
56
- help='increase output verbosity', default=0
57
- )
58
48
  parser.add_argument (
59
49
  '-n', '--dry-run', dest='dry_run', action='store_true',
60
50
  help='dry run: do not upload any images',
@@ -83,95 +73,93 @@ def build_list (args):
83
73
 
84
74
  for filename in args.images:
85
75
  try:
86
- gt = read_sidecar_file (filename)
87
- except (OSError, json.decoder.JSONDecodeError) as e:
76
+ gt = mt.read_sidecar_file (filename)
77
+ except mt.MapillaryError as e:
88
78
  errors += 1
89
- sys.stderr.write (str (e))
79
+ logging.exception (e)
90
80
  continue
91
81
 
92
82
  if gt.get ('status_code', 0) == 204:
93
83
  discarded += 1
94
- if args.verbose:
95
- sys.stderr.write ("Image %s was already uploaded\n" % filename)
84
+ logging.warning ('%s was already uploaded', filename)
96
85
  continue
97
86
 
98
87
  if 'MAPLatitude' not in gt or 'MAPCaptureTime' not in gt:
99
88
  discarded += 1
100
- if args.verbose:
101
- sys.stderr.write ("No GPS data for file %s\n" % filename)
89
+ logging.warning ('%s has no GPS data', filename)
102
90
  continue
103
91
 
104
92
  if gt.get ('MAPDOP', 1.0) > args.dop_max:
105
93
  discarded += 1
106
- if args.verbose:
107
- sys.stderr.write ("Max GPS DOP exceeded in file %s\n" % filename)
94
+ logging.warning ('%s exceeds max GPS DOP', filename)
108
95
  continue
109
96
 
110
97
  geotags.append (gt)
111
98
 
112
- if args.verbose:
113
- print ("%d images were discarded and %d had errors"
114
- % (discarded, errors))
99
+ logging.info ('%d images were discarded and %d had errors', discarded, errors)
115
100
 
116
101
  return geotags
117
102
 
118
103
 
119
- def upload_image_file (args, session, geotags):
104
+ def upload_image (args, session, geotags):
120
105
  ''' Upload one image. '''
121
106
 
122
- try:
123
- upload_image (session, geotags, args.dry_run)
124
- except (requests.exceptions.HTTPError, ValueError) as e:
125
- sys.stderr.write ("Error uploading image: %s\n" % geotags['filename'])
126
- sys.stderr.write (str (e) + '\n')
107
+ geotags = mt.upload_image (session, geotags, args.dry_run)
127
108
  args.pbar.update ()
128
109
 
110
+ return geotags
129
111
 
130
112
 
131
- def upload_sequence (args, sequence):
113
+ def upload_sequence (args, sequence_id, sequence):
132
114
  ''' Upload a sequence of images. '''
133
115
 
134
- upload_token = args.config[args.config.sections()[0]]['user_upload_token']
135
- with managed_session (upload_token, args.dry_run) as session:
136
- with Pool (UPLOAD_THREADS) as pool:
137
- pool.starmap (upload_image_file, zip (
138
- itertools.repeat (args),
139
- itertools.repeat (session),
140
- sequence,
141
- ))
116
+ try:
117
+ upload_token = args.config[args.config.sections()[0]]['user_upload_token']
118
+ with mt.managed_session (upload_token, args.dry_run) as session:
119
+ with Pool (UPLOAD_THREADS) as pool:
120
+ sequence = pool.starmap (upload_image, zip (
121
+ itertools.repeat (args),
122
+ itertools.repeat (session),
123
+ sequence,
124
+ ))
125
+
126
+ # write status into sidecars only if the sequence was successful
127
+ if not args.dry_run:
128
+ for geotags in sequence:
129
+ del geotags['sequence_id']
130
+ mt.write_sidecar_file (geotags['filename'], geotags)
131
+
132
+ except mt.SequenceUploadError as e:
133
+ logging.exception (e)
134
+
135
+ return sequence
142
136
 
143
137
 
144
138
  def main ():
145
139
  ''' Main function. '''
146
140
 
147
141
  args = build_parser ().parse_args ()
148
-
149
- args.config = read_config_file ()
142
+ mt.init_logging (args.verbose)
143
+ args.config = mt.read_config_file ()
150
144
 
151
145
  geotags = build_list (args)
152
146
  if len (geotags) == 0:
153
- if args.verbose:
154
- print ("No images to upload.")
147
+ logging.info ('No images to upload.')
155
148
  return
156
149
 
157
- geotags = cut_sequences (geotags, args.t_max, args.d_max)
158
- sequences = [list (v) for k, v in itertools.groupby (
150
+ geotags = mt.cut_sequences (geotags, args.t_max, args.d_max)
151
+ parameters = [(args, k, list (v)) for k, v in itertools.groupby (
159
152
  geotags, operator.itemgetter ('sequence_id'))]
160
153
 
161
- if args.verbose:
162
- print ("Sequenced %d images in %d sequences" % (len (geotags), len (sequences)))
154
+ logging.info ('Sequenced %d images in %d sequences', len (geotags), len (parameters))
163
155
 
164
156
  with tqdm (total = len (geotags), desc = 'Dry run' if args.dry_run else 'Uploading',
165
- unit = 'image', disable = args.verbose == 0) as args.pbar:
157
+ unit = 'image', smoothing = 0, disable = args.verbose == 0) as args.pbar:
166
158
 
167
159
  with Pool (SEQUENCE_THREADS) as seq_pool:
168
- seq_pool.starmap (upload_sequence, zip (
169
- itertools.repeat (args),
170
- sequences,
171
- ))
160
+ seq_pool.starmap (upload_sequence, parameters)
172
161
 
173
- if args.verbose:
174
- print ("Done.")
162
+ logging.info ('Done.')
175
163
 
176
164
 
177
165
  if __name__ == '__main__':
@@ -1,20 +1,24 @@
1
- ''' Global Configuration '''
1
+ ''' Package '''
2
2
 
3
+ import argparse
3
4
  import configparser
4
5
  from contextlib import contextmanager
5
6
  import datetime
6
7
  import getpass
7
8
  import io
8
9
  import json
10
+ import logging
9
11
  import operator
10
12
  import os
11
13
  import re
14
+ from urllib3.util.retry import Retry
12
15
  import uuid
13
16
 
14
17
  from geopy.distance import geodesic
15
18
  import piexif
16
19
  import requests
17
20
 
21
+
18
22
  CLIENT_ID = os.getenv (
19
23
  'MAPILLARY_CLIENT_ID',
20
24
  'MkJKbDA0bnZuZlcxeTJHTmFqN3g1dzo1YTM0NjRkM2EyZGU5MzBh'
@@ -32,32 +36,88 @@ LOGIN_ENDPOINT = API_ENDPOINT + '/v2/ua/login'
32
36
  UPLOAD_TOKENS_ENDPOINT = API_ENDPOINT + '/v3/users/{user_key}/upload_tokens'
33
37
  UPLOAD_ENDPOINT = API_ENDPOINT + '/v3/me/uploads'
34
38
 
39
+ API_TIMEOUT = 60.0
40
+ API_RETRIES = 5
41
+
42
+ # make requests retry on failure
43
+
44
+ api = requests.Session ()
45
+ api.mount ('https://', requests.adapters.HTTPAdapter (max_retries = Retry (
46
+ total = API_RETRIES,
47
+ backoff_factor = 1,
48
+ method_whitelist = ['GET', 'POST', 'PUT', 'DELETE'],
49
+ status_forcelist = [429, 500, 502, 503, 504],
50
+ )))
51
+
52
+ # api = requests
53
+
54
+ args = argparse.Namespace ()
55
+
56
+ class MapillaryError (Exception):
57
+ '''A more general error.'''
58
+
59
+
60
+ class ImageUploadError (MapillaryError):
61
+ '''Error while uploading an image.'''
62
+
63
+
64
+ class SequenceUploadError (MapillaryError):
65
+ '''Error while uploading a sequence.'''
66
+
67
+
68
+ def build_parser (description):
69
+ ''' Build the commandline parser. '''
70
+
71
+ parser = argparse.ArgumentParser (
72
+ description = description,
73
+ formatter_class = argparse.RawDescriptionHelpFormatter, # don't wrap my description
74
+ fromfile_prefix_chars = '@'
75
+ )
76
+
77
+ parser.add_argument (
78
+ '-v', '--verbose', dest='verbose', action='count',
79
+ help='increase output verbosity', default=0
80
+ )
81
+ return parser
82
+
83
+
84
+ def init_logging (verbose):
85
+ ''' Init the logger. '''
86
+
87
+ level = logging.WARNING
88
+ if verbose == 1:
89
+ level = logging.INFO
90
+ if verbose > 1:
91
+ level = logging.DEBUG
92
+ logging.basicConfig (level = level, format = '%(message)s')
93
+
35
94
 
36
95
  def get_auth_tokens (user_name, user_email, user_password):
37
96
  ''' Get the authorization tokens from the Mapillary API. '''
38
97
 
39
98
  try:
40
99
  # get user key
41
- message = 'Error getting user key: No user by name %s.' % user_name
100
+ message = 'Error getting user key: No user by name {user}.'.format (user = user_name)
42
101
  params = { 'client_id' : CLIENT_ID, 'usernames' : user_name }
43
- r = requests.get (USERS_ENDPOINT, params = params)
102
+ r = api.get (USERS_ENDPOINT, params = params, timeout = API_TIMEOUT)
44
103
  r.raise_for_status ()
45
104
  user_key = r.json ()[0]['key']
46
105
 
47
106
  # get upload token
48
- message = 'Error getting upload token: Wrong email %s or wrong password.' % user_email
107
+ message = 'Error getting upload token: Wrong email {email} or wrong password.'.format (
108
+ email = user_email)
49
109
  params = { 'client_id' : CLIENT_ID }
50
110
  data = { 'email' : user_email, 'password' : user_password }
51
- r = requests.post (LOGIN_ENDPOINT, params = params, data = data)
111
+ r = api.post (LOGIN_ENDPOINT, params = params, data = data, timeout = API_TIMEOUT)
52
112
  r.raise_for_status ()
53
113
  upload_token = r.json ()['token']
54
114
 
55
115
  # get user hashes
56
116
  message = 'Error getting user hashes.'
57
117
  params = { 'client_id' : CLIENT_ID }
58
- headers = { 'Authorization' : 'Bearer %s' % upload_token }
59
- r = requests.get (UPLOAD_TOKENS_ENDPOINT.format (user_key = user_key),
60
- params = params, headers = headers)
118
+ headers = { 'Authorization' : 'Bearer ' + upload_token }
119
+ r = api.get (UPLOAD_TOKENS_ENDPOINT.format (user_key = user_key),
120
+ params = params, headers = headers, timeout = API_TIMEOUT)
61
121
  r.raise_for_status ()
62
122
  j = r.json ()
63
123
 
@@ -70,15 +130,15 @@ def get_auth_tokens (user_name, user_email, user_password):
70
130
  'aws_access_key_id' : j.get ('aws_access_key_id', ''),
71
131
  }
72
132
 
73
- except (IndexError, KeyError, requests.exceptions.HTTPError):
74
- print (message)
75
- raise
133
+ except (IndexError, KeyError, requests.exceptions.RequestException) as e:
134
+ raise MapillaryError (message) from e
76
135
 
77
136
 
78
137
  def gps_ascii (value):
79
138
  ''' Decode ascii bytes to string. '''
80
139
  return value.decode ('ascii')
81
140
 
141
+
82
142
  def gps_rational (values):
83
143
  ''' Decode exif rational value to float. '''
84
144
  return float (values[0]) / float (values [1])
@@ -134,32 +194,47 @@ def get_image_geotags (filename):
134
194
  piexif.ExifIFD.SubSecTimeOriginal,
135
195
  piexif.ExifIFD.SubSecTimeDigitized]:
136
196
  if f in t:
137
- subseconds = float (b"0." + t[f])
197
+ subseconds = float (b'0.' + t[f])
138
198
  break
139
199
 
140
200
  for f in [piexif.ImageIFD.DateTime,
141
201
  piexif.ExifIFD.DateTimeOriginal,
142
202
  piexif.ExifIFD.DateTimeDigitized]:
143
203
  if f in t:
144
- dt = " ".join (re.sub ('[^0-9]', ' ', str (t[f])).split ())
204
+ dt = ' '.join (re.sub ('[^0-9]', ' ', str (t[f])).split ())
145
205
  dt = datetime.datetime.strptime (dt, '%Y %m %d %H %M %S')
146
206
  dt += datetime.timedelta (seconds = subseconds)
147
- geotags['MAPCaptureTime'] = dt.strftime ("%Y_%m_%d_%H_%M_%S_%f")
207
+ geotags['MAPCaptureTime'] = dt.strftime ('%Y_%m_%d_%H_%M_%S_%f')
148
208
  break
149
209
 
150
210
  # heading
151
211
  try:
152
212
  heading = gps_rational (t[piexif.GPSIFD.GPSImgDirection]) % 360.0
153
213
  ref = 'True' if gps_ascii (t[piexif.GPSIFD.GPSImgDirectionRef]) == 'T' else 'Magnetic'
154
- geotags["MAPCompassHeading"] = {
214
+ geotags['MAPCompassHeading'] = {
155
215
  ref + 'Heading' : heading,
156
216
  }
157
217
  except KeyError:
158
218
  pass
159
219
 
220
+ try:
221
+ # piexif is not very forgiving when asked to write wrong data types
222
+ # this is a test that lets us discover bogus images early
223
+ patch_exif_dict (exif_data, geotags)
224
+ piexif.dump (exif_data)
225
+ except ValueError as e:
226
+ raise MapillaryError (
227
+ '{filename} has invalid Exif data\n'.format (filename = filename)) from e
228
+
160
229
  return geotags
161
230
 
162
231
 
232
+ def set_uuid (geotags):
233
+ ''' Set uuid '''
234
+
235
+ geotags.setdefault ('MAPPhotoUUID', str (uuid.uuid4 ()))
236
+
237
+
163
238
  def cut_sequences (geotags, time_max, dist_max):
164
239
  ''' Build image sequences '''
165
240
 
@@ -195,6 +270,14 @@ def get_upload_filename (filename, geotags):
195
270
  ch = ch, ext = ext, **geotags)
196
271
 
197
272
 
273
+ def patch_exif_dict (exif_dict, geotags):
274
+ ''' put a json dump of the geotags into the image description field '''
275
+
276
+ filtered_geotags = { k: v for k, v in geotags.items () if k.startswith ('MAP') }
277
+ exif_dict['0th'][piexif.ImageIFD.ImageDescription] = json.dumps (filtered_geotags)
278
+ return exif_dict
279
+
280
+
198
281
  def upload_image (session, geotags, dry_run):
199
282
  ''' Upload one image to the server. '''
200
283
 
@@ -205,11 +288,8 @@ def upload_image (session, geotags, dry_run):
205
288
  data['key'] = session['key_prefix'] + upload_filename
206
289
 
207
290
  with open (filename, 'rb') as fpin:
208
-
209
- # put a json dump of the geotags into the image description field
210
291
  exif_dict = piexif.load (fpin.read ())
211
- filtered_geotags = { k: v for k, v in geotags.items () if k.startswith ('MAP') }
212
- exif_dict['0th'][piexif.ImageIFD.ImageDescription] = json.dumps (filtered_geotags)
292
+ patch_exif_dict (exif_dict, geotags)
213
293
 
214
294
  # insert exif data into memory buffer
215
295
  fpout = io.BytesIO ()
@@ -217,65 +297,99 @@ def upload_image (session, geotags, dry_run):
217
297
  try:
218
298
  piexif.insert (piexif.dump (exif_dict), fpin.read (), fpout)
219
299
  except ValueError as e:
220
- raise Exception ("Error: invalid Exif data in image: %s\n" % filename) from e
300
+ raise ImageUploadError (
301
+ '{filename} has invalid Exif data\n'.format (filename = filename)) from e
221
302
 
222
303
  # and upload the memory buffer
223
- if dry_run:
224
- geotags['status_code'] = 204
225
- else:
226
- r = requests.post (session['url'], data = data,
227
- files = { 'file' : (upload_filename, fpout) })
228
- geotags['status_code'] = r.status_code
229
- geotags['uploaded_timestamp'] = datetime.datetime.utcnow ().isoformat ()
230
- geotags['uploaded_filename'] = upload_filename
231
- geotags['uploaded_sequence_key'] = session['key']
232
- del geotags['sequence_id']
233
- write_sidecar_file (filename, geotags)
234
- r.raise_for_status ()
304
+ try:
305
+ if dry_run:
306
+ geotags['status_code'] = 204
307
+ else:
308
+ r = api.post (session['url'], data = data, timeout = API_TIMEOUT,
309
+ files = { 'file' : (upload_filename, fpout) })
310
+ geotags['status_code'] = r.status_code
311
+ geotags['uploaded_timestamp'] = datetime.datetime.utcnow ().isoformat ()
312
+ geotags['uploaded_filename'] = upload_filename
313
+ geotags['uploaded_sequence_key'] = session['key']
314
+ r.raise_for_status ()
315
+ except requests.exceptions.RequestException as e:
316
+ raise ImageUploadError (
317
+ '{filename} server error while uploading\n'.format (filename = filename)) from e
235
318
 
236
- return True
319
+ return geotags
237
320
 
238
321
 
239
322
  def create_session (upload_token):
240
323
  ''' Create a new upload session. '''
241
324
 
242
- params = { "client_id" : CLIENT_ID }
243
- headers = { "Authorization" : "Bearer " + upload_token }
244
- payload = { "type" : "images/sequence" }
325
+ try:
326
+ params = { 'client_id' : CLIENT_ID }
327
+ headers = { 'Authorization' : 'Bearer ' + upload_token }
328
+ payload = { 'type' : 'images/sequence' }
329
+
330
+ r = api.post (UPLOAD_ENDPOINT, params = params, headers = headers,
331
+ json = payload, timeout = API_TIMEOUT)
332
+ r.raise_for_status ()
333
+
334
+ except requests.exceptions.RequestException as e:
335
+ raise SequenceUploadError ('Error creating session.') from e
245
336
 
246
- r = requests.post (UPLOAD_ENDPOINT, params = params, headers = headers, json = payload)
247
- r.raise_for_status()
248
337
  return r.json ()
249
338
 
250
339
 
251
340
  def close_session (session, upload_token):
252
341
  ''' Close an upload session. '''
253
342
 
254
- url = '{endpoint}/{key}/closed'.format (endpoint = UPLOAD_ENDPOINT, key = session['key'])
255
- params = { "client_id" : CLIENT_ID }
256
- headers = { "Authorization" : "Bearer " + upload_token }
343
+ try:
344
+ url = '{endpoint}/{key}/closed'.format (endpoint = UPLOAD_ENDPOINT, key = session['key'])
345
+ params = { 'client_id' : CLIENT_ID }
346
+ headers = { 'Authorization' : 'Bearer ' + upload_token }
347
+
348
+ r = api.put (url, params = params, headers = headers, timeout = API_TIMEOUT)
349
+ r.raise_for_status ()
350
+
351
+ except requests.exceptions.RequestException as e:
352
+ raise SequenceUploadError ('Error closing session.') from e
257
353
 
258
- r = requests.put (url, params = params, headers = headers)
259
- r.raise_for_status()
354
+
355
+ def abort_session (session, upload_token):
356
+ ''' Abort an upload session. '''
357
+
358
+ try:
359
+ url = '{endpoint}/{key}'.format (endpoint = UPLOAD_ENDPOINT, key = session['key'])
360
+ params = { 'client_id' : CLIENT_ID }
361
+ headers = { 'Authorization' : 'Bearer ' + upload_token }
362
+
363
+ r = api.delete (url, params = params, headers = headers, timeout = API_TIMEOUT)
364
+ r.raise_for_status ()
365
+
366
+ except requests.exceptions.RequestException as e:
367
+ raise SequenceUploadError ('Error aborting session.') from e
260
368
 
261
369
 
262
370
  @contextmanager
263
371
  def managed_session (upload_token, dry_run):
264
372
  ''' A context manager for sessions. '''
265
373
 
266
- if dry_run:
267
- session = {
268
- 'fields' : {},
269
- 'key_prefix' : '',
270
- }
271
- else:
272
- session = create_session (upload_token)
273
-
374
+ is_open = False
274
375
  try:
376
+ if dry_run:
377
+ session = {
378
+ 'fields' : {},
379
+ 'key_prefix' : '',
380
+ }
381
+ else:
382
+ session = create_session (upload_token)
383
+ is_open = True
275
384
  yield session
276
-
385
+ except ImageUploadError as e:
386
+ if is_open:
387
+ is_open = False
388
+ abort_session (session, upload_token)
389
+ raise SequenceUploadError (
390
+ 'Error: aborting sequence because one of its images did not upload.') from e
277
391
  finally:
278
- if not dry_run:
392
+ if is_open:
279
393
  close_session (session, upload_token)
280
394
 
281
395
 
@@ -319,7 +433,7 @@ def read_sidecar_file (image_filename):
319
433
  except OSError as e:
320
434
  return { 'filename' : image_filename }
321
435
  except json.decoder.JSONDecodeError:
322
- raise Exception ("JSON syntax error in sidecar file %s\n" % sidecar) from e
436
+ raise MapillaryError ('{sidecar} JSON syntax error\n'.format (sidecar = sidecar)) from e
323
437
 
324
438
 
325
439
  def write_sidecar_file (image_filename, geotags):
@@ -327,3 +441,12 @@ def write_sidecar_file (image_filename, geotags):
327
441
 
328
442
  with open (get_sidecar_filename (image_filename), 'w') as fp:
329
443
  fp.write (json.dumps (geotags, sort_keys = True, indent = 4))
444
+
445
+
446
+ def delete_sidecar_file (image_filename):
447
+ ''' Delete the sidecar file for image. '''
448
+
449
+ try:
450
+ os.remove (get_sidecar_filename (image_filename))
451
+ except OSError:
452
+ pass
@@ -0,0 +1,108 @@
1
+ Metadata-Version: 2.1
2
+ Name: mapillary-tools
3
+ Version: 0.0.2
4
+ Summary: Mapillary Commandline Image Uploader
5
+ Home-page: https://github.com/MarcelloPerathoner/mapillary-tools
6
+ Author: Marcello Perathoner
7
+ Author-email: marcello@perathoner.de
8
+ License: GPL3
9
+ Description: ======================================
10
+ Mapillary Commandline Image Uploader
11
+ ======================================
12
+
13
+
14
+ Commandline tools to upload discrete images to Mapillary.
15
+ Does not support videos.
16
+
17
+
18
+ Rewrite
19
+ =======
20
+
21
+ This is a rewrite of a portion of the official Mapillary tools found at:
22
+ https://github.com/mapillary/mapillary_tools
23
+
24
+ The official Mapillary tools are broken because Python 2 is dead. Major
25
+ distributions have already removed Python 2 from their standard installation.
26
+ An official Python 3 port has been requested in May 2018 but has not
27
+ materialized so far.
28
+
29
+ A quick look at the official code shows that it is not worth porting anyway.
30
+ That code was obviously written by lots of different people with varying
31
+ understanding of Python and then duct-taped together to make it look like one
32
+ application. It also pulls in way too many dependencies, like a full image
33
+ manipulation library, 2 different Exif libraries (one of them custom patched)
34
+ and an advanced keystore.
35
+
36
+ The rewritten code uses Python 3 and only a few external libraries.
37
+
38
+ This code does not change your image files, neither does it copy them. All data
39
+ is put into small sidecar files. This keeps your disk lean and your backups
40
+ small.
41
+
42
+
43
+ Usage
44
+ =====
45
+
46
+ 1. Authorize: This will prompt for your Mapillary password:
47
+
48
+ .. code-block:: shell
49
+
50
+ mapillary_auth.py --user_name <your_username> --user_email <your_email>
51
+
52
+ This step is needed only once. Your credentials are now stored in the file
53
+ :code:`~/.config/mapillary/configs/<CLIENT_ID>`. Keep this file secret.
54
+ Mapillary credentials do not expire.
55
+
56
+
57
+ 2. Pre-process the images:
58
+
59
+ .. code-block:: shell
60
+
61
+ mapillary_process.py ~/Pictures/Mapillary/*.jpg
62
+
63
+ This step extracts GPS data from your images and stores it in sidecar files.
64
+
65
+
66
+ 3. Upload the images:
67
+
68
+ .. code-block:: shell
69
+
70
+ mapillary_upload.py ~/Pictures/Mapillary/*.jpg
71
+
72
+ The script remembers which images were successfully uploaded. If you run the
73
+ upload script on the same images again, the ones already uploaded will not be
74
+ uploaded again.
75
+
76
+ 4. Cleanup:
77
+
78
+ .. code-block:: shell
79
+
80
+ mapillary_process.py --clean ~/Pictures/Mapillary/*.jpg
81
+
82
+ Delete all sidecar files. Caution: This will erase all memory about which
83
+ files where already uploaded.
84
+
85
+ Run the scripts with '-h' to see more options.
86
+
87
+
88
+ GPS Tracks
89
+ ==========
90
+
91
+ This code has no support for GPS tracks.
92
+
93
+ To use GPS information stored in a GPS track use exiftool to insert GPS
94
+ information into your image files, then proces them.
95
+
96
+ https://exiftool.org/geotag.html
97
+
98
+ Keywords: mapillary commandline console image upload
99
+ Platform: UNKNOWN
100
+ Classifier: Development Status :: 3 - Alpha
101
+ Classifier: Environment :: Console
102
+ Classifier: Intended Audience :: End Users/Desktop
103
+ Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
104
+ Classifier: Operating System :: POSIX :: Linux
105
+ Classifier: Programming Language :: Python :: 3 :: Only
106
+ Classifier: Topic :: Utilities
107
+ Requires-Python: >=3.5.0
108
+ Description-Content-Type: text/x-rst
@@ -1,3 +1,4 @@
1
+ MANIFEST.in
1
2
  README.rst
2
3
  setup.py
3
4
  bin/mapillary_auth.py
@@ -2,29 +2,36 @@
2
2
 
3
3
  from setuptools import setup
4
4
 
5
+ VERSION='0.0.2'
6
+
7
+ with open ('README.rst', 'r') as fh:
8
+ long_description = fh.read ()
9
+
10
+ with open ('requirements.txt', 'r') as fh:
11
+ requirements = fh.readlines ()
12
+
5
13
  setup (
6
14
  name='mapillary_tools',
7
- version='0.0.1',
15
+ version=VERSION,
8
16
  description='Mapillary Commandline Image Uploader',
17
+ long_description=long_description,
18
+ long_description_content_type='text/x-rst',
9
19
  keywords='mapillary commandline console image upload',
10
- url='https://github.com/MarcelloPerathoner/mapillary_tools',
20
+ url='https://github.com/MarcelloPerathoner/mapillary-tools',
11
21
  author='Marcello Perathoner',
12
22
  author_email='marcello@perathoner.de',
13
23
  license='GPL3',
14
24
  python_requires='>=3.5.0',
25
+ install_requires=requirements,
15
26
  packages=['mapillary_tools'],
27
+ # See: https://pypi.org/pypi?%3Aaction=list_classifiers
16
28
  classifiers=[
17
29
  'Development Status :: 3 - Alpha',
18
- 'License :: OSI Approved :: MIT License',
19
30
  'Environment :: Console',
20
31
  'Intended Audience :: End Users/Desktop',
21
32
  'License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)',
22
33
  'Operating System :: POSIX :: Linux',
23
- 'Programming Language :: Python :: 3.5',
24
- 'Programming Language :: Python :: 3.6',
25
- 'Programming Language :: Python :: 3.7',
26
- 'Programming Language :: Python :: 3.8',
27
- 'Programming Language :: Python :: 3.9',
34
+ 'Programming Language :: Python :: 3 :: Only',
28
35
  'Topic :: Utilities',
29
36
  ],
30
37
  scripts=[
@@ -32,10 +39,4 @@ setup (
32
39
  'bin/mapillary_process.py',
33
40
  'bin/mapillary_upload.py',
34
41
  ],
35
- install_requires=[
36
- 'geopy',
37
- 'piexif',
38
- 'requests',
39
- 'tqdm',
40
- ],
41
42
  )
@@ -1,24 +0,0 @@
1
- Metadata-Version: 1.2
2
- Name: mapillary_tools
3
- Version: 0.0.1
4
- Summary: Mapillary Commandline Image Uploader
5
- Home-page: https://github.com/MarcelloPerathoner/mapillary_tools
6
- Author: Marcello Perathoner
7
- Author-email: marcello@perathoner.de
8
- License: GPL3
9
- Description: UNKNOWN
10
- Keywords: mapillary commandline console image upload
11
- Platform: UNKNOWN
12
- Classifier: Development Status :: 3 - Alpha
13
- Classifier: License :: OSI Approved :: MIT License
14
- Classifier: Environment :: Console
15
- Classifier: Intended Audience :: End Users/Desktop
16
- Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
17
- Classifier: Operating System :: POSIX :: Linux
18
- Classifier: Programming Language :: Python :: 3.5
19
- Classifier: Programming Language :: Python :: 3.6
20
- Classifier: Programming Language :: Python :: 3.7
21
- Classifier: Programming Language :: Python :: 3.8
22
- Classifier: Programming Language :: Python :: 3.9
23
- Classifier: Topic :: Utilities
24
- Requires-Python: >=3.5.0
@@ -1,62 +0,0 @@
1
- #!/usr/bin/python3
2
-
3
- '''Extract EXIF info from discrete image files.
4
-
5
- Extract Latitude, Longitude, Altitude, DateTime and Heading from the image files
6
- and put the information into sidecar files (same filename with an added extension
7
- of .mapillary).
8
-
9
- Process images from a directory:
10
-
11
- mapillary_process.py ~/Pictures/Mapillary/*.jpg
12
-
13
- Process images from a list of images (one complete filepath per line):
14
-
15
- mapillary_process.py @picture_list.txt
16
- find ~/Pictures/Mapillary -name "*.jpg" | mapillary_process.py @/dev/stdin
17
-
18
- '''
19
-
20
- import argparse
21
- import sys
22
- import uuid
23
-
24
- from mapillary_tools import get_image_geotags, read_sidecar_file, write_sidecar_file
25
-
26
-
27
- def build_parser ():
28
- ''' Build the commandline parser. '''
29
-
30
- parser = argparse.ArgumentParser (
31
- description = __doc__,
32
- formatter_class = argparse.RawDescriptionHelpFormatter, # don't wrap my description
33
- fromfile_prefix_chars = '@'
34
- )
35
-
36
- parser.add_argument (
37
- 'images', metavar='FILENAME', type=str, nargs='+',
38
- help='the images to process'
39
- )
40
-
41
- parser.add_argument (
42
- '-v', '--verbose', dest='verbose', action='count',
43
- help='increase output verbosity', default=0
44
- )
45
- return parser
46
-
47
-
48
- if __name__ == '__main__':
49
- args = build_parser ().parse_args ()
50
-
51
- for image in args.images:
52
- geotags = read_sidecar_file (image)
53
- geotags.setdefault ('MAPPhotoUUID', str (uuid.uuid4 ()))
54
- geotags.update (get_image_geotags (image))
55
-
56
- if args.verbose:
57
- print (image, geotags)
58
-
59
- write_sidecar_file (image, geotags)
60
-
61
- if 'MAPLatitude' not in geotags or 'MAPCaptureTime' not in geotags:
62
- sys.stderr.write ('No GPS data found in {filename}\n'.format (filename = image))
@@ -1,24 +0,0 @@
1
- Metadata-Version: 1.2
2
- Name: mapillary-tools
3
- Version: 0.0.1
4
- Summary: Mapillary Commandline Image Uploader
5
- Home-page: https://github.com/MarcelloPerathoner/mapillary_tools
6
- Author: Marcello Perathoner
7
- Author-email: marcello@perathoner.de
8
- License: GPL3
9
- Description: UNKNOWN
10
- Keywords: mapillary commandline console image upload
11
- Platform: UNKNOWN
12
- Classifier: Development Status :: 3 - Alpha
13
- Classifier: License :: OSI Approved :: MIT License
14
- Classifier: Environment :: Console
15
- Classifier: Intended Audience :: End Users/Desktop
16
- Classifier: License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)
17
- Classifier: Operating System :: POSIX :: Linux
18
- Classifier: Programming Language :: Python :: 3.5
19
- Classifier: Programming Language :: Python :: 3.6
20
- Classifier: Programming Language :: Python :: 3.7
21
- Classifier: Programming Language :: Python :: 3.8
22
- Classifier: Programming Language :: Python :: 3.9
23
- Classifier: Topic :: Utilities
24
- Requires-Python: >=3.5.0