sciris 3.5.1__tar.gz → 3.5.2__tar.gz

{sciris-3.5.1 → sciris-3.5.2}/CHANGELOG.md

@@ -4,6 +4,12 @@ All major updates to Sciris are documented here.
 
 By import convention, components of the Sciris library are listed beginning with `sc.`, e.g. `sc.odict()`.
 
+## Version 3.5.2 (2026-10-16)
+
+1. `sc.urlopen()` and `sc.download()` have a new `timeout` argument (default 30 seconds; `None` waits forever, as before), and a new `progress` argument to show a progress bar.
+1. `sc.urlopen()` now adds the `prefix` to any URL without a scheme (previously, e.g. `httpbin.org` was left alone because it starts with "http"), accepts a string or bytes for `data`, and closes the connection after reading.
+1. `sc.urlopen()` and `sc.download()` no longer include query strings in default filenames (e.g. `data?a=1` is now saved as `data`).
+
 ## Version 3.5.1 (2026-10-15)
 
 1. `sc.iterobj()` and `sc.search()` are about twice as fast; `sc.iterobj()` no longer descends into classes or NumPy dtypes stored as attributes (e.g. `self.dtype = np.int64`), since their attributes belong to the class rather than the object. To restore the previous behavior, pass `atomic=sc.sc_nested.atomic_with_tuple`.

{sciris-3.5.1/sciris.egg-info → sciris-3.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sciris
-Version: 3.5.1
+Version: 3.5.2
 Summary: Fast, flexible tools to simplify scientific Python
 Author-email: Sciris Development Team <info@sciris.org>
 License-Expression: MIT

{sciris-3.5.1 → sciris-3.5.2}/sciris/sc_utils.py

@@ -756,7 +756,7 @@ def asciify(string, form='NFKD', encoding='ascii', errors='ignore', **kwargs):
 __all__ += ['urlopen', 'wget', 'download', 'htmlify']
 
 def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
-            prefix='http', convert=True, die=False, response='text', verbose=False):
+            prefix='http', convert=True, die=False, response='text', timeout=30, progress=False, verbose=False):
     """
     Download a single URL.
 
@@ -769,11 +769,13 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
         save (bool): if supplied instead of `filename`, then use the default filename
         headers (dict): a dictionary of headers to pass
         params (dict): a dictionary of parameters to pass to the GET request
-        data (dict) a dictionary of parameters to pass to a POST request
-        prefix (str): the string to ensure the URL starts with (else, add it)
+        data (dict/str/bytes): a dictionary of parameters to pass to a POST request (or the already-encoded body)
+        prefix (str): the scheme to add if the URL doesn't have one (e.g. 'http' turns 'wikipedia.org' into 'http://wikipedia.org')
         convert (bool): whether to convert from bytes to string
         die (bool): whether to raise an exception if converting to text failed
         response (str): what to return: 'text' (default), 'json' (dictionary version of the data), 'status' (the HTTP status), or 'full' (the full response object)
+        timeout (float): how many seconds to wait for the server to respond before raising an error (None to wait forever); this limits each connection or read attempt, not the whole download
+        progress (bool): whether to show a progress bar while downloading (a running total if the server doesn't give the size)
         verbose (bool): whether to print progress
 
     **Examples**:
@@ -783,12 +785,15 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
     sc.urlopen('http://wikipedia.org', filename='wikipedia.html') # Save to file wikipedia.html
     sc.urlopen('https://wikipedia.org', save=True, headers={'User-Agent':'Custom agent'}) # Save to the default filename (here, wikipedia.org), with headers
     sc.urlopen('wikipedia.org', response='status') # Only return the HTTP status of the site
+    sc.urlopen('wikipedia.org', timeout=5) # Give up if the server doesn't respond within 5 seconds
+    sc.urlopen('https://example.com/bigfile.zip', save=True, progress=True) # Show a progress bar
     ```
 
     - *New in version 2.0.0:* renamed from `wget` to `urlopen`; new arguments
     - *New in version 2.0.1:* creates folders by default if they do not exist
     - *New in version 2.0.4:* "prefix" argument, e.g. prepend "http://" if not present
     - *New in version 3.1.4:* renamed "return_response" to "response"; additional options
+    - *New in version 3.5.2:* "timeout" and "progress" arguments; "data" can be a string or bytes; the prefix is added to any URL without a scheme (e.g. "httpbin.org")
     """
     from urllib import request as ur # Need to import these directly, not via urllib
     from urllib import parse as up
@@ -805,24 +810,29 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
 
     # Handle parameters and data
     full_url = url
-    if prefix is not None:
-        if not full_url.startswith(prefix):
-            full_url = prefix + '://' + full_url # Leaves https alone, but adds http:// otherwise
+    if prefix is not None and '://' not in full_url:
+        full_url = prefix + '://' + full_url # Leaves e.g. https:// alone, but adds http:// otherwise
     if params is not None:
         full_url += ('&' if '?' in full_url else '?') + up.urlencode(params) # Append to an existing query string if present
-    if data is not None:
+    if isinstance(data, str):
+        data = data.encode('utf-8')
+    elif data is not None and not isinstance(data, bytes):
         data = up.urlencode(data).encode(encoding='utf-8', errors='ignore')
 
     if verbose: print(f'Downloading {url}...')
     request = ur.Request(full_url, headers=headers, data=data)
     try:
-        resp = ur.urlopen(request) # Actually open the URL
+        resp = ur.urlopen(request, timeout=timeout) # Actually open the URL
     except ue.HTTPError as E: # e.g. 404
         if response == 'status':
+            E.close()
             return E.code # Return the error status rather than raising
         raise
+    if response != 'full': # Read what we need now, then close the connection
+        with resp:
+            body = _readurl(resp, progress) if response in ['text', 'json'] else None
     if response in ['text', 'json']:
-        output = resp.read()
+        output = body
         if response == 'json':
             try:
                 output = sc.loadjson(string=output)
@@ -851,7 +861,7 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
     if filename is None and save:
         filename = resp.headers.get_filename() # From the "Content-Disposition" header, if present
         if not filename:
-            filename = url.rstrip('/').split('/')[-1] # Remove trailing /, then pull out the last chunk
+            filename = _url_filename(url)
 
     if filename is not None and save is not False:
         if verbose: print(f'Saving to {filename}...')
@@ -875,7 +885,32 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
 wget = urlopen
 
 
-def download(url, *args, filename=None, save=True, parallel=True, die=True, verbose=True, **kwargs):
+def _readurl(resp, progress=False, chunksize=2**16):
+    """ Read an HTTP response, optionally with a progress bar """
+    if not progress:
+        return resp.read()
+    total = resp.length # From the Content-Length header; None if the server doesn't give it
+    chunks = []
+    done = 0
+    while chunk := resp.read(chunksize):
+        chunks.append(chunk)
+        done += len(chunk)
+        label = f'Downloaded {sc.humanize_bytes(done):>10}' # Fixed width, so the line is fully overwritten
+        if total:
+            sc.progressbar(done, total, label=label, flush=True) # Ends the line when done == total
+        else:
+            print(f'\r{label}', end='', flush=True)
+    if done != total:
+        print() # End the line
+    return b''.join(chunks)
+
+
+def _url_filename(url):
+    """ Turn e.g. 'http://mysite.com/tree/index.html?a=1' into 'index.html' """
+    return url.split('?')[0].split('#')[0].rstrip('/').split('/')[-1]
+
+
+def download(url, *args, filename=None, save=True, parallel=True, die=True, progress=False, verbose=True, **kwargs):
     """
     Download one or more URLs in parallel and return output or save them to disk.
 
@@ -888,8 +923,9 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
         save (bool): if supplied instead of `filename`, then use the default filename
         parallel (bool): whether to download multiple URLs in parallel
         die (bool): whether to raise an exception if a URL can't be retrieved (default true)
+        progress (bool): whether to show a progress bar (of bytes for a single URL, or of URLs completed for several)
         verbose (bool): whether to print progress (if verbose=2, print extra detail on each downloaded URL)
-        **kwargs (dict): passed to `sc.urlopen()`
+        **kwargs (dict): passed to `sc.urlopen()` (e.g. `timeout`)
 
     **Examples**:
 
@@ -899,12 +935,14 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
     sc.download({'sciris.html':'http://sciris.org', 'covasim.html':'http://covasim.org'}) # Download two and save to disk
     sc.download(['http://sciris.org', 'http://covasim.org'], filename=['sciris.html', 'covasim.html']) # Ditto
     data = sc.download(dict(sciris='http://sciris.org', covasim='http://covasim.org'), save=False) # Download and store in memory
+    sc.download('https://example.com/bigfile.zip', progress=True, timeout=10) # Show a progress bar, and give up if the server stops responding for 10 seconds
     ```
 
     - *New in version 2.0.0.*
     - *New in version 3.0.0:* "die" argument
     - *New in version 3.1.1:* default order switched from URL:filename to filename:URL pairs
     - *New in version 3.1.3:* output as objdict instead of odict
+    - *New in version 3.5.2:* "progress" argument
     """
     T = sc.timer()
 
@@ -927,7 +965,7 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
         else: # Otherwise, turn the URL into a filename
             keys = []
             for url in urls: # Turn e.g. 'http://mysite.com/tree/index.html' into 'index.html', or 'index.html1' if it's a duplicate
-                keys.append(uniquename(url.split('/')[-1], keys))
+                keys.append(uniquename(_url_filename(url), keys))
     elif n_keys != n_urls: # pragma: no cover
         errormsg = f'Cannot process {n_urls} URLs and {n_keys} filenames'
         raise ValueError(errormsg)
@@ -938,12 +976,12 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
     # Get results in parallel
     wget_verbose = (verbose>1) or (verbose and n_urls == 1) # By default, don't print progress on each download
     iterkwargs = dict(url=urls, filename=keys)
-    func_kwargs = mergedicts(dict(save=save, verbose=wget_verbose), kwargs)
+    func_kwargs = mergedicts(dict(save=save, verbose=wget_verbose, progress=progress and n_urls == 1), kwargs) # For several URLs, show progress by URL instead of by byte
     if n_urls > 1 and parallel:
-        outputs = sc.parallelize(urlopen, iterkwargs=iterkwargs, kwargs=func_kwargs, parallelizer='thread', die=die)
+        outputs = sc.parallelize(urlopen, iterkwargs=iterkwargs, kwargs=func_kwargs, parallelizer='thread', progress=progress, die=die)
     else:
         outputs = []
-        for key,url in zip(keys, urls):
+        for i,(key,url) in enumerate(zip(keys, urls)):
             try:
                 output = urlopen(url=url, filename=key, **func_kwargs)
             except Exception as E:
@@ -955,6 +993,8 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
                     output = E
 
             outputs.append(output)
+            if progress and n_urls > 1:
+                sc.progressbar(i+1, n_urls, label=f'URL {i+1}/{n_urls}', flush=True)
 
     # If we're returning the data rather than saving the files, convert to an odict
     if not save:

{sciris-3.5.1 → sciris-3.5.2}/sciris/sc_version.py

@@ -4,6 +4,6 @@ Version and license information.
 
 __all__ = ['__version__', '__versiondate__', '__license__']
 
-__version__      = '3.5.1'
-__versiondate__  = '2026-10-15'
+__version__      = '3.5.2'
+__versiondate__  = '2026-10-16'
 __license__      = f'Sciris {__version__} ({__versiondate__}) – © 2014-2026 by the Sciris Development Team'

{sciris-3.5.1 → sciris-3.5.2/sciris.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sciris
-Version: 3.5.1
+Version: 3.5.2
 Summary: Fast, flexible tools to simplify scientific Python
 Author-email: Sciris Development Team <info@sciris.org>
 License-Expression: MIT

{sciris-3.5.1 → sciris-3.5.2}/sciris.egg-info/SOURCES.txt

@@ -10,7 +10,6 @@ README.md
 STYLE_GUIDE.md
 context7.json
 pyproject.toml
-sciris_issues.md
 .claude-plugin/marketplace.json
 .github/workflows/publish_docs.yaml
 .github/workflows/pypi_release.yaml

{sciris-3.5.1 → sciris-3.5.2}/tests/pytest.ini

@@ -1,5 +1,6 @@
 [pytest]
 filterwarnings =
     ignore::pytest.PytestReturnNotNoneWarning
+faulthandler_timeout = 300
 env =
     SCIRIS_BACKEND=agg

{sciris-3.5.1 → sciris-3.5.2}/tests/test_utils.py

@@ -91,11 +91,17 @@ def test_urlopen_local():
     print('\nTesting urlopen with a local server')
 
     class Handler(BaseHTTPRequestHandler):
+        def log_message(self, *args):
+            pass # Don't print each request
+
         def do_GET(self):
             if self.path == '/missing':
                 self.send_response(404)
                 self.end_headers()
                 return
+            if self.path == '/slow':
+                sc.timedsleep(1)
+                return # The client has timed out by now, so don't reply
             self.send_response(200)
             if self.path == '/latin1':
                 self.send_header('Content-Type', 'text/plain; charset=iso-8859-1')
@@ -103,6 +109,14 @@ def test_urlopen_local():
             else:
                 self.send_header('Content-Disposition', 'attachment; filename="temp_report.csv"')
                 body = self.path.encode()
+            if self.path == '/sized':
+                self.send_header('Content-Length', str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_POST(self):
+            body = self.rfile.read(int(self.headers['Content-Length']))
+            self.send_response(200)
             self.end_headers()
             self.wfile.write(body)
 
@@ -113,8 +127,14 @@ def test_urlopen_local():
         assert sc.urlopen(url + '/missing', response='status') == 404
         assert sc.urlopen(url + '/api?a=1', params=dict(b=2)) == '/api?a=1&b=2'
         assert sc.urlopen(url + '/latin1') == 'café'
-        data = sc.download(f'http://{url}/v1/data', f'http://{url}/v2/data', save=False)
-        assert list(data.values()) == ['/v1/data', '/v2/data'] # Duplicate filenames are not merged
+        assert sc.urlopen(url + '/post', data=b'raw') == 'raw'
+        assert sc.urlopen(url + '/sized', progress=True) == '/sized' # With Content-Length
+        assert sc.urlopen(url + '/unsized', progress=True) == '/unsized' # Without
+        with pytest.raises(TimeoutError):
+            sc.urlopen(url + '/slow', timeout=0.1)
+        data = sc.download(f'http://{url}/v1/data', f'http://{url}/v2/data?a=1', save=False, progress=True)
+        assert list(data.keys()) == ['data', 'data1'] # Query strings are not part of the filename
+        assert list(data.values()) == ['/v1/data', '/v2/data?a=1'] # Duplicate filenames are not merged
         filename = sc.urlopen(url + '/report', save=True) # Filename from the Content-Disposition header
         assert sc.path(filename).name == 'temp_report.csv'
         sc.rmpath(filename)

sciris-3.5.1/sciris_issues.md

@@ -1,21 +0,0 @@
-# Sciris issues
-
-Found while speeding up how Starsim v4 builds a sim (Sciris as installed in the `v4` env). Nothing here has been changed in Sciris.
-
-## `sc.search()` builds the whole tree before matching
-
-`sc.search()` calls `sc.iterobj()` with the default function, which stores every object it visits in `output`, keyed by `tuple(trace)` (an `sc.objdict` `__setitem__` per object), and only then filters that tree for matches. For a search by type, almost all of that work is thrown away. Building a Starsim sim does five such searches (for distributions and time parameters), and they cost about 3 µs per object visited. That was most of the time to build a small sim (about 8 of 13 ms before the changes in Starsim, and about 4 of 7 ms after them).
-
-Suggested fix: give `iterobj()` a mode in which the function only records matches (or let `search()` pass a function that keeps only the objects that match, and not store the rest), so a search costs only the walk.
-
-## `IterObj` deep-copies the `skip` dict on every call
-
-`IterObj.__init__` does `skip = sc.dcp(skip)` when `skip` is a dict. Starsim passes a list of a few hundred object IDs in `skip['ids']`, so each search starts by deep-copying it. A shallow copy (or `dict(skip)`, since the values are only read) would do.
-
-## `sc.iterobj()` descends into classes and NumPy dtypes
-
-With the default `atomic`, a class object (e.g. `np.int64` stored as an attribute `dtype`) is treated as an object and its `__dict__` is walked: 49 extra visits for one such attribute. Classes and `np.dtype` instances could be atomic by default, as strings and arrays are. (Starsim now skips them itself.)
-
-## `sc.yeartodate()` is slow
-
-About 65 µs a call, most of it in `sc.datedelta()`, which sanitizes its arguments with `sc.dcp()` and goes through `dateutil.relativedelta`. `ss.date.from_year()` called it once per time point, which made building a sim with weekly steps over 30 years take about 70 ms. Starsim now computes the date directly (1 January of the year, plus `round(fraction × days in the year)` days, which gives identical dates on 95,445 test years). The same shortcut in `yeartodate()` itself would help other callers. A vectorized form, for an array of years, would help more.