sciris 3.5.0__tar.gz → 3.5.2__tar.gz

{sciris-3.5.0 → sciris-3.5.2}/CHANGELOG.md

@@ -4,6 +4,18 @@ All major updates to Sciris are documented here.
 
 By import convention, components of the Sciris library are listed beginning with `sc.`, e.g. `sc.odict()`.
 
+## Version 3.5.2 (2026-10-16)
+
+1. `sc.urlopen()` and `sc.download()` have a new `timeout` argument (default 30 seconds; `None` waits forever, as before), and a new `progress` argument to show a progress bar.
+1. `sc.urlopen()` now adds the `prefix` to any URL without a scheme (previously, e.g. `httpbin.org` was left alone because it starts with "http"), accepts a string or bytes for `data`, and closes the connection after reading.
+1. `sc.urlopen()` and `sc.download()` no longer include query strings in default filenames (e.g. `data?a=1` is now saved as `data`).
+
+## Version 3.5.1 (2026-10-15)
+
+1. `sc.iterobj()` and `sc.search()` are about twice as fast; `sc.iterobj()` no longer descends into classes or NumPy dtypes stored as attributes (e.g. `self.dtype = np.int64`), since their attributes belong to the class rather than the object. To restore the previous behavior, pass `atomic=sc.sc_nested.atomic_with_tuple`.
+1. `sc.search()` no longer matches the root object when searching by key.
+1. `sc.yeartodate()` is about 6 times faster, and now accepts a list or array of years, returning a list of dates (about 40 ns per date).
+
 ## Version 3.5.0 (2026-10-13)
 
 1. `sc.obj()` is a new, shorter name for `sc.objdict()`, e.g. `pars = sc.obj(a=1, b=2)`. It is the preferred name for new code.

{sciris-3.5.0/sciris.egg-info → sciris-3.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sciris
-Version: 3.5.0
+Version: 3.5.2
 Summary: Fast, flexible tools to simplify scientific Python
 Author-email: Sciris Development Team <info@sciris.org>
 License-Expression: MIT

{sciris-3.5.0 → sciris-3.5.2}/sciris/sc_datetime.py

@@ -662,29 +662,41 @@ def yeartodate(year, as_date=True, **kwargs):
     Convert a decimal year to a date
 
     Args:
-        year (int, float):  The numerical year to convert to a DateTime
-        as_date (bool): If True (default), return an `sc.date` object, otherwise return a string
+        year (int, float, list, array):  The numerical year (or years) to convert to a date
+        as_date (bool): If True (default), return a `datetime.date` object, otherwise return a string
 
     Returns:
-        An `sc.date` object (default) or string, depending on the `as_date` argument
+        A `datetime.date` object (default) or string, depending on the `as_date` argument; a list of these if a list or array of years is supplied
 
-    **Example**:
+    **Examples**:
 
     ```python
     sc.yeartodate(2010.5) # Returns datetime.date(2010, 7, 2)
+    sc.yeartodate(np.arange(2020, 2021, 1/52)) # Returns a list of weekly dates
     ```
 
     - *New in version 3.2.1.*
+    - *New in version 3.5.1:* faster; list and array inputs
     """
     as_date = kwargs.pop('asdate', as_date) # Handle with or without underscore
-    full_years = int(year)
-    remainder = year - full_years
-    year_days = _get_year_length(full_years).days
-    days = int(np.round(remainder*year_days))
-    base = dt.date(year=full_years, month=1, day=1)
-    out = datedelta(base, days=days)
-    if not as_date:
-        out = str(out)
+
+    if isinstance(year, (list, tuple, np.ndarray)): # Handle multiple years
+        years = np.asarray(year, dtype=float)
+        full_years = years.astype(int) # Truncate, like int()
+        starts = (full_years - 1970).astype('datetime64[Y]') # Start of each year
+        year_days = ((starts + 1).astype('datetime64[D]') - starts.astype('datetime64[D]')).astype(int)
+        days = np.round((years - full_years)*year_days).astype(int)
+        out = (starts.astype('datetime64[D]') + days).tolist() # Convert to a list of datetime.date objects
+        if not as_date:
+            out = [str(d) for d in out]
+    else:
+        full_years = int(year)
+        remainder = year - full_years
+        year_days = _get_year_length(full_years).days
+        days = int(np.round(remainder*year_days))
+        out = dt.date(year=full_years, month=1, day=1) + dt.timedelta(days=days)
+        if not as_date:
+            out = str(out)
     return out
 
 

{sciris-3.5.0 → sciris-3.5.2}/sciris/sc_nested.py

@@ -20,6 +20,7 @@ from .sc_utils import _not_given, _is_given # Allows searching for actual None v
 # Define objects for which it doesn't make sense to descend further -- used here and sc.equal()
 atomic_classes = [np.ndarray, pd.Series, pd.DataFrame, pd.core.indexes.base.Index]
 atomic_with_tuple = atomic_classes + [tuple]
+atomic_types = [type, np.dtype] # Classes and dtypes are also not descended into by sc.iterobj(), but are compared normally by sc.equal()
 
 
 ##############################################################################
@@ -411,15 +412,15 @@ class IterObj:
         atomic_list = sc.tolist(self.atomic)
         if 'default' in atomic_list: # Handle objects to not descend into
             atomic_list.remove('default')
-            base_atomic = atomic_with_tuple
+            base_atomic = atomic_with_tuple + atomic_types
         if 'default-tuple' in atomic_list:
             atomic_list.remove('default-tuple')
-            base_atomic = atomic_classes
+            base_atomic = atomic_classes + atomic_types
         self.atomic = tuple(base_atomic + atomic_list)
 
         # Handle objects to skip
         if isinstance(skip, dict):
-            skip = sc.dcp(skip)
+            skip = dict(skip) # Shallow copy since the values are only read
             skip_keys       = sc.tolist(skip.pop('keys', None))
             skip_ids        = sc.tolist(skip.pop('ids', None))
             skip_subclasses = sc.tolist(skip.pop('subclasses', None))
@@ -481,9 +482,10 @@ class IterObj:
             print(space*len(self._trace) + string)
         return
 
-    def iteritems(self, parent, trace):
+    def iteritems(self, parent, trace, itertype=None):
         """ Return an iterator over items in this object """
-        itertype = self.check_iter_type(parent)
+        if itertype is None:
+            itertype = self.check_iter_type(parent)
         out = None
         if self.custom_iter:
             out = self.custom_iter(parent)
@@ -505,9 +507,7 @@ class IterObj:
                         out = [] # Return nothing if doesn't have __dict__.items() or __slots__, e.g. a weird wrapped function
             else:
                 out = [] # Return nothing if not recognized
-        out = list(out)
-        for i in range(len(out)):
-            out[i] = [parent, trace, *list(out[i])] # Prepend parent and trace to the arguments
+        out = [(parent, trace, k, v) for k,v in out] # Prepend parent and trace to the arguments
         return out
 
     def getitem(self, key, parent):
@@ -542,12 +542,14 @@ class IterObj:
         """ Shortcut to check_iter_type() """
         return check_iter_type(obj, known=self.atomic, custom=self.custom_type)
 
-    def check_proceed(self, key, subobj, newid):
+    def check_proceed(self, key, subobj, newid, itertype=None):
         """ Check if we should process the object, and if so, whether to descend into it """
 
         # If we've already parsed this (iterable) object, skip it, or if aliases=True, process it but don't descend into it again
         in_memo = (newid in self._memo) and (self._memo[newid] > self.recursion)
-        repeat = in_memo and self.check_iter_type(subobj)
+        if in_memo and itertype is None:
+            itertype = self.check_iter_type(subobj)
+        repeat = in_memo and bool(itertype)
         memo_skip = repeat and not self.aliases
         descend = not repeat
         if repeat and self.aliases:
@@ -560,22 +562,23 @@ class IterObj:
         instance_skip = isinstance(subobj, self._skip_instances)
 
         # Finalize
-        skips = [memo_skip, key_skip, id_skip, subclass_skip, instance_skip]
-        proceed = not any(skips)
+        proceed = not (memo_skip or key_skip or id_skip or subclass_skip or instance_skip)
 
         if not proceed and self.verbose: # Just for debugging
+            skips = [memo_skip, key_skip, id_skip, subclass_skip, instance_skip]
             labels = ['memo', 'key', 'id', 'subclass', 'instance']
             pairs = [f'{label}_skip=True' for label,skip in zip(labels, skips) if skip]
             self.indent(f'Skipping "{key}" because {sc.strjoin(pairs)}')
 
         return proceed, descend
 
-    def process_obj(self, parent, trace, key, subobj, newid):
+    def process_obj(self, parent, trace, key, subobj, newid, itertype=None):
         """ Process a single object, returning its trace and its new value """
         self._memo[newid] += 1
         trace = trace + [key]
-        subitertype = self.check_iter_type(subobj)
-        self.indent(f'{len(self)} Trace {trace} | Type "{str(subitertype)}" | {type(subobj)}')
+        subitertype = self.check_iter_type(subobj) if itertype is None else itertype
+        if self.verbose: # Check here to avoid formatting the string
+            self.indent(f'{len(self)} Trace {trace} | Type "{str(subitertype)}" | {type(subobj)}')
         newobj = subobj
         if not (self.leaf and subitertype):
             newobj = self.func(subobj, **self.func_kw)
@@ -604,11 +607,15 @@ class IterObj:
         while queue:
             parent,trace,key,subobj = queue.popleft()
             newid = id(subobj)
-            proceed, descend = self.check_proceed(key, subobj, newid)
+            itertype = self.check_iter_type(subobj) # Calculate once, since it's needed several times
+            proceed, descend = self.check_proceed(key, subobj, newid, itertype=itertype)
             if proceed:
-                newtrace, newobj = self.process_obj(parent, trace, key, subobj, newid) # Process the object
+                newtrace, newobj = self.process_obj(parent, trace, key, subobj, newid, itertype=itertype) # Process the object
                 if descend: # If modifying in place, descend into the new object, since that's what's in the parent now
-                    newitems = self.iteritems(newobj if self.inplace else subobj, newtrace)
+                    if newobj is subobj or not self.inplace:
+                        newitems = self.iteritems(subobj, newtrace, itertype=itertype)
+                    else:
+                        newitems = self.iteritems(newobj, newtrace)
                     if self.depthfirst:
                         queue.extendleft(reversed(newitems)) # extendleft() swaps order, so swap back
                     else:
@@ -676,7 +683,7 @@ def iterobj(obj, func=None, inplace=False, copy=False, leaf=False, recursion=0,
         leaf (bool): whether to apply the function only to leaf nodes of the object
         recursion (int): number of recursive steps to allow, i.e. parsing the same objects multiple times (default 0)
         depthfirst (bool): whether to parse the object depth-first (default) or breadth-first
-        atomic (list): a list of known classes to treat as atomic (do not descend into); if 'default', use defaults (e.g. tuple, `np.array`, `pd.DataFrame`); if 'default-tuple', use defaults except for tuples
+        atomic (list): a list of known classes to treat as atomic (do not descend into); if 'default', use defaults (e.g. tuple, `np.array`, `pd.DataFrame`, classes, and NumPy dtypes); if 'default-tuple', use defaults except for tuples
         skip (list/dict): a list of objects to skip over entirely; can also be a dict with "keys", "ids", "subclasses", and/or "instances", which skip each of those
         rootkey (str): the key to list as the root of the object (default `'root'`)
         verbose (bool): whether to print progress
@@ -717,6 +724,7 @@ def iterobj(obj, func=None, inplace=False, copy=False, leaf=False, recursion=0,
     - *New in version 3.1.5:* "recursion" argument; better handling of atomic classes
     - *New in version 3.1.6:* "skip", "depthfirst", "to_df", and "flatten" arguments
     - *New in version 3.4.0:* "aliases" argument
+    - *New in version 3.5.1:* classes and NumPy dtypes are atomic by default
     """
     # Create the object
     io = IterObj(obj=obj, func=func, inplace=inplace, copy=copy, leaf=leaf, recursion=recursion, aliases=aliases, depthfirst=depthfirst,
@@ -963,23 +971,15 @@ def search(obj, query=_not_given, key=_not_given, value=_not_given, type=_not_gi
     flatten = kwargs.pop('flatten', False) # Don't flatten because that will disrupt the matching
     tree = iterobj(obj, **kwargs) # For key matching
 
-    # Do the matching
-    matches = []
-
-    # Match keys
-    if _is_given(key):
-        for k in tree.keys():
-            if check_match(k[-1], key): # Only want the last key of the trace
-                matches.append(k)
-
-    # Match values (including types)
-    if _is_given(value):
-        for k,v in tree.items():
-            if check_match(v, value):
-                matches.append(k)
-
-    # Reassemble dict to maintain order
-    out = sc.objdict({k:v for k,v in tree.items() if k in matches})
+    # Do the matching, keeping the order of the tree
+    match_key = _is_given(key)
+    match_value = _is_given(value)
+    out = {}
+    for k,v in tree.items():
+        is_root = not isinstance(k, tuple) # The root has no key to match
+        if (match_key and not is_root and check_match(k[-1], key)) or (match_value and check_match(v, value)): # Only want the last key of the trace
+            out[k] = v
+    out = sc.objdict(out)
 
     if flatten:
         out = flatten_traces(out)

{sciris-3.5.0 → sciris-3.5.2}/sciris/sc_utils.py

@@ -756,7 +756,7 @@ def asciify(string, form='NFKD', encoding='ascii', errors='ignore', **kwargs):
 __all__ += ['urlopen', 'wget', 'download', 'htmlify']
 
 def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
-            prefix='http', convert=True, die=False, response='text', verbose=False):
+            prefix='http', convert=True, die=False, response='text', timeout=30, progress=False, verbose=False):
     """
     Download a single URL.
 
@@ -769,11 +769,13 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
         save (bool): if supplied instead of `filename`, then use the default filename
         headers (dict): a dictionary of headers to pass
         params (dict): a dictionary of parameters to pass to the GET request
-        data (dict) a dictionary of parameters to pass to a POST request
-        prefix (str): the string to ensure the URL starts with (else, add it)
+        data (dict/str/bytes): a dictionary of parameters to pass to a POST request (or the already-encoded body)
+        prefix (str): the scheme to add if the URL doesn't have one (e.g. 'http' turns 'wikipedia.org' into 'http://wikipedia.org')
         convert (bool): whether to convert from bytes to string
         die (bool): whether to raise an exception if converting to text failed
         response (str): what to return: 'text' (default), 'json' (dictionary version of the data), 'status' (the HTTP status), or 'full' (the full response object)
+        timeout (float): how many seconds to wait for the server to respond before raising an error (None to wait forever); this limits each connection or read attempt, not the whole download
+        progress (bool): whether to show a progress bar while downloading (a running total if the server doesn't give the size)
         verbose (bool): whether to print progress
 
     **Examples**:
@@ -783,12 +785,15 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
     sc.urlopen('http://wikipedia.org', filename='wikipedia.html') # Save to file wikipedia.html
     sc.urlopen('https://wikipedia.org', save=True, headers={'User-Agent':'Custom agent'}) # Save to the default filename (here, wikipedia.org), with headers
     sc.urlopen('wikipedia.org', response='status') # Only return the HTTP status of the site
+    sc.urlopen('wikipedia.org', timeout=5) # Give up if the server doesn't respond within 5 seconds
+    sc.urlopen('https://example.com/bigfile.zip', save=True, progress=True) # Show a progress bar
     ```
 
     - *New in version 2.0.0:* renamed from `wget` to `urlopen`; new arguments
     - *New in version 2.0.1:* creates folders by default if they do not exist
     - *New in version 2.0.4:* "prefix" argument, e.g. prepend "http://" if not present
     - *New in version 3.1.4:* renamed "return_response" to "response"; additional options
+    - *New in version 3.5.2:* "timeout" and "progress" arguments; "data" can be a string or bytes; the prefix is added to any URL without a scheme (e.g. "httpbin.org")
     """
     from urllib import request as ur # Need to import these directly, not via urllib
     from urllib import parse as up
@@ -805,24 +810,29 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
 
     # Handle parameters and data
     full_url = url
-    if prefix is not None:
-        if not full_url.startswith(prefix):
-            full_url = prefix + '://' + full_url # Leaves https alone, but adds http:// otherwise
+    if prefix is not None and '://' not in full_url:
+        full_url = prefix + '://' + full_url # Leaves e.g. https:// alone, but adds http:// otherwise
     if params is not None:
         full_url += ('&' if '?' in full_url else '?') + up.urlencode(params) # Append to an existing query string if present
-    if data is not None:
+    if isinstance(data, str):
+        data = data.encode('utf-8')
+    elif data is not None and not isinstance(data, bytes):
         data = up.urlencode(data).encode(encoding='utf-8', errors='ignore')
 
     if verbose: print(f'Downloading {url}...')
     request = ur.Request(full_url, headers=headers, data=data)
     try:
-        resp = ur.urlopen(request) # Actually open the URL
+        resp = ur.urlopen(request, timeout=timeout) # Actually open the URL
     except ue.HTTPError as E: # e.g. 404
         if response == 'status':
+            E.close()
             return E.code # Return the error status rather than raising
         raise
+    if response != 'full': # Read what we need now, then close the connection
+        with resp:
+            body = _readurl(resp, progress) if response in ['text', 'json'] else None
     if response in ['text', 'json']:
-        output = resp.read()
+        output = body
         if response == 'json':
             try:
                 output = sc.loadjson(string=output)
@@ -851,7 +861,7 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
     if filename is None and save:
         filename = resp.headers.get_filename() # From the "Content-Disposition" header, if present
         if not filename:
-            filename = url.rstrip('/').split('/')[-1] # Remove trailing /, then pull out the last chunk
+            filename = _url_filename(url)
 
     if filename is not None and save is not False:
         if verbose: print(f'Saving to {filename}...')
@@ -875,7 +885,32 @@ def urlopen(url, filename=None, save=None, headers=None, params=None, data=None,
 wget = urlopen
 
 
-def download(url, *args, filename=None, save=True, parallel=True, die=True, verbose=True, **kwargs):
+def _readurl(resp, progress=False, chunksize=2**16):
+    """ Read an HTTP response, optionally with a progress bar """
+    if not progress:
+        return resp.read()
+    total = resp.length # From the Content-Length header; None if the server doesn't give it
+    chunks = []
+    done = 0
+    while chunk := resp.read(chunksize):
+        chunks.append(chunk)
+        done += len(chunk)
+        label = f'Downloaded {sc.humanize_bytes(done):>10}' # Fixed width, so the line is fully overwritten
+        if total:
+            sc.progressbar(done, total, label=label, flush=True) # Ends the line when done == total
+        else:
+            print(f'\r{label}', end='', flush=True)
+    if done != total:
+        print() # End the line
+    return b''.join(chunks)
+
+
+def _url_filename(url):
+    """ Turn e.g. 'http://mysite.com/tree/index.html?a=1' into 'index.html' """
+    return url.split('?')[0].split('#')[0].rstrip('/').split('/')[-1]
+
+
+def download(url, *args, filename=None, save=True, parallel=True, die=True, progress=False, verbose=True, **kwargs):
     """
     Download one or more URLs in parallel and return output or save them to disk.
 
@@ -888,8 +923,9 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
         save (bool): if supplied instead of `filename`, then use the default filename
         parallel (bool): whether to download multiple URLs in parallel
         die (bool): whether to raise an exception if a URL can't be retrieved (default true)
+        progress (bool): whether to show a progress bar (of bytes for a single URL, or of URLs completed for several)
         verbose (bool): whether to print progress (if verbose=2, print extra detail on each downloaded URL)
-        **kwargs (dict): passed to `sc.urlopen()`
+        **kwargs (dict): passed to `sc.urlopen()` (e.g. `timeout`)
 
     **Examples**:
 
@@ -899,12 +935,14 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
     sc.download({'sciris.html':'http://sciris.org', 'covasim.html':'http://covasim.org'}) # Download two and save to disk
     sc.download(['http://sciris.org', 'http://covasim.org'], filename=['sciris.html', 'covasim.html']) # Ditto
     data = sc.download(dict(sciris='http://sciris.org', covasim='http://covasim.org'), save=False) # Download and store in memory
+    sc.download('https://example.com/bigfile.zip', progress=True, timeout=10) # Show a progress bar, and give up if the server stops responding for 10 seconds
     ```
 
     - *New in version 2.0.0.*
     - *New in version 3.0.0:* "die" argument
     - *New in version 3.1.1:* default order switched from URL:filename to filename:URL pairs
     - *New in version 3.1.3:* output as objdict instead of odict
+    - *New in version 3.5.2:* "progress" argument
     """
     T = sc.timer()
 
@@ -927,7 +965,7 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
         else: # Otherwise, turn the URL into a filename
             keys = []
             for url in urls: # Turn e.g. 'http://mysite.com/tree/index.html' into 'index.html', or 'index.html1' if it's a duplicate
-                keys.append(uniquename(url.split('/')[-1], keys))
+                keys.append(uniquename(_url_filename(url), keys))
     elif n_keys != n_urls: # pragma: no cover
         errormsg = f'Cannot process {n_urls} URLs and {n_keys} filenames'
         raise ValueError(errormsg)
@@ -938,12 +976,12 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
     # Get results in parallel
     wget_verbose = (verbose>1) or (verbose and n_urls == 1) # By default, don't print progress on each download
     iterkwargs = dict(url=urls, filename=keys)
-    func_kwargs = mergedicts(dict(save=save, verbose=wget_verbose), kwargs)
+    func_kwargs = mergedicts(dict(save=save, verbose=wget_verbose, progress=progress and n_urls == 1), kwargs) # For several URLs, show progress by URL instead of by byte
     if n_urls > 1 and parallel:
-        outputs = sc.parallelize(urlopen, iterkwargs=iterkwargs, kwargs=func_kwargs, parallelizer='thread', die=die)
+        outputs = sc.parallelize(urlopen, iterkwargs=iterkwargs, kwargs=func_kwargs, parallelizer='thread', progress=progress, die=die)
     else:
         outputs = []
-        for key,url in zip(keys, urls):
+        for i,(key,url) in enumerate(zip(keys, urls)):
             try:
                 output = urlopen(url=url, filename=key, **func_kwargs)
             except Exception as E:
@@ -955,6 +993,8 @@ def download(url, *args, filename=None, save=True, parallel=True, die=True, verb
                     output = E
 
             outputs.append(output)
+            if progress and n_urls > 1:
+                sc.progressbar(i+1, n_urls, label=f'URL {i+1}/{n_urls}', flush=True)
 
     # If we're returning the data rather than saving the files, convert to an odict
     if not save:

{sciris-3.5.0 → sciris-3.5.2}/sciris/sc_version.py

@@ -4,6 +4,6 @@ Version and license information.
 
 __all__ = ['__version__', '__versiondate__', '__license__']
 
-__version__      = '3.5.0'
-__versiondate__  = '2026-10-13'
+__version__      = '3.5.2'
+__versiondate__  = '2026-10-16'
 __license__      = f'Sciris {__version__} ({__versiondate__}) – © 2014-2026 by the Sciris Development Team'

{sciris-3.5.0 → sciris-3.5.2/sciris.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: sciris
-Version: 3.5.0
+Version: 3.5.2
 Summary: Fast, flexible tools to simplify scientific Python
 Author-email: Sciris Development Team <info@sciris.org>
 License-Expression: MIT

{sciris-3.5.0 → sciris-3.5.2}/sciris.egg-info/SOURCES.txt

@@ -9,7 +9,6 @@ MANIFEST.in
 README.md
 STYLE_GUIDE.md
 context7.json
-objdict_performance.md
 pyproject.toml
 .claude-plugin/marketplace.json
 .github/workflows/publish_docs.yaml

{sciris-3.5.0 → sciris-3.5.2}/tests/pytest.ini

@@ -1,5 +1,6 @@
 [pytest]
 filterwarnings =
     ignore::pytest.PytestReturnNotNoneWarning
+faulthandler_timeout = 300
 env =
     SCIRIS_BACKEND=agg

{sciris-3.5.0 → sciris-3.5.2}/tests/test_datetime.py

@@ -126,6 +126,13 @@ def test_dates():
     o.year = sc.datetoyear('2010-07-01')
     assert sc.datetoyear(dt.datetime(2010, 7, 1, 12)) > o.year # Keeps the time of day
 
+    print('\nTesting yeartodate')
+    o.date = sc.yeartodate(2010.5)
+    assert o.date == dt.date(2010, 7, 2)
+    assert sc.yeartodate(2010.5, as_date=False) == '2010-07-02'
+    years = [2010.5, 2012.999, 2024.2]
+    assert sc.yeartodate(np.array(years)) == [sc.yeartodate(y) for y in years] # Arrays give the same dates
+
     print('\nTesting datedelta')
     o.dd = sc.datedelta('2021-07-07', 3) # Add 3 days
     assert o.dd == '2021-07-10'

{sciris-3.5.0 → sciris-3.5.2}/tests/test_nested.py

@@ -238,6 +238,7 @@ def test_search():
     assert len(valmatches) == 1
     assert sc.getnested(nested, valmatches[0]) == val # Get from the original nested object
     assert sc.search(dict(a=np.arange(5), b=3), value=3).keys() == [('b',)] # Arrays are skipped rather than raising an error
+    assert not len(sc.search(dict(a=1), key='t')) # The root has no key, so doesn't match
 
     return o
 
@@ -297,6 +298,7 @@ def test_iterobj():
     d = dict(t=(1,(2,3)))
     sc.iterobj(d, lambda x: list(x) if isinstance(x, tuple) else x, atomic='default-tuple', inplace=True)
     assert d == dict(t=[1,[2,3]]) # Nested tuples are converted too, since it descends into the new object
+    assert len(sc.iterobj(dict(a=np.int64, b=np.dtype(float)))) == 3 # Classes and dtypes aren't descended into
 
     return out1
 

{sciris-3.5.0 → sciris-3.5.2}/tests/test_utils.py

@@ -91,11 +91,17 @@ def test_urlopen_local():
     print('\nTesting urlopen with a local server')
 
     class Handler(BaseHTTPRequestHandler):
+        def log_message(self, *args):
+            pass # Don't print each request
+
         def do_GET(self):
             if self.path == '/missing':
                 self.send_response(404)
                 self.end_headers()
                 return
+            if self.path == '/slow':
+                sc.timedsleep(1)
+                return # The client has timed out by now, so don't reply
             self.send_response(200)
             if self.path == '/latin1':
                 self.send_header('Content-Type', 'text/plain; charset=iso-8859-1')
@@ -103,6 +109,14 @@ def test_urlopen_local():
             else:
                 self.send_header('Content-Disposition', 'attachment; filename="temp_report.csv"')
                 body = self.path.encode()
+            if self.path == '/sized':
+                self.send_header('Content-Length', str(len(body)))
+            self.end_headers()
+            self.wfile.write(body)
+
+        def do_POST(self):
+            body = self.rfile.read(int(self.headers['Content-Length']))
+            self.send_response(200)
             self.end_headers()
             self.wfile.write(body)
 
@@ -113,8 +127,14 @@ def test_urlopen_local():
         assert sc.urlopen(url + '/missing', response='status') == 404
         assert sc.urlopen(url + '/api?a=1', params=dict(b=2)) == '/api?a=1&b=2'
         assert sc.urlopen(url + '/latin1') == 'café'
-        data = sc.download(f'http://{url}/v1/data', f'http://{url}/v2/data', save=False)
-        assert list(data.values()) == ['/v1/data', '/v2/data'] # Duplicate filenames are not merged
+        assert sc.urlopen(url + '/post', data=b'raw') == 'raw'
+        assert sc.urlopen(url + '/sized', progress=True) == '/sized' # With Content-Length
+        assert sc.urlopen(url + '/unsized', progress=True) == '/unsized' # Without
+        with pytest.raises(TimeoutError):
+            sc.urlopen(url + '/slow', timeout=0.1)
+        data = sc.download(f'http://{url}/v1/data', f'http://{url}/v2/data?a=1', save=False, progress=True)
+        assert list(data.keys()) == ['data', 'data1'] # Query strings are not part of the filename
+        assert list(data.values()) == ['/v1/data', '/v2/data?a=1'] # Duplicate filenames are not merged
         filename = sc.urlopen(url + '/report', save=True) # Filename from the Content-Disposition header
         assert sc.path(filename).name == 'temp_report.csv'
         sc.rmpath(filename)

sciris-3.5.0/objdict_performance.md

@@ -1,89 +0,0 @@
-# `sc.objdict` attribute access is slow
-
-## Summary
-
-Reading a key from an `sc.objdict` as an attribute (`obj.a`) takes about 530 ns, roughly 35 times slower than `types.SimpleNamespace` or `sc.dictobj` (15 ns) and 6 times slower than reading the same key as an item (`obj['a']`, 90 ns). Writing via an attribute (`obj.a = 1`) takes about 1150 ns, roughly 60 times slower than `SimpleNamespace`. This is invisible in normal use, but it adds up when an `objdict` is read inside a hot loop.
-
-## How it was found
-
-In the Starsim v4 ODE solver (`starsim/solvers.py` on the `v4` branch), the right-hand side function read its per-compartment and per-flow records from `objdict`s. Profiling showed attribute access on those objects was about a third of the right-hand side's run time. Swapping the records read in the hot loop for `types.SimpleNamespace` removed the cost. The solver still uses `objdict` everywhere outside the hot loop.
-
-## Measurements
-
-Sciris 3.4.0, Python 3.13, Linux; best of 5 runs of 1,000,000 operations each.
-
-| Object | `obj.a` | `obj['a']` | `obj.a = 1.0` |
-|---|---|---|---|
-| `dict` | n/a | 10 ns | n/a |
-| `types.SimpleNamespace` | 15 ns | n/a | 19 ns |
-| `sc.odict` | n/a | 65 ns | n/a |
-| `sc.dictobj` | 15 ns | 81 ns | 20 ns |
-| `sc.objdict` | 531 ns | 91 ns | 1148 ns |
-
-```python
-import timeit
-import types
-import sciris as sc
-
-n = 1_000_000
-data = dict(a=1.0, b=2.0, c=3.0)
-objs = dict(
-    SimpleNamespace = types.SimpleNamespace(**data),
-    dictobj         = sc.dictobj(data),
-    objdict         = sc.objdict(data),
-)
-for name,obj in objs.items():
-    for stmt in ['obj.a', 'obj.a = 1.0']:
-        t = min(timeit.repeat(stmt, globals=dict(obj=obj), number=n, repeat=5))/n*1e9
-        print(f'{name:16s} {stmt:12s} {t:6.0f} ns')
-```
-
-## Cause
-
-`objdict.__getattribute__` (in `sciris/sc_odict.py`) tries the name as a real attribute first and falls back to the dict only if that fails:
-
-```python
-try: # First, try to get the attribute as an attribute
-    return object.__getattribute__(self, attr)
-except Exception as E: # If that fails, try to get it as a dict item, but pass along the original exception
-    return self.__getitem__(attr, exception=E)
-```
-
-So every successful read of a key raises and catches an `AttributeError` before reaching the dict, and then goes through two more Python-level calls (`objdict.__getitem__`, then `odict.__getitem__`). Creating and catching the exception is most of the cost.
-
-`objdict.__setattr__` does the same thing: it calls `object.__getattribute__` to check that the name is not an existing attribute, which raises for every ordinary key, and then calls `__setitem__`.
-
-Reading a method or other real attribute (`obj.keys`) does not hit the exception and takes about 90 ns.
-
-## Possible fix
-
-Check the dict first, and use it directly when the name is a key and is not also an attribute of the class. Methods still win over keys of the same name, so behavior is unchanged; everything else falls through to the existing code.
-
-```python
-def __getattribute__(self, attr):
-    if dict.__contains__(self, attr) and not hasattr(type(self), attr): # A key that doesn't shadow a method
-        return dict.__getitem__(self, attr)
-    ... # Existing code
-
-def __setattr__(self, name, value):
-    if dict.__contains__(self, name) and not hasattr(type(self), name): # An existing key
-        return dict.__setitem__(self, name, value)
-    ... # Existing code
-```
-
-Timings for a prototype of this, written as a subclass of `sc.objdict`:
-
-| Operation | Current | Prototype |
-|---|---|---|
-| `obj.a` (existing key) | 532 ns | 62 ns |
-| `obj.a = 1.0` (existing key) | 1178 ns | 101 ns |
-| `obj.keys` (method) | 89 ns | 140 ns |
-| `obj.new = 1.0; del obj['new']` (new key) | 1493 ns | 1819 ns |
-
-Reading and writing existing keys become about 9 and 12 times faster. Method lookups and setting new keys become slower (by about 50 ns and 330 ns), because they now pay for the extra check before falling through; some of that is the overhead of the subclass calling the parent method, which a change made in `objdict` itself would not have.
-
-The prototype passed a few quick checks (a key named `keys` does not shadow the method, integer indexing still works, `defaultdict='nested'` still works), but it has not been run against the Sciris test suite, and nothing in Sciris has been changed.
-
-## Workaround
-
-In a hot loop, read `objdict` keys as items (`obj['a']`), or hold the records in a `types.SimpleNamespace` or `sc.dictobj` instead.