omlish 0.0.0.dev299__tar.gz → 0.0.0.dev301__tar.gz

{omlish-0.0.0.dev299/omlish.egg-info → omlish-0.0.0.dev301}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omlish
-Version: 0.0.0.dev299
+Version: 0.0.0.dev301
 Summary: omlish
 Author: wrmsr
 License: BSD-3-Clause

omlish-0.0.0.dev301/README.md

@@ -0,0 +1,162 @@
+# Overview
+
+Core utilities and foundational code. It's relatively large but completely self-contained.
+
+# Notable packages
+
+- **[lang](lang)** - The standard library of this standard library. Usually imported as a whole `as lang`, it contains
+  an array of general purpose utilities used practically everywhere. It is kept relatively lightweight: its heaviest
+  import is stdlib dataclasses and its transitives. Some of its contents include:
+
+  - **[cached](lang/cached)** - The standard `cached_function` / `cached_property` tools, which are more capable than
+    [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache).
+  - **[imports](lang/imports.py)** - Import tools like `proxy_import` for late-loaded imports and `proxy_init` for
+    late-loaded module globals.
+  - **[classes](lang/classes)** - Class tools and bases, such as `Abstract` (which checks at subclass definition not
+    instantiation), `Sealed` / `PackageSealed`, and `Final`.
+  - **[maybes](lang/maybes.py)** - A simple, nestable formalization of the presence or absence of an object, as in
+    [many](https://en.cppreference.com/w/cpp/utility/optional)
+    [other](https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html)
+    [languages](https://doc.rust-lang.org/std/option/).
+
+- **[bootstrap](bootstrap)** - A centralized, configurable, all-in-one collection of various process-initialization
+  minutiae like resource limiting, profiling, remote debugging, log configuration, environment variables, et cetera.
+  Usable as a context manager or via its [cli](bootstrap/main.py).
+
+- **[collections](collections)** - A handful of collection utilities and simple implementations, including:
+
+  - **[cache](collections/cache)** - A configurable LRU / LFU cache with options like ttl and  max size / weight.
+  - **[hasheq](collections/hasheq.py)** - A dict taking an external `__hash__` / `__eq__` implementation.
+  - **[identity](collections/identity.py)** - Identity-keyed collections.
+  - **[sorted](collections/sorted)** - interfaces for value-sorted collections and mappings, and a simple but correct
+    skiplist-backed implementation.
+  - **[persistent](collections/persistent)** - Interfaces for [persistent](https://en.wikipedia.org/wiki/Persistent_data_structure)
+    maps, and a simple but correct treap-backed implementation.
+
+- **[dataclasses](dataclasses)** - A fully-compatible reimplementation of stdlib [dataclasses](https://docs.python.org/3/library/dataclasses.html)
+  with numerous enhancements and additional features. The [full stdlib test suite](dataclasses/tests/cpython) is run
+  against it ensuring compatibility - they *are* dataclasses. Current enhancements include:
+
+  - Simple field coercion and validation.
+  - Any number of `@dc.init` or `@dc.validate` methods, not just a central `__post_init__`.
+  - Optional generic type parameter substitution in generated `__init__` methods, enabling accurate reflection.
+  - An optional [metaclass](dataclasses/metaclass) which removes the need for re-decorating subclasses (with support for
+    inheritance of dataclass parameters like `frozen`), and some basic [base classes](dataclasses/metaclass/bases.py).
+  - (Nearly finished) support for ahead-of-time / build-time code generation, greatly reducing import times.
+
+  The stdlib-equivalent API is exported in such a way as to be direct aliases for the stdlib API itself, simplifying
+  tool support.
+
+- **[dispatch](dispatch)** - A beefed-up version of [functools.singledispatch](https://docs.python.org/3/library/functools.html#functools.singledispatch),
+  most notably supporting MRO-honoring method impl dispatch.
+
+- **[formats](formats)** - Tools for various data formats, including:
+
+  - **[json](formats/json)** - Tools for json, including abstraction over various backends and a self-contained streaming
+    / incremental parser.
+  - **[json5](formats/json5)** - A self-contained and tested [Json5](https://json5.org/) parser.
+  - **[toml](formats/toml)** - Toml tools, including a [lite](#lite-code) version of the stdlib parser (for use in older
+    pythons).
+
+- **[http](http)** - HTTP code, including:
+
+  - **[clients](http/clients)** - An abstraction over HTTP clients, with urllib and httpx implementations.
+  - **[coro](http/coro)** - Coroutine / [sans-io](https://sans-io.readthedocs.io/) style reformulation of some stdlib
+    http machinery - namely `http.server` (and soon `http.client`). This style of code can run the same in sync, async,
+    or [any](https://docs.python.org/3/library/selectors.html) [other](asyncs/bluelet) context.
+
+- **[inject](inject)** - A [guice](https://github.com/google/guice)-style dependency injector.
+
+- **[io](io)** - IO tools, including:
+
+  - **[compress](io/compress)** - Abstraction over various compression schemes, with particular attention to incremental
+    operation. For example it includes an incremental reformulation of stdlib's [gzip](io/compress/gzip.py).
+  - **[coro](io/coro)** - Utilities for coroutine / sans-io style code.
+  - **[fdio](io/fdio)** - An implementation of classic [selector](https://docs.python.org/3/library/selectors.html)-style
+    IO dispatch, akin to the deprecated [asyncore](https://docs.python.org/3.11/library/asyncore.html). While more
+    modern asyncio style code is generally preferred, it nearly always involves
+    [background threads](https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Lib/asyncio/unix_events.py#L1349)
+    making it [unsuitable for forking processes](https://rachelbythebay.com/w/2011/06/07/forked/) like
+    [process supervisors](https://github.com/wrmsr/omlish/tree/master/ominfra/supervisor).
+
+- **[jmespath](specs/jmespath)** - A vendoring of [jmespath community edition](https://github.com/jmespath-community/python-jmespath),
+  modernized and adapted to this codebase.
+
+- **[marshal](marshal)** - A [jackson](https://github.com/FasterXML/jackson)-style serde system.
+
+- **[manifests](manifests)** - A system for sharing lightweight metadata within / across codebases.
+
+- **[reflect](reflect)** - Reflection utilities, including primarily a formalization of stdlib type annotations for use
+  at runtime, decoupled from stdlib impl detail. Keeping this working is notoriously difficult across python versions
+  (one of the primary reasons for only supporting 3.12+).
+
+- **[sql](sql)** - A collection of SQL utilities, including:
+
+  - **[alchemy](sql/alchemy)** - SQLAlchemy utilities. The codebase is moving away from SQLAlchemy however in favor of
+    its own internal SQL api.
+  - **[api](sql/api)** - An abstracted API for SQL interaction, with support for dbapi compatible drivers (and a 
+    SQLAlchemy adapter).
+  - **[queries](sql/queries)** - A SQL query builder with a fluent interface.
+
+- **[testing](testing)** - Test - primarily pytest - helpers, including:
+
+  - **['harness'](testing/pytest/inject/harness.py)** - An all-in-one fixture marrying it to the codebase's dependency
+    injector.
+  - **[plugins/async](testing/pytest/plugins/asyncs)** - An in-house async-backend abstraction plugin, capable of
+    handling all of asyncio / trio / trio-asyncio / *any-future-event-loop-impl* without having multiple fighting plugins 
+    (*[I know, I know](https://xkcd.com/927/)*).
+  - **[plugins](testing/pytest/plugins)** - Various other plugins.
+
+- **[lite](lite)** - The standard library of 'lite' code, this is the only package beneath `lang`, and parts of it are
+  re-exported by it for deduplication. On top of miscellaneous utilities it contains a handful of independent,
+  self-contained, significantly simplified 'lite' equivalents of some major core packages:
+
+  - **[lite/inject.py](lite/inject.py)** - The lite injector, which is more conservative with features and reflection
+    than the core injector.
+  - **[lite/marshal.py](lite/marshal.py)** - The lite marshalling system, which is a classic canned setup of simple
+    type-specific 2-method classes and limited generic handling.
+
+# Lite code
+
+A subset of this codebase is written in a 'lite' style - while most of the code is written for python 3.12+, 'lite' code
+is written for 3.8+, and is written in a style conducive to
+[amalgamation](https://github.com/wrmsr/omlish/tree/master/omdev#amalgamation), in which multiple python source files are
+stitched together into one single self-contained python script.
+
+Code written in this style has notable differences from standard code, including (but not limited to):
+
+- No name mangling is done in this process, which means (among other things) that code must be written expecting to be
+  all dumped into the same giant namespace. Where a non-lite class might be [`omlish.inject.keys.Key`](inject/keys.py),
+  a lite equivalent might be [`omlish.lite.inject.InjectorKey`](lite/inject.py).
+- All internal imports import each individual item out of modules rather than importing the modules and referencing
+  their contents. Where non-lite code would `from .. import x; x.y`, lite code would `from ..x import y`. As a result
+  there are frequently 'API' non-instantiated namespace classes serving the purpose of modules - just handy bags of
+  stuff with shortened names.
+- As lite code is tested in 3.8+ but core code requires 3.12+, packages containing lite code can't import anything
+  non-lite in their (and their ancestors') `__init__.py`'s. Furthermore, `__init__.py` files are omitted outright in
+  amalgamation, so they effectively must be empty in any package containing any lite code. As a result there are
+  frequently [`all.py`](configs/all.py) files in mixed-lite packages which serve the purpose of `__init__.py` for
+  non-lite usage - where importing non-lite packages from non-lite code would be done via `from .. import lang`,
+  importing mixed-lite packages from non-lite code would be done via `from ..configs import all as cfgs`.
+
+# Dependencies
+
+This package has no required dependencies of any kind, but there are numerous optional integrations - see
+[`__about__.py`](__about__.py) for a full list, but some specific examples are:
+
+- **anyio** - While lite code must use only asyncio, non-trivial async non-lite code prefers to be written to anyio.
+- **pytest** - What is used for all non-lite testing - as lite code has no dependencies of any kind its testing uses
+  stdlib's [unittest](https://docs.python.org/3/library/unittest.html).
+- **sqlalchemy** - Parts of the codebase use SQLAlchemy for db stuff, but it is being migrated away from in favor of the
+  internal api. It will however likely still remain as an optional dep for the api adapter.
+- **asttokens / executing** - For getting runtime source representations of function call arguments, an optional
+  capability of [check](check.py).
+- **wrapt** - For (optionally-enabled) injector circular proxies.
+- **greenlet** - For some gnarly stuff like the [sync<->async bridge](asyncs/bridge.py) and the
+  [io trampoline](io/trampoline.py).
+
+Additionally, some catchall dep categories include:
+
+- **compression** - Various preferred compression backends like lz4, python-snappy, zstandard, and brotli.
+- **formats** - Various preferred data format backends like orjson/ujson, pyyaml, cbor2, and cloudpickle.
+- **sql drivers** - Various preferred and tested sql drivers.

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/omlish/__about__.py

@@ -1,5 +1,5 @@
-__version__ = '0.0.0.dev299'
-__revision__ = '73b5419aa5d43e554a35a615d906559440e76d1b'
+__version__ = '0.0.0.dev301'
+__revision__ = '1567cf674d01bddcf7c1ad426e2677fe25730bd0'
 
 
 #

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/omlish/asyncs/asyncio/subprocesses.py

@@ -191,7 +191,7 @@ class AsyncioSubprocesses(AbstractAsyncSubprocesses):
         async with self.popen(*run.cmd, **kwargs) as proc:
             stdout, stderr = await self.communicate(proc, run.input, run.timeout)
 
-        if check and proc.returncode:
+        if run.check and proc.returncode:
             raise subprocess.CalledProcessError(
                 proc.returncode,
                 run.cmd,

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/omlish/check.py

@@ -1,3 +1,8 @@
+"""
+While the core functionality lives in `lite/check.py` and is available in lite code, this module re-exports the contents
+of its `check` API object as module-globals for convenience. It also sets up some non-lite integration, most notably
+`ArgsRenderer` for rendering the AST's and values of failed checks (if the necessary optional dependencies are present).
+"""
 import typing as ta
 
 from .lite.check import CheckArgsRenderer as ArgsRenderer  # noqa

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/omlish/formats/toml/parser.py

@@ -95,6 +95,7 @@ class TomlDecodeError(ValueError):
 
 def toml_load(fp: ta.BinaryIO, /, *, parse_float: TomlParseFloat = float) -> ta.Dict[str, ta.Any]:
     """Parse TOML from a binary file object."""
+
     b = fp.read()
     try:
         s = b.decode()

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/omlish/lang/collections.py

@@ -12,16 +12,27 @@ V = ta.TypeVar('V')
 def yield_dict_init(*args: ta.Any, **kwargs: ta.Any) -> ta.Iterable[tuple[ta.Any, ta.Any]]:
     if len(args) > 1:
         raise TypeError
+
     if args:
         [src] = args
+
+        # Prefer .items() as it's potentially faster.
         if isinstance(src, collections.abc.Mapping):
-            for k in src:
+            yield from src.items()
+
+        # Support keys() duck-typed dict init behavior:
+        #  https://docs.python.org/3/library/stdtypes.html#dict
+        #  https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Objects/dictobject.c#L2671
+        elif hasattr(src, 'keys'):
+            # https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Objects/dictobject.c#L2927
+            for k in src.keys():  # noqa
                 yield (k, src[k])
+
         else:
             for k, v in src:
                 yield (k, v)
-    for k, v in kwargs.items():
-        yield (k, v)
+
+    yield from kwargs.items()
 
 
 def merge_dicts(

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301/omlish.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omlish
-Version: 0.0.0.dev299
+Version: 0.0.0.dev301
 Summary: omlish
 Author: wrmsr
 License: BSD-3-Clause

{omlish-0.0.0.dev299 → omlish-0.0.0.dev301}/pyproject.toml

@@ -12,7 +12,7 @@ authors = [
 urls = {source = 'https://github.com/wrmsr/omlish'}
 license = {text = 'BSD-3-Clause'}
 requires-python = '>=3.12'
-version = '0.0.0.dev299'
+version = '0.0.0.dev301'
 classifiers = [
     'License :: OSI Approved :: BSD License',
     'Development Status :: 2 - Pre-Alpha',

omlish-0.0.0.dev299/README.md

@@ -1,31 +0,0 @@
-It's like my previous python monorepo-ey thing [`omnibus`](https://github.com/wrmsr/omnibus/tree/wrmsr_exp_split)... ish.
-
-Core packages begin with `om`, scratch app is in `app`, temp / dump code is in `x`.
-
-----
-
-The core packages are:
-
-- **omlish**: core foundational code
-- **omdev**: development utilities
-- **omserv**: production web server
-- **ominfra**: infrastructure and cloud code
-- **ommlx**: ml / ai code
-
-----
-
-Core packages installable from pypi, or from git via:
-
-```bash
-pip install 'git+https://github.com/wrmsr/omlish@master#subdirectory=.pkg/<pkg>'
-```
-
-Core packages have no required dependencies, but numerous optional ones - see their respective `pyproject.toml` files for details.
-
-The CLI is installable through uvx or pipx via:
-
-```bash
-curl -LsSf 'https://raw.githubusercontent.com/wrmsr/omlish/master/omdev/cli/install.py' | python3 -
-```
-
-Additional deps to be injected may be appended to the command.