omlish 0.0.0.dev300__tar.gz → 0.0.0.dev302__tar.gz

{omlish-0.0.0.dev300/omlish.egg-info → omlish-0.0.0.dev302}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omlish
-Version: 0.0.0.dev300
+Version: 0.0.0.dev302
 Summary: omlish
 Author: wrmsr
 License: BSD-3-Clause

omlish-0.0.0.dev302/README.md

@@ -0,0 +1,163 @@
+# Overview
+
+Core utilities and foundational code. It's relatively large but completely self-contained.
+
+# Notable packages
+
+- **[lang](lang)** - The standard library of this standard library. Usually imported as a whole `as lang`, it contains
+  an array of general purpose utilities used practically everywhere. It is kept relatively lightweight: its heaviest
+  import is stdlib dataclasses and its transitives. Some of its contents include:
+
+  - **[cached](lang/cached)** - The standard `cached_function` / `cached_property` tools, which are more capable than
+    [`functools.lru_cache`](https://docs.python.org/3/library/functools.html#functools.lru_cache).
+  - **[imports](lang/imports.py)** - Import tools like `proxy_import` for late-loaded imports and `proxy_init` for
+    late-loaded module globals.
+  - **[classes](lang/classes)** - Class tools and bases, such as `Abstract` (which checks at subclass definition not
+    instantiation), `Sealed` / `PackageSealed`, and `Final`.
+  - **[maybes](lang/maybes.py)** - A simple, nestable formalization of the presence or absence of an object, as in
+    [many](https://en.cppreference.com/w/cpp/utility/optional)
+    [other](https://docs.oracle.com/javase/8/docs/api/java/util/Optional.html)
+    [languages](https://doc.rust-lang.org/std/option/).
+
+- **[bootstrap](bootstrap)** - A centralized, configurable, all-in-one collection of various process-initialization
+  minutiae like resource limiting, profiling, remote debugging, log configuration, environment variables, et cetera.
+  Usable as a context manager or via its [cli](bootstrap/main.py).
+
+- **[collections](collections)** - A handful of collection utilities and simple implementations, including:
+
+  - **[cache](collections/cache)** - A configurable LRU / LFU cache with options like ttl and  max size / weight.
+  - **[hasheq](collections/hasheq.py)** - A dict taking an external `__hash__` / `__eq__` implementation.
+  - **[identity](collections/identity.py)** - Identity-keyed collections.
+  - **[sorted](collections/sorted)** - Interfaces for value-sorted collections and mappings, and a simple but correct
+    skiplist-backed implementation.
+  - **[persistent](collections/persistent)** - Interfaces for [persistent](https://en.wikipedia.org/wiki/Persistent_data_structure)
+    maps, and a simple but correct treap-backed implementation.
+
+- **[dataclasses](dataclasses)** - A fully-compatible reimplementation of stdlib [dataclasses](https://docs.python.org/3/library/dataclasses.html)
+  with numerous enhancements and additional features. The [full stdlib test suite](dataclasses/tests/cpython) is run
+  against it ensuring compatibility - they *are* dataclasses. Current enhancements include:
+
+  - Simple field coercion and validation.
+  - Any number of `@dc.init` or `@dc.validate` methods, not just a central `__post_init__`.
+  - Optional generic type parameter substitution in generated `__init__` methods, enabling accurate reflection.
+  - An optional [metaclass](dataclasses/metaclass) which removes the need for re-decorating subclasses (with support for
+    inheritance of dataclass parameters like `frozen`), and some basic [base classes](dataclasses/metaclass/bases.py).
+  - (Nearly finished) support for ahead-of-time / build-time code generation, greatly reducing import times.
+
+  The stdlib-equivalent api is exported in such a way as to be direct aliases for the stdlib api itself, simplifying
+  tool support.
+
+- **[dispatch](dispatch)** - A beefed-up version of [functools.singledispatch](https://docs.python.org/3/library/functools.html#functools.singledispatch),
+  most notably supporting MRO-honoring method impl dispatch.
+
+- **[formats](formats)** - Tools for various data formats, including:
+
+  - **[json](formats/json)** - Tools for json, including abstraction over various backends and a self-contained streaming
+    / incremental parser.
+  - **[json5](formats/json5)** - A self-contained and tested [Json5](https://json5.org/) parser.
+  - **[toml](formats/toml)** - Toml tools, including a [lite](#lite-code) version of the stdlib parser (for use in older
+    pythons).
+
+- **[http](http)** - HTTP code, including:
+
+  - **[clients](http/clients)** - An abstraction over HTTP clients, with urllib and httpx implementations.
+  - **[coro](http/coro)** - Coroutine / [sans-io](https://sans-io.readthedocs.io/) style reformulation of some stdlib
+    http machinery - namely `http.server` (and soon `http.client`). This style of code can run the same in sync, async,
+    or [any](https://docs.python.org/3/library/selectors.html) [other](asyncs/bluelet) context.
+
+- **[inject](inject)** - A [guice](https://github.com/google/guice)-style dependency injector.
+
+- **[io](io)** - IO tools, including:
+
+  - **[compress](io/compress)** - Abstraction over various compression schemes, with particular attention to incremental
+    operation. For example it includes [an incremental reformulation of stdlib's gzip](io/compress/gzip.py).
+  - **[coro](io/coro)** - Utilities for coroutine / sans-io style code.
+  - **[fdio](io/fdio)** - An implementation of classic [selector](https://docs.python.org/3/library/selectors.html)-style
+    IO dispatch, akin to the deprecated [asyncore](https://docs.python.org/3.11/library/asyncore.html). While more
+    modern asyncio style code is generally preferred, it nearly always involves
+    [background threads](https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Lib/asyncio/unix_events.py#L1349)
+    making it [unsuitable for forking processes](https://rachelbythebay.com/w/2011/06/07/forked/) like
+    [process supervisors](https://github.com/wrmsr/omlish/tree/master/ominfra/supervisor).
+
+- **[jmespath](specs/jmespath)** - A vendoring of [jmespath community edition](https://github.com/jmespath-community/python-jmespath),
+  modernized and adapted to this codebase.
+
+- **[marshal](marshal)** - A [jackson](https://github.com/FasterXML/jackson)-style serde system.
+
+- **[manifests](manifests)** - A system for sharing lightweight metadata within / across codebases.
+
+- **[reflect](reflect)** - Reflection utilities, including primarily a formalization of stdlib type annotations for use
+  at runtime, decoupled from stdlib impl detail. Keeping this working is notoriously difficult across python versions
+  (one of the primary reasons for only supporting 3.12+).
+
+- **[sql](sql)** - A collection of SQL utilities, including:
+
+  - **[alchemy](sql/alchemy)** - SQLAlchemy utilities. The codebase is moving away from SQLAlchemy however in favor of
+    its own internal SQL api.
+  - **[api](sql/api)** - An abstracted api for SQL interaction, with support for dbapi compatible drivers (and a 
+    SQLAlchemy adapter).
+  - **[queries](sql/queries)** - A SQL query builder with a fluent interface.
+
+- **[testing](testing)** - Test - primarily pytest - helpers, including:
+
+  - **['harness'](testing/pytest/inject/harness.py)** - An all-in-one fixture marrying it to the codebase's dependency
+    injector.
+  - **[plugins/async](testing/pytest/plugins/asyncs)** - An in-house async-backend abstraction plugin, capable of
+    handling all of asyncio / trio / trio-asyncio / *any-future-event-loop-impl* without having multiple fighting plugins 
+    (*[I know, I know](https://xkcd.com/927/)*).
+  - **[plugins](testing/pytest/plugins)** - Various other plugins.
+
+- **[lite](lite)** - The standard library of 'lite' code. This is the only package beneath `lang`, and parts of it are
+  re-exported by it for deduplication. On top of miscellaneous utilities it contains a handful of independent,
+  self-contained, significantly simplified 'lite' equivalents of some major core packages:
+
+  - **[lite/inject.py](lite/inject.py)** - The lite injector, which is more conservative with features and reflection
+    than the core injector. The codebase's
+    [MiniGuice](https://github.com/google/guice/commit/70248eafa90cd70a68b293763e53f6aec656e73c).
+  - **[lite/marshal.py](lite/marshal.py)** - The lite marshalling system, which is a classic canned setup of simple
+    type-specific 2-method classes and limited generic handling.
+
+# Lite code
+
+A subset of this codebase is written in a 'lite' style. While most of the code is written for python 3.12+, 'lite' code
+is written for 3.8+, and is written in a style conducive to
+[amalgamation](https://github.com/wrmsr/omlish/tree/master/omdev#amalgamation) in which multiple python source files are
+stitched together into one single self-contained python script.
+
+Code written in this style has notable differences from standard code, including (but not limited to):
+
+- No name mangling is done in amalgamation, which means (among other things) that code must be written expecting to be
+  all dumped into the same giant namespace. Where a non-lite class might be [`omlish.inject.keys.Key`](inject/keys.py),
+  a lite equivalent might be [`omlish.lite.inject.InjectorKey`](lite/inject.py).
+- All internal imports `import` each individual item out of modules rather than importing the modules and referencing
+  their contents. Where non-lite code would `from .. import x; x.y`, lite code would `from ..x import y; y`. As a result
+  there are frequently 'api' non-instantiated namespace classes serving the purpose of modules - just handy bags of
+  stuff with shortened names.
+- As lite code is tested in 3.8+ but core code requires 3.12+, packages containing lite code can't import anything
+  non-lite in their (and their ancestors') `__init__.py`'s. Furthermore, `__init__.py` files are omitted outright in
+  amalgamation, so they effectively must be empty in any package containing any lite code. As a result there are
+  frequently [`all.py`](configs/all.py) files in mixed-lite packages which serve the purpose of `__init__.py` for
+  non-lite usage - where importing non-lite packages from non-lite code would be done via `from .. import lang`,
+  importing mixed-lite packages from non-lite code would be done via `from ..configs import all as cfgs`.
+
+# Dependencies
+
+This library has no required dependencies of any kind, but there are numerous optional integrations - see
+[`__about__.py`](__about__.py) for a full list, but some specific examples are:
+
+- **anyio** - While lite code must use only asyncio, non-trivial async non-lite code prefers to be written to anyio.
+- **pytest** - What is used for all non-lite testing - as lite code has no dependencies of any kind its testing uses
+  stdlib's [unittest](https://docs.python.org/3/library/unittest.html).
+- **sqlalchemy** - Parts of the codebase use SQLAlchemy for db stuff, but it is being migrated away from in favor of the
+  internal api. It will however likely still remain as an optional dep for the api adapter.
+- **asttokens / executing** - For getting runtime source representations of function call arguments, an optional
+  capability of [check](check.py).
+- **wrapt** - For (optionally-enabled) injector circular proxies.
+- **greenlet** - For some gnarly stuff like the [sync<->async bridge](asyncs/bridge.py) and the
+  [io trampoline](io/trampoline.py).
+
+Additionally, some catchall dep categories include:
+
+- **compression** - Various preferred compression backends like lz4, python-snappy, zstandard, and brotli.
+- **formats** - Various preferred data format backends like orjson/ujson, pyyaml, cbor2, and cloudpickle.
+- **sql drivers** - Various preferred and tested sql drivers.

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/__about__.py

@@ -1,5 +1,5 @@
-__version__ = '0.0.0.dev300'
-__revision__ = 'ac78b9a6312ffbb6d660f974c26df0aabc8c9570'
+__version__ = '0.0.0.dev302'
+__revision__ = '0b4f820252a0f7a3e020da512ead1e112d150cd2'
 
 
 #

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/asyncs/anyio/streams.py

@@ -54,8 +54,8 @@ def staple_memory_object_stream(
 ) -> MemoryStapledObjectStream[T]:
     send, receive = args
     return MemoryStapledObjectStream(
-        check.isinstance(send, MemoryObjectSendStream),  # type: ignore
-        check.isinstance(receive, MemoryObjectReceiveStream),  # type: ignore
+        check.isinstance(send, MemoryObjectSendStream),
+        check.isinstance(receive, MemoryObjectReceiveStream),
     )
 
 
@@ -64,6 +64,6 @@ def staple_memory_object_stream(
 def staple_memory_object_stream2[T](max_buffer_size: float = 0) -> MemoryStapledObjectStream[T]:
     send, receive = anyio.create_memory_object_stream[T](max_buffer_size)
     return MemoryStapledObjectStream(
-        check.isinstance(send, MemoryObjectSendStream),  # type: ignore
-        check.isinstance(receive, MemoryObjectReceiveStream),  # type: ignore
+        check.isinstance(send, MemoryObjectSendStream),
+        check.isinstance(receive, MemoryObjectReceiveStream),
     )

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/asyncs/asyncio/subprocesses.py

@@ -191,7 +191,7 @@ class AsyncioSubprocesses(AbstractAsyncSubprocesses):
         async with self.popen(*run.cmd, **kwargs) as proc:
             stdout, stderr = await self.communicate(proc, run.input, run.timeout)
 
-        if check and proc.returncode:
+        if run.check and proc.returncode:
             raise subprocess.CalledProcessError(
                 proc.returncode,
                 run.cmd,

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/asyncs/bridge.py

@@ -1,10 +1,21 @@
 """
-https://github.com/kubernetes/kubernetes/blob/60c4c2b2521fb454ce69dee737e3eb91a25e0535/pkg/controller/volume/persistentvolume/pv_controller.go#L60-L63
+A system for bridging sync and async code. Supports nesting / reentrancy. Robust, but not the most efficient
+implementation - its primary usecase are heavy and init-time ops like lock management and connection establishment, not
+so much for packet-level ops.
 
-==================================================================
-PLEASE DO NOT ATTEMPT TO SIMPLIFY THIS CODE.
-KEEP THE SPACE SHUTTLE FLYING.
-==================================================================
+==
+
+The code is written in a very dumb, simple, explicit style to keep the bookkeeping center-stage and front-of-mind. To
+quote kubernetes:
+
+  https://github.com/kubernetes/kubernetes/blob/60c4c2b2521fb454ce69dee737e3eb91a25e0535/pkg/controller/volume/persistentvolume/pv_controller.go#L60-L63
+
+  ==================================================================
+  PLEASE DO NOT ATTEMPT TO SIMPLIFY THIS CODE.
+  KEEP THE SPACE SHUTTLE FLYING.
+  ==================================================================
+
+==
 
 TODO:
  - reuse greenlet if nested somehow?

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/check.py

@@ -1,3 +1,8 @@
+"""
+While the core functionality lives in `lite/check.py` and is available in lite code, this module re-exports the contents
+of its `check` API object as module-globals for convenience. It also sets up some non-lite integration, most notably
+`ArgsRenderer` for rendering the AST's and values of failed checks (if the necessary optional dependencies are present).
+"""
 import typing as ta
 
 from .lite.check import CheckArgsRenderer as ArgsRenderer  # noqa

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/formats/toml/parser.py

@@ -95,6 +95,7 @@ class TomlDecodeError(ValueError):
 
 def toml_load(fp: ta.BinaryIO, /, *, parse_float: TomlParseFloat = float) -> ta.Dict[str, ta.Any]:
     """Parse TOML from a binary file object."""
+
     b = fp.read()
     try:
         s = b.decode()

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/io/fdio/handlers.py

@@ -7,6 +7,9 @@ from ...lite.check import check
 from ...sockets.addresses import SocketAddress
 
 
+##
+
+
 class FdioHandler(abc.ABC):
     @abc.abstractmethod
     def fd(self) -> int:

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/io/fdio/kqueue.py

@@ -7,6 +7,9 @@ import typing as ta
 from .pollers import FdioPoller
 
 
+##
+
+
 KqueueFdioPoller: ta.Optional[ta.Type[FdioPoller]]
 if sys.platform == 'darwin' or sys.platform.startswith('freebsd'):
 

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/io/fdio/manager.py

@@ -5,6 +5,9 @@ from .handlers import FdioHandler
 from .pollers import FdioPoller
 
 
+##
+
+
 class FdioManager:
     def __init__(
             self,

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/io/trampoline.py

@@ -24,8 +24,8 @@ T = ta.TypeVar('T')
 
 BytesLike: ta.TypeAlias = ta.Any
 
-BufferedReader = io.BufferedReader
-# BufferedReader = pyio.BufferedReader
+BufferedReader: ta.TypeAlias = io.BufferedReader
+# BufferedReader: ta.TypeAlias = pyio.BufferedReader
 
 
 ##

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/lang/collections.py

@@ -12,16 +12,29 @@ V = ta.TypeVar('V')
 def yield_dict_init(*args: ta.Any, **kwargs: ta.Any) -> ta.Iterable[tuple[ta.Any, ta.Any]]:
     if len(args) > 1:
         raise TypeError
+
     if args:
         [src] = args
+
+        # Prefer .items() as it's potentially faster.
         if isinstance(src, collections.abc.Mapping):
-            for k in src:
+            # Not yield-from, explicitly enforce 2ple-ness
+            for k, v in src.items():
+                yield (k, v)
+
+        # Support keys() duck-typed dict init behavior:
+        #  https://docs.python.org/3/library/stdtypes.html#dict
+        #  https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Objects/dictobject.c#L2671
+        elif hasattr(src, 'keys'):
+            # https://github.com/python/cpython/blob/95d9dea1c4ed1b1de80074b74301cee0b38d5541/Objects/dictobject.c#L2927
+            for k in src.keys():  # noqa
                 yield (k, src[k])
+
         else:
             for k, v in src:
                 yield (k, v)
-    for k, v in kwargs.items():
-        yield (k, v)
+
+    yield from kwargs.items()
 
 
 def merge_dicts(

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/lang/imports.py

@@ -1,6 +1,7 @@
 """
 TODO:
  - proxy_init 'as' alias support - attrs of (src, dst)
+ - use importlib.util.resolve_name
 """
 import contextlib
 import functools
@@ -189,6 +190,7 @@ def try_import(spec: str) -> types.ModuleType | None:
 
 
 def resolve_import_name(name: str, package: str | None = None) -> str:
+    # FIXME: importlib.util.resolve_name
     level = 0
 
     if name.startswith('.'):

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/lite/check.py

@@ -143,7 +143,15 @@ class Checks:
             spec = (object,)
         return spec
 
-    def isinstance(self, v: ta.Any, spec: ta.Union[ta.Type[T], tuple], msg: CheckMessage = None) -> T:  # noqa
+    @ta.overload
+    def isinstance(self, v: ta.Any, spec: ta.Type[T], msg: CheckMessage = None) -> T:
+        ...
+
+    @ta.overload
+    def isinstance(self, v: ta.Any, spec: ta.Any, msg: CheckMessage = None) -> ta.Any:
+        ...
+
+    def isinstance(self, v, spec, msg=None):
         if not isinstance(v, self._unpack_isinstance_spec(spec)):
             self._raise(
                 TypeError,
@@ -155,7 +163,15 @@ class Checks:
 
         return v
 
-    def of_isinstance(self, spec: ta.Union[ta.Type[T], tuple], msg: CheckMessage = None) -> ta.Callable[[ta.Any], T]:
+    @ta.overload
+    def of_isinstance(self, spec: ta.Type[T], msg: CheckMessage = None) -> ta.Callable[[ta.Any], T]:
+        ...
+
+    @ta.overload
+    def of_isinstance(self, spec: ta.Any, msg: CheckMessage = None) -> ta.Callable[[ta.Any], ta.Any]:
+        ...
+
+    def of_isinstance(self, spec, msg=None):
         def inner(v):
             return self.isinstance(v, self._unpack_isinstance_spec(spec), msg)
 

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/omlish/subprocesses/editor.py

@@ -1,5 +1,9 @@
 # ruff: noqa: UP006 UP007
 # @omlish-lite
+"""
+TODO:
+  - git var GIT_EDITOR ?
+"""
 import dataclasses as dc
 import os
 import tempfile
@@ -16,6 +20,18 @@ from .run import SubprocessRunOutput
 ##
 
 
+DEFAULT_USER_TEXT_EDITOR = 'vi'
+
+
+def get_user_text_editor(default: ta.Optional[str] = None) -> str:
+    if default is None:
+        default = DEFAULT_USER_TEXT_EDITOR
+    return os.environ.get('EDITOR', default)
+
+
+##
+
+
 class EditTextWithUserEditor(SubprocessRunnable, ExitStacked):
     def __init__(self, initial_text: str = '') -> None:
         super().__init__()
@@ -46,11 +62,10 @@ class EditTextWithUserEditor(SubprocessRunnable, ExitStacked):
 
     def make_run(self) -> SubprocessRun:
         tf = self._temp_file()
-
-        editor = os.environ.get('EDITOR', 'vi')
+        ed = get_user_text_editor()
 
         return SubprocessRun.of(
-            editor,
+            ed,
             tf.path,
         )
 

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302/omlish.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: omlish
-Version: 0.0.0.dev300
+Version: 0.0.0.dev302
 Summary: omlish
 Author: wrmsr
 License: BSD-3-Clause

{omlish-0.0.0.dev300 → omlish-0.0.0.dev302}/pyproject.toml

@@ -12,7 +12,7 @@ authors = [
 urls = {source = 'https://github.com/wrmsr/omlish'}
 license = {text = 'BSD-3-Clause'}
 requires-python = '>=3.12'
-version = '0.0.0.dev300'
+version = '0.0.0.dev302'
 classifiers = [
     'License :: OSI Approved :: BSD License',
     'Development Status :: 2 - Pre-Alpha',

omlish-0.0.0.dev300/README.md

@@ -1,31 +0,0 @@
-It's like my previous python monorepo-ey thing [`omnibus`](https://github.com/wrmsr/omnibus/tree/wrmsr_exp_split)... ish.
-
-Core packages begin with `om`, scratch app is in `app`, temp / dump code is in `x`.
-
-----
-
-The core packages are:
-
-- **omlish**: core foundational code
-- **omdev**: development utilities
-- **omserv**: production web server
-- **ominfra**: infrastructure and cloud code
-- **ommlx**: ml / ai code
-
-----
-
-Core packages installable from pypi, or from git via:
-
-```bash
-pip install 'git+https://github.com/wrmsr/omlish@master#subdirectory=.pkg/<pkg>'
-```
-
-Core packages have no required dependencies, but numerous optional ones - see their respective `pyproject.toml` files for details.
-
-The CLI is installable through uvx or pipx via:
-
-```bash
-curl -LsSf 'https://raw.githubusercontent.com/wrmsr/omlish/master/omdev/cli/install.py' | python3 -
-```
-
-Additional deps to be injected may be appended to the command.