tostr 0.1.0__tar.gz

tostr-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Avery Brown
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tostr-0.1.0/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.4
+Name: tostr
+Version: 0.1.0
+Summary: Token-Optimized Syntax Tree String IR Generator
+Author: Avery Brown
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: tree-sitter>=0.21.0
+Requires-Dist: tree-sitter-java
+Requires-Dist: tree-sitter-c-sharp
+Requires-Dist: tree-sitter-python
+Requires-Dist: google-genai
+Requires-Dist: pydantic
+Requires-Dist: typer
+Requires-Dist: watchfiles
+Requires-Dist: loguru
+Requires-Dist: fastmcp
+Requires-Dist: pathspec
+Dynamic: license-file
+
+<p align="center">
+    <a href="https://toastedtools.com/"><img src="./logo.png" alt="Tostr Logo" width="816"></a>
+</p>
+
+<h1 align="center">
+Pre-computing Agentic AI Code Context
+</h1>
+
+<!-- usage gif goes here -->
+
+<p align="center">
+Tostr is a CLI and MCP agent context engine which greatly reduces token costs and context bloat for agentic LLM coding assistants by pre-computing an llm-described AST in the .tost format
+</p>
+
+# Features
+### Pre-computed Abstract Syntax Tree
+Tostr scrapes your project on initialization, building a comprehensive Abstract Syntax Tree IR (Intermediate Representation) of the entire OOP code structure and stores it in a local SQLite database.
+
+### Semantic Dependency Graph Resolution
+Tostr resolves dependencies between structures in your code, building a dependency graph to allow agents to traverse inbound or outbound method calls efficiently.
+
+### MCP and CLI access
+Tostr has both a CLI and MCP interface, allowing llms to boot up the mcp server for larger development sessions, while allowing agents or human developers to utilize the CLI for individual actions or quick, manual AST traversals.
+
+### Automatic Incremental Change Diffs
+While the MCP server is running, Tostr identifies the subtree of the AST which was updated on file save, add, or delete, then re-scrapes and re-describes exactly the section that was updated, ensuring that the AST is instantly up-to-date during development.
+
+### Lightweight SQLite Cache
+The AST IR and Dependency Graph is cached to an on-drive SQLite .db file to vastly increase efficiency of agent AST traversal requests, as well as allow the AST to be directly queried via sql commands.
+
+# Quick Start
+<!-- downnload and install TBD -->
+
+## Initializing Tostr
+Before being able to use Tostr, the repository must be initialized using the CLI or MCP.
+
+To manually initialize the repository, cd to the root of the project and run:
+```
+% tostr init . --ignore 'default'
+```
+<!--
+This creates the .Tostr directory and initializes the default *.toastignore* to exclude environment files, node_modules, build artifacts, and other files which are not needed in the project AST
+
+In the */.tostr* directory, you will also find the *config.toml* file to configure the other adjustable parameters (The full list of configuration parameters can be found ***Here***)
+-->
+## Parsing the project
+Once the project itself is initialized and configured, the AST cache has to be initialized. To do this manually, cd to the project root (where you initialized the .Tostr files) and run:
+```
+% tostr parse .
+```
+This will take anywhere from a few seconds to a few minutes depending on the size of your repository, as the CLI parses the repository using tree-sitter, then passes the AST concurrently to your configured LLM provider for describing and embedding. Projects with particularly large individual classes will take longer to parse, since the description generation is blocked by class.
+> The time it takes to parse during this step is one time per project, as the incremental diffing allows further parses to only update the cache invalidations
+
+## Using the CLI
+Now that the project is initialized and parsed, Tostr is ready to go!
+
+
+### Project Skeleton
+To test it, navigate to the project root and run:
+```
+% tostr skeleton . --depth 1
+```
+You should see Tostr print the AST skeleton of your root and its direct children directories to your console. 
+> The 'depth' parameter can be adjusted to determine how many layers into the file tree should be skeletonized and printed (default is infinite, or the whole subtree of the path provided)
+
+```
+% tostr skeleton . --depth 1
+
+/src/project/foo.py
+    C-1234 | class Foo(Bar)
+    //  Description of the Foo class, outlining usage and purpose
+        rather than syntax
+
+/src/project/child_dir/fizz.py
+    C-1235 | class Fizz
+    //  Description of the Fizz class...
+    C-1236 | class Buzz(Fizz)
+    //  Description of the Buzz class...
+
+```
+
+### Inspecting Structs
+
+Each of the structs (files, classes, methods) can be inspected further to see more details about them:
+
+```
+% tostr inspect 'C-1234' --pretty
+
+/src/project/foo.py
+C-1234 | class Foo(Bar)
+// Description of the Foo class, outlining usage and purpose
+   rather than syntax
+fields: 
+        int num1, num2; Fizz field3; Buzz field4, field5
+methods:
+        M-12345 @L10-12 | def async foobar(num1: int = 0) -> int
+        // Description of the foobar method...
+        <  child.Fizz#outbound_dependency1(), ~M-12346|#foo(num: int), 
+           ~M-12347|#bar(num:int)
+
+        M-12346 @L22-24 | def foo(num: int) -> int
+        // Description of the foo method...
+
+        M-12347 @L27-30 | def bar(num: int) -> int
+        // Description of the bar method...
+```
+
+You can also use flags to expand the detail of the output:
+* `-v` / `--verbose`: Increases the verbosity of inspect commands to include more information, such as inbound dependencies (`>`) and impact scores.
+* `-b` / `--body`: Attaches the body source code of the root struct being inspected to the bottom of the output.
+*  `-r` / `--raw`: Disables pretty printing, or the indentation and line-wrapping configured in `.Tostr/config.toml`. Pretty printing is active by default for CLI commands but inactive for MCP commands.
+
+```
+% tostr inspect 'M-12345' -v -b
+/src/project/foo.py
+C-1234 | Project.Foo
+M-12345 @L10-20 | def async foobar(num1: int = 0) -> int
+// Description of the foobar method non-truncated
+<  child.Fizz#outbound_dependency1(), ~M-12346|#foo(num: int), 
+   ~M-12347|#bar(num:int)
+>  child.Fizz#inbound_dependency1(num1: int)
+# impact score: 5
+
+'''
+self.field3.inbound_dependency1(num1)
+return num1 + self.foo(num1) + self.bar(num1)
+'''
+```

tostr-0.1.0/README.md

@@ -0,0 +1,128 @@
+<p align="center">
+    <a href="https://toastedtools.com/"><img src="./logo.png" alt="Tostr Logo" width="816"></a>
+</p>
+
+<h1 align="center">
+Pre-computing Agentic AI Code Context
+</h1>
+
+<!-- usage gif goes here -->
+
+<p align="center">
+Tostr is a CLI and MCP agent context engine which greatly reduces token costs and context bloat for agentic LLM coding assistants by pre-computing an llm-described AST in the .tost format
+</p>
+
+# Features
+### Pre-computed Abstract Syntax Tree
+Tostr scrapes your project on initialization, building a comprehensive Abstract Syntax Tree IR (Intermediate Representation) of the entire OOP code structure and stores it in a local SQLite database.
+
+### Semantic Dependency Graph Resolution
+Tostr resolves dependencies between structures in your code, building a dependency graph to allow agents to traverse inbound or outbound method calls efficiently.
+
+### MCP and CLI access
+Tostr has both a CLI and MCP interface, allowing llms to boot up the mcp server for larger development sessions, while allowing agents or human developers to utilize the CLI for individual actions or quick, manual AST traversals.
+
+### Automatic Incremental Change Diffs
+While the MCP server is running, Tostr identifies the subtree of the AST which was updated on file save, add, or delete, then re-scrapes and re-describes exactly the section that was updated, ensuring that the AST is instantly up-to-date during development.
+
+### Lightweight SQLite Cache
+The AST IR and Dependency Graph is cached to an on-drive SQLite .db file to vastly increase efficiency of agent AST traversal requests, as well as allow the AST to be directly queried via sql commands.
+
+# Quick Start
+<!-- downnload and install TBD -->
+
+## Initializing Tostr
+Before being able to use Tostr, the repository must be initialized using the CLI or MCP.
+
+To manually initialize the repository, cd to the root of the project and run:
+```
+% tostr init . --ignore 'default'
+```
+<!--
+This creates the .Tostr directory and initializes the default *.toastignore* to exclude environment files, node_modules, build artifacts, and other files which are not needed in the project AST
+
+In the */.tostr* directory, you will also find the *config.toml* file to configure the other adjustable parameters (The full list of configuration parameters can be found ***Here***)
+-->
+## Parsing the project
+Once the project itself is initialized and configured, the AST cache has to be initialized. To do this manually, cd to the project root (where you initialized the .Tostr files) and run:
+```
+% tostr parse .
+```
+This will take anywhere from a few seconds to a few minutes depending on the size of your repository, as the CLI parses the repository using tree-sitter, then passes the AST concurrently to your configured LLM provider for describing and embedding. Projects with particularly large individual classes will take longer to parse, since the description generation is blocked by class.
+> The time it takes to parse during this step is one time per project, as the incremental diffing allows further parses to only update the cache invalidations
+
+## Using the CLI
+Now that the project is initialized and parsed, Tostr is ready to go!
+
+
+### Project Skeleton
+To test it, navigate to the project root and run:
+```
+% tostr skeleton . --depth 1
+```
+You should see Tostr print the AST skeleton of your root and its direct children directories to your console. 
+> The 'depth' parameter can be adjusted to determine how many layers into the file tree should be skeletonized and printed (default is infinite, or the whole subtree of the path provided)
+
+```
+% tostr skeleton . --depth 1
+
+/src/project/foo.py
+    C-1234 | class Foo(Bar)
+    //  Description of the Foo class, outlining usage and purpose
+        rather than syntax
+
+/src/project/child_dir/fizz.py
+    C-1235 | class Fizz
+    //  Description of the Fizz class...
+    C-1236 | class Buzz(Fizz)
+    //  Description of the Buzz class...
+
+```
+
+### Inspecting Structs
+
+Each of the structs (files, classes, methods) can be inspected further to see more details about them:
+
+```
+% tostr inspect 'C-1234' --pretty
+
+/src/project/foo.py
+C-1234 | class Foo(Bar)
+// Description of the Foo class, outlining usage and purpose
+   rather than syntax
+fields: 
+        int num1, num2; Fizz field3; Buzz field4, field5
+methods:
+        M-12345 @L10-12 | def async foobar(num1: int = 0) -> int
+        // Description of the foobar method...
+        <  child.Fizz#outbound_dependency1(), ~M-12346|#foo(num: int), 
+           ~M-12347|#bar(num:int)
+
+        M-12346 @L22-24 | def foo(num: int) -> int
+        // Description of the foo method...
+
+        M-12347 @L27-30 | def bar(num: int) -> int
+        // Description of the bar method...
+```
+
+You can also use flags to expand the detail of the output:
+* `-v` / `--verbose`: Increases the verbosity of inspect commands to include more information, such as inbound dependencies (`>`) and impact scores.
+* `-b` / `--body`: Attaches the body source code of the root struct being inspected to the bottom of the output.
+*  `-r` / `--raw`: Disables pretty printing, or the indentation and line-wrapping configured in `.Tostr/config.toml`. Pretty printing is active by default for CLI commands but inactive for MCP commands.
+
+```
+% tostr inspect 'M-12345' -v -b
+/src/project/foo.py
+C-1234 | Project.Foo
+M-12345 @L10-20 | def async foobar(num1: int = 0) -> int
+// Description of the foobar method non-truncated
+<  child.Fizz#outbound_dependency1(), ~M-12346|#foo(num: int), 
+   ~M-12347|#bar(num:int)
+>  child.Fizz#inbound_dependency1(num1: int)
+# impact score: 5
+
+'''
+self.field3.inbound_dependency1(num1)
+return num1 + self.foo(num1) + self.bar(num1)
+'''
+```

tostr-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tostr"
+version = "0.1.0"
+authors = [
+    { name = "Avery Brown" }
+]
+description = "Token-Optimized Syntax Tree String IR Generator"
+readme = "README.md"
+requires-python = ">=3.9"
+dependencies = [
+    "tree-sitter>=0.21.0",
+    "tree-sitter-java",
+    "tree-sitter-c-sharp",
+    "tree-sitter-python",
+    "google-genai",
+    "pydantic",
+    "typer",
+    "watchfiles",
+    "loguru",
+    "fastmcp",
+    "pathspec"
+]
+
+[project.scripts]
+tostr = "tostr.cli:app"
+
+[tool.pytest.ini_options]
+pythonpath = ["src"]

tostr-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tostr-0.1.0/src/tostr/cli.py

@@ -0,0 +1,329 @@
+import asyncio
+import time
+from pathlib import Path
+import typer
+from typing import Annotated, List
+from loguru import logger
+from tostr.exceptions import TostrError
+
+from tostr.commands import (
+    init_async, 
+    inspect_async, 
+    skeleton_async, 
+    watch_async, 
+    clean_db,
+    resolve_uid_to_id
+)
+
+from tostr.server import mcp
+
+from tostr.core.utils.logger import configure_cli_logging
+
+import multiprocessing
+
+
+# Initialize the Typer app
+app = typer.Typer(
+    name="tostr",
+    help="AST scraper for LLM RAG context generation.",
+    add_completion=False # Optional: Turns off the auto-generated completion install command for cleaner help menus
+)
+
+def _run_watcher_thread(target_path: Path):
+    """
+    Sets up an isolated async environment for the background thread.
+    """
+    loop = asyncio.new_event_loop()
+    asyncio.set_event_loop(loop)
+    
+    try:
+        logger.info(f"Background watcher started on {target_path}")
+        loop.run_until_complete(watch_async(target_path))
+    except Exception as e:
+        logger.exception(f"Fatal error in background watcher: {e}")
+    finally:
+        loop.close()
+        logger.info("Background watcher shut down.")
+
+@app.command("start-mcp")
+def start_mcp():
+    """Start the bare MCP server. Awaits agent initialization."""
+    mcp.run()
+
+@app.command()
+def watch(
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,       # Typer automatically checks if the path exists
+        file_okay=False,   # Typer blocks files, only allowing directories
+        dir_okay=True,
+        resolve_path=True  # Converts relative paths to absolute paths automatically
+    ),
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False
+):
+    """Watch for changes to files and update the SQLite database."""
+    configure_cli_logging(debug)
+    try:
+        asyncio.run(watch_async(path))
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+
+@app.command()
+def clean(
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,       # Typer automatically checks if the path exists
+        file_okay=False,   # Typer blocks files, only allowing directories
+        dir_okay=True,
+        resolve_path=True  # Converts relative paths to absolute paths automatically
+    ),
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False
+):
+    """Clean the SQLite database."""
+    configure_cli_logging(debug)
+    try:
+        clean_db(path)
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+
+@app.command()
+def init(
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,       # Typer automatically checks if the path exists
+        file_okay=False,   # Typer blocks files, only allowing directories
+        dir_okay=True,
+        resolve_path=True  # Converts relative paths to absolute paths automatically
+    ),
+    use_cache: Annotated[
+        bool, 
+        typer.Option(
+            "--use-cache/--no-cache", 
+            help="Load cache if it exists"
+            )
+        ] = True,
+    ignore: Annotated[
+        str,
+        typer.Option(
+            "--ignore",
+            "-i",
+            help="Add a default ignore template to the project folder (e.g., 'java', 'default')"
+        )
+    ] = None,
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False
+):
+    """Parse files and setup SQLite database."""
+    configure_cli_logging(debug)
+    start_time = time.perf_counter()
+    try:
+        asyncio.run(init_async(path, use_cache, ignore))
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+    
+    end_time = time.perf_counter()
+    elapsed_time = end_time - start_time
+    logger.debug(f"Finished in {elapsed_time:.4f} seconds.")
+
+
+@app.command()
+def resolve(
+    uid: Annotated[
+        str, 
+        typer.Argument(help="The UID to resolve to an ID")
+    ],
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,
+        file_okay=False,
+        dir_okay=True,
+        resolve_path=True
+    ),
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False
+):
+    """Resolve a UID to its corresponding struct ID."""
+    configure_cli_logging(debug)
+    try:
+        result = resolve_uid_to_id(uid, path)
+        if result:
+            print(result)
+        else:
+            typer.secho(f"❌ Error: No struct found with UID '{uid}'", fg="red", err=True)
+            raise typer.Exit(code=1)
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+
+@app.command()
+def inspect(
+    ids: Annotated[
+        List[str], 
+        typer.Argument(help="List of struct IDs or UIDs to inspect")
+    ],
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,       # Typer automatically checks if the path exists
+        file_okay=False,   # Typer blocks files, only allowing directories
+        dir_okay=True,
+        resolve_path=True  # Converts relative paths to absolute paths automatically
+    ),
+    include_body: Annotated[
+        bool, 
+        typer.Option(
+            "--body/--no-body", 
+            help="Include code body in output"
+            )
+        ] = False,
+    pretty: Annotated[
+        bool,
+        typer.Option(
+            "--pretty/--raw",
+            help="Pretty format output with line wrapping and indentation (disable for raw output)"
+        )
+    ] = True,
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False,
+    max_lines: Annotated[
+        int,
+        typer.Option(
+            "--max-lines",
+            "-m",
+            help="Maximum number of lines to include in the output (default: 500)"
+        )
+    ] = 500
+):
+    """Output the AST details for specific struct IDs or UIDs."""
+    configure_cli_logging(debug)
+    
+    start_time = time.perf_counter()
+    try:
+        result = asyncio.run(inspect_async(ids, path, include_body=include_body, pretty=pretty))
+        lines = result.splitlines()
+        if len(lines) > max_lines:
+            result = "\n".join(lines[:max_lines]) + "\n...[OUTPUT TRUNCATED AT 500 LINES] - Use a higher '--max-lines <N>' to see more."
+        print(result)
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+    
+    end_time = time.perf_counter()
+    elapsed_time = end_time - start_time
+    logger.debug(f"Finished in {elapsed_time:.4f} seconds.")
+
+
+@app.command()
+def skeleton(
+    subpath: Annotated[
+        str, 
+        typer.Argument(help="File or directory path relative to the project root to generate a skeleton for")
+    ] = ".",
+    path: Path = typer.Argument(
+        ".", 
+        help="Path to the project directory to scan",
+        exists=True,
+        file_okay=False,
+        dir_okay=True,
+        resolve_path=True
+    ),
+    pretty: Annotated[
+        bool,
+        typer.Option(
+            "--pretty/--raw",
+            help="Pretty format output with line wrapping and indentation (disable for raw output)"
+        )
+    ] = True,
+    debug: Annotated[
+        bool, 
+        typer.Option(
+            "--debug/--no-debug", 
+            "-d/-nd",
+            help="Enable debug logging"
+            )
+    ] = False,
+    depth: Annotated[
+        int,
+        typer.Option(
+            "--depth",
+            "-d",
+            help="Depth to traverse for skeleton generation (default: 7)"
+        )
+    ] = 7, 
+    files_only: Annotated[
+        bool,
+        typer.Option(
+            "--files-only",
+            "-f",
+            help="Only generate skeleton for files (default: False)"
+        )
+    ] = False,
+    max_lines: Annotated[
+        int,
+        typer.Option(
+            "--max-lines",
+            "-m",
+            help="Maximum number of lines to include in the output (default: 500)"
+        )
+    ] = 500
+    
+):
+    """Output the .tost skeleton format for all files matching a specific subpath."""
+    configure_cli_logging(debug)
+    
+    start_time = time.perf_counter()
+    try:
+        result = asyncio.run(skeleton_async(subpath, path, pretty=pretty, depth=depth, files_only=files_only))
+        lines = result.splitlines()
+        if len(lines) > max_lines:
+            result = "\n".join(lines[:max_lines]) + "\n...[OUTPUT TRUNCATED AT 500 LINES] - Use a higher '--max-lines <N>' to see more."
+        print(result)
+    except TostrError as e:
+        typer.secho(f"❌ Error: {e}", fg="red", err=True)
+        raise typer.Exit(code=1)
+    end_time = time.perf_counter()
+    elapsed_time = end_time - start_time
+    logger.debug(f"Finished in {elapsed_time:.4f} seconds.")
+
+if __name__ == "__main__":
+    multiprocessing.freeze_support()
+    app()