graphitect 0.2.0__py3-none-any.whl

graphify/extractors/MIGRATION.md

@@ -0,0 +1,107 @@
+# Migrating a language extractor out of extract.py
+
+`graphify/extract.py` is being split into this package, one language per PR
+(upstream issue #1212). This is the playbook for porting ONE language. It is
+written so an AI agent can execute it in a single session.
+
+## Status
+
+| module | migrated |
+|---|---|
+| blade | yes |
+| zig | yes |
+| elixir | yes |
+| razor | yes |
+| dart | yes |
+| rust | yes |
+| go | yes |
+| powershell (ps1 + psd1 manifest) | yes |
+| fortran | yes |
+| sql | yes |
+| dm (dm/dmm/dmi/dmf) | yes |
+| bash | yes |
+| apex | yes |
+| terraform | yes |
+| sln | yes |
+| pascal_forms (dfm + lfm) | yes |
+| json_config | yes |
+| (config-driven core: python, js, java, c, cpp, csharp, kotlin, scala, php, lua, swift, groovy, vue, svelte, astro, xaml, groovy) | no — shared _extract_generic core, move as one batch |
+| (other bespoke: julia, verilog, markdown, objc, csproj, slnx, lazarus_package, pascal) | no |
+
+Note: config-driven extractors (python, js, java, c, cpp, ruby, csharp,
+kotlin, scala, php, lua, swift, groovy) depend on the shared
+`_extract_generic` core (~1,300 lines). Do NOT port them one-by-one; the core
+must move first as its own coordinated batch. Pick a bespoke extractor.
+
+## Invariants (non-negotiable)
+
+1. **Verbatim moves only.** No renames, no docstring edits, no reformatting,
+   no added annotations, no "improvements". Verify: save the block before
+   cutting and confirm the pasted block is byte-identical.
+2. **One language per PR.** Small diffs keep review trivial and avoid
+   conflicts with other in-flight ports.
+3. **Facade re-export is mandatory.** `extract.py` must keep exporting every
+   moved name (`from graphify.extractors.<mod> import extract_<lang>  # noqa: F401`
+   in the marked migration block, kept alphabetical). Existing importers
+   (`__main__.py`, `watch.py`, `pg_introspect.py`, tests) must not change.
+4. **Never import from `graphify.extract` inside this package.** Import
+   direction is strictly extract.py -> extractors/. If you need a helper that
+   lives in extract.py, classify it (below) and move it.
+5. **Zero test edits** outside `tests/test_extractors_registry.py`. The
+   untouched language tests passing IS the proof of behavior preservation.
+
+## Helper classification
+
+For every `_name` your function references that is defined OUTSIDE it:
+
+- run `grep -c '_name' graphify/extract.py` AFTER your candidate move;
+- remaining uses > 0 -> **shared**: move it to `base.py` and add it to the
+  facade re-import in extract.py;
+- remaining uses = 0 -> **private**: move it into your language module.
+
+Closures, constants, and `import` statements defined INSIDE your function
+move with it for free — leave them exactly where they are. Only add a
+module-header import for names the pasted code references at module scope
+that are not satisfied internally, and verify each header import is used.
+
+## Pre-flight
+
+1. Check upstream for conflicts: open PRs/issues mentioning your language,
+   and churn: `git log --oneline --since="3 months ago" upstream/<default> | grep -i <lang>`.
+   High churn -> pick another language.
+2. Confirm your extractor is bespoke (its `extract_<lang>` is a full function,
+   not a 5-line `_extract_generic(path, LanguageConfig(...))` wrapper).
+3. Check whether tests/ exercises your language's behavior (grep for
+   `test_<lang>`). If it has no behavioral tests, the byte-identity check in
+   step 3 below is the ENTIRE proof of preservation — include the
+   `git diff --color-moved` evidence in your PR description.
+
+## Steps
+
+1. Append a failing test to `tests/test_extractors_registry.py`:
+   module import + facade identity + registry identity (copy an existing
+   `test_<lang>_migrated` as the template).
+2. `grep -n 'def extract_<lang>' graphify/extract.py`; the span ends at the
+   line before the next top-level statement (`^def ` or `^_CONST`). Beware
+   neighbors: top-level constants AFTER your function may belong to the NEXT
+   function (e.g. `_CONFIG_JSON_*` sit after where extract_razor used to be
+   but were never razor's).
+3. Save the span to a temp file. Create `graphify/extractors/<lang>.py` with
+   module docstring (`"""<Lang> extractor. Moved verbatim from graphify/extract.py."""`),
+   `from __future__ import annotations`, minimal stdlib imports, base imports,
+   then paste the function. Verify byte-identity against the temp file.
+4. Delete the span from extract.py, leaving exactly two blank lines between
+   the now-adjacent top-level definitions; add the facade re-import; add the
+   registry entry in `__init__.py` (alphabetical); update the Status table
+   above.
+5. `uv run pytest -q` -> 0 failures, no test file changed except the registry
+   test. If ImportError/NameError: a helper was misclassified — go to
+   Helper classification.
+6. One commit: `refactor(extract): move extract_<lang> to extractors/<lang>.py (verbatim)`.
+
+## What NOT to do
+
+- Do not rewire dispatch, add classes, or add lazy imports — mechanism layers
+  come later, by separate agreement (see #1212 discussion).
+- Do not port two languages in one PR "while you're at it".
+- Do not touch `__main__.py`.

graphify/extractors/__init__.py

@@ -0,0 +1,66 @@
+"""Per-language extractors, incrementally migrated out of graphify/extract.py.
+
+Dispatch still flows through graphify.extract (the facade re-exports every
+moved name), so importing from graphify.extract keeps working unchanged.
+LANGUAGE_EXTRACTORS is the registry seed; wiring dispatch through it is a
+later, separate step. See MIGRATION.md for how to port another language.
+"""
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Callable
+
+from graphify.extractors.apex import extract_apex
+from graphify.extractors.bash import extract_bash
+from graphify.extractors.blade import extract_blade
+from graphify.extractors.commonlisp import extract_commonlisp
+from graphify.extractors.dart import extract_dart
+from graphify.extractors.dm import extract_dm, extract_dmf, extract_dmi, extract_dmm
+from graphify.extractors.elixir import extract_elixir
+from graphify.extractors.fortran import extract_fortran
+from graphify.extractors.go import extract_go
+from graphify.extractors.json_config import extract_json
+from graphify.extractors.julia import extract_julia
+from graphify.extractors.markdown import extract_markdown
+from graphify.extractors.objc import extract_objc
+from graphify.extractors.pascal import extract_pascal
+from graphify.extractors.pascal_forms import extract_delphi_form, extract_lazarus_form
+from graphify.extractors.powershell import extract_powershell, extract_powershell_manifest
+from graphify.extractors.razor import extract_razor
+from graphify.extractors.rust import extract_rust
+from graphify.extractors.sln import extract_sln
+from graphify.extractors.sql import extract_sql
+from graphify.extractors.terraform import extract_terraform
+from graphify.extractors.verilog import extract_verilog
+from graphify.extractors.zig import extract_zig
+
+LANGUAGE_EXTRACTORS: dict[str, Callable[[Path], dict]] = {
+    "apex": extract_apex,
+    "bash": extract_bash,
+    "blade": extract_blade,
+    "commonlisp": extract_commonlisp,
+    "dart": extract_dart,
+    "delphi_form": extract_delphi_form,
+    "dm": extract_dm,
+    "dmf": extract_dmf,
+    "dmi": extract_dmi,
+    "dmm": extract_dmm,
+    "elixir": extract_elixir,
+    "fortran": extract_fortran,
+    "go": extract_go,
+    "json": extract_json,
+    "julia": extract_julia,
+    "lazarus_form": extract_lazarus_form,
+    "markdown": extract_markdown,
+    "objc": extract_objc,
+    "pascal": extract_pascal,
+    "powershell": extract_powershell,
+    "powershell_manifest": extract_powershell_manifest,
+    "razor": extract_razor,
+    "rust": extract_rust,
+    "sln": extract_sln,
+    "sql": extract_sql,
+    "terraform": extract_terraform,
+    "verilog": extract_verilog,
+    "zig": extract_zig,
+}

graphify/extractors/apex.py

@@ -0,0 +1,215 @@
+"""Apex extractor. Moved verbatim from graphify/extract.py."""
+from __future__ import annotations
+
+
+from pathlib import Path
+from graphify.extractors.base import _file_stem, _make_id
+
+
+def extract_apex(path: Path) -> dict:
+    """Extract classes, interfaces, enums, methods, and Salesforce constructs from
+    Apex .cls and .trigger files using regex (no tree-sitter grammar on PyPI)."""
+    import re as _re
+    try:
+        source = path.read_text(encoding="utf-8", errors="replace")
+    except OSError:
+        return {"nodes": [], "edges": []}
+
+    str_path = str(path)
+    stem = _file_stem(path)
+    file_nid = _make_id(str_path)
+
+    nodes: list[dict] = []
+    edges: list[dict] = []
+    seen_ids: set[str] = set()
+
+    def add_node(nid: str, label: str, line: int) -> None:
+        if nid not in seen_ids:
+            seen_ids.add(nid)
+            nodes.append({
+                "id": nid,
+                "label": label,
+                "file_type": "code",
+                "source_file": str_path,
+                "source_location": f"L{line}",
+            })
+
+    def add_edge(src: str, tgt: str, relation: str, line: int,
+                 confidence: str = "EXTRACTED") -> None:
+        edges.append({
+            "source": src,
+            "target": tgt,
+            "relation": relation,
+            "confidence": confidence,
+            "source_file": str_path,
+            "source_location": f"L{line}",
+            "weight": 1.0,
+        })
+
+    add_node(file_nid, path.name, 1)
+
+    lines = source.splitlines()
+
+    _ACCESS = r"(?:public|private|protected|global|webService)?"
+    _SHARING = r"(?:\s+(?:with|without|inherited)\s+sharing)?"
+    _MOD = r"(?:\s+(?:abstract|virtual|override|static|final|transient|testMethod))?"
+    _ANNOTATION = r"(?:\s*@\w+(?:\s*\([^)]*\))?\s*)*"
+
+    cls_re = _re.compile(
+        rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*class\s+(\w+)"
+        rf"(?:\s+extends\s+(\w+))?(?:\s+implements\s+([\w,\s]+))?\s*\{{?",
+        _re.IGNORECASE,
+    )
+    iface_re = _re.compile(
+        rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*interface\s+(\w+)"
+        rf"(?:\s+extends\s+([\w,\s]+))?\s*\{{?",
+        _re.IGNORECASE,
+    )
+    enum_re = _re.compile(
+        rf"^{_ANNOTATION}\s*{_ACCESS}{_SHARING}{_MOD}\s*enum\s+(\w+)\s*\{{?",
+        _re.IGNORECASE,
+    )
+    trigger_re = _re.compile(
+        r"^\s*trigger\s+(\w+)\s+on\s+(\w+)\s*\(",
+        _re.IGNORECASE,
+    )
+    method_re = _re.compile(
+        rf"^{_ANNOTATION}\s*{_ACCESS}{_MOD}\s*(?:static\s+)?[\w<>\[\]]+\s+(\w+)\s*\([^)]*\)\s*(?:throws\s+\w+\s*)?\{{?",
+        _re.IGNORECASE,
+    )
+    annotation_re = _re.compile(r"@(\w+)", _re.IGNORECASE)
+    soql_re = _re.compile(r"\[\s*SELECT\b[^\]]+FROM\s+(\w+)", _re.IGNORECASE)
+    dml_re = _re.compile(r"\b(insert|update|delete|upsert|merge|undelete)\s+\w", _re.IGNORECASE)
+
+    _CONTROL_FLOW = frozenset({
+        "if", "else", "for", "while", "do", "switch", "try", "catch",
+        "finally", "return", "throw", "new", "void", "null",
+        "true", "false", "this", "super", "class", "interface", "enum",
+        "trigger", "on",
+    })
+
+    current_class_nid: str | None = None
+    pending_annotations: list[str] = []
+
+    for lineno, line_text in enumerate(lines, start=1):
+        stripped = line_text.strip()
+
+        if stripped.startswith("@"):
+            for m in annotation_re.finditer(stripped):
+                pending_annotations.append(m.group(1).lower())
+            continue
+
+        tm = trigger_re.match(stripped)
+        if tm:
+            trig_name, sobject = tm.group(1), tm.group(2)
+            trig_nid = _make_id(stem, trig_name)
+            add_node(trig_nid, trig_name, lineno)
+            add_edge(file_nid, trig_nid, "contains", lineno)
+            sob_nid = _make_id(sobject)
+            if sob_nid not in seen_ids:
+                add_node(sob_nid, sobject, lineno)
+            add_edge(trig_nid, sob_nid, "uses", lineno, confidence="INFERRED")
+            current_class_nid = trig_nid
+            pending_annotations = []
+            continue
+
+        cm = cls_re.match(stripped)
+        if cm:
+            class_name = cm.group(1)
+            if class_name.lower() in _CONTROL_FLOW:
+                pending_annotations = []
+                continue
+            class_nid = _make_id(stem, class_name)
+            add_node(class_nid, class_name, lineno)
+            add_edge(file_nid, class_nid, "contains", lineno)
+            if cm.group(2):
+                base = cm.group(2).strip()
+                base_nid = _make_id(stem, base)
+                if base_nid not in seen_ids:
+                    base_nid = _make_id(base)
+                if base_nid not in seen_ids:
+                    add_node(base_nid, base, lineno)
+                add_edge(class_nid, base_nid, "extends", lineno, confidence="INFERRED")
+            if cm.group(3):
+                for iface in cm.group(3).split(","):
+                    iface = iface.strip()
+                    if iface:
+                        iface_nid = _make_id(stem, iface)
+                        if iface_nid not in seen_ids:
+                            iface_nid = _make_id(iface)
+                        if iface_nid not in seen_ids:
+                            add_node(iface_nid, iface, lineno)
+                        add_edge(class_nid, iface_nid, "implements", lineno, confidence="INFERRED")
+            current_class_nid = class_nid
+            pending_annotations = []
+            continue
+
+        im = iface_re.match(stripped)
+        if im:
+            iface_name = im.group(1)
+            if iface_name.lower() in _CONTROL_FLOW:
+                pending_annotations = []
+                continue
+            iface_nid = _make_id(stem, iface_name)
+            add_node(iface_nid, iface_name, lineno)
+            add_edge(file_nid if current_class_nid is None else current_class_nid,
+                     iface_nid, "contains", lineno)
+            if im.group(2):
+                for parent in im.group(2).split(","):
+                    parent = parent.strip()
+                    if parent:
+                        parent_nid = _make_id(stem, parent)
+                        if parent_nid not in seen_ids:
+                            parent_nid = _make_id(parent)
+                        if parent_nid not in seen_ids:
+                            add_node(parent_nid, parent, lineno)
+                        add_edge(iface_nid, parent_nid, "extends", lineno, confidence="INFERRED")
+            pending_annotations = []
+            continue
+
+        em = enum_re.match(stripped)
+        if em:
+            enum_name = em.group(1)
+            if enum_name.lower() in _CONTROL_FLOW:
+                pending_annotations = []
+                continue
+            enum_nid = _make_id(stem, enum_name)
+            add_node(enum_nid, enum_name, lineno)
+            add_edge(file_nid if current_class_nid is None else current_class_nid,
+                     enum_nid, "contains", lineno)
+            pending_annotations = []
+            continue
+
+        if current_class_nid is not None:
+            mm = method_re.match(stripped)
+            if mm:
+                method_name = mm.group(1)
+                if method_name.lower() not in _CONTROL_FLOW:
+                    method_nid = _make_id(current_class_nid, method_name)
+                    method_label = f".{method_name}()"
+                    add_node(method_nid, method_label, lineno)
+                    add_edge(current_class_nid, method_nid, "method", lineno)
+                    if "auraenabled" in pending_annotations or "invocablemethod" in pending_annotations:
+                        add_edge(file_nid, method_nid, "contains", lineno, confidence="INFERRED")
+                    pending_annotations = []
+                    continue
+
+        pending_annotations = []
+
+        for sm in soql_re.finditer(line_text):
+            sobject = sm.group(1)
+            sob_nid = _make_id(sobject)
+            if sob_nid not in seen_ids:
+                add_node(sob_nid, sobject, lineno)
+            src = current_class_nid or file_nid
+            add_edge(src, sob_nid, "uses", lineno, confidence="INFERRED")
+
+        for dm in dml_re.finditer(line_text):
+            dml_op = dm.group(1).lower()
+            dml_nid = _make_id(f"dml_{dml_op}")
+            if dml_nid not in seen_ids:
+                add_node(dml_nid, dml_op, lineno)
+            src = current_class_nid or file_nid
+            add_edge(src, dml_nid, "uses", lineno, confidence="INFERRED")
+
+    return {"nodes": nodes, "edges": edges}

graphify/extractors/base.py

@@ -0,0 +1,85 @@
+# DO NOT import from graphify.extract here — direction is extract.py → extractors/ only.
+from __future__ import annotations
+
+from pathlib import Path
+
+from graphify.ids import make_id
+
+# Language built-in globals that AST may classify as call targets when used as
+# constructors or coercion functions (e.g. String(x), Number(x), Boolean(x)).
+# Without this filter they become god-nodes accumulating spurious edges from
+# every call site. Filter applied at same-file and cross-file resolution.
+# See issue #726.
+_LANGUAGE_BUILTIN_GLOBALS: frozenset[str] = frozenset({
+    # JavaScript / TypeScript ECMAScript built-ins
+    "String", "Number", "Boolean", "Object", "Array", "Symbol", "BigInt",
+    "Date", "RegExp", "Error", "TypeError", "RangeError", "SyntaxError",
+    "ReferenceError", "EvalError", "URIError",
+    "Promise", "Map", "Set", "WeakMap", "WeakSet", "JSON", "Math",
+    "Reflect", "Proxy", "Intl",
+    "parseInt", "parseFloat", "isNaN", "isFinite",
+    "encodeURIComponent", "decodeURIComponent", "encodeURI", "decodeURI",
+    # Browser / Node common globals
+    "URL", "URLSearchParams", "FormData", "Blob", "File",
+    "Headers", "Request", "Response", "AbortController", "AbortSignal",
+    "TextEncoder", "TextDecoder", "console",
+    # Python built-in callables
+    "str", "int", "float", "bool", "list", "dict", "set", "tuple", "bytes",
+    "len", "range", "enumerate", "zip", "map", "filter", "sum", "min", "max",
+    "print", "open", "isinstance", "type", "super", "sorted", "reversed",
+    "any", "all", "abs", "round", "next", "iter", "hash", "id", "repr",
+    "callable", "getattr", "setattr", "hasattr", "delattr", "vars", "dir",
+    # Swift standard library / Foundation / SwiftUI (#2147). Value-type
+    # initializers (Data(x), Int(x), UUID()) and protocol conformance targets
+    # appear from virtually every file of a Swift codebase, exactly like the
+    # ECMAScript constructors above. String/Date/URL/Error are already listed.
+    "Int", "Int8", "Int16", "Int32", "Int64",
+    "UInt", "UInt8", "UInt16", "UInt32", "UInt64",
+    "Double", "Float", "Bool", "Character",
+    "Sendable", "Codable", "Decodable", "Encodable", "Equatable", "Hashable",
+    "Identifiable", "Comparable", "CaseIterable", "RawRepresentable",
+    "CustomStringConvertible", "CustomDebugStringConvertible", "AnyObject",
+    "LocalizedError",
+    "Data", "UUID", "Decimal", "Calendar", "Locale", "TimeZone", "Bundle",
+    "IndexPath", "IndexSet", "NotificationCenter", "UserDefaults",
+    "FileManager", "URLSession", "URLRequest", "URLComponents",
+    "JSONDecoder", "JSONEncoder", "DateFormatter", "NumberFormatter",
+    "ISO8601DateFormatter",
+    "NSObject", "NSString", "NSError", "NSLock", "NSAttributedString",
+    "DispatchQueue", "DispatchGroup", "OperationQueue", "RunLoop",
+    "View", "Color", "Font",
+})
+
+
+def _make_id(*parts: str) -> str:
+    return make_id(*parts)
+
+
+def _file_stem(path: Path) -> str:
+    """Stem used as the node-ID prefix for a file and its symbols.
+
+    The full path (extension dropped) is preserved as path segments; ``make_id``
+    later collapses the separators to underscores. Using every segment — not just
+    the immediate parent dir (#1504) — means same-named files in different
+    directories get distinct IDs instead of colliding into one
+    last-writer-wins node:
+
+        docs/v1/api/README.md -> docs/v1/api/README -> docs_v1_api_readme
+        docs/v2/api/README.md -> docs/v2/api/README -> docs_v2_api_readme
+
+    Top-level files keep a bare stem (``setup.py`` -> ``setup``). When passed an
+    absolute path the whole path is encoded; the extract() id-remap post-pass
+    re-derives the canonical repo-relative form from ``source_file`` so the on-disk
+    location can't leak into the persisted IDs (#502).
+
+    Returns "" for a path with no name (``Path('.')`` — a source_file that equals
+    the scan root, so it has no per-file stem). Guarding here keeps
+    ``path.with_suffix("")`` from raising ``ValueError: '.' has an empty name`` and
+    protects every caller, not just ``_semantic_id_remap`` (#1618)."""
+    if not path.name:
+        return ""
+    return path.with_suffix("").as_posix()
+
+
+def _read_text(node, source: bytes) -> str:
+    return source[node.start_byte:node.end_byte].decode("utf-8", errors="replace")