semanticembed 0.2.1__tar.gz → 0.7.0__tar.gz

semanticembed-0.7.0/.github/workflows/publish.yml

@@ -0,0 +1,51 @@
+name: Publish to PyPI
+
+# Triggers on a tag push of the form `v*` — e.g. `git tag v0.6.0 && git push --tags`.
+# Uses PyPI Trusted Publishing (OIDC) so no API token is stored as a GitHub secret.
+# One-time setup: pypi.org -> Account -> Publishing -> Add pending publisher
+#   PyPI Project Name: semanticembed
+#   Owner:             jmurray10
+#   Repository:        semanticembed-sdk
+#   Workflow name:     publish.yml
+#   Environment:       pypi   (matches the `environment:` line below)
+
+on:
+  push:
+    tags:
+      - "v*"
+  workflow_dispatch:  # allow manual run from the Actions tab
+
+jobs:
+  build-and-publish:
+    name: Build and publish
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # required for trusted publishing
+      contents: read
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Install build tooling
+        run: python -m pip install --upgrade build
+
+      - name: Verify the tag matches the package version
+        run: |
+          TAG_VERSION="${GITHUB_REF#refs/tags/v}"
+          PKG_VERSION=$(python -c "import tomllib; print(tomllib.loads(open('pyproject.toml','rb').read().decode())['project']['version'])")
+          if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
+            echo "Tag version ($TAG_VERSION) does not match pyproject.toml ($PKG_VERSION)" >&2
+            exit 1
+          fi
+
+      - name: Build sdist + wheel
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

semanticembed-0.7.0/CHANGELOG.md

@@ -0,0 +1,101 @@
+# Changelog
+
+All notable changes to the SemanticEmbed SDK.
+
+## 0.7.0 — Honeycomb + Datadog live connectors
+
+- New: `semanticembed.live.from_honeycomb(dataset, api_key)` — runs a
+  Honeycomb Query API request that breaks down spans by `trace.span_id`,
+  `trace.parent_id`, and `service.name`, then derives parent-child
+  service edges. Polls the result endpoint until complete.
+  Override `api_url` for EU tenants.
+- New: `semanticembed.live.from_datadog(api_key, app_key)` — calls the
+  Spans Search API (`POST /api/v2/spans/events/search`), paginates via
+  `meta.page.after`, applies the same span-row → service-edge join.
+  Optional `env` and `service` filters; `site` override for EU/US3/US5.
+- Both raise clear `ValueError`s when credentials are missing; both fall
+  back to canonical env vars (`HONEYCOMB_API_KEY` / `HONEYCOMB_DATASET`,
+  `DD_API_KEY` / `DD_APP_KEY`).
+
+## 0.6.0 — Infrastructure-as-code parsers
+
+- New: `se.extract.from_cloudformation(template)` — parses YAML/JSON templates,
+  honors `DependsOn` plus implicit `Ref`/`Fn::GetAtt`/`Fn::Sub` references.
+  Custom YAML loader handles short intrinsic tags (`!Ref`, `!GetAtt` etc.).
+- New: `se.extract.from_aws_cdk(file)` — Python CDK construct kwarg references.
+- New: `se.extract.from_pulumi(file)` — Python Pulumi resource kwarg references.
+- `from_directory()` auto-detects all three.
+- TypeScript CDK/Pulumi tracked for a future release.
+
+## 0.5.1 — Agent packaging fix
+
+- Move `agent/` under `src/semanticembed/agent/` so it actually ships in the
+  wheel. Previous installs (0.2.x) declared the agent's dependencies but
+  didn't install the agent itself.
+- New console scripts: `semanticembed-agent`, `semanticembed-gemini-agent`.
+  Also reachable as `python -m semanticembed.agent`.
+
+## 0.5.0 — Live observability connectors
+
+- New: `semanticembed.live.from_dynatrace(env_url, api_token)` — pulls
+  service-to-service call edges from Smartscape via the Environment API v2.
+  Pagination, env-var auth fallback, header-based authentication.
+- Honeycomb / Datadog connectors planned.
+
+## 0.4.1 — Service-boundary rollup + encode cache
+
+- `from_python_imports(depth=N)` rolls module names up to the first N path
+  components. `depth=2` is useful for `services/<svc>/...` monorepos —
+  cuts edge-list noise ~10x.
+- `encode(cache=True)` opts into an in-process LRU cache (max 64 entries,
+  order-independent edge-set keys). `clear_encode_cache()` for explicit
+  eviction. Off by default to preserve current behavior.
+
+## 0.4.0 — AI agent framework parsers
+
+- New: `from_langgraph(file)` — parses `add_edge`, `add_conditional_edges`
+  with explicit `path_map`, `set_entry_point`, `set_finish_point`.
+- New: `from_crewai(file)` — `Task(agent=...)`, `Task(context=[...])`,
+  `Crew(manager_agent=...)` patterns.
+- New: `from_autogen(file)` — `GroupChat(agents=[...])` with explicit
+  `GroupChatManager` produces a star; without a manager produces a
+  fully-connected subgraph. Plus `initiate_chat` edges.
+- Pure AST parsers — no need to `pip install` the framework being analyzed.
+- `from_directory()` auto-detects all three by scanning Python file headers
+  for the relevant `import` statements.
+
+## 0.3.0 — OpenTelemetry traces + dedupe
+
+- New: `from_otel_traces(path)` — auto-detects OTLP / Jaeger / Zipkin JSON.
+  Edges emit at the **service** level (intra-service spans roll up).
+- `from_directory()` picks up `traces.json` / `otel.json` / `jaeger.json`
+  at the repo root, plus any `*.json` under `traces/`.
+- New: `se.dedupe_edges(edges, normalize, aliases, drop_self_loops)` — for
+  blending edges from multiple sources cleanly. Handles the
+  `AuthService` / `auth-svc` / `auth_svc` convergence problem. Modes:
+  `none` / `snake` / `lower` / `kebab`.
+
+## 0.2.3 — Modernized skill (Ollama removed)
+
+- The Claude Code skill at `skill/analyze.py` no longer requires Ollama.
+  Uses the SDK directly via `se.find_edges()` + `se.encode()`. The parent
+  Claude Code agent does any natural-language extraction natively.
+- Single dependency: `pip install 'semanticembed[extract]'`.
+
+## 0.2.2 — `find_edges()` + preflight + retry + disclosure
+
+- New: `se.find_edges(path, provider, max_nodes)` — programmatic agent
+  hook. Tries `extract.from_directory` first (deterministic, no network).
+  Falls through to Claude or Gemini only if no infra files are recognized.
+- `encode()` now raises `NodeLimitError` before the HTTP call when no
+  license key is set and the graph exceeds the 50-node free tier.
+- `encode()` retries once on `ConnectError` / `ReadTimeout` / 502 / 503 /
+  504 with a 0.5s backoff.
+- README adds a "What gets sent where" subsection clarifying that the
+  agent paths send file contents to Anthropic / Google APIs.
+
+## 0.2.1 and earlier
+
+Initial PyPI releases. See git history for the original surface
+(`encode`, `extract.from_docker_compose`, `extract.from_kubernetes`,
+`extract.from_terraform`, `extract.from_python_imports`, `explain`, `ask`).

{semanticembed-0.2.1 → semanticembed-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: semanticembed
-Version: 0.2.1
+Version: 0.7.0
 Summary: 6D structural intelligence for directed graphs. Six numbers per node. Sub-millisecond.
 Project-URL: Homepage, https://github.com/jmurray10/semanticembed-sdk
 Project-URL: Documentation, https://github.com/jmurray10/semanticembed-sdk
@@ -30,6 +30,10 @@ Provides-Extra: all
 Requires-Dist: claude-agent-sdk; extra == 'all'
 Requires-Dist: google-genai; extra == 'all'
 Requires-Dist: pyyaml>=6.0; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: pyyaml>=6.0; extra == 'dev'
+Requires-Dist: respx>=0.20; extra == 'dev'
 Provides-Extra: extract
 Requires-Dist: pyyaml>=6.0; extra == 'extract'
 Description-Content-Type: text/markdown
@@ -194,12 +198,29 @@ edges = se.extract.from_github_actions(".github/workflows")
 # From Terraform
 edges = se.extract.from_terraform("infra/")
 
+# From CloudFormation (YAML or JSON)
+edges = se.extract.from_cloudformation("template.yaml")
+
+# From AWS CDK (Python)
+edges = se.extract.from_aws_cdk("app.py")
+
+# From Pulumi (Python)
+edges = se.extract.from_pulumi("__main__.py")
+
 # From Python imports (module dependency graph)
 edges = se.extract.from_python_imports("src/")
 
 # From Node.js monorepo (inter-package dependencies)
 edges = se.extract.from_package_json_workspaces(".")
 
+# From OpenTelemetry traces (OTLP / Jaeger / Zipkin JSON)
+edges = se.extract.from_otel_traces("traces.json")
+
+# From AI agent frameworks (AST-only — no need to install the framework)
+edges = se.extract.from_langgraph("workflow.py")   # StateGraph.add_edge / add_conditional_edges / set_entry_point
+edges = se.extract.from_crewai("crew.py")          # Task(agent=...) / Task(context=...) / Crew(manager_agent=...)
+edges = se.extract.from_autogen("agents.py")       # GroupChat(agents=...) / initiate_chat(...)
+
 # Auto-detect everything in a directory
 edges, sources = se.extract.from_directory(".")
 print(f"Found {len(edges)} edges from {sources}")
@@ -209,7 +230,95 @@ result = se.encode(edges)
 print(result.table)
 ```
 
-Requires `pyyaml` for YAML parsing: `pip install pyyaml`
+Requires `pyyaml` for YAML parsing: `pip install 'semanticembed[extract]'`
+
+### Trace ingestion (highest-fidelity edges)
+
+Compose / k8s / Terraform describe deployment, not actual call edges. **Real
+runtime traces are the only source with the actual call graph.** v0.3.0 ships
+a deterministic parser for the three common JSON formats:
+
+- **OTLP** (OpenTelemetry Collector / SDK exports): `{"resourceSpans": [...]}`
+- **Jaeger** (`jaeger-query` API, `jaeger-cli`): `{"data": [{"spans": [...]}]}`
+- **Zipkin** (Zipkin v2 API): top-level array with `localEndpoint.serviceName`
+
+Edges are emitted at the **service level** — same-service spans roll up. Place
+a `traces.json` (or `otel.json` / `jaeger.json` / `zipkin.json`) at your repo
+root and `from_directory()` will pick it up.
+
+### Live observability connectors
+
+Static analysis is great for repos. For *running* infra, pull traces directly:
+
+```python
+from semanticembed import live
+
+# Dynatrace — Smartscape services + call relationships
+edges = live.from_dynatrace(
+    env_url="https://abc12345.live.dynatrace.com",
+    api_token=os.environ["DYNATRACE_API_TOKEN"],
+)
+
+# Honeycomb — Query API over a dataset
+edges = live.from_honeycomb(
+    dataset="my-app-prod",
+    api_key=os.environ["HONEYCOMB_API_KEY"],
+    lookback_seconds=900,
+)
+
+# Datadog — Spans Search API
+edges = live.from_datadog(
+    api_key=os.environ["DD_API_KEY"],
+    app_key=os.environ["DD_APP_KEY"],
+    env="prod",
+    lookback="now-30m",
+)
+```
+
+### AI agent frameworks
+
+The three popular Python agent frameworks each have an explicit graph-building
+API. Static AST parsing extracts the actual call graph the framework will run.
+The SDK does **not** import or run the framework — you don't need
+`pip install langgraph` to extract from a LangGraph script.
+
+**LangGraph** — `g.add_edge`, `g.add_conditional_edges` (with explicit
+`path_map`), `g.set_entry_point`, `g.set_finish_point`. The sentinels `START`
+and `END` are emitted as literal node names.
+
+**CrewAI** — `Task(agent=researcher)` produces `researcher -> task_var`;
+`Task(context=[t1, t2])` produces `t1 -> task_var` / `t2 -> task_var`;
+`Crew(manager_agent=mgr)` adds a `mgr -> agent` fan-out.
+
+**AutoGen** — `GroupChat(agents=[a, b, c])` with an explicit
+`GroupChatManager` produces a star (`manager -> a`, `-> b`, `-> c`).
+Without a manager, it's fully connected. `x.initiate_chat(y)` always
+produces `x -> y`.
+
+`from_directory()` auto-detects these by scanning Python files for the
+relevant `import` statements and only running the matching parser on those
+files (cheap and accurate vs. walking the whole tree).
+
+### Blending sources cleanly
+
+Combining traces + compose + Python imports usually produces the same logical
+service under several names (`auth-svc`, `auth_svc`, `AuthService`). Use
+`dedupe_edges` to canonicalize:
+
+```python
+compose_edges, _ = se.extract.from_directory(".")
+trace_edges = se.extract.from_otel_traces("traces.json")
+
+edges = se.dedupe_edges(
+    list(compose_edges) + trace_edges,
+    normalize="snake",                          # AuthService -> auth_service
+    aliases={"auth_svc": "auth_service"},       # explicit overrides
+)
+result = se.encode(edges)
+```
+
+Modes: `"none"` (default), `"snake"`, `"lower"`, `"kebab"`. Self-loops are
+dropped by default.
 
 ---
 
@@ -254,21 +363,33 @@ for node, deltas in changes.items():
 An autonomous agent that scans your repo, extracts edges, encodes, and explains results interactively. Choose your LLM backend:
 
 ```bash
-# Claude agent
-pip install claude-agent-sdk
+# Claude agent (installs the agent code + the Anthropic agent SDK)
+pip install 'semanticembed[agent-claude]'
 export ANTHROPIC_API_KEY=sk-ant-...
-python -m agent
+semanticembed-agent              # interactive
+semanticembed-agent --ask "What is my biggest SPOF?"
 
 # Gemini agent
-pip install google-genai
+pip install 'semanticembed[agent-gemini]'
 export GOOGLE_API_KEY=...
-python -m agent.gemini_agent
-
-# Single question
-python -m agent --ask "What is my biggest SPOF?"
+semanticembed-gemini-agent
 ```
 
-The agent has 7 tools: scan, extract (docker-compose, k8s, Python imports), encode, diff, and simulate architecture changes. See [agent/README.md](agent/README.md) for full docs.
+Both binaries are also reachable as `python -m semanticembed.agent` /
+`python -m semanticembed.agent.gemini_agent`.
+
+The agent has 7 tools: scan, extract (docker-compose, k8s, Python imports), encode, diff, and simulate architecture changes. See [src/semanticembed/agent/README.md](src/semanticembed/agent/README.md) for full docs.
+
+### What gets sent where
+
+Be explicit about data egress before pointing the agent at private architecture:
+
+- **Claude agent** (`python -m agent`): the LLM reads tool outputs as conversation context, so the contents of `docker-compose.yml`, Kubernetes manifests, Terraform `.tf` files, Python source, and `package.json` files in your project go to **Anthropic's API** along with your prompts. Conversation history is governed by Anthropic's data-use policies.
+- **Gemini agent** (`python -m agent.gemini_agent`): same data flow, sent to **Google's API** instead.
+- **Skill** (`skill/analyze.py`): runs **Ollama on your machine**. Raw input never leaves localhost unless you set `SEMBED_OLLAMA_URL` to a remote host.
+- **Cloud `encode()` call**: only the **edge list** (node names, e.g. `["frontend", "auth"]`) goes to the SemanticEmbed Railway endpoint. File contents are never sent.
+
+If your topology is sensitive, prefer the skill (local Ollama) or pre-extract edges deterministically with `se.extract.from_directory()` and call `se.encode()` directly — that path sends only the edge list.
 
 ---
 

{semanticembed-0.2.1 → semanticembed-0.7.0}/README.md

@@ -158,12 +158,29 @@ edges = se.extract.from_github_actions(".github/workflows")
 # From Terraform
 edges = se.extract.from_terraform("infra/")
 
+# From CloudFormation (YAML or JSON)
+edges = se.extract.from_cloudformation("template.yaml")
+
+# From AWS CDK (Python)
+edges = se.extract.from_aws_cdk("app.py")
+
+# From Pulumi (Python)
+edges = se.extract.from_pulumi("__main__.py")
+
 # From Python imports (module dependency graph)
 edges = se.extract.from_python_imports("src/")
 
 # From Node.js monorepo (inter-package dependencies)
 edges = se.extract.from_package_json_workspaces(".")
 
+# From OpenTelemetry traces (OTLP / Jaeger / Zipkin JSON)
+edges = se.extract.from_otel_traces("traces.json")
+
+# From AI agent frameworks (AST-only — no need to install the framework)
+edges = se.extract.from_langgraph("workflow.py")   # StateGraph.add_edge / add_conditional_edges / set_entry_point
+edges = se.extract.from_crewai("crew.py")          # Task(agent=...) / Task(context=...) / Crew(manager_agent=...)
+edges = se.extract.from_autogen("agents.py")       # GroupChat(agents=...) / initiate_chat(...)
+
 # Auto-detect everything in a directory
 edges, sources = se.extract.from_directory(".")
 print(f"Found {len(edges)} edges from {sources}")
@@ -173,7 +190,95 @@ result = se.encode(edges)
 print(result.table)
 ```
 
-Requires `pyyaml` for YAML parsing: `pip install pyyaml`
+Requires `pyyaml` for YAML parsing: `pip install 'semanticembed[extract]'`
+
+### Trace ingestion (highest-fidelity edges)
+
+Compose / k8s / Terraform describe deployment, not actual call edges. **Real
+runtime traces are the only source with the actual call graph.** v0.3.0 ships
+a deterministic parser for the three common JSON formats:
+
+- **OTLP** (OpenTelemetry Collector / SDK exports): `{"resourceSpans": [...]}`
+- **Jaeger** (`jaeger-query` API, `jaeger-cli`): `{"data": [{"spans": [...]}]}`
+- **Zipkin** (Zipkin v2 API): top-level array with `localEndpoint.serviceName`
+
+Edges are emitted at the **service level** — same-service spans roll up. Place
+a `traces.json` (or `otel.json` / `jaeger.json` / `zipkin.json`) at your repo
+root and `from_directory()` will pick it up.
+
+### Live observability connectors
+
+Static analysis is great for repos. For *running* infra, pull traces directly:
+
+```python
+from semanticembed import live
+
+# Dynatrace — Smartscape services + call relationships
+edges = live.from_dynatrace(
+    env_url="https://abc12345.live.dynatrace.com",
+    api_token=os.environ["DYNATRACE_API_TOKEN"],
+)
+
+# Honeycomb — Query API over a dataset
+edges = live.from_honeycomb(
+    dataset="my-app-prod",
+    api_key=os.environ["HONEYCOMB_API_KEY"],
+    lookback_seconds=900,
+)
+
+# Datadog — Spans Search API
+edges = live.from_datadog(
+    api_key=os.environ["DD_API_KEY"],
+    app_key=os.environ["DD_APP_KEY"],
+    env="prod",
+    lookback="now-30m",
+)
+```
+
+### AI agent frameworks
+
+The three popular Python agent frameworks each have an explicit graph-building
+API. Static AST parsing extracts the actual call graph the framework will run.
+The SDK does **not** import or run the framework — you don't need
+`pip install langgraph` to extract from a LangGraph script.
+
+**LangGraph** — `g.add_edge`, `g.add_conditional_edges` (with explicit
+`path_map`), `g.set_entry_point`, `g.set_finish_point`. The sentinels `START`
+and `END` are emitted as literal node names.
+
+**CrewAI** — `Task(agent=researcher)` produces `researcher -> task_var`;
+`Task(context=[t1, t2])` produces `t1 -> task_var` / `t2 -> task_var`;
+`Crew(manager_agent=mgr)` adds a `mgr -> agent` fan-out.
+
+**AutoGen** — `GroupChat(agents=[a, b, c])` with an explicit
+`GroupChatManager` produces a star (`manager -> a`, `-> b`, `-> c`).
+Without a manager, it's fully connected. `x.initiate_chat(y)` always
+produces `x -> y`.
+
+`from_directory()` auto-detects these by scanning Python files for the
+relevant `import` statements and only running the matching parser on those
+files (cheap and accurate vs. walking the whole tree).
+
+### Blending sources cleanly
+
+Combining traces + compose + Python imports usually produces the same logical
+service under several names (`auth-svc`, `auth_svc`, `AuthService`). Use
+`dedupe_edges` to canonicalize:
+
+```python
+compose_edges, _ = se.extract.from_directory(".")
+trace_edges = se.extract.from_otel_traces("traces.json")
+
+edges = se.dedupe_edges(
+    list(compose_edges) + trace_edges,
+    normalize="snake",                          # AuthService -> auth_service
+    aliases={"auth_svc": "auth_service"},       # explicit overrides
+)
+result = se.encode(edges)
+```
+
+Modes: `"none"` (default), `"snake"`, `"lower"`, `"kebab"`. Self-loops are
+dropped by default.
 
 ---
 
@@ -218,21 +323,33 @@ for node, deltas in changes.items():
 An autonomous agent that scans your repo, extracts edges, encodes, and explains results interactively. Choose your LLM backend:
 
 ```bash
-# Claude agent
-pip install claude-agent-sdk
+# Claude agent (installs the agent code + the Anthropic agent SDK)
+pip install 'semanticembed[agent-claude]'
 export ANTHROPIC_API_KEY=sk-ant-...
-python -m agent
+semanticembed-agent              # interactive
+semanticembed-agent --ask "What is my biggest SPOF?"
 
 # Gemini agent
-pip install google-genai
+pip install 'semanticembed[agent-gemini]'
 export GOOGLE_API_KEY=...
-python -m agent.gemini_agent
-
-# Single question
-python -m agent --ask "What is my biggest SPOF?"
+semanticembed-gemini-agent
 ```
 
-The agent has 7 tools: scan, extract (docker-compose, k8s, Python imports), encode, diff, and simulate architecture changes. See [agent/README.md](agent/README.md) for full docs.
+Both binaries are also reachable as `python -m semanticembed.agent` /
+`python -m semanticembed.agent.gemini_agent`.
+
+The agent has 7 tools: scan, extract (docker-compose, k8s, Python imports), encode, diff, and simulate architecture changes. See [src/semanticembed/agent/README.md](src/semanticembed/agent/README.md) for full docs.
+
+### What gets sent where
+
+Be explicit about data egress before pointing the agent at private architecture:
+
+- **Claude agent** (`python -m agent`): the LLM reads tool outputs as conversation context, so the contents of `docker-compose.yml`, Kubernetes manifests, Terraform `.tf` files, Python source, and `package.json` files in your project go to **Anthropic's API** along with your prompts. Conversation history is governed by Anthropic's data-use policies.
+- **Gemini agent** (`python -m agent.gemini_agent`): same data flow, sent to **Google's API** instead.
+- **Skill** (`skill/analyze.py`): runs **Ollama on your machine**. Raw input never leaves localhost unless you set `SEMBED_OLLAMA_URL` to a remote host.
+- **Cloud `encode()` call**: only the **edge list** (node names, e.g. `["frontend", "auth"]`) goes to the SemanticEmbed Railway endpoint. File contents are never sent.
+
+If your topology is sensitive, prefer the skill (local Ollama) or pre-extract edges deterministically with `se.extract.from_directory()` and call `se.encode()` directly — that path sends only the edge list.
 
 ---
 

semanticembed-0.7.0/examples/sample_pipeline.json

@@ -0,0 +1,14 @@
+{
+  "name": "Sample Pipeline",
+  "description": "Minimal 7-node pipeline used as the default starter example in notebook 04. Replace the edges with your own architecture to try the SDK on a real graph.",
+  "edges": [
+    {"source": "frontend", "target": "auth"},
+    {"source": "frontend", "target": "catalog"},
+    {"source": "frontend", "target": "checkout"},
+    {"source": "auth", "target": "db"},
+    {"source": "catalog", "target": "db"},
+    {"source": "checkout", "target": "payments"},
+    {"source": "checkout", "target": "db"},
+    {"source": "payments", "target": "stripe"}
+  ]
+}