archsight 0.1.2 → 0.1.3

data/lib/archsight/web/doc/import.md

@@ -0,0 +1,458 @@
+# Import System
+
+The import system allows you to declaratively define data imports that generate architecture resources from external sources like GitLab, GitHub, and git repositories.
+
+## Overview
+
+Imports are defined as YAML resources with kind `Import`. Each import specifies:
+
+- A **handler** that knows how to fetch and process data
+- **Configuration** specific to that handler
+- Optional **caching** to avoid re-running unchanged imports
+
+Dependencies between imports are automatically derived from the `generates` relation. When a parent import generates child imports, those children depend on the parent and will run after it completes.
+
+The import executor runs imports concurrently in dependency order, with a visual progress display showing completion percentage and ETA.
+
+## Running Imports
+
+```bash
+# Run all pending imports
+archsight import
+
+# Verbose output
+archsight import -v
+
+# Show execution plan without running
+archsight import --dry-run
+
+# Force re-run all imports (ignore cache)
+archsight import --force
+```
+
+### Progress Display
+
+In TTY mode, imports show a live progress display:
+
+```
+Overall ████████████░░░░░░░░ 60% [30/50] ETA: 2:15
+Import:Repo:project-a - Analyzing code
+Import:Repo:project-b - Cloning repository
+Import:Repo:project-c - Done
+```
+
+## Defining an Import
+
+Create YAML files in the `imports/` directory:
+
+```yaml
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:MyData
+  annotations:
+    import/handler: <handler-name>
+    import/priority: "10"
+    import/cacheTime: "24h"
+    import/config/key: value
+spec: {}
+```
+
+### Core Annotations
+
+| Annotation | Description |
+|------------|-------------|
+| `import/handler` | Handler to execute (required) |
+| `import/enabled` | Set to "false" to disable |
+| `import/priority` | Execution order (lower runs first, default: 0) |
+| `import/cacheTime` | Cache duration: "30m", "1h", "24h", "7d", or "never" |
+| `import/outputPath` | Output file path relative to resources directory |
+
+### Caching
+
+Imports can be cached to avoid re-running when data hasn't changed:
+
+```yaml
+import/cacheTime: "24h"  # Re-run after 24 hours
+```
+
+Supported duration formats:
+- `30m` - 30 minutes
+- `1h` - 1 hour
+- `24h` - 24 hours
+- `7d` - 7 days
+- `never` - Always run (default)
+
+The cache uses the `generated/at` annotation written by handlers. When an import completes, it writes a marker with the current timestamp. On subsequent runs, the executor checks if `generated/at + cacheTime > now` to skip cached imports.
+
+### Configuration Pattern
+
+Handler-specific configuration uses `import/config/*` annotations:
+
+```yaml
+import/config/host: gitlab.company.com
+import/config/fallbackTeam: "Team:Platform"
+```
+
+## Available Handlers
+
+### gitlab
+
+Lists repositories from a GitLab instance and generates child Import resources.
+
+**Configuration:**
+- `host` - GitLab host (required)
+- `exploreGroups` - If "true", explore all visible groups (default: false)
+- `repoOutputPath` - Output path for repository handler results
+- `fallbackTeam` - Default team when no contributor match found
+- `botTeam` - Team for bot-only repositories
+
+**Environment:**
+- `GITLAB_TOKEN` - Personal access token (required)
+
+**Output:** Generates `Import:Repo:gitlab:*` resources for each repository.
+
+### github
+
+Lists repositories from a GitHub organization and generates child Import resources.
+
+**Configuration:**
+- `org` - GitHub organization (required)
+- `repoOutputPath` - Output path for repository handler results
+
+**Environment:**
+- `GITHUB_TOKEN` - GitHub Personal Access Token (required)
+  - Create at: https://github.com/settings/tokens
+  - Required scopes: `repo` (private repos) or `public_repo` (public only)
+  - If you have `gh` CLI authenticated: `export GITHUB_TOKEN=$(gh auth token)`
+
+**Output:** Generates `Import:Repo:github:*` resources for each repository.
+
+### repository
+
+Analyzes a single git repository and generates a TechnologyArtifact resource.
+
+**Configuration:**
+- `path` - Local repository path (required)
+- `gitUrl` - Git URL to clone from (if not already cloned)
+- `archived` - If "true", mark as archived
+- `visibility` - Repository visibility: internal, public, open-source
+- `sccPath` - Path to scc binary (default: scc)
+- `fallbackTeam` - Default team when no contributor match found
+- `botTeam` - Team for bot-only repositories
+
+**Output:** Generates one TechnologyArtifact resource with:
+- Code analysis metrics (languages, LOC, estimated cost)
+- Git activity metrics (commits, contributors, bus factor)
+- Team matching based on contributor history
+- Deployment artifact detection (containers, charts, etc.)
+- Agentic tool detection (Claude, Cursor, etc.)
+
+**Special Cases:**
+
+The repository handler creates minimal artifacts for repositories that can't be fully analyzed:
+
+| Status | Reason |
+|--------|--------|
+| `inaccessible` | Clone failed due to access denied or auth errors |
+| `empty` | Repository has no commits |
+| `no-code` | No analyzable source code (only config, docs, etc.) |
+
+These artifacts include `activity/status` and `activity/reason` annotations documenting why full analysis wasn't possible.
+
+### rest-api-index
+
+Fetches an API index JSON and generates child Import resources for each API.
+
+**Configuration:**
+- `indexUrl` - URL to fetch API index JSON (required)
+- `baseUrl` - Base URL for spec files (optional, derived from indexUrl)
+- `interfaceOutputPath` - Shared output path for ApplicationInterface resources
+- `dataObjectOutputPath` - Shared output path for DataObject resources
+- `skipVisibility` - Comma-separated visibilities to skip (e.g., "public-preview,beta")
+
+**Expected Index Format:**
+```json
+[
+  {
+    "name": "compute",
+    "version": "6.0",
+    "visibility": "private",
+    "specPath": "/rest-api/compute/openapi.yaml",
+    "redocPath": "/rest-api/compute/redoc.html",
+    "gate": "GA"
+  }
+]
+```
+
+**Output:** Generates `Import:RestApi:*` resources for each API in the index.
+
+### rest-api
+
+Downloads an OpenAPI spec and generates ApplicationInterface and DataObject resources.
+
+**Configuration:**
+- `name` - API name (required)
+- `version` - API version (default: "1.0")
+- `visibility` - API visibility (default: "private")
+- `specUrl` - Full URL to OpenAPI spec - http, https, or file:// (required)
+- `htmlUrl` - Full URL to HTML documentation (optional)
+- `gate` - Release gate: "GA", "BETA", etc. (default: "GA")
+- `interfaceOutputPath` - Output path for ApplicationInterface resources
+- `dataObjectOutputPath` - Output path for DataObject resources
+
+**Output:**
+- ApplicationInterface resource with detected auth methods (JWT, Basic, API Key, OAuth2, OIDC)
+- DataObject resources extracted from OpenAPI schema definitions
+
+**Interface Naming:** `{Visibility}:{ApiName}:v{MajorVersion}:RestAPI`
+- Example: `Private:Compute:v6:RestAPI`
+
+### Example: REST API Multi-Stage Import
+
+```yaml
+# imports/rest-api.yaml
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:RestApi:Index
+  annotations:
+    import/handler: rest-api-index
+    import/priority: "1"
+    import/cacheTime: "24h"
+    import/config/indexUrl: https://api.example.com/index.json
+    import/config/interfaceOutputPath: generated/rest-api-interfaces.yaml
+    import/config/dataObjectOutputPath: generated/rest-api-data-objects.yaml
+    import/config/skipVisibility: public-preview
+spec: {}
+```
+
+After running, this generates child imports like:
+
+```yaml
+# generated/Import_RestApi_Index.yaml
+---
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:RestApi:compute
+  annotations:
+    import/handler: rest-api
+    import/config/name: compute
+    import/config/version: "6.0"
+    import/config/visibility: private
+    import/config/specUrl: https://api.example.com/rest-api/compute/openapi.yaml
+    import/config/htmlUrl: https://api.example.com/rest-api/compute/redoc.html
+    import/config/gate: GA
+    import/config/interfaceOutputPath: generated/rest-api-interfaces.yaml
+    import/config/dataObjectOutputPath: generated/rest-api-data-objects.yaml
+spec: {}
+---
+# Parent import tracks what it generates
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:RestApi:Index
+spec:
+  generates:
+    imports:
+      - Import:RestApi:compute
+```
+
+The dependency of `Import:RestApi:compute` on `Import:RestApi:Index` is automatically derived from the `generates` relation above.
+
+## Multi-Stage Import Pattern
+
+Imports can generate other Import resources, enabling multi-stage workflows:
+
+1. **GitLab import** runs and discovers repositories
+2. Generates `Import:Repo:*` for each repository
+3. Database reloads, discovers new Import resources
+4. **Repository imports** run concurrently (up to 20)
+5. Each generates TechnologyArtifact resources
+6. Loop continues until no pending imports remain
+
+### Example: GitLab Multi-Stage Import
+
+```yaml
+# imports/gitlab.yaml
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:GitLab
+  annotations:
+    import/handler: gitlab
+    import/priority: "1"
+    import/cacheTime: "24h"
+    import/config/host: gitlab.company.com
+    import/config/repoOutputPath: generated/repositories.yaml
+    import/config/fallbackTeam: "Team:Platform"
+spec: {}
+```
+
+After running, this generates:
+
+```yaml
+# generated/gitlab-imports.yaml
+---
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:Repo:gitlab:company:my-service
+  annotations:
+    import/handler: repository
+    import/config/path: ~/.cache/archsight/git/gitlab/company/my-service
+    import/config/gitUrl: git@gitlab.company.com:company/my-service.git
+    import/config/archived: "false"
+    import/config/visibility: internal
+    import/outputPath: generated/repositories.yaml
+spec: {}
+---
+# Parent import tracks what it generates
+apiVersion: architecture/v1alpha1
+kind: Import
+metadata:
+  name: Import:GitLab
+spec:
+  generates:
+    imports:
+      - Import:Repo:gitlab:company:my-service
+```
+
+The child import depends on `Import:GitLab` because the parent's `generates` relation includes it.
+
+## Generated Annotations
+
+### TechnologyArtifact Annotations
+
+Repository analysis generates these annotations:
+
+**Repository Info:**
+- `artifact/type` - Always "repo" for repositories
+- `repository/git` - Git URL
+- `repository/visibility` - internal, public, open-source
+- `repository/recentTags` - Recent git tags (releases)
+- `repository/accessible` - "false" if repo couldn't be accessed
+- `repository/error` - Error message for inaccessible repos
+
+**Code Metrics (from scc):**
+- `scc/languages` - Comma-separated language list
+- `scc/estimatedCost` - COCOMO cost estimate
+- `scc/estimatedPeople` - Estimated team size
+- `scc/language/*/loc` - Lines of code per language
+
+**Activity Metrics:**
+- `activity/status` - active, abandoned, bot-only, archived, inaccessible, empty, no-code
+- `activity/reason` - Explanation for non-standard statuses
+- `activity/commits` - Monthly commit counts (12 months)
+- `activity/contributors` - Monthly contributor counts
+- `activity/contributors/6m` - Unique contributors (6 months)
+- `activity/contributors/total` - Total unique contributors
+- `activity/busFactor` - Risk assessment: high, medium, low
+- `activity/createdAt` - First commit date
+- `activity/lastHumanCommit` - Last non-bot commit date
+
+**Deployment Detection:**
+- `repository/artifacts` - Detected artifact types (container, chart, debian, rpm)
+- `deployment/images` - OCI container image names
+- `workflow/platforms` - CI/CD platforms (github-actions, gitlab-ci)
+- `workflow/types` - Workflow types (build, test, deploy, etc.)
+- `agentic/tools` - AI coding tools detected (claude, cursor, aider)
+
+## Troubleshooting
+
+### Deadlock Error
+
+If you see "Deadlock: pending imports have unsatisfied dependencies", check that:
+
+1. All dependencies exist as Import resources
+2. Dependencies don't form a circular chain
+3. Dependent imports haven't failed
+
+### Access Denied Errors
+
+When a repository can't be cloned due to access issues, the handler creates a minimal artifact with:
+- `activity/status: inaccessible`
+- `repository/accessible: false`
+- `repository/error: <error message>`
+
+This allows the import to continue processing other repositories.
+
+### Cached Imports Not Updating
+
+If an import isn't re-running when expected:
+1. Check `import/cacheTime` annotation value
+2. Check `generated/at` annotation on the Import resource
+3. Use `archsight import --force` to bypass cache for all imports
+4. Alternatively, use `import/cacheTime: never` to always re-run a specific import
+
+**Note:** The cache is automatically invalidated when:
+- The output file (specified by `import/outputPath`) is deleted or missing
+- The import source YAML file has been modified since the last run
+
+## Creating Custom Handlers
+
+To create a new handler:
+
+1. Create a Ruby file in `lib/archsight/import/handlers/`
+2. Inherit from `Archsight::Import::Handler`
+3. Implement the `execute` method
+4. Register with `Registry.register("name", YourHandler)`
+
+```ruby
+# Repository handler - clones/syncs and analyzes a git repository
+class Archsight::Import::Handlers::Custom < Archsight::Import::Handler
+  def execute
+    # Read configuration
+    url = config("url")
+
+    # Update progress
+    progress.update("Fetching data")
+
+    # Fetch and process data
+    data = fetch_data(url)
+
+    # Generate resources
+    resources = data.map { |item| build_resource(item) }
+
+    # Write output with self-marker for caching
+    yaml_content = resources_to_yaml(resources) + YAML.dump(self_marker)
+    write_yaml(yaml_content)
+  end
+
+  # Marker for cache timestamp
+  def self_marker
+    {
+      "apiVersion" => "architecture/v1alpha1",
+      "kind" => "Import",
+      "metadata" => {
+        "name" => import_resource.name,
+        "annotations" => { "generated/at" => Time.now.utc.iso8601 }
+      },
+      "spec" => {}
+    }
+  end
+
+  private
+
+  def build_resource(item)
+    resource_yaml(
+      kind: "TechnologyArtifact",
+      name: item["name"],
+      annotations: { "custom/field" => item["value"] }
+    )
+  end
+end
+
+Archsight::Import::Registry.register("custom", Archsight::Import::Handlers::Custom)
+```
+
+### Handler Helper Methods
+
+| Method | Description |
+|--------|-------------|
+| `config(key, default:)` | Get configuration value |
+| `progress.update(msg)` | Update progress display |
+| `write_yaml(content)` | Write YAML to output path |
+| `resource_yaml(kind:, name:, ...)` | Build resource hash |
+| `import_yaml(name:, handler:, ...)` | Build child import hash |

data/lib/archsight/web/doc/index.md.erb

@@ -7,6 +7,7 @@ Welcome to the Architecture Documentation System. This documentation covers how
 - [Tool Guide](/doc/tool) - How to use the architecture tool
 - [Query Syntax](/doc/search) - Search and filter resources
 - [Architecture Modeling](/doc/modeling) - How to model your architecture
+- [REST API Documentation](/api/docs) - JSON API for programmatic access
 
 ## Resource Model
 

data/lib/archsight/web/public/css/artifact.css

@@ -129,6 +129,16 @@
   color: var(--color);
 }
 
+.estimate-note {
+  margin-left: 0.5rem;
+  color: var(--muted-color);
+  cursor: help;
+}
+
+.estimate-note i {
+  font-size: 0.875rem;
+}
+
 /* Activity summary */
 .activity-summary {
   display: flex;

data/lib/archsight/web/public/css/graph.css

@@ -104,3 +104,17 @@
         fill: #666666 !important;
     }
 }
+
+
+/* ===== Graph Too Large Message ===== */
+
+.graph-too-large {
+    color: var(--pico-muted-color);
+    font-size: 0.875rem;
+    margin: 0.5rem 0;
+}
+
+.graph-too-large i {
+    margin-right: 0.25rem;
+    opacity: 0.7;
+}