x-fidelity 1.6.1 → 1.8.0

package/CHANGELOG.md

@@ -1,3 +1,17 @@
+# [1.8.0](https://github.com/zotoio/x-fidelity/compare/v1.7.0...v1.8.0) (2024-08-01)
+
+
+### Features
+
+* Add system architecture diagram to README.md ([ce80b79](https://github.com/zotoio/x-fidelity/commit/ce80b795031c366b232e51c0cf0e651fa77c364c))
+
+# [1.7.0](https://github.com/zotoio/x-fidelity/compare/v1.6.1...v1.7.0) (2024-07-30)
+
+
+### Features
+
+* **extensibility:** archtypes as replacable json files and docs ([ce7b06d](https://github.com/zotoio/x-fidelity/commit/ce7b06df4b3ae76c4d5094b478e7425579b585b2))
+
 ## [1.6.1](https://github.com/zotoio/x-fidelity/compare/v1.6.0...v1.6.1) (2024-07-27)
 
 

package/README.md

@@ -1,6 +1,6 @@
 # x-fidelity
 
-x-fidelity is an advanced CLI tool designed to enforce opinionated framework adherence checks within a codebase. It provides a flexible and extensible way to ensure your projects follow specific standards and best practices.
+x-fidelity is an advanced CLI tool and paired config server designed to perform opinionated framework adherence checks within a codebase. It provides a flexible and extensible way to ensure your projects are using specific standards, tools and best practices.
 
 ```
 =====================================
@@ -38,29 +38,32 @@ x-fidelity is an advanced CLI tool designed to enforce opinionated framework adh
 
 1. [Intent and Purpose](#intent-and-purpose)
 2. [Key Features](#key-features)
-3. [Installation](#installation)
-4. [Usage](#usage)
-5. [Configuration](#configuration)
-6. [Extending x-fidelity](#extending-x-fidelity)
-7. [OpenAI Integration](#openai-integration)
-8. [Hosting Config Servers](#hosting-config-servers)
-9. [Best Practices](#best-practices)
-10. [Linting](#linting)
-11. [Contributing](#contributing)
-12. [License](#license)
+3. [System Architecture](#system-architecture)
+4. [Installation](#installation)
+5. [Usage](#usage)
+6. [Configuration](#configuration)
+7. [Extending x-fidelity](#extending-x-fidelity)
+8. [OpenAI Integration](#openai-integration)
+9. [Hosting Config Servers](#hosting-config-servers)
+10. [Best Practices](#best-practices)
+11. [Linting](#linting)
+12. [Contributing](#contributing)
+13. [License](#license)
 
 ## Intent and Purpose
 
 x-fidelity aims to streamline the process of maintaining code quality and consistency across projects. By providing a flexible, rule-based system, it allows teams to:
 
-- Enforce coding standards and best practices
-- Ensure consistent project structures
-- Maintain up-to-date dependencies
+- Enforce bespoke coding standards and best practices
+- Ensure consistent project archetype structures
+- Maintain up-to-date private dependencies
 - Catch potential issues early in the development process
-- Integrate advanced code analysis using AI (via OpenAI)
+- Integrate GenAI-based advanced code analysis (experimental)
 
 The tool is designed to be highly customizable, allowing teams to define their own archetypes, rules, and checks tailored to their specific needs and tech stacks.
 
+> x-fidelity is not a replacement for standard linting more generalised code analysis tools.  it is intended to help with management of bespoke requirements and as a simple way to experiment with GenAI based code reviews.
+
 ## Key Features
 
 - **Flexible Archetype System**: Define custom project archetypes with specific rules and configurations.
@@ -72,6 +75,69 @@ The tool is designed to be highly customizable, allowing teams to define their o
 - **OpenAI Integration**: Leverage AI for advanced code analysis and suggestions.
 - **Extensible Architecture**: Easily add new operators, facts, and rules to suit your needs.
 
+## System Architecture
+
+The following diagram illustrates the overall architecture of the x-fidelity system:
+
+```mermaid
+graph TD
+    subgraph "Client Environments"
+        CI[CI Environment]
+        Local[Local Development]
+    end
+
+    subgraph "x-fidelity Core"
+        Engine[Analysis Engine]
+        CLI[CLI Interface]
+        ConfigMgr[Config Manager]
+    end
+
+    subgraph "x-fidelity Infrastructure"
+        CS[Config Server]
+        TS[Telemetry Server]
+    end
+
+    subgraph "External Services"
+        GH[GitHub]
+        OAI[OpenAI API]
+    end
+
+    subgraph "Data Sources"
+        Files[Repository Files]
+        Deps[Dependencies]
+    end
+
+    CI -->|Use| Engine
+    Local -->|Use| Engine
+    CI -->|Use| CLI
+    Local -->|Use| CLI
+
+    CLI -->|Initialize| ConfigMgr
+    Engine -->|Use| ConfigMgr
+
+    ConfigMgr -->|Fetch config| CS
+    Engine -->|Send telemetry| TS
+
+    Engine -->|Analyze| Files
+    Engine -->|Check| Deps
+
+    CS -->|Optional: Fetch rules| GH
+    TS -->|Optional: Store data| GH
+
+    Engine -.->|Optional: AI analysis| OAI
+
+    classDef optional stroke-dasharray: 5 5
+    class OAI optional
+```
+
+This diagram shows the main components of x-fidelity and how they interact:
+
+- **Client Environments**: Where x-fidelity is used (CI systems or local development).
+- **x-fidelity Core**: The main components of the system, including the analysis engine, CLI interface, and configuration manager.
+- **x-fidelity Infrastructure**: Servers for configuration and telemetry.
+- **External Services**: GitHub for repository interaction and optional OpenAI integration.
+- **Data Sources**: The files and dependencies that x-fidelity analyzes.
+
 ## Installation
 
 Install x-fidelity using Node.js 18+ and Yarn:
@@ -104,7 +170,7 @@ xfidelity [-d --dir <directory>] [-c --configServer <url>] [-a --archetype <arch
 - `-d --dir <directory>`: Specify the root directory to analyze (default: current directory)
 - `-c --configServer <url>`: URL to fetch the configuration from. eg. https://localhost:8888
 - `-a --archetype <archetype>`: Archetype to use for analysis (default: 'node-fullstack')
-- `-m --mode <mode>`: Run mode: 'cli' or 'server' (default: 'cli')
+- `-m --mode <mode>`: Run mode: 'client' or 'server' (default: 'client')
 - `-p --port <port>`: Port number for server mode (default: 8888)
 - `-o --openaiEnabled <boolean>`: Enable OpenAI analysis (default: false)
 - `-t --telemetryCollector <url>`: The URL telemetry data will be sent to for usage analysis
@@ -119,120 +185,199 @@ xfidelity --configServer https://localhost:8888
 # Analyze parent directory with java-microservice archetype and enable OpenAI analysis
 xfidelity -d .. -a java-microservice -c https://localhost:8888 -o true
 
-# Run in server mode with custom port and specify telemetry collector
+#Run in server mode with custom port and specify telemetry collector
 xfidelity --mode server --port 9999 -t https://telemetry.example.com
 
 # Use local config and rules
 xfidelity -l /path/to/local/config
-
 ```
 
-### Configuration Server
+## Configuration
 
-Start the built-in configuration server:
+x-fidelity uses archetypes to define project-specific configurations. Archetypes are now managed as JSON files, which can be stored locally or on a remote server.
 
-```sh
-yarn start-server
-```
+### Archetype Structure
 
-Or use the CLI:
+Archetypes specify:
 
-```sh
-xfidelity --mode server
+- Rules to apply
+- Operators to use
+- Facts to gather
+- Dependency version requirements
+- Standard directory structure
+- File patterns to include or exclude
+
+Example archetype JSON structure:
+
+```json
+{
+    "rules": ["rule1", "rule2"],
+    "operators": ["operator1", "operator2"],
+    "facts": ["fact1", "fact2"],
+    "config": {
+        "minimumDependencyVersions": {
+            "dependency1": "^1.0.0",
+            "dependency2": "^2.0.0"
+        },
+        "standardStructure": {
+            "src": {
+                "components": null,
+                "utils": null
+            },
+            "tests": null
+        },
+        "blacklistPatterns": [".*\\/\\..*", ".*\\/(dist|build)(\\/.*|$)"],
+        "whitelistPatterns": [".*\\.(ts|tsx|js|jsx)$"]
+    }
+}
 ```
 
-Set a custom port:
+### Local Configuration
+
+To use local configuration files for archetypes and rules, use the `-l` or `--localConfig` option:
 
 ```sh
-xfidelity --mode server --port 9999
+xfidelity -l /path/to/local/config
 ```
 
-You can also set the port using an environment variable:
+The local config directory should contain:
+- Archetype JSON files (e.g., `node-fullstack.json`)
+- A `rules` subdirectory containing rule JSON files
+
+You can override default archetypes or add new ones by placing the corresponding JSON files in the local config directory.
+
+### Remote Configuration
+
+To use a remote configuration server, use the `-c` or `--configServer` option:
 
 ```sh
-export XFI_SERVER_PORT=8888
-xfidelity --mode server
+xfidelity -c https://config-server.example.com
 ```
 
-## Configuration
+The remote server should provide endpoints to serve archetype and rule configurations.
 
-x-fidelity uses archetypes to define project-specific configurations. Archetypes specify:
+## Hosting Config Servers
 
-- Rules to apply
-- Operators to use
-- Facts to gather
-- Dependency version requirements
-- Standard directory structure
-- File patterns to include or exclude
+x-fidelity allows for centrally managed, hot-updatable custom rulesets that can be executed within managed CI pipelines and locally, ensuring consistency of applied rules. Here's an overview of the setup required:
 
-Example archetype structure:
-
-```typescript
-interface ArchetypeConfig {
-    rules: string[];
-    operators: string[];
-    facts: string[];
-    configUrl?: string;
-    config: {
-        minimumDependencyVersions: Record<string, string>;
-        standardStructure: Record<string, any>;
-        blacklistPatterns: string[];
-        whitelistPatterns: string[];
-    };
-}
-```
+1. Set up a Node.js host environment (Docker containerization recommended).
+2. Create a GitHub repository to host your archetypes and rules.
+3. Clone the GitHub repository to the server filesystem.
+4. Install the x-fidelity CLI on the server.
+5. Configure the CLI to:
+   - Run on startup in server mode (`--mode server`)
+   - Point to your rules directory cloned from GitHub (`--localConfig ../rule-repo/config`)
+   - Optionally set the port to listen on (`--port <port>`)
+6. Create a simple CI pipeline step 'framework fidelity' after git repo clone to workspace:
+   - Install the x-fidelity CLI
+   - Run the CLI on the checked-out repo, pointing to the server (`--configServer http://my-server:8888`)
+
+### Docker Example
+
+Here's a basic Docker setup for hosting an x-fidelity config server:
 
-## Telemetry
+```dockerfile
+FROM node:18
 
-x-fidelity now includes telemetry functionality to help improve the tool. Telemetry data is sent to a specified collector URL and includes information about the analysis process, such as:
+# Install x-fidelity
+RUN yarn global add x-fidelity
 
-- Archetype used
-- Repository path (anonymized)
-- File count
-- Failure count
-- Host information (platform, CPU, memory)
-- User information (anonymized username, home directory, shell)
+# Clone your rules repository
+RUN git clone https://github.com/your-org/x-fidelity-rules.git /rules
 
-To specify a telemetry collector, use the `-t` or `--telemetryCollector` option:
+# Set up the start command
+CMD ["x-fidelity", "--mode", "server", "--localConfig", "/rules", "--port", "8888"]
+```
+
+Build and run the Docker container:
 
 ```sh
-xfidelity -t https://telemetry.example.com
+docker build -t x-fidelity-server .
+docker run -p 8888:8888 x-fidelity-server
+```
+
+### CI Pipeline Integration
+
+In your CI pipeline (e.g., GitHub Actions, GitLab CI, Jenkins), add a step to run x-fidelity:
+
+```yaml
+steps:
+  - name: Check out code
+    uses: actions/checkout@v2
+
+  - name: Install x-fidelity
+    run: yarn global add x-fidelity
+
+  - name: Run x-fidelity
+    run: xfidelity --configServer http://x-fidelity-server:8888
 ```
 
-If no telemetry collector is specified, telemetry data will not be sent.
+This setup allows you to maintain a centralized set of rules and archetypes that can be easily updated and applied across all your projects.
 
 ## Extending x-fidelity
 
-x-fidelity is designed to be highly extensible:
+x-fidelity is designed to be highly extensible. You can add custom rules, operators, facts, and archetypes:
+
+1. **Custom Rules**: Add new JSON rule files in the `rules` subdirectory of your local config or on your config server.
+2. **Custom Operators**: Implement new operators and add them to your x-fidelity fork or plugin.
+3. **Custom Facts**: Create new fact providers and add them to your x-fidelity fork or plugin.
+4. **New Archetypes**: Define new archetypes as JSON files in your local config directory or on your config server.
+
+Example of a custom rule JSON file (`my-custom-rule.json`):
+
+```json
+{
+    "name": "my-custom-rule",
+    "conditions": {
+        "all": [
+            {
+                "fact": "fileData",
+                "path": "$.fileContent",
+                "operator": "fileContains",
+                "value": "TODO:"
+            }
+        ]
+    },
+    "event": {
+        "type": "violation",
+        "params": {
+            "message": "TODO comments should be resolved before committing",
+            "level": "warning",
+            "details": {
+                "fact": "fileData",
+                "path": "$.filePath"
+            }
+        }
+    }
+}
+```
 
-1. **Custom Rules**: Add new JSON rule files in `src/rules`.
-2. **Custom Operators**: Implement new operators in `src/operators` and add them to `src/operators/index.ts`.
-3. **Custom Facts**: Create new fact providers in `src/facts` and add them to `src/facts/index.ts`.
-4. **New Archetypes**: Define new archetypes in `src/archetypes` and include them in `src/archetypes/index.ts`.
+Note on rule event types:
+- Events of type "violation" are treated as warnings and do not cause the tool to return an error code.
+- Events of type "fatality" are strictly enforced and will cause the tool to return an error code 1.
 
-Example of creating a new archetype:
+Example of a custom archetype JSON file (`my-custom-archetype.json`):
 
-```typescript
-// src/archetypes/myNewArchetype.ts
-export const myNewArchetype: ArchetypeConfig = {
-    rules: ['myCustomRule', 'standardRule1', 'standardRule2'],
-    operators: ['myCustomOperator', 'standardOperator1'],
-    facts: ['myCustomFact', 'standardFact1'],
-    config: {
-        minimumDependencyVersions: {
-            'my-framework': '^2.0.0'
+```json
+{
+    "rules": ["myCustomRule", "standardRule1", "standardRule2"],
+    "operators": ["myCustomOperator", "standardOperator1"],
+    "facts": ["myCustomFact", "standardFact1"],
+    "config": {
+        "minimumDependencyVersions": {
+            "my-framework": "^2.0.0"
         },
-        standardStructure: {
-            src: {
-                components: null,
-                utils: null
+        "standardStructure": {
+            "src": {
+                "components": null,
+                "utils": null
             },
-            tests: null
+            "tests": null
         },
-        blacklistPatterns: ['.*\\/\\..*', '.*\\/(dist|build)(\\/.*|$)'],
-        whitelistPatterns: ['.*\\.(ts|tsx|js|jsx)$']
+        "blacklistPatterns": [".*\\/\\..*", ".*\\/(dist|build)(\\/.*|$)"],
+        "whitelistPatterns": [".*\\.(ts|tsx|js|jsx)$"]
     }
-};
+}
 ```
 
 ## OpenAI Integration
@@ -261,78 +406,21 @@ export OPENAI_MODEL=gpt-4  # Optional, default is gpt-4o
 > [!IMPORTANT]
 > Be aware of potential costs and data privacy concerns when using OpenAI's API.
 
-## Local Configuration
-
-You can now use local configuration files for archetypes and rules. To use local configuration, use the `-l` or `--localConfig` option:
-
-```sh
-xfidelity -l /path/to/local/config
-```
-
-The local config directory should contain:
-
-- Archetype configuration files (e.g., `node-fullstack.json`)
-- A `rules` subdirectory containing rule files
-
-## Hosting Config Servers
-
-To host a config server for x-fidelity:
-
-1. Set up a Node.js server (e.g., Express.js)
-2. Implement endpoints for archetype configurations and rules
-3. Ensure security, scalability, and performance
-4. Use HTTPS and implement proper authentication
-5. Consider using a CDN for global distribution
-
-Example server setup:
-
-```javascript
-const express = require('express');
-const app = express();
-
-app.get('/archetypes/:archetype', (req, res) => {
-    // Fetch and return archetype configuration
-});
-
-app.get('/archetypes/:archetype/rules/:rule', (req, res) => {
-    // Fetch and return specific rule
-});
-
-app.listen(8888, () => {
-    console.log('Config server running on port 8888');
-});
-```
-
 ## Best Practices
 
-1. **Version Control**: Keep your x-fidelity configurations in version control.
+1. **Version Control**: Keep your x-fidelity configurations (archetypes and rules) in version control.
 2. **Continuous Integration**: Integrate x-fidelity checks into your CI/CD pipeline.
 3. **Regular Updates**: Keep your archetypes, rules, and dependencies up to date.
 4. **Documentation**: Document custom rules, operators, and archetypes for your team.
 5. **Gradual Implementation**: When introducing x-fidelity to an existing project, start with basic checks and gradually increase strictness.
 6. **Team Alignment**: Ensure your team understands and agrees on the rules being enforced.
 7. **Performance**: Be mindful of the performance impact, especially for large codebases.
+8. **Centralized Management**: Use a config server to manage and distribute your archetypes and rules across projects.
 
 ## Contributing
 
 Contributions to x-fidelity are welcome! Please refer to the `CONTRIBUTING.md` file for guidelines on how to contribute to this project.
 
-## Linting
-
-This project uses ESLint for static code analysis. To run the linter:
-
-```sh
-yarn lint
-```
-
-To automatically fix linting issues:
-
-```sh
-yarn lint:fix
-```
-
-ESLint is also integrated into the CI pipeline and runs alongside unit tests in GitHub Actions.
-
 ## License
 
 This project is licensed under the MIT License. See the `LICENSE` file for details.

package/dist/archetypes/index.js

@@ -1,62 +1,47 @@
 "use strict";
+var __createBinding = (this && this.__createBinding) || (Object.create ? (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    var desc = Object.getOwnPropertyDescriptor(m, k);
+    if (!desc || ("get" in desc ? !m.__esModule : desc.writable || desc.configurable)) {
+      desc = { enumerable: true, get: function() { return m[k]; } };
+    }
+    Object.defineProperty(o, k2, desc);
+}) : (function(o, m, k, k2) {
+    if (k2 === undefined) k2 = k;
+    o[k2] = m[k];
+}));
+var __setModuleDefault = (this && this.__setModuleDefault) || (Object.create ? (function(o, v) {
+    Object.defineProperty(o, "default", { enumerable: true, value: v });
+}) : function(o, v) {
+    o["default"] = v;
+});
+var __importStar = (this && this.__importStar) || function (mod) {
+    if (mod && mod.__esModule) return mod;
+    var result = {};
+    if (mod != null) for (var k in mod) if (k !== "default" && Object.prototype.hasOwnProperty.call(mod, k)) __createBinding(result, mod, k);
+    __setModuleDefault(result, mod);
+    return result;
+};
+var __importDefault = (this && this.__importDefault) || function (mod) {
+    return (mod && mod.__esModule) ? mod : { "default": mod };
+};
 Object.defineProperty(exports, "__esModule", { value: true });
 exports.archetypes = void 0;
-exports.archetypes = {
-    'node-fullstack': {
-        rules: ['sensitiveLogging-iterative', 'outdatedFramework-global', 'noDatabases-iterative', 'nonStandardDirectoryStructure-global', 'openaiAnalysisTop5-global', 'openaiAnalysisA11y-global'],
-        operators: ['fileContains', 'outdatedFramework', 'nonStandardDirectoryStructure', 'openaiAnalysisHighSeverity'],
-        facts: ['repoFilesystemFacts', 'repoDependencyFacts', 'openaiAnalysisFacts'],
-        config: {
-            minimumDependencyVersions: {
-                commander: '^2.0.0',
-                nodemon: '^3.9.0'
-            },
-            standardStructure: {
-                app: {
-                    frontend: null,
-                    server: null
-                }
-            },
-            blacklistPatterns: [
-                '.*\\/\\..*', // dot files
-                '.*\\.(log|lock)$', // file extensions blacklisted
-                '.*\\/(dist|coverage|build|node_modules)(\\/.*|$)' // directory names blacklisted
-            ],
-            whitelistPatterns: [
-                '.*\\.(ts|tsx|js|jsx|md)$' // file extensions whitelisted
-            ]
-        }
-    },
-    'java-microservice': {
-        rules: ['sensitiveLogging-iterative', 'outdatedFramework-global', 'noDatabases-iterative', 'nonStandardDirectoryStructure-global'],
-        operators: ['fileContains', 'outdatedFramework', 'nonStandardDirectoryStructure'],
-        facts: ['repoFilesystemFacts', 'repoDependencyFacts'],
-        config: {
-            minimumDependencyVersions: {
-                'spring-boot-starter': '^2.5.0',
-                'spring-boot-starter-web': '^2.5.0'
-            },
-            standardStructure: {
-                src: {
-                    main: {
-                        java: null,
-                        resources: null
-                    },
-                    test: {
-                        java: null,
-                        resources: null
-                    }
-                }
-            },
-            blacklistPatterns: [
-                '.*\\/\\..*', // dot files
-                '.*\\.(log|lock)$', // file extensions blacklisted
-                '.*\\/(target|build|out|dist|coverage|build|node_modules)(\\/.*|$)' // directory names blacklisted
-            ],
-            whitelistPatterns: [
-                '.*\\.(java|xml|properties|yml)$',
-                '.*\\/pom\\.xml$'
-            ]
-        }
+const fs = __importStar(require("fs"));
+const path = __importStar(require("path"));
+const logger_1 = __importDefault(require("../utils/logger"));
+function loadArchetypeFromJson(fileName) {
+    const filePath = path.join(__dirname, fileName);
+    const fileContent = fs.readFileSync(filePath, 'utf8');
+    try {
+        return JSON.parse(fileContent);
+    }
+    catch (error) {
+        logger_1.default.error(`Error parsing JSON in file ${fileName}: ${error}`);
+        return {}; // Return an empty object as a fallback
     }
+}
+exports.archetypes = {
+    'node-fullstack': loadArchetypeFromJson('node-fullstack.json'),
+    'java-microservice': loadArchetypeFromJson('java-microservice.json')
 };

package/dist/archetypes/java-microservice.json

@@ -0,0 +1,44 @@
+{
+    "rules": [
+        "sensitiveLogging-iterative",
+        "outdatedFramework-global",
+        "noDatabases-iterative",
+        "nonStandardDirectoryStructure-global"
+    ],
+    "operators": [
+        "fileContains",
+        "outdatedFramework",
+        "nonStandardDirectoryStructure"
+    ],
+    "facts": [
+        "repoFilesystemFacts",
+        "repoDependencyFacts"
+    ],
+    "config": {
+        "minimumDependencyVersions": {
+            "spring-boot-starter": "^2.5.0",
+            "spring-boot-starter-web": "^2.5.0"
+        },
+        "standardStructure": {
+            "src": {
+                "main": {
+                    "java": null,
+                    "resources": null
+                },
+                "test": {
+                    "java": null,
+                    "resources": null
+                }
+            }
+        },
+        "blacklistPatterns": [
+            ".*\\/\\..*",
+            ".*\\.(log|lock)$",
+            ".*\\/(target|build|out|dist|coverage|build|node_modules)(\\/.*|$)"
+        ],
+        "whitelistPatterns": [
+            ".*\\.(java|xml|properties|yml)$",
+            ".*\\/pom\\.xml$"
+        ]
+    }
+}