dslop 1.6.0 → 1.7.0

package/CHANGELOG.md

@@ -1,5 +1,30 @@
 # Changelog
 
+## v1.7.0
+
+[compare changes](https://github.com/turf-sports/dslop/compare/v1.6.1...v1.7.0)
+
+### 🚀 Enhancements
+
+- AST-only output, filter same-file dupes, sort by occurrences ([53acc46](https://github.com/turf-sports/dslop/commit/53acc46))
+
+### ❤️ Contributors
+
+- Siddharth Sharma <sharmasiddharthcs@gmail.com>
+
+## v1.6.1
+
+[compare changes](https://github.com/turf-sports/dslop/compare/v1.6.0...v1.6.1)
+
+### 🩹 Fixes
+
+- Suppress all logs when using --json output ([59e30f1](https://github.com/turf-sports/dslop/commit/59e30f1))
+- Compact JSON output (90K→4K lines) ([3e93895](https://github.com/turf-sports/dslop/commit/3e93895))
+
+### ❤️ Contributors
+
+- Siddharth Sharma <sharmasiddharthcs@gmail.com>
+
 ## v1.6.0
 
 [compare changes](https://github.com/turf-sports/dslop/compare/v1.5.2...v1.6.0)

package/README.md

@@ -1,12 +1,92 @@
 # dslop
 
-Find duplicate code in your codebase.
+Find duplicate functions, types, and code in your codebase using AST analysis.
 
 ```bash
 npx dslop
 ```
 
-By default, checks your branch changes against the codebase. If no changes found, automatically does a full scan.
+## What it finds
+
+**Real duplicates, not noise.** dslop uses AST parsing to find semantically identical code - functions with the same logic but different variable names, types defined in multiple places, copy-pasted utilities.
+
+```bash
+$ dslop --all --json
+
+{
+  "summary": { "duplicateGroups": 101 },
+  "duplicates": [
+    {
+      "type": "function",
+      "name": "loadEnv",
+      "occurrences": 6,
+      "locations": [
+        { "name": "loadEnv", "file": "scripts/migrate.ts", "line": 8 },
+        { "name": "loadEnv", "file": "scripts/seed.ts", "line": 12 },
+        { "name": "loadEnv", "file": "lib/db/migrate.ts", "line": 7 },
+        { "name": "loadEnv", "file": "lib/db/check-state.ts", "line": 9 }
+      ]
+    },
+    {
+      "type": "interface",
+      "name": "Params",
+      "occurrences": 6,
+      "locations": [
+        { "name": "Params", "file": "app/api/games/[id]/route.ts", "line": 5 },
+        { "name": "Params", "file": "app/api/games/[id]/state/route.ts", "line": 8 },
+        { "name": "Params", "file": "app/api/games/[id]/details/route.ts", "line": 12 }
+      ]
+    },
+    {
+      "type": "type",
+      "name": "LeaderboardPlayer",
+      "occurrences": 3,
+      "locations": [
+        { "name": "LeaderboardPlayer", "file": "packages/types/leaderboard.ts", "line": 45, "exported": true },
+        { "name": "LeaderboardPlayer", "file": "apps/mobile/types.ts", "line": 52, "exported": false },
+        { "name": "LeaderboardPlayer", "file": "apps/web/lib/types.ts", "line": 30, "exported": false }
+      ]
+    }
+  ]
+}
+```
+
+### Monorepo cross-package duplicates
+
+Find types and functions duplicated across packages:
+
+```bash
+$ dslop --all --cross-package
+
+Found 48 duplicate functions/types
+
+# Types duplicated between packages/types and apps/
+PrizeDistribution    packages/types/game.ts ↔ packages/db/schema/game.ts
+DevicePlatform       apps/web/lib/notifications.ts ↔ apps/listener/lib/notifications.ts
+TeamColors           apps/mobile/store/types.ts ↔ apps/web/lib/types/colors.ts
+
+# Functions copy-pasted between apps/
+subscribeToChannel   apps/web/lib/subscriptions.ts ↔ apps/listener/lib/subscriptions.ts
+getTeamLogoUrl       packages/shared/logos.ts → also in apps/web (3 copies)
+normalizeError       apps/web/sentry.config.ts ↔ apps/listener/lib/sentry.ts
+```
+
+### PR review mode
+
+By default, dslop checks your branch changes against the existing codebase:
+
+```bash
+$ dslop
+
+Scanning...
+  Mode: checking changed lines in 3 files
+
+Found 2 duplicate functions/types
+
+# You're adding code that already exists elsewhere:
+getUserDisplayName   your change: app/profile/page.tsx:19
+                     exists in: components/sidebar.tsx:50
+```
 
 ## Install
 
@@ -18,9 +98,10 @@ npm i -g dslop
 
 ```bash
 dslop                        # check PR changes (or full scan if none)
-dslop ./apps/web             # scan apps/web (full if no changes there)
-dslop -c                     # changes only, exit if none found
-dslop --cross-package        # cross-package dupes (monorepos)
+dslop --all                  # full codebase scan
+dslop --all --json           # JSON output for tooling
+dslop --cross-package        # only cross-package dupes (monorepos)
+dslop ./apps/web             # scan specific directory
 ```
 
 ## Options
@@ -32,47 +113,47 @@ dslop --cross-package        # cross-package dupes (monorepos)
 | `-m, --min-lines` | min lines per block (default: 4) |
 | `-s, --similarity` | similarity threshold 0-100 (default: 70) |
 | `-e, --extensions` | file extensions (default: ts,tsx,js,jsx) |
-| `--cross-package` | only show dupes across packages |
-| `--json` | json output |
+| `--cross-package` | only show dupes across packages/apps |
+| `--json` | JSON output |
 
 ## How it works
 
-dslop uses two detection methods in parallel:
-
-### 1. AST-based detection (functions/classes)
+dslop parses TypeScript/JavaScript with Babel and extracts functions, classes, types, and interfaces. It normalizes the AST by replacing all identifiers with generic placeholders (`$0`, `$1`, etc.), preserving only the code structure.
 
-Parses TypeScript/JavaScript with Babel to extract functions and classes. Normalizes the AST by replacing all identifiers with generic placeholders (`$0`, `$1`, etc.), preserving only the code structure.
-
-**This catches:**
+This catches:
 - Functions with identical logic but different variable names
-- Renamed copies of existing functions
-- Structurally identical classes
-
-Example: `calculateSum(numbers)` and `computeTotal(items)` with the same loop structure will match.
-
-### 2. Text-based detection (code blocks)
-
-Sliding window over source files extracts overlapping blocks at sizes 4, 6, 9, 13... lines. Before hashing, code is normalized:
-- String literals → `"<STRING>"`
-- Numbers → `<NUMBER>`
-- Whitespace collapsed
-- Comments preserved (intentional - comments often indicate copy-paste)
-
-Exact hash matches = exact duplicates. For similar (non-exact) matches, uses character-level similarity.
+- Types/interfaces defined in multiple places
+- Copy-pasted utilities across packages
+
+Example: these two functions are detected as duplicates:
+
+```ts
+// apps/web/utils.ts
+function getUserInitials(user: User): string {
+  const first = user.firstName?.[0] ?? '';
+  const last = user.lastName?.[0] ?? '';
+  return (first + last).toUpperCase();
+}
+
+// apps/admin/helpers.ts  
+function getInitials(person: Person): string {
+  const f = person.firstName?.[0] ?? '';
+  const l = person.lastName?.[0] ?? '';
+  return (f + l).toUpperCase();
+}
+```
 
-### Smart defaults
+### What it ignores
 
-1. If you have branch changes → checks those against the codebase
-2. If no changes found → automatically scans the entire target path
-3. Use `-c` to force changes-only mode (useful in CI)
+- Same-file duplicates (patterns within a single file)
+- Tiny functions (configurable via `--min-lines`)
+- Common patterns from UI libraries (shadcn components, etc.)
 
 ## Limitations
 
-- **TypeScript/JavaScript only for AST:** AST parsing uses Babel with TS/JSX plugins. Other languages fall back to text-based only.
-- **No cross-language:** Won't detect a Python function duplicated in TypeScript.
-- **Comments affect text matching:** Intentional tradeoff. Copy-pasted code often includes comments.
-- **Minimum 4 lines:** Shorter duplicates ignored to reduce noise. Use `-m 2` for stricter.
-- **Memory:** Loads all blocks in memory. Very large codebases (>1M lines) may be slow.
+- **TypeScript/JavaScript only:** AST parsing uses Babel with TS/JSX plugins
+- **No cross-language:** Won't detect duplication across languages
+- **Memory:** Loads all AST nodes in memory. Very large codebases (>1M lines) may be slow
 
 ## License
 

package/dist/index.cjs

@@ -43616,9 +43616,6 @@ var TEMPLATE_PLACEHOLDER = "<TEMPLATE>";
 var NUMBER_PLACEHOLDER = "<NUMBER>";
 var COLOR_PLACEHOLDER = "<COLOR>";
 var MAX_PATH_DISPLAY_LENGTH = 60;
-var CODE_PREVIEW_CONTEXT_LINES = 3;
-var MAX_MATCHES_IN_SUMMARY = 5;
-var MAX_GROUPS_DETAILED = 20;
 var SECTION_SEPARATOR = "\u2500".repeat(80);
 var COLORS = {
   reset: "\x1B[0m",
@@ -44066,12 +44063,17 @@ function findDuplicates(blocks, minSimilarity, basePath) {
   const filteredSimilar = similarDuplicates.filter((g) => g.similarity >= minSimilarity);
   const allDuplicates = [...exactDuplicates, ...filteredSimilar];
   allDuplicates.sort((a, b) => {
-    const impactA = a.occurrences * a.lineCount;
-    const impactB = b.occurrences * b.lineCount;
-    return impactB - impactA;
+    if (b.occurrences !== a.occurrences) {
+      return b.occurrences - a.occurrences;
+    }
+    return b.lineCount - a.lineCount;
   });
   const dedupedGroups = deduplicateGroups(allDuplicates);
-  return dedupedGroups.map((group) => ({
+  const crossFileGroups = dedupedGroups.filter((group) => {
+    const uniqueFiles = new Set(group.matches.map((m) => m.filePath));
+    return uniqueFiles.size > 1;
+  });
+  return crossFileGroups.map((group) => ({
     ...group,
     suggestion: generateRefactoringSuggestion(group, basePath)
   }));
@@ -44713,7 +44715,11 @@ function findASTDuplicates(blocks, _minSimilarity) {
     });
   }
   groups.sort((a, b) => b.matches.length - a.matches.length);
-  return groups;
+  const crossFileGroups = groups.filter((group) => {
+    const uniqueFiles = new Set(group.matches.map((m) => m.filePath));
+    return uniqueFiles.size > 1;
+  });
+  return crossFileGroups;
 }
 
 // src/formatter.ts
@@ -44736,175 +44742,6 @@ function truncatePath(filePath, basePath) {
   }
   return relativePath;
 }
-function getSimilarityBadge(similarity) {
-  const percent = Math.round(similarity * 100);
-  if (similarity === 1) {
-    return `${red}${bold}EXACT${reset}`;
-  } else if (similarity >= 0.9) {
-    return `${yellow}${percent}%${reset}`;
-  } else {
-    return `${cyan}${percent}%${reset}`;
-  }
-}
-function formatGroup(group, index, basePath) {
-  const lines = [];
-  const badge = getSimilarityBadge(group.similarity);
-  const impact = group.occurrences * group.lineCount;
-  lines.push(`${bold}Group ${index + 1}${reset} \u2502 ${badge} \u2502 ${group.lineCount} lines \xD7 ${group.occurrences} occurrences = ${green}${impact} lines${reset} of duplication`);
-  lines.push("");
-  const matchesToShow = group.matches.slice(0, MAX_MATCHES_IN_SUMMARY);
-  const hasMore = group.matches.length > MAX_MATCHES_IN_SUMMARY;
-  for (const match2 of matchesToShow) {
-    const displayPath = truncatePath(match2.filePath, basePath);
-    lines.push(`  ${dim}\u251C\u2500${reset} ${displayPath}:${yellow}${match2.startLine}${reset}-${yellow}${match2.endLine}${reset}`);
-  }
-  if (hasMore) {
-    lines.push(`  ${dim}\u2514\u2500${reset} ${gray}... and ${group.matches.length - MAX_MATCHES_IN_SUMMARY} more${reset}`);
-  }
-  const firstMatch = group.matches[0];
-  if (firstMatch) {
-    lines.push("");
-    lines.push(`  ${dim}Code preview:${reset}`);
-    const previewLines = firstMatch.content.split("\n").slice(0, CODE_PREVIEW_CONTEXT_LINES).map((line) => `  ${gray}\u2502${reset} ${dim}${line.slice(0, 80)}${line.length > 80 ? "..." : ""}${reset}`);
-    lines.push(...previewLines);
-    if (firstMatch.content.split("\n").length > CODE_PREVIEW_CONTEXT_LINES) {
-      lines.push(`  ${gray}\u2502${reset} ${dim}...${reset}`);
-    }
-  }
-  if (group.suggestion) {
-    lines.push("");
-    const confidenceColor = group.suggestion.confidence === "high" ? green : group.suggestion.confidence === "medium" ? yellow : gray;
-    lines.push(`  ${magenta}\u2192 Suggestion:${reset} Move to ${cyan}${group.suggestion.targetLocation}${reset}`);
-    if (group.suggestion.suggestedName) {
-      lines.push(`  ${dim}  Name: ${group.suggestion.suggestedName}${reset}`);
-    }
-    lines.push(`  ${dim}  ${group.suggestion.reason} ${confidenceColor}[${group.suggestion.confidence}]${reset}`);
-  }
-  return lines.join("\n");
-}
-function formatOutput(groups, basePath) {
-  if (groups.length === 0) {
-    return `${green}No duplicates found!${reset}`;
-  }
-  const lines = [];
-  lines.push(SECTION_SEPARATOR);
-  lines.push(`${bold}DUPLICATE CODE DETECTED${reset}`);
-  lines.push(SECTION_SEPARATOR);
-  lines.push("");
-  const groupsToShow = groups.slice(0, MAX_GROUPS_DETAILED);
-  for (let i = 0; i < groupsToShow.length; i++) {
-    const group = groupsToShow[i];
-    if (group) {
-      lines.push(formatGroup(group, i, basePath));
-      lines.push("");
-    }
-  }
-  if (groups.length > MAX_GROUPS_DETAILED) {
-    lines.push(`${dim}... and ${groups.length - MAX_GROUPS_DETAILED} more groups${reset}`);
-    lines.push("");
-  }
-  lines.push(SECTION_SEPARATOR);
-  return lines.join("\n");
-}
-function formatStats(groups) {
-  if (groups.length === 0) {
-    return "";
-  }
-  const totalDuplicateLines = groups.reduce(
-    (sum, g) => sum + g.lineCount * g.occurrences,
-    0
-  );
-  const exactMatches = groups.filter((g) => g.similarity === 1).length;
-  const similarMatches = groups.length - exactMatches;
-  const avgSimilarity = groups.reduce((sum, g) => sum + g.similarity, 0) / groups.length;
-  const uniqueFiles = /* @__PURE__ */ new Set();
-  for (const group of groups) {
-    for (const match2 of group.matches) {
-      uniqueFiles.add(match2.filePath);
-    }
-  }
-  const lines = [];
-  lines.push(`${bold}SUMMARY${reset}`);
-  lines.push(SECTION_SEPARATOR);
-  lines.push(`  Total duplicate groups:    ${bold}${groups.length}${reset}`);
-  lines.push(`  Exact matches:             ${bold}${exactMatches}${reset}`);
-  lines.push(`  Similar matches:           ${bold}${similarMatches}${reset}`);
-  lines.push(`  Files affected:            ${bold}${uniqueFiles.size}${reset}`);
-  lines.push(`  Total duplicate lines:     ${red}${bold}${totalDuplicateLines.toLocaleString()}${reset}`);
-  lines.push(`  Average similarity:        ${bold}${Math.round(avgSimilarity * 100)}%${reset}`);
-  lines.push(SECTION_SEPARATOR);
-  lines.push("");
-  lines.push(`${dim}Tip: Use --json for machine-readable output${reset}`);
-  return lines.join("\n");
-}
-var TYPE_LABELS = {
-  type: "Type",
-  interface: "Interface",
-  function: "Function",
-  class: "Class",
-  const: "Constant",
-  enum: "Enum"
-};
-function formatDeclarationGroup(group, index, basePath) {
-  const lines = [];
-  const typeLabel = TYPE_LABELS[group.type] || group.type;
-  const simBadge = group.similarity >= 0.95 ? `${red}${bold}EXACT${reset}` : `${yellow}${Math.round(group.similarity * 100)}%${reset}`;
-  lines.push(`${bold}${typeLabel} ${index + 1}${reset} \u2502 ${simBadge} \u2502 ${group.matches.length} occurrences`);
-  if (group.nameSimilarity > 0 && group.nameSimilarity < 1) {
-    lines.push(`  ${dim}Name similarity: ${Math.round(group.nameSimilarity * 100)}%${reset}`);
-  }
-  lines.push("");
-  for (const match2 of group.matches.slice(0, 5)) {
-    const displayPath = truncatePath(match2.filePath, basePath);
-    const exportBadge = match2.exported ? `${green}exported${reset}` : `${gray}local${reset}`;
-    lines.push(`  ${dim}\u251C\u2500${reset} ${cyan}${match2.name}${reset} [${exportBadge}]`);
-    lines.push(`     ${displayPath}:${yellow}${match2.startLine}${reset}-${yellow}${match2.endLine}${reset}`);
-  }
-  if (group.matches.length > 5) {
-    lines.push(`  ${dim}\u2514\u2500${reset} ${gray}... and ${group.matches.length - 5} more${reset}`);
-  }
-  lines.push("");
-  lines.push(`  ${magenta}\u2192${reset} ${group.suggestion}`);
-  return lines.join("\n");
-}
-function formatDeclarations(groups, basePath) {
-  if (groups.length === 0) {
-    return "";
-  }
-  const lines = [];
-  lines.push("");
-  lines.push(SECTION_SEPARATOR);
-  lines.push(`${bold}DUPLICATE DECLARATIONS${reset}`);
-  lines.push(SECTION_SEPARATOR);
-  lines.push("");
-  const byType = /* @__PURE__ */ new Map();
-  for (const group of groups) {
-    const existing = byType.get(group.type) || [];
-    existing.push(group);
-    byType.set(group.type, existing);
-  }
-  let globalIndex = 0;
-  for (const [type, typeGroups] of byType) {
-    const typeLabel = TYPE_LABELS[type] || type;
-    lines.push(`${bold}${typeLabel}s (${typeGroups.length})${reset}`);
-    lines.push("");
-    for (const group of typeGroups.slice(0, 10)) {
-      lines.push(formatDeclarationGroup(group, globalIndex++, basePath));
-      lines.push("");
-    }
-    if (typeGroups.length > 10) {
-      lines.push(`${dim}... and ${typeGroups.length - 10} more ${typeLabel.toLowerCase()}s${reset}`);
-      lines.push("");
-    }
-  }
-  lines.push(SECTION_SEPARATOR);
-  const totalDups = groups.reduce((sum, g) => sum + g.matches.length, 0);
-  lines.push(`${bold}Declaration Summary${reset}`);
-  lines.push(`  Duplicate groups: ${bold}${groups.length}${reset}`);
-  lines.push(`  Total occurrences: ${bold}${totalDups}${reset}`);
-  lines.push(SECTION_SEPARATOR);
-  return lines.join("\n");
-}
 var AST_TYPE_LABELS = {
   function: "Function",
   arrow: "Arrow Function",
@@ -51724,7 +51561,7 @@ async function scanDirectory(targetPath, options, enableAST = true) {
 }
 
 // index.ts
-var VERSION = process.env.npm_package_version || "1.6.0";
+var VERSION = process.env.npm_package_version || "1.7.0";
 function parseDiffOutput(diff, cwd) {
   const changes = /* @__PURE__ */ new Map();
   let currentFile = null;
@@ -51993,16 +51830,14 @@ Scanning ${targetPath}...`);
       astDuplicates = filterCrossPackage(astDuplicates);
       declDuplicates = filterCrossPackage(declDuplicates);
     }
-    const totalGroups = duplicates.length + astDuplicates.length + declDuplicates.length;
-    console.log(`Found ${totalGroups} duplicate groups in ${Math.round(detectTime)}ms`);
-    if (astDuplicates.length > 0 || declDuplicates.length > 0) {
-      console.log(`  (${duplicates.length} blocks, ${astDuplicates.length} AST, ${declDuplicates.length} declarations)
+    if (!jsonOutput) {
+      console.log(`Found ${astDuplicates.length} duplicate functions/types in ${Math.round(detectTime)}ms
 `);
-    } else {
-      console.log();
     }
-    if (totalGroups === 0) {
-      if (!scanAll) {
+    if (astDuplicates.length === 0) {
+      if (jsonOutput) {
+        console.log(JSON.stringify({ summary: { duplicateGroups: 0 }, duplicates: [] }, null, 2));
+      } else if (!scanAll) {
         console.log("No duplicates in your changes. You're good!");
       } else if (crossPackage) {
         console.log("No cross-package duplicates found!");
@@ -52012,17 +51847,28 @@ Scanning ${targetPath}...`);
       process.exit(0);
     }
     if (jsonOutput) {
-      console.log(JSON.stringify({ duplicates, ast: astDuplicates, declarations: declDuplicates }, null, 2));
+      const compactAST = astDuplicates.slice(0, 50).map((a) => ({
+        type: a.type,
+        name: a.matches[0]?.name || "unknown",
+        occurrences: a.matches.length,
+        locations: a.matches.map((m) => ({
+          name: m.name,
+          file: m.filePath.replace(process.cwd() + "/", ""),
+          line: m.startLine,
+          exported: m.exported
+        }))
+      }));
+      console.log(JSON.stringify({
+        summary: {
+          duplicateGroups: astDuplicates.length
+        },
+        duplicates: compactAST
+      }, null, 2));
     } else {
       if (astDuplicates.length > 0) {
         console.log(formatASTDuplicates(astDuplicates, targetPath));
-      }
-      if (duplicates.length > 0) {
-        console.log(formatOutput(duplicates, targetPath));
-        console.log(formatStats(duplicates));
-      }
-      if (declDuplicates.length > 0) {
-        console.log(formatDeclarations(declDuplicates, targetPath));
+      } else {
+        console.log("No duplicate functions or types found!");
       }
     }
   } catch (error) {

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "dslop",
-  "version": "1.6.0",
+  "version": "1.7.0",
   "description": "Detect Similar/Duplicate Lines Of Programming - Find code duplication in your codebase",
   "type": "module",
   "main": "./dist/index.cjs",