purecontext-mcp 1.1.1 → 1.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. package/package.json +1 -1
  2. package/docs/dev/API_STABILITY.md +0 -319
  3. package/docs/dev/DECISIONS.md +0 -22
  4. package/docs/dev/DOCUMENTATION_PLAN.md +0 -113
  5. package/docs/dev/PHASE10_TASKS.md +0 -476
  6. package/docs/dev/PHASE11_TASKS.md +0 -385
  7. package/docs/dev/PHASE12_TASKS.md +0 -335
  8. package/docs/dev/PHASE13_TASKS.md +0 -381
  9. package/docs/dev/PHASE14_TASKS.md +0 -371
  10. package/docs/dev/PHASE15_TASKS.md +0 -256
  11. package/docs/dev/PHASE16_TASKS.md +0 -314
  12. package/docs/dev/PHASE17_TASKS.md +0 -321
  13. package/docs/dev/PHASE18_TASKS.md +0 -345
  14. package/docs/dev/PHASE19_TASKS.md +0 -261
  15. package/docs/dev/PHASE1_TASKS.md +0 -443
  16. package/docs/dev/PHASE20_TASKS.md +0 -280
  17. package/docs/dev/PHASE21_TASKS.md +0 -355
  18. package/docs/dev/PHASE22_TASKS.md +0 -371
  19. package/docs/dev/PHASE23_TASKS.md +0 -274
  20. package/docs/dev/PHASE24_TASKS.md +0 -326
  21. package/docs/dev/PHASE25_TASKS.md +0 -452
  22. package/docs/dev/PHASE26_TASKS.md +0 -253
  23. package/docs/dev/PHASE27_TASKS.md +0 -410
  24. package/docs/dev/PHASE2_TASKS.md +0 -328
  25. package/docs/dev/PHASE3_TASKS.md +0 -571
  26. package/docs/dev/PHASE4_TASKS.md +0 -531
  27. package/docs/dev/PHASE5_TASKS.md +0 -835
  28. package/docs/dev/PHASE6_TASKS.md +0 -347
  29. package/docs/dev/PHASE7_TASKS.md +0 -257
  30. package/docs/dev/PHASE8_TASKS.md +0 -299
  31. package/docs/dev/PHASE9_TASKS.md +0 -320
  32. package/docs/dev/PureContext_MCP_PRD_v1.0.docx +0 -0
  33. package/docs/dev/SELF_HOSTING.md +0 -142
  34. package/docs/dev/TEAM_SETUP.md +0 -316
  35. package/docs/dev/TELEMETRY.md +0 -99
  36. package/docs/dev/feature-analysis.md +0 -305
  37. package/docs/dev/phase-1-notes.md +0 -3
@@ -1,299 +0,0 @@
1
- # Phase 8 — Task Breakdown
2
-
3
- **Goal**: Implement a token savings tracker that measures and reports how many tokens PureContext saves compared to reading raw files. This provides tangible value metrics for users and enables cost-aware decision-making.
4
-
5
- **Scope rationale**: Token efficiency is the core value proposition of PureContext — agents retrieve precisely the code they need instead of entire files. Phase 8 makes this value visible and measurable. The implementation mirrors jCodeMunch's approach (which has proven effective in production) but is adapted for PureContext's TypeScript architecture and SQLite storage.
6
-
7
- **Approach**: Tasks are sequenced to build the foundation first (storage and estimation), then integrate tracking into each tool, and finally add reporting capabilities.
8
-
9
- ---
10
-
11
- ## Task 70: Token Tracker Core
12
-
13
- Implement the core token tracking infrastructure: persistent storage, estimation logic, and cost calculations.
14
-
15
- **Deliverables:**
16
-
17
- - `src/core/token-tracker.ts`
18
- - `BYTES_PER_TOKEN = 4` — rough but consistent approximation (cl100k_base encoding)
19
- - `estimateSavings(rawBytes: number, responseBytes: number): number`
20
- - Returns `Math.max(0, Math.floor((rawBytes - responseBytes) / BYTES_PER_TOKEN))`
21
- - Never returns negative values
22
- - `recordSavings(tokensSaved: number): number`
23
- - Adds to the running total
24
- - Returns the new cumulative total
25
- - Uses an in-memory accumulator with periodic disk flush (every 5 calls)
26
- - `getTotalSaved(): number`
27
- - Returns current cumulative total without modifying it
28
- - `costAvoided(tokensSaved: number, totalTokensSaved: number): CostEstimates`
29
- - Returns cost estimates based on model pricing tiers:
30
- ```typescript
31
- interface CostEstimates {
32
- cost_avoided: Record<string, number>; // This call
33
- total_cost_avoided: Record<string, number>; // Cumulative
34
- }
35
- ```
36
- - Pricing tiers (USD per million input tokens):
37
- - `claude_opus_4`: $15.00
38
- - `claude_sonnet_4`: $3.00
39
- - `claude_haiku_4`: $0.80
40
- - `gpt4o`: $2.50
41
- - `gpt4o_mini`: $0.15
42
- - Values rounded to 4 decimal places
43
-
44
- - `src/core/db/savings-store.ts`
45
- - Storage file: `~/.purecontext/_savings.json`
46
- - Schema:
47
- ```json
48
- {
49
- "total_tokens_saved": 123456,
50
- "anon_id": "uuid-v4",
51
- "last_updated": "2026-04-14T10:30:00Z"
52
- }
53
- ```
54
- - `loadSavings(): SavingsData`
55
- - `saveSavings(data: SavingsData): void`
56
- - File created on first save if it doesn't exist
57
- - Uses atomic write (write to temp file, then rename)
58
-
59
- - In-memory state management:
60
- - `_State` class holds:
61
- - `total: number` — cumulative total (disk + unflushed)
62
- - `unflushed: number` — delta not yet written to disk
63
- - `callCount: number` — calls since last flush
64
- - `loaded: boolean` — whether disk state has been loaded
65
- - Lazy loading: disk state loaded on first `recordSavings()` or `getTotalSaved()` call
66
- - Flush triggers:
67
- - Every 5 calls (`FLUSH_INTERVAL = 5`)
68
- - On process exit (`process.on('exit')`)
69
- - On SIGTERM/SIGINT (graceful shutdown)
70
-
71
- **Key technical notes:**
72
- - The in-memory accumulator avoids disk I/O on every tool call — critical for performance
73
- - SIGTERM handling is important because MCP servers are commonly killed via pipe close
74
- - The `anon_id` field is reserved for optional future telemetry (not implemented in Phase 8)
75
- - Token estimation is deliberately simple (bytes/4) — matches jCodeMunch and is consistent
76
-
77
- **Verify:** Call `recordSavings(1000)` five times. Verify file is created after fifth call. Kill process with SIGTERM, restart, verify total persists. Verify `costAvoided()` returns correct USD values for various token counts.
78
-
79
- **Tests:** `estimateSavings`: raw > response (positive), raw < response (zero, not negative), raw == response (zero). `recordSavings`: accumulation, flush trigger at interval. `costAvoided`: verify arithmetic for each pricing tier. Persistence: write, kill process, reload, verify total.
80
-
81
- ---
82
-
83
- ## Task 71: Integrate Token Tracking into Retrieval Tools
84
-
85
- Add token savings tracking to all tools that retrieve symbol or file content.
86
-
87
- **Deliverables:**
88
-
89
- - Update `src/server/tools/get-symbol-source.ts`:
90
- - Before returning response:
91
- 1. Calculate `rawBytes`: file size of the source file (from `files` table or `fs.statSync`)
92
- 2. Calculate `responseBytes`: `symbol.endByte - symbol.startByte`
93
- 3. Call `estimateSavings(rawBytes, responseBytes)`
94
- 4. Call `recordSavings(tokensSaved)`
95
- 5. Add to response `_meta`:
96
- ```json
97
- {
98
- "tokens_saved": 1234,
99
- "total_tokens_saved": 56789,
100
- "cost_avoided": { "claude_opus_4": 0.0185, ... },
101
- "total_cost_avoided": { "claude_opus_4": 0.8518, ... }
102
- }
103
- ```
104
-
105
- - Update `src/server/tools/search-symbols.ts`:
106
- - For each returned symbol:
107
- - Accumulate `rawBytes` from unique files containing results
108
- - Accumulate `responseBytes` from symbol byte lengths
109
- - Calculate aggregate savings across all results
110
- - Add `_meta` fields as above
111
-
112
- - Update `src/server/tools/get-file-outline.ts`:
113
- - `rawBytes`: full file size
114
- - `responseBytes`: sum of all symbol signatures + metadata (not full source)
115
- - Lower savings ratio expected (outlines are compact)
116
-
117
- - Update `src/server/tools/get-context-bundle.ts`:
118
- - `rawBytes`: sum of all files that would need to be read for full context
119
- - `responseBytes`: actual bundle size (symbol + imports)
120
-
121
- - Update `src/server/tools/find-importers.ts` and `src/server/tools/get-blast-radius.ts`:
122
- - `rawBytes`: sum of all files in the traversal
123
- - `responseBytes`: actual response JSON size
124
-
125
- **Key technical notes:**
126
- - File size lookup should use cached values from the `files` table when available
127
- - For multi-symbol responses, deduplicate file sizes (same file counted once even if multiple symbols from it)
128
- - `responseBytes` calculation varies by tool — some return source, some return metadata only
129
-
130
- **Verify:** Call `get-symbol-source` → verify `_meta.tokens_saved` is positive and plausible. Call `search-symbols` with 5 results → verify aggregate savings. Verify `total_tokens_saved` increases across multiple calls.
131
-
132
- **Tests:** Per-tool: mock file sizes and symbol ranges, verify correct savings calculation. Multi-symbol deduplication: two symbols from same file, file size counted once. Cumulative total: multiple tool calls, verify running total increases.
133
-
134
- ---
135
-
136
- ## Task 72: Token Savings Statistics Tool
137
-
138
- Add a dedicated tool for querying token savings statistics.
139
-
140
- **Deliverables:**
141
-
142
- - `src/server/tools/get-savings-stats.ts`
143
- - Tool name: `get-savings-stats`
144
- - Input schema:
145
- ```json
146
- {
147
- "reset": { "type": "boolean", "default": false, "description": "Reset the counter to zero" }
148
- }
149
- ```
150
- - Output:
151
- ```json
152
- {
153
- "total_tokens_saved": 123456,
154
- "equivalent_context_windows": {
155
- "claude_200k": 0.62,
156
- "gpt4_128k": 0.96
157
- },
158
- "total_cost_avoided": {
159
- "claude_opus_4": 1.8518,
160
- "claude_sonnet_4": 0.3704,
161
- "claude_haiku_4": 0.0988,
162
- "gpt4o": 0.3086,
163
- "gpt4o_mini": 0.0185
164
- },
165
- "session_start": "2026-04-14T08:00:00Z",
166
- "_meta": {
167
- "powered_by": "PureContext MCP"
168
- }
169
- }
170
- ```
171
- - `equivalent_context_windows`: tokens saved divided by model context window sizes
172
- - Claude 200k: 200,000 tokens
173
- - GPT-4 128k: 128,000 tokens
174
- - If `reset: true`: clear the counter, return `{ "reset": true, "previous_total": 123456 }`
175
- - Register in `src/server/mcp-server.ts`
176
-
177
- - Update `src/core/token-tracker.ts`:
178
- - Add `resetSavings(): number`
179
- - Returns the previous total
180
- - Resets in-memory and disk state to zero
181
- - Add `getSessionStart(): Date`
182
- - Returns timestamp when the current session started (first `recordSavings()` call)
183
- - Stored in `_savings.json` as `session_start`
184
-
185
- **Key technical notes:**
186
- - The reset functionality is useful for starting fresh tracking sessions
187
- - Context window equivalents give users an intuitive sense of scale
188
- - Session start timestamp helps contextualize the savings (per-session vs all-time)
189
-
190
- **Verify:** After multiple tool calls, run `get-savings-stats` → verify total matches accumulated savings. Run with `reset: true` → verify counter resets. Make more calls → verify new accumulation starts from zero.
191
-
192
- **Tests:** Stats retrieval: mock accumulated total, verify output fields. Reset: verify previous total returned, verify new calls start from zero. Context window calculations: verify arithmetic.
193
-
194
- ---
195
-
196
- ## Task 73: Response Envelope Enhancement
197
-
198
- Standardize the `_meta` envelope across all tools to include consistent token tracking fields.
199
-
200
- **Deliverables:**
201
-
202
- - `src/server/tools/_meta.ts`
203
- - `buildMeta(options: MetaOptions): MetaEnvelope`
204
- ```typescript
205
- interface MetaOptions {
206
- timingMs: number;
207
- rawBytes?: number;
208
- responseBytes?: number;
209
- // ... other tool-specific fields
210
- }
211
-
212
- interface MetaEnvelope {
213
- timing_ms: number;
214
- tokens_saved?: number;
215
- total_tokens_saved?: number;
216
- cost_avoided?: Record<string, number>;
217
- total_cost_avoided?: Record<string, number>;
218
- powered_by: string;
219
- }
220
- ```
221
- - Automatically calculates and records savings when `rawBytes` and `responseBytes` are provided
222
- - Adds `powered_by: 'PureContext MCP'` to every response
223
-
224
- - Update all tool handlers to use `buildMeta()`:
225
- - `get-symbol-source.ts`
226
- - `search-symbols.ts`
227
- - `get-file-outline.ts`
228
- - `get-repo-outline.ts`
229
- - `get-context-bundle.ts`
230
- - `find-importers.ts`
231
- - `get-blast-radius.ts`
232
- - `search-text.ts`
233
- - `get-file-tree.ts`
234
- - `index-folder.ts` (no savings — indexing generates, doesn't retrieve)
235
- - `list-repos.ts` (no savings — metadata only)
236
-
237
- - Tools that don't retrieve content (indexing, listing) should still use `buildMeta()` for consistent `timing_ms` and `powered_by` fields, but omit savings fields.
238
-
239
- **Key technical notes:**
240
- - Centralizing meta construction ensures consistency and reduces duplication
241
- - The `powered_by` field provides attribution in tool responses
242
- - Savings fields are optional — tools that don't retrieve content omit them
243
-
244
- **Verify:** Run each tool, verify all responses have consistent `_meta` structure. Verify retrieval tools include savings fields. Verify non-retrieval tools omit savings but include timing.
245
-
246
- **Tests:** `buildMeta` with savings: verify all fields present. `buildMeta` without savings: verify savings fields absent. Verify `powered_by` always present.
247
-
248
- ---
249
-
250
- ## Task 74: Phase 8 Integration Tests
251
-
252
- Validate the complete token tracking pipeline end-to-end.
253
-
254
- **Deliverables:**
255
-
256
- - Integration tests `test/integration/phase8.test.ts`:
257
- 1. Index a fixture project, call `get-symbol-source` for a symbol → verify `tokens_saved > 0`
258
- 2. Verify `tokens_saved` calculation: `(fileSize - symbolByteLength) / 4` matches returned value
259
- 3. Call `search-symbols`, retrieve 5 results → verify aggregate savings across results
260
- 4. Call `get-file-outline` → verify savings reflect outline vs full file
261
- 5. Make 10 tool calls, call `get-savings-stats` → verify `total_tokens_saved` equals sum of individual calls
262
- 6. Call `get-savings-stats` with `reset: true` → verify reset works
263
- 7. Kill process (SIGTERM simulation), restart, verify savings persisted
264
- 8. Verify `cost_avoided` arithmetic: `tokens_saved * (rate / 1_000_000)` for each tier
265
- 9. Verify `equivalent_context_windows` arithmetic
266
- 10. Full suite regression: all Phase 1–7 tests still green
267
-
268
- - Performance verification:
269
- - Token tracking overhead: < 1ms per tool call
270
- - Disk flush: < 5ms (atomic write to small JSON file)
271
- - Memory overhead: < 1KB for in-memory state
272
-
273
- **Verify:** `npm run test` passes entirely — all Phase 1–8 tests green. Token tracking adds negligible overhead to tool response times.
274
-
275
- ---
276
-
277
- ## Order of Execution
278
-
279
- ```
280
- Task 70: Token tracker core ███░░░░░░░ Foundation
281
- Task 71: Integrate into retrieval tools ██████░░░░ Integration
282
- Task 72: Savings statistics tool ████████░░ Tool
283
- Task 73: Response envelope enhancement █████████░ Standardization
284
- Task 74: Integration tests ██████████ Polish
285
- ```
286
-
287
- Tasks proceed in order: core infrastructure (70) must be complete before integration (71), which must be complete before the stats tool (72). Task 73 can begin after Task 71 is complete. Task 74 validates everything.
288
-
289
- ---
290
-
291
- ## Post-Phase 8: What Comes Next
292
-
293
- Phase 8 establishes token efficiency as a measurable, visible feature. Future phases continue expanding PureContext capabilities:
294
-
295
- - **Phase 9**: Additional language handlers (Elixir, Haskell, Scala, R)
296
- - **Phase 10**: Additional framework adapters (Spring, Hibernate, SQLAlchemy, Django ORM, Rust/Java web frameworks)
297
- - **Phase 11**: Semantic search (HNSW) for repos with > 50k symbols
298
- - **Phase 12**: Rate limiting and multi-tenant auth for hosted deployments
299
- - **Phase 13**: Web UI for exploring the symbol graph visually
@@ -1,320 +0,0 @@
1
- # Phase 9 — Task Breakdown
2
-
3
- **Goal**: Expand language coverage to Elixir, Haskell, Scala, and R — completing PureContext's coverage of popular functional and statistical languages. Each language has a distinct syntax and paradigm, requiring careful handler implementation.
4
-
5
- **Scope rationale**: These four languages represent important niches: Elixir (Erlang VM, Phoenix framework), Haskell (pure functional, type theory), Scala (JVM functional-OOP hybrid, big data), and R (statistical computing, data science). Adding them broadens PureContext's applicability to functional programming and data science teams.
6
-
7
- **Approach**: Tasks are sequenced by complexity: Elixir and Scala have more familiar syntax, while Haskell's type system and R's syntax require more care. Each handler is independent and can be developed in parallel.
8
-
9
- ---
10
-
11
- ## Task 75: Elixir Language Handler
12
-
13
- Add Elixir as a new language handler. Elixir's macro system and module-based architecture map cleanly to PureContext's symbol model.
14
-
15
- **Deliverables:**
16
-
17
- - Download and bundle `grammars/tree-sitter-elixir.wasm`
18
- - Source: `tree-sitter-elixir` npm package
19
- - Build: `npx tree-sitter build-wasm node_modules/tree-sitter-elixir`
20
-
21
- - `src/handlers/elixir.ts` — implements `LanguageHandler`
22
- - `extensions()`: `['.ex', '.exs']`
23
- - `grammarPath()`: path to `tree-sitter-elixir.wasm`
24
- - `extractSymbols(tree, source)`:
25
- - `call` with identifier `defmodule` → kind `'class'` (modules as classes)
26
- - `call` with identifier `def` → kind `'function'` (public function)
27
- - `call` with identifier `defp` → skip (private function)
28
- - `call` with identifier `defmacro` → kind `'function'` with `frameworkMeta.elixir_macro: true`
29
- - `call` with identifier `defmacrop` → skip (private macro)
30
- - `call` with identifier `defstruct` → kind `'type'`
31
- - `call` with identifier `defprotocol` → kind `'interface'`
32
- - `call` with identifier `defimpl` → kind `'class'` with `frameworkMeta.elixir_impl: true`
33
- - `@callback` attribute → kind `'method'` (behaviour callback)
34
- - `@type` attribute → kind `'type'`
35
- - `@spec` attribute → capture as signature for the following function
36
- - Module attributes with `@moduledoc` or `@doc` → capture as docstring
37
- - `extractImports(tree, source)`:
38
- - `alias` directive: `alias Foo.Bar` → specifier = `Foo.Bar`
39
- - `import` directive: `import Foo.Bar` → specifier = `Foo.Bar`
40
- - `use` directive: `use GenServer` → specifier = `GenServer`
41
- - `require` directive: `require Logger` → specifier = `Logger`
42
- - `resolvedPath`: `null` (Elixir module resolution is external)
43
- - `isTypeOnly`: `false`
44
- - `extractDocstring(node)`:
45
- - Look for `@doc` attribute preceding function definition
46
- - Look for `@moduledoc` attribute at module start
47
- - Strip heredoc syntax (`"""`) and leading whitespace
48
-
49
- **Signature building:**
50
- - Modules: `defmodule ModuleName`
51
- - Functions: `def name(params) :: return_type` (from `@spec` if present)
52
- - Functions without spec: `def name(params)` (with arity)
53
- - Macros: `defmacro name(params)`
54
- - Protocols: `defprotocol Name`
55
- - Structs: `defstruct [:field1, :field2, ...]`
56
- - Keep signatures under 120 chars
57
-
58
- **Key technical notes:**
59
- - Elixir's `defmodule` creates nested scopes — qualify function names with module path
60
- - Pattern matching in function heads creates multiple function clauses — emit one symbol per head, not per clause
61
- - Guards (`when guard`) should be included in the signature
62
- - Elixir's `@spec` type annotations provide rich signature information
63
- - `.ex` files are compiled code; `.exs` files are scripts (both indexed)
64
-
65
- **Verify:** Index `test/fixtures/elixir-project/`. Verify modules, public functions, macros, protocols, and structs extracted. Verify private functions (`defp`) are skipped. Verify `@doc` captured as docstring.
66
-
67
- **Tests:** `defmodule`, `def` (public), `defp` (skipped), `defmacro`, `defstruct`, `defprotocol`, `defimpl`, `alias/import/use/require`, `@doc` docstring, function with `@spec`, function with guards.
68
-
69
- ---
70
-
71
- ## Task 76: Haskell Language Handler
72
-
73
- Add Haskell as a new language handler. Haskell's pure functional paradigm and advanced type system require special attention to type signatures and typeclass instances.
74
-
75
- **Deliverables:**
76
-
77
- - Download and bundle `grammars/tree-sitter-haskell.wasm`
78
- - Source: `tree-sitter-haskell` npm package
79
- - Build: `npx tree-sitter build-wasm node_modules/tree-sitter-haskell`
80
- - Note: Haskell grammar is complex; verify parsing of common constructs before proceeding
81
-
82
- - `src/handlers/haskell.ts` — implements `LanguageHandler`
83
- - `extensions()`: `['.hs', '.lhs']`
84
- - `grammarPath()`: path to `tree-sitter-haskell.wasm`
85
- - `extractSymbols(tree, source)`:
86
- - `function` node (top-level binding) → kind `'function'`
87
- - `signature` node (type signature) → associate with following function
88
- - `type_synonym` (`type X = Y`) → kind `'type'`
89
- - `newtype` declaration → kind `'type'`
90
- - `data` declaration → kind `'class'` (algebraic data type)
91
- - `class` declaration (typeclass) → kind `'interface'`
92
- - `instance` declaration → kind `'class'` with `frameworkMeta.haskell_instance: true`
93
- - Data constructors within `data` → kind `'const'`
94
- - Record fields within `data` → kind `'const'`
95
- - Pattern bindings (`x = value`) → kind `'const'`
96
- - Skip functions/bindings in `where` clauses (local scope)
97
- - `extractImports(tree, source)`:
98
- - `import` declaration: `import Data.List`, `import qualified Data.Map as M`
99
- - `importedNames`: extract from `import Data.List (sort, nub)`
100
- - `isTypeOnly`: `false` (Haskell doesn't distinguish)
101
- - Handle `qualified`, `hiding`, and `as` modifiers
102
- - `extractDocstring(node)`:
103
- - Haddock documentation: `-- |` or `{- | ... -}` preceding declarations
104
- - Also `-- ^` for argument documentation
105
- - Strip comment markers and leading whitespace
106
-
107
- **Signature building:**
108
- - Functions with signature: `functionName :: Type -> Type -> ReturnType`
109
- - Functions without signature: `functionName arg1 arg2 = ...` (pattern from definition)
110
- - Data types: `data TypeName = Constructor1 | Constructor2`
111
- - Newtypes: `newtype Name = Name WrappedType`
112
- - Type synonyms: `type Name = ExistingType`
113
- - Typeclasses: `class ClassName a where`
114
- - Instances: `instance ClassName Type where`
115
- - Keep signatures under 120 chars; truncate complex types
116
-
117
- **Key technical notes:**
118
- - Haskell type signatures are separate from function definitions — correlate by name
119
- - Multiple equations for the same function (pattern matching) → emit one symbol, use first equation's position
120
- - Typeclass instances are important for navigation — include the type being instanced
121
- - Literate Haskell (`.lhs`) uses `>` prefix for code lines — strip before parsing
122
- - GADTs and type families are advanced features — basic extraction sufficient for Phase 9
123
-
124
- **Verify:** Index `test/fixtures/haskell-project/`. Verify functions with type signatures extracted. Verify `data`, `newtype`, `type` declarations. Verify typeclass and instances. Verify Haddock comments captured.
125
-
126
- **Tests:** Function with signature, function without signature, `data` with constructors, `newtype`, `type` synonym, `class` declaration, `instance` declaration, `import` qualified, Haddock `-- |` comment, `.lhs` literate file.
127
-
128
- ---
129
-
130
- ## Task 77: Scala Language Handler
131
-
132
- Add Scala as a new language handler. Scala's hybrid OOP-FP paradigm on the JVM makes it popular for big data (Spark) and backend services.
133
-
134
- **Deliverables:**
135
-
136
- - Download and bundle `grammars/tree-sitter-scala.wasm`
137
- - Source: `tree-sitter-scala` npm package
138
- - Build: `npx tree-sitter build-wasm node_modules/tree-sitter-scala`
139
-
140
- - `src/handlers/scala.ts` — implements `LanguageHandler`
141
- - `extensions()`: `['.scala', '.sc']`
142
- - `grammarPath()`: path to `tree-sitter-scala.wasm`
143
- - `extractSymbols(tree, source)`:
144
- - `class_definition` → kind `'class'`
145
- - `trait_definition` → kind `'interface'`
146
- - `object_definition` → kind `'class'` with `frameworkMeta.scala_object: true`
147
- - `case_class_definition` → kind `'class'` with `frameworkMeta.scala_case_class: true`
148
- - `function_definition` (top-level or in object) → kind `'function'`
149
- - `function_definition` in class/trait → kind `'method'`
150
- - `val_definition` at top-level or in object → kind `'const'`
151
- - `type_definition` → kind `'type'`
152
- - `enum_definition` (Scala 3) → kind `'enum'`
153
- - `given_definition` (Scala 3 implicits) → kind `'const'` with `frameworkMeta.scala_given: true`
154
- - `extension_definition` (Scala 3) → kind `'class'` with `frameworkMeta.scala_extension: true`
155
- - Skip `private` and `protected` without `[package]` qualifier
156
- - `extractImports(tree, source)`:
157
- - `import_declaration`: `import scala.collection.mutable._`
158
- - Handle Scala's `{ A, B => C }` renaming syntax
159
- - `isTypeOnly`: `false`
160
- - `extractDocstring(node)`:
161
- - Scaladoc: `/** ... */` preceding declarations
162
- - Strip `*` prefixes and `@param`, `@return` tags for summary
163
-
164
- **Signature building:**
165
- - Functions: `def name(params: Types): ReturnType`
166
- - Classes: `class Name(params) extends Parent with Trait`
167
- - Case classes: `case class Name(field: Type, ...)`
168
- - Objects: `object Name extends Trait`
169
- - Traits: `trait Name extends Parent`
170
- - Types: `type Name = Alias` or `type Name[A] <: Upper`
171
- - Implicit/given: `given name: Type = ...`
172
- - Keep signatures under 120 chars
173
-
174
- **Key technical notes:**
175
- - Scala 2 and Scala 3 syntax differ significantly — handle both
176
- - `object` declarations are singletons, not classes — special metadata
177
- - Case classes are immutable data containers — primary constructor in signature
178
- - Scala's implicit parameters and givens are important for API understanding
179
- - Companion objects (same name as class) are common — don't deduplicate
180
-
181
- **Verify:** Index `test/fixtures/scala-project/`. Verify classes, traits, objects, case classes extracted. Verify methods distinguished from functions. Verify Scala 3 `enum`, `given`, `extension` extracted.
182
-
183
- **Tests:** `class`, `trait`, `object`, `case class`, `def` method, `def` function, `val`, `type`, `enum` (Scala 3), `given` (Scala 3), `extension` (Scala 3), `import` with renaming, Scaladoc comment, `private` method (skipped).
184
-
185
- ---
186
-
187
- ## Task 78: R Language Handler
188
-
189
- Add R as a new language handler. R's syntax for statistical computing is unique but maps to the symbol model with appropriate handling.
190
-
191
- **Deliverables:**
192
-
193
- - Download and bundle `grammars/tree-sitter-r.wasm`
194
- - Source: `tree-sitter-r` npm package
195
- - Build: `npx tree-sitter build-wasm node_modules/tree-sitter-r`
196
-
197
- - `src/handlers/r.ts` — implements `LanguageHandler`
198
- - `extensions()`: `['.r', '.R', '.Rmd']` (note: R is case-sensitive on some systems)
199
- - `grammarPath()`: path to `tree-sitter-r.wasm`
200
- - `extractSymbols(tree, source)`:
201
- - `left_assignment` where RHS is `function(...)` → kind `'function'`
202
- - Pattern: `name <- function(args) { ... }`
203
- - `right_assignment` where LHS is `function(...)` → kind `'function'`
204
- - Pattern: `function(args) { ... } -> name`
205
- - `equals_assignment` where RHS is `function(...)` → kind `'function'`
206
- - Pattern: `name = function(args) { ... }`
207
- - `left_assignment` / `equals_assignment` with non-function RHS → kind `'const'`
208
- - S3 method: function named `generic.class` → kind `'method'`
209
- - S4 class: `setClass("ClassName", ...)` call → kind `'class'`
210
- - S4 method: `setMethod("generic", ...)` call → kind `'method'`
211
- - R6 class: `R6Class("ClassName", ...)` call → kind `'class'`
212
- - Skip local function assignments (inside other functions)
213
- - `extractImports(tree, source)`:
214
- - `library(package)` and `require(package)` calls → specifier = package name
215
- - `import::from(package, func1, func2)` → specifier = package, importedNames = [func1, func2]
216
- - `::` namespace access in code (e.g., `dplyr::filter`) → log but don't emit import record
217
- - `source("file.R")` → specifier = file path (local file import)
218
- - `extractDocstring(node)`:
219
- - Roxygen2 comments: `#'` prefix lines preceding function
220
- - Strip `#' ` prefix, capture `@title`, `@description`, and first paragraph
221
-
222
- **Signature building:**
223
- - Functions: `name <- function(arg1, arg2 = default)` (preserve defaults)
224
- - S3 methods: `generic.class <- function(x, ...)`
225
- - S4 classes: `setClass("ClassName", slots = c(...))`
226
- - Constants: `NAME <- value` (truncate value if long)
227
- - Keep signatures under 120 chars
228
-
229
- **Key technical notes:**
230
- - R uses `<-` for assignment more than `=` (convention)
231
- - Function names with `.` are S3 methods if they match `generic.class` pattern
232
- - R's lazy evaluation means default argument expressions can be complex
233
- - `.Rmd` files are R Markdown — extract R code chunks only
234
- - R packages use NAMESPACE files for exports — out of scope for handler, extract all top-level
235
-
236
- **Verify:** Index `test/fixtures/r-project/`. Verify functions with `<-`, `=`, and `->` assignment extracted. Verify S3 methods detected. Verify `library()` imports captured. Verify Roxygen2 comments captured.
237
-
238
- **Tests:** `<-` function, `=` function, `->` function (rare but valid), constant assignment, S3 method (`print.myclass`), `library()` import, `require()` import, Roxygen2 `#'` comment, nested function (skipped), `.Rmd` code chunk.
239
-
240
- ---
241
-
242
- ## Task 79: Phase 9 Test Fixtures and Integration Tests
243
-
244
- Validate all four new language handlers end-to-end.
245
-
246
- **Deliverables:**
247
-
248
- - `test/fixtures/elixir-project/` — representative Elixir project:
249
- - `lib/my_app.ex` — main module with functions
250
- - `lib/my_app/user.ex` — struct definition
251
- - `lib/my_app/repo.ex` — behaviour and callbacks
252
- - `lib/my_app/macros.ex` — macro definitions
253
- - `test/my_app_test.exs` — test file
254
- - `mix.exs` — project config
255
-
256
- - `test/fixtures/haskell-project/` — representative Haskell project:
257
- - `src/Main.hs` — main module with functions and types
258
- - `src/Types.hs` — data types and newtypes
259
- - `src/Typeclass.hs` — typeclass definition and instances
260
- - `app/Main.hs` — executable entry point
261
- - `stack.yaml` or `cabal.project` — project config
262
-
263
- - `test/fixtures/scala-project/` — representative Scala project:
264
- - `src/main/scala/Main.scala` — object with main method
265
- - `src/main/scala/models/User.scala` — case class
266
- - `src/main/scala/services/UserService.scala` — class with methods
267
- - `src/main/scala/traits/Repository.scala` — trait definition
268
- - `build.sbt` — project config
269
-
270
- - `test/fixtures/r-project/` — representative R project:
271
- - `R/functions.R` — function definitions
272
- - `R/classes.R` — S3/S4/R6 classes
273
- - `R/utils.R` — utility functions with Roxygen2 docs
274
- - `inst/examples/example.R` — example script
275
- - `DESCRIPTION` — package config
276
-
277
- - Integration tests `test/integration/phase9.test.ts`:
278
- 1. Elixir: `defmodule` extracted as class
279
- 2. Elixir: `def` extracted, `defp` skipped
280
- 3. Elixir: `@doc` captured as docstring
281
- 4. Elixir: `alias`, `import`, `use` captured as imports
282
- 5. Haskell: function with type signature extracted
283
- 6. Haskell: `data` type with constructors
284
- 7. Haskell: `class` and `instance` declarations
285
- 8. Haskell: Haddock `-- |` captured
286
- 9. Scala: `class`, `object`, `trait` extracted
287
- 10. Scala: `case class` with scala_case_class metadata
288
- 11. Scala: Scala 3 `enum` and `given` (if grammar supports)
289
- 12. R: function with `<-` assignment
290
- 13. R: S3 method detection (`print.myclass`)
291
- 14. R: `library()` captured as import
292
- 15. R: Roxygen2 `#'` captured
293
- 16. Full suite regression: all Phase 1–8 tests still green
294
-
295
- **Verify:** `npm run test` passes entirely — all Phase 1–9 tests green. Verify token tracking works correctly for new language handlers.
296
-
297
- ---
298
-
299
- ## Order of Execution
300
-
301
- ```
302
- Task 75: Elixir language handler ██░░░░░░░░ Languages
303
- Task 76: Haskell language handler ████░░░░░░ Languages
304
- Task 77: Scala language handler ██████░░░░ Languages
305
- Task 78: R language handler ████████░░ Languages
306
- Task 79: Fixtures + integration tests ██████████ Polish
307
- ```
308
-
309
- Tasks 75–78 are fully independent and can be developed in parallel. Task 79 validates everything once all handlers are complete.
310
-
311
- ---
312
-
313
- ## Post-Phase 9: What Comes Next
314
-
315
- Phase 9 completes PureContext's language expansion to 18 languages. Future phases add framework adapters and advanced features:
316
-
317
- - **Phase 10**: Additional framework adapters (Spring, Hibernate, SQLAlchemy, Django ORM, Axum, Actix-web, Rocket, Spring Boot, Micronaut, Quarkus)
318
- - **Phase 11**: Semantic search (HNSW) for repos with > 50k symbols
319
- - **Phase 12**: Rate limiting and multi-tenant auth for hosted deployments
320
- - **Phase 13**: Web UI for exploring the symbol graph visually