@dizzlkheinz/ynab-mcpb 0.16.0 → 0.17.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.code/agents/0098661e-0fa3-4990-beb9-c0cbf3f123aa/status.txt +1 -0
- package/.code/agents/1324/exec-call_tIpx9uV1TpARbAMZonRQm8AO.txt +757 -0
- package/.code/agents/1572/exec-call_GjVFBFOWcY7lE0idc5nWlLNh.txt +781 -0
- package/.code/agents/1846/exec-call_1YNAVD18RjrMN7JnfkkQhUP3.txt +766 -0
- package/.code/agents/1846/exec-call_lh3lDzE4WJAh1lFiomiiZ73D.txt +766 -0
- package/.code/agents/2038/exec-call_DYwOukaYsL8VCONWmV2rUW5u.txt +766 -0
- package/.code/agents/2038/exec-call_c7fOQ7UrpVcTtvdfGBRM146V.txt +652 -0
- package/.code/agents/2038/exec-call_ySNyq9Mm55jWE480s54r5QcA.txt +766 -0
- package/.code/agents/2256/exec-call_AtPcRWPmFPMcmX6qOFm1fCEY.txt +766 -0
- package/.code/agents/2454/exec-call_aFJpupwjfZeOBm7ixI5Vc8z2.txt +766 -0
- package/.code/agents/2454/exec-call_wogZ4HfXTodTEXvdgXlVUBpv.txt +766 -0
- package/.code/agents/2e905864-aa07-4314-bcf9-c5b32277e4ac/result.txt +36 -0
- package/.code/agents/3073/exec-call_Peeagc9DxGYLgE6pNdMZhqIE.txt +766 -0
- package/.code/agents/3073/exec-call_d2YSE3hXF08KRSoUM3qd8Z3x.txt +766 -0
- package/.code/agents/335aa031-466d-4fb7-925f-3cd864e264d0/result.txt +191 -0
- package/.code/agents/3364/exec-call_NbhIrsM5HhyDZDmJZG5CuCYL.txt +766 -0
- package/.code/agents/3364/exec-call_cKtJg0NrXiwXEFwlsE3uPZRA.txt +766 -0
- package/.code/agents/36d98414-5cde-4d9d-9a67-a240a18c1f07/result.txt +189 -0
- package/.code/agents/4604e866-b7b8-44f5-992f-2f683b0a523b/status.txt +1 -0
- package/.code/agents/5f8dc01c-47b3-4163-b0b3-aa31be89fcdc/status.txt +1 -0
- package/.code/agents/7/exec-call_HltHpkDox0Zm1vGEjdksUgpE.txt +1120 -0
- package/.code/agents/7/exec-call_LCATrOPPAgbxW9Q1z0XaVi2E.txt +2646 -0
- package/.code/agents/7/exec-call_W8DeRfNG9hvbgVFvf0clBf6R.txt +2646 -0
- package/.code/agents/94a0ddf3-a304-4ec3-913e-3cceef509948/error.txt +1 -0
- package/.code/agents/e2c752b7-711d-423a-af57-f53c809deb84/result.txt +160 -0
- package/.code/agents/e6601719-c31f-4a0e-8c71-d70787d0ab71/status.txt +1 -0
- package/.code/agents/f250b7ed-5bd5-4036-aa8c-ce63caee7d61/result.txt +20 -0
- package/AGENTS.md +1 -36
- package/CLAUDE.md +131 -51
- package/NUL +0 -1
- package/README.md +27 -14
- package/dist/bundle/index.cjs +41 -41
- package/dist/server/YNABMCPServer.js +28 -381
- package/dist/server/config.d.ts +2 -0
- package/dist/server/config.js +1 -0
- package/dist/tools/accountTools.d.ts +2 -0
- package/dist/tools/accountTools.js +45 -0
- package/dist/tools/adapters.d.ts +12 -0
- package/dist/tools/adapters.js +25 -0
- package/dist/tools/budgetTools.d.ts +2 -0
- package/dist/tools/budgetTools.js +30 -0
- package/dist/tools/categoryTools.d.ts +2 -0
- package/dist/tools/categoryTools.js +45 -0
- package/dist/tools/monthTools.d.ts +2 -0
- package/dist/tools/monthTools.js +32 -0
- package/dist/tools/payeeTools.d.ts +2 -0
- package/dist/tools/payeeTools.js +32 -0
- package/dist/tools/reconciliation/index.d.ts +2 -0
- package/dist/tools/reconciliation/index.js +33 -0
- package/dist/tools/schemas/common.d.ts +3 -0
- package/dist/tools/schemas/common.js +3 -0
- package/dist/tools/schemas/outputs/comparisonOutputs.d.ts +1 -1
- package/dist/tools/transactionTools.d.ts +2 -0
- package/dist/tools/transactionTools.js +129 -0
- package/dist/tools/utilityTools.d.ts +3 -1
- package/dist/tools/utilityTools.js +32 -2
- package/dist/types/index.d.ts +1 -0
- package/dist/types/toolRegistration.d.ts +27 -0
- package/dist/types/toolRegistration.js +1 -0
- package/package.json +2 -2
- package/scripts/run-domain-integration-tests.js +4 -1
- package/src/__tests__/workflows.e2e.test.ts +1 -7
- package/src/server/YNABMCPServer.ts +33 -519
- package/src/server/__tests__/toolRegistration.test.ts +236 -0
- package/src/server/config.ts +1 -0
- package/src/tools/__tests__/adapters.test.ts +113 -0
- package/src/tools/__tests__/transactionTools.test.ts +90 -17
- package/src/tools/__tests__/utilityTools.test.ts +7 -7
- package/src/tools/accountTools.ts +53 -0
- package/src/tools/adapters.ts +74 -0
- package/src/tools/budgetTools.ts +37 -0
- package/src/tools/categoryTools.ts +53 -0
- package/src/tools/monthTools.ts +39 -0
- package/src/tools/payeeTools.ts +39 -0
- package/src/tools/reconciliation/index.ts +45 -0
- package/src/tools/schemas/common.ts +18 -0
- package/src/tools/transactionTools.ts +150 -0
- package/src/tools/utilityTools.ts +42 -2
- package/src/types/index.ts +3 -0
- package/src/types/toolRegistration.ts +88 -0
- package/.dxtignore +0 -57
- package/.github/workflows/pr-description-check.yml +0 -88
- package/CODEREVIEW_RESPONSE.md +0 -128
- package/SCHEMA_IMPROVEMENT_SUMMARY.md +0 -120
- package/TESTING_NOTES.md +0 -217
- package/accountactivity-merged.csv +0 -149
- package/bundle-analysis.html +0 -13110
- package/docs/README.md +0 -72
- package/docs/getting-started/CONFIGURATION.md +0 -175
- package/docs/getting-started/INSTALLATION.md +0 -333
- package/docs/getting-started/QUICKSTART.md +0 -282
- package/docs/guides/ARCHITECTURE.md +0 -533
- package/docs/guides/DEPLOYMENT.md +0 -189
- package/docs/guides/INTEGRATION_TESTING.md +0 -730
- package/docs/guides/TESTING.md +0 -591
- package/docs/plans/2025-11-20-reloadable-config-token-validation.md +0 -93
- package/docs/plans/2025-11-21-fix-transaction-cached-property.md +0 -362
- package/docs/plans/2025-11-21-reconciliation-error-handling.md +0 -90
- package/docs/plans/2025-11-21-v014-hardening.md +0 -153
- package/docs/plans/reconciliation-v2-redesign.md +0 -1571
- package/docs/reconciliation-flow.md +0 -83
- package/docs/reference/EXAMPLES.md +0 -946
- package/docs/reference/TOOLS.md +0 -348
- package/docs/reference/TROUBLESHOOTING.md +0 -481
- package/fix-types.sh +0 -17
- package/test-csv-sample.csv +0 -28
- package/test-exports/sample_bank_statement.csv +0 -7
- package/test-reconcile-autodetect.js +0 -40
- package/test-reconcile-tool.js +0 -152
- package/test-reconcile-with-csv.cjs +0 -89
- package/test-statement.csv +0 -8
- package/test_debug.js +0 -47
- package/test_mcp_tools.mjs +0 -75
- package/test_simple.mjs +0 -16
|
@@ -1,730 +0,0 @@
|
|
|
1
|
-
# Integration Testing Strategy for YNAB MCP Server
|
|
2
|
-
|
|
3
|
-
## Philosophy
|
|
4
|
-
|
|
5
|
-
**Integration tests should test real API interactions, not mocks.** This project uses a tiered testing strategy that works within YNAB's rate limits while maintaining test quality and fast feedback loops.
|
|
6
|
-
|
|
7
|
-
## YNAB API Constraints
|
|
8
|
-
|
|
9
|
-
- **Rate Limit**: 200 requests per access token per hour (rolling window)
|
|
10
|
-
- **No Official Sandbox**: YNAB does not provide a test/sandbox environment for API development
|
|
11
|
-
- **Real Data**: Tests run against real YNAB budgets (recommend using a dedicated test budget)
|
|
12
|
-
|
|
13
|
-
---
|
|
14
|
-
|
|
15
|
-
## Testing Tiers
|
|
16
|
-
|
|
17
|
-
### Tier 1: Core/Smoke Tests (~10-15 API calls, <3 minutes)
|
|
18
|
-
|
|
19
|
-
**Purpose**: Essential functionality verification during active development
|
|
20
|
-
|
|
21
|
-
**What's Tested**:
|
|
22
|
-
- List budgets
|
|
23
|
-
- Get single budget by ID
|
|
24
|
-
- List accounts for a budget
|
|
25
|
-
- Basic transaction CRUD (create, read, update, delete - one of each)
|
|
26
|
-
- Get user info
|
|
27
|
-
|
|
28
|
-
**When to Run**:
|
|
29
|
-
- Before every commit
|
|
30
|
-
- During active development (frequent)
|
|
31
|
-
- In CI/CD pipelines
|
|
32
|
-
|
|
33
|
-
**Rate Limit Impact**: Negligible (~10-15 calls = ~7-8% of hourly limit)
|
|
34
|
-
|
|
35
|
-
**NPM Script**: `npm run test:integration:core`
|
|
36
|
-
|
|
37
|
-
---
|
|
38
|
-
|
|
39
|
-
### Tier 2: Domain Tests (~30-50 API calls per domain, 5-10 minutes)
|
|
40
|
-
|
|
41
|
-
**Purpose**: Comprehensive testing of specific feature areas you're actively working on
|
|
42
|
-
|
|
43
|
-
**Domains**:
|
|
44
|
-
- **Budget Tools**: Budget listing, retrieval, settings
|
|
45
|
-
- **Account Tools**: Account management, balances, reconciliation
|
|
46
|
-
- **Transaction Tools**: Full CRUD, bulk operations, filtering
|
|
47
|
-
- **Category Tools**: Category management, activity tracking
|
|
48
|
-
- **Payee Tools**: Payee management and transactions
|
|
49
|
-
- **Month Tools**: Monthly budget data, rollover calculations
|
|
50
|
-
- **Delta Operations**: Server knowledge tracking, incremental updates
|
|
51
|
-
- **Reconciliation**: Account reconciliation workflows, recommendation engine
|
|
52
|
-
|
|
53
|
-
**When to Run**:
|
|
54
|
-
- When working on a specific domain (selective testing)
|
|
55
|
-
- Before committing changes to a domain
|
|
56
|
-
- In PR validation for changed domains only
|
|
57
|
-
|
|
58
|
-
**Rate Limit Impact**: Moderate (one domain = ~15-25% of hourly limit)
|
|
59
|
-
|
|
60
|
-
**NPM Scripts**:
|
|
61
|
-
- `npm run test:integration:budgets`
|
|
62
|
-
- `npm run test:integration:accounts`
|
|
63
|
-
- `npm run test:integration:transactions`
|
|
64
|
-
- `npm run test:integration:categories`
|
|
65
|
-
- `npm run test:integration:payees`
|
|
66
|
-
- `npm run test:integration:reconciliation`
|
|
67
|
-
- `npm run test:integration:delta`
|
|
68
|
-
|
|
69
|
-
**Throttling**: Light delays between test files (5-10 seconds) to avoid bursts
|
|
70
|
-
|
|
71
|
-
---
|
|
72
|
-
|
|
73
|
-
### Tier 3: Full Integration Suite (~200 API calls, 2-3 hours)
|
|
74
|
-
|
|
75
|
-
**Purpose**: Comprehensive validation of entire system before releases or major changes
|
|
76
|
-
|
|
77
|
-
**What's Tested**: All integration tests across all domains
|
|
78
|
-
|
|
79
|
-
**When to Run**:
|
|
80
|
-
- Before releases (manual trigger)
|
|
81
|
-
- Nightly or weekly (scheduled)
|
|
82
|
-
- After major refactoring
|
|
83
|
-
- On-demand for comprehensive validation
|
|
84
|
-
|
|
85
|
-
**Rate Limit Impact**: Uses full hourly quota (~100% of rate limit)
|
|
86
|
-
|
|
87
|
-
**NPM Script**: `npm run test:integration:full`
|
|
88
|
-
|
|
89
|
-
**Throttling**: Intelligent rate-limit-aware pacing:
|
|
90
|
-
- Monitors API rate limit headers (`X-Rate-Limit-Remaining`)
|
|
91
|
-
- Automatically delays test execution to stay under 200 requests/hour
|
|
92
|
-
- Spreads tests over 2-3 hours with smart scheduling
|
|
93
|
-
- Safe to run unattended - will never hit rate limits
|
|
94
|
-
|
|
95
|
-
---
|
|
96
|
-
|
|
97
|
-
## Test Organization
|
|
98
|
-
|
|
99
|
-
### File Structure
|
|
100
|
-
|
|
101
|
-
```
|
|
102
|
-
src/
|
|
103
|
-
├── __tests__/
|
|
104
|
-
│ ├── unit/ # Unit tests (mocked, no API)
|
|
105
|
-
│ └── testUtils.ts # Shared test utilities
|
|
106
|
-
├── server/__tests__/
|
|
107
|
-
│ ├── *.test.ts # Unit tests
|
|
108
|
-
│ └── *.integration.test.ts # Integration tests (tagged by tier)
|
|
109
|
-
└── tools/__tests__/
|
|
110
|
-
├── *.test.ts # Unit tests
|
|
111
|
-
├── *.integration.test.ts # Integration tests (tagged by tier)
|
|
112
|
-
└── *.delta.integration.test.ts # Delta-specific integration tests
|
|
113
|
-
```
|
|
114
|
-
|
|
115
|
-
### Test Tagging
|
|
116
|
-
|
|
117
|
-
Integration tests are tagged with metadata to enable tier-based filtering:
|
|
118
|
-
|
|
119
|
-
```typescript
|
|
120
|
-
import { describe, it, expect } from 'vitest';
|
|
121
|
-
|
|
122
|
-
describe('Budget Tools Integration', () => {
|
|
123
|
-
it('should list budgets', {
|
|
124
|
-
meta: { tier: 'core', domain: 'budgets' }
|
|
125
|
-
}, async () => {
|
|
126
|
-
// Test implementation
|
|
127
|
-
});
|
|
128
|
-
|
|
129
|
-
it('should get budget settings', {
|
|
130
|
-
meta: { tier: 'domain', domain: 'budgets' }
|
|
131
|
-
}, async () => {
|
|
132
|
-
// Test implementation
|
|
133
|
-
});
|
|
134
|
-
});
|
|
135
|
-
```
|
|
136
|
-
|
|
137
|
-
---
|
|
138
|
-
|
|
139
|
-
## Rate Limit Management
|
|
140
|
-
|
|
141
|
-
### Throttling Implementation
|
|
142
|
-
|
|
143
|
-
**Core Tests**: No throttling needed (fast execution)
|
|
144
|
-
|
|
145
|
-
**Domain Tests**: Light throttling between test files
|
|
146
|
-
```typescript
|
|
147
|
-
// Delay between test files in same domain
|
|
148
|
-
const DOMAIN_TEST_DELAY_MS = 5000; // 5 seconds
|
|
149
|
-
```
|
|
150
|
-
|
|
151
|
-
**Full Suite**: Intelligent throttling based on rate limit headers
|
|
152
|
-
```typescript
|
|
153
|
-
// Rate limit aware test scheduler
|
|
154
|
-
class RateLimitScheduler {
|
|
155
|
-
async scheduleTest(testFn: () => Promise<void>) {
|
|
156
|
-
const remainingCalls = this.getRemainingFromHeaders();
|
|
157
|
-
const estimatedCallsInTest = this.estimateAPICalls(testFn);
|
|
158
|
-
|
|
159
|
-
if (remainingCalls < estimatedCallsInTest + 20) { // 20 call buffer
|
|
160
|
-
const waitTime = this.calculateWaitTime();
|
|
161
|
-
console.log(`Rate limit approaching, waiting ${waitTime}ms...`);
|
|
162
|
-
await this.sleep(waitTime);
|
|
163
|
-
}
|
|
164
|
-
|
|
165
|
-
await testFn();
|
|
166
|
-
}
|
|
167
|
-
|
|
168
|
-
private calculateWaitTime(): number {
|
|
169
|
-
// Calculate time until rate limit window resets
|
|
170
|
-
// Based on X-Rate-Limit-Reset header
|
|
171
|
-
return this.getResetTime() - Date.now();
|
|
172
|
-
}
|
|
173
|
-
}
|
|
174
|
-
```
|
|
175
|
-
|
|
176
|
-
### Best Practices
|
|
177
|
-
|
|
178
|
-
1. **Monitor Rate Limit Headers**: Always check `X-Rate-Limit-Remaining` and `X-Rate-Limit-Reset` in responses
|
|
179
|
-
2. **Fail Gracefully**: If rate limit hit, pause tests and resume after reset window
|
|
180
|
-
3. **Estimate API Calls**: Track approximate API calls per test for scheduling
|
|
181
|
-
4. **Use Delta Requests**: Prefer delta/incremental API calls when possible to reduce response size
|
|
182
|
-
5. **Cache Aggressively**: Use cached data in tests when fresh data isn't required
|
|
183
|
-
|
|
184
|
-
---
|
|
185
|
-
|
|
186
|
-
## Vitest Configuration
|
|
187
|
-
|
|
188
|
-
### Project Setup
|
|
189
|
-
|
|
190
|
-
```typescript
|
|
191
|
-
// vitest.config.ts
|
|
192
|
-
import { defineConfig } from 'vitest/config';
|
|
193
|
-
|
|
194
|
-
const integrationFiles = ['src/**/*.integration.test.ts'];
|
|
195
|
-
|
|
196
|
-
export default defineConfig({
|
|
197
|
-
test: {
|
|
198
|
-
environment: 'node',
|
|
199
|
-
globals: true,
|
|
200
|
-
setupFiles: ['src/__tests__/setup.ts'],
|
|
201
|
-
projects: [
|
|
202
|
-
{
|
|
203
|
-
test: {
|
|
204
|
-
name: 'unit',
|
|
205
|
-
include: ['src/**/*.{test,spec}.ts'],
|
|
206
|
-
exclude: [
|
|
207
|
-
'src/**/*.integration.test.ts',
|
|
208
|
-
'src/**/*.e2e.test.ts',
|
|
209
|
-
'src/server/__tests__/YNABMCPServer.test.ts',
|
|
210
|
-
],
|
|
211
|
-
},
|
|
212
|
-
},
|
|
213
|
-
{
|
|
214
|
-
test: {
|
|
215
|
-
name: 'integration:core',
|
|
216
|
-
include: integrationFiles,
|
|
217
|
-
env: { INTEGRATION_TEST_TIER: 'core' },
|
|
218
|
-
testTimeout: 30000,
|
|
219
|
-
hookTimeout: 10000,
|
|
220
|
-
},
|
|
221
|
-
},
|
|
222
|
-
{
|
|
223
|
-
test: {
|
|
224
|
-
name: 'integration:domain',
|
|
225
|
-
include: integrationFiles,
|
|
226
|
-
env: { INTEGRATION_TEST_TIER: 'domain' },
|
|
227
|
-
testTimeout: 60000,
|
|
228
|
-
hookTimeout: 15000,
|
|
229
|
-
},
|
|
230
|
-
},
|
|
231
|
-
{
|
|
232
|
-
test: {
|
|
233
|
-
name: 'integration:full',
|
|
234
|
-
include: integrationFiles,
|
|
235
|
-
env: { INTEGRATION_TEST_TIER: 'full' },
|
|
236
|
-
testTimeout: 120000,
|
|
237
|
-
hookTimeout: 30000,
|
|
238
|
-
fileParallelism: false,
|
|
239
|
-
maxWorkers: 1,
|
|
240
|
-
},
|
|
241
|
-
},
|
|
242
|
-
{
|
|
243
|
-
test: {
|
|
244
|
-
name: 'e2e',
|
|
245
|
-
include: ['src/**/*.e2e.test.ts'],
|
|
246
|
-
},
|
|
247
|
-
},
|
|
248
|
-
],
|
|
249
|
-
},
|
|
250
|
-
});
|
|
251
|
-
```
|
|
252
|
-
|
|
253
|
-
---
|
|
254
|
-
|
|
255
|
-
## NPM Scripts
|
|
256
|
-
|
|
257
|
-
```json
|
|
258
|
-
{
|
|
259
|
-
"scripts": {
|
|
260
|
-
"test": "vitest run --project unit",
|
|
261
|
-
"test:unit": "vitest run --project unit",
|
|
262
|
-
"test:watch": "vitest --project unit",
|
|
263
|
-
|
|
264
|
-
"test:integration": "npm run test:integration:core",
|
|
265
|
-
"test:integration:core": "vitest run --project integration:core",
|
|
266
|
-
"test:integration:full": "node scripts/run-throttled-integration-tests.js",
|
|
267
|
-
|
|
268
|
-
"test:integration:budgets": "node scripts/run-domain-integration-tests.js budgets",
|
|
269
|
-
"test:integration:accounts": "node scripts/run-domain-integration-tests.js accounts",
|
|
270
|
-
"test:integration:transactions": "node scripts/run-domain-integration-tests.js transactions",
|
|
271
|
-
"test:integration:categories": "node scripts/run-domain-integration-tests.js categories",
|
|
272
|
-
"test:integration:payees": "node scripts/run-domain-integration-tests.js payees",
|
|
273
|
-
"test:integration:months": "node scripts/run-domain-integration-tests.js months",
|
|
274
|
-
"test:integration:reconciliation": "node scripts/run-domain-integration-tests.js reconciliation",
|
|
275
|
-
"test:integration:delta": "node scripts/run-domain-integration-tests.js delta",
|
|
276
|
-
|
|
277
|
-
"test:all": "npm run test:unit && npm run test:integration:core",
|
|
278
|
-
"test:coverage": "vitest run --coverage --project unit"
|
|
279
|
-
}
|
|
280
|
-
}
|
|
281
|
-
```
|
|
282
|
-
|
|
283
|
-
Each domain script routes through `scripts/run-domain-integration-tests.js`, which sets the `INTEGRATION_TEST_DOMAINS` environment variable so only tests tagged with those domains execute. You can target multiple domains at once (for example `node scripts/run-domain-integration-tests.js budgets accounts`) or pass additional Vitest flags after `--`.
|
|
284
|
-
|
|
285
|
-
---
|
|
286
|
-
|
|
287
|
-
## CI/CD Integration
|
|
288
|
-
|
|
289
|
-
### GitHub Actions Example
|
|
290
|
-
|
|
291
|
-
```yaml
|
|
292
|
-
name: CI Tests
|
|
293
|
-
|
|
294
|
-
on: [push, pull_request]
|
|
295
|
-
|
|
296
|
-
jobs:
|
|
297
|
-
unit-tests:
|
|
298
|
-
runs-on: ubuntu-latest
|
|
299
|
-
steps:
|
|
300
|
-
- uses: actions/checkout@v4
|
|
301
|
-
- uses: actions/setup-node@v4
|
|
302
|
-
with:
|
|
303
|
-
node-version: '20'
|
|
304
|
-
- run: npm ci
|
|
305
|
-
- run: npm run test:unit
|
|
306
|
-
|
|
307
|
-
integration-core:
|
|
308
|
-
runs-on: ubuntu-latest
|
|
309
|
-
needs: unit-tests
|
|
310
|
-
steps:
|
|
311
|
-
- uses: actions/checkout@v4
|
|
312
|
-
- uses: actions/setup-node@v4
|
|
313
|
-
with:
|
|
314
|
-
node-version: '20'
|
|
315
|
-
- run: npm ci
|
|
316
|
-
- run: npm run test:integration:core
|
|
317
|
-
env:
|
|
318
|
-
YNAB_ACCESS_TOKEN: ${{ secrets.YNAB_ACCESS_TOKEN }}
|
|
319
|
-
|
|
320
|
-
integration-full:
|
|
321
|
-
runs-on: ubuntu-latest
|
|
322
|
-
# Only run on schedule (nightly) or manual trigger
|
|
323
|
-
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
|
|
324
|
-
steps:
|
|
325
|
-
- uses: actions/checkout@v4
|
|
326
|
-
- uses: actions/setup-node@v4
|
|
327
|
-
with:
|
|
328
|
-
node-version: '20'
|
|
329
|
-
- run: npm ci
|
|
330
|
-
- run: npm run test:integration:full
|
|
331
|
-
env:
|
|
332
|
-
YNAB_ACCESS_TOKEN: ${{ secrets.YNAB_ACCESS_TOKEN }}
|
|
333
|
-
```
|
|
334
|
-
|
|
335
|
-
**CI Strategy**:
|
|
336
|
-
- **Every Push**: Unit tests only
|
|
337
|
-
- **Every PR**: Unit tests + core integration tests (~10-15 API calls)
|
|
338
|
-
- **Nightly/Weekly**: Full integration suite (scheduled)
|
|
339
|
-
- **Manual**: Full integration suite (on-demand via workflow_dispatch)
|
|
340
|
-
|
|
341
|
-
---
|
|
342
|
-
|
|
343
|
-
## Environment Variables
|
|
344
|
-
|
|
345
|
-
```bash
|
|
346
|
-
# .env.test
|
|
347
|
-
# YNAB API Configuration
|
|
348
|
-
YNAB_ACCESS_TOKEN=your_token_here
|
|
349
|
-
|
|
350
|
-
# Test Configuration
|
|
351
|
-
TEST_BUDGET_ID=optional_specific_budget_id
|
|
352
|
-
TEST_ACCOUNT_ID=optional_specific_account_id
|
|
353
|
-
|
|
354
|
-
# Integration Test Behavior
|
|
355
|
-
INTEGRATION_TEST_TIER=core # core | domain | full
|
|
356
|
-
INTEGRATION_TEST_DOMAINS= # comma-separated domains for domain tier
|
|
357
|
-
SKIP_INTEGRATION_TESTS=false # Set to true to skip all integration tests
|
|
358
|
-
INTEGRATION_THROTTLE_MS=5000 # Delay between domain tests (milliseconds)
|
|
359
|
-
|
|
360
|
-
# Rate Limit Configuration
|
|
361
|
-
RATE_LIMIT_PER_HOUR=200 # Base rate limit before applying buffer
|
|
362
|
-
RATE_LIMIT_BUFFER=20 # Reserve N calls before throttling
|
|
363
|
-
RATE_LIMIT_WINDOW_MS=3600000 # Window length (1 hour)
|
|
364
|
-
RATE_LIMIT_MAX_WAIT_MS=3600000 # Max wait time for rate limit reset (1 hour)
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
---
|
|
368
|
-
|
|
369
|
-
## Tier Assignment Guidelines
|
|
370
|
-
|
|
371
|
-
### Core Tier (Fast, Frequent)
|
|
372
|
-
|
|
373
|
-
Assign tests to **core** tier if they:
|
|
374
|
-
- ✅ Test fundamental, critical-path functionality
|
|
375
|
-
- ✅ Are needed for basic confidence before committing
|
|
376
|
-
- ✅ Make 1-2 API calls per test
|
|
377
|
-
- ✅ Run in under 5 seconds (including network latency)
|
|
378
|
-
- ✅ Don't depend on complex state or multiple resources
|
|
379
|
-
|
|
380
|
-
**Example Core Tests**:
|
|
381
|
-
- `GET /budgets` - List all budgets
|
|
382
|
-
- `GET /budgets/{budget_id}` - Get single budget
|
|
383
|
-
- `GET /budgets/{budget_id}/accounts` - List accounts
|
|
384
|
-
- `POST /budgets/{budget_id}/transactions` - Create transaction
|
|
385
|
-
- `GET /user` - Get user info
|
|
386
|
-
|
|
387
|
-
### Domain Tier (Selective, Thorough)
|
|
388
|
-
|
|
389
|
-
Assign tests to **domain** tier if they:
|
|
390
|
-
- ✅ Test comprehensive functionality within a specific domain
|
|
391
|
-
- ✅ Are needed when working on that specific feature area
|
|
392
|
-
- ✅ Make 3-10 API calls per test
|
|
393
|
-
- ✅ Test edge cases, error handling, or complex workflows
|
|
394
|
-
- ✅ May depend on multiple resources or state
|
|
395
|
-
|
|
396
|
-
**Example Domain Tests**:
|
|
397
|
-
- Full transaction CRUD workflow
|
|
398
|
-
- Account reconciliation with recommendations
|
|
399
|
-
- Category group management and reordering
|
|
400
|
-
- Delta request handling with server_knowledge
|
|
401
|
-
- Bulk transaction operations
|
|
402
|
-
- Month-to-month rollover calculations
|
|
403
|
-
|
|
404
|
-
### Full Suite (Comprehensive, Scheduled)
|
|
405
|
-
|
|
406
|
-
All integration tests run in **full** tier, including:
|
|
407
|
-
- ✅ All core tests
|
|
408
|
-
- ✅ All domain tests
|
|
409
|
-
- ✅ Performance tests
|
|
410
|
-
- ✅ Stress tests
|
|
411
|
-
- ✅ Long-running scenarios
|
|
412
|
-
|
|
413
|
-
---
|
|
414
|
-
|
|
415
|
-
## Delta Integration Tests
|
|
416
|
-
|
|
417
|
-
Delta tests (`*.delta.integration.test.ts`) require special handling due to their stateful nature:
|
|
418
|
-
|
|
419
|
-
### Challenge
|
|
420
|
-
Delta requests track `server_knowledge` - a monotonically increasing value that represents server state. Sequential delta requests depend on previous responses.
|
|
421
|
-
|
|
422
|
-
### Strategy
|
|
423
|
-
1. **Initialize Fresh**: Each delta test starts with a fresh API client (no cached server_knowledge)
|
|
424
|
-
2. **Test Sequence**: Test both initial fetch AND subsequent delta request in same test
|
|
425
|
-
3. **Validate Incremental**: Verify that delta responses are smaller than full responses
|
|
426
|
-
4. **Reset State**: Clean up any created test data to avoid polluting future delta tests
|
|
427
|
-
|
|
428
|
-
### Example Delta Test
|
|
429
|
-
|
|
430
|
-
```typescript
|
|
431
|
-
describe('Delta Integration', () => {
|
|
432
|
-
it('should fetch initial data and then delta updates', {
|
|
433
|
-
meta: { tier: 'domain', domain: 'delta' }
|
|
434
|
-
}, async () => {
|
|
435
|
-
// Initial fetch (no server_knowledge)
|
|
436
|
-
const initial = await api.budgets.getBudgetById(budgetId);
|
|
437
|
-
const serverKnowledge = initial.data.server_knowledge;
|
|
438
|
-
|
|
439
|
-
// Create a change
|
|
440
|
-
await api.transactions.createTransaction(budgetId, { /* ... */ });
|
|
441
|
-
|
|
442
|
-
// Delta fetch (with server_knowledge)
|
|
443
|
-
const delta = await api.budgets.getBudgetById(budgetId, serverKnowledge);
|
|
444
|
-
|
|
445
|
-
// Verify delta is smaller than full response
|
|
446
|
-
expect(delta.data.transactions.length).toBeLessThan(initial.data.transactions.length);
|
|
447
|
-
expect(delta.data.server_knowledge).toBeGreaterThan(serverKnowledge);
|
|
448
|
-
});
|
|
449
|
-
});
|
|
450
|
-
```
|
|
451
|
-
|
|
452
|
-
---
|
|
453
|
-
|
|
454
|
-
## Throttled Test Runner
|
|
455
|
-
|
|
456
|
-
The full integration suite uses a custom test runner that intelligently throttles API calls:
|
|
457
|
-
|
|
458
|
-
```typescript
|
|
459
|
-
// scripts/run-throttled-integration-tests.js
|
|
460
|
-
import { spawn } from 'child_process';
|
|
461
|
-
import { readFileSync } from 'fs';
|
|
462
|
-
|
|
463
|
-
class ThrottledTestRunner {
|
|
464
|
-
constructor() {
|
|
465
|
-
this.rateLimit = 200; // requests per hour
|
|
466
|
-
this.rateLimitWindow = 3600000; // 1 hour in ms
|
|
467
|
-
this.buffer = 20; // safety buffer
|
|
468
|
-
this.requestHistory = [];
|
|
469
|
-
}
|
|
470
|
-
|
|
471
|
-
async run() {
|
|
472
|
-
// Get all integration test files
|
|
473
|
-
const testFiles = this.getIntegrationTests();
|
|
474
|
-
|
|
475
|
-
for (const testFile of testFiles) {
|
|
476
|
-
await this.runWithThrottling(testFile);
|
|
477
|
-
}
|
|
478
|
-
}
|
|
479
|
-
|
|
480
|
-
async runWithThrottling(testFile) {
|
|
481
|
-
// Check if we need to throttle
|
|
482
|
-
const recentRequests = this.getRecentRequests();
|
|
483
|
-
|
|
484
|
-
if (recentRequests >= this.rateLimit - this.buffer) {
|
|
485
|
-
const waitTime = this.calculateWaitTime();
|
|
486
|
-
console.log(`⏳ Rate limit approaching (${recentRequests}/${this.rateLimit} calls)`);
|
|
487
|
-
console.log(` Waiting ${Math.round(waitTime / 60000)} minutes...`);
|
|
488
|
-
await this.sleep(waitTime);
|
|
489
|
-
}
|
|
490
|
-
|
|
491
|
-
// Run test
|
|
492
|
-
console.log(`▶️ Running ${testFile}...`);
|
|
493
|
-
await this.runTest(testFile);
|
|
494
|
-
|
|
495
|
-
// Track request count (estimate based on test file)
|
|
496
|
-
const estimatedCalls = this.estimateAPICalls(testFile);
|
|
497
|
-
this.requestHistory.push({
|
|
498
|
-
timestamp: Date.now(),
|
|
499
|
-
calls: estimatedCalls,
|
|
500
|
-
});
|
|
501
|
-
}
|
|
502
|
-
|
|
503
|
-
getRecentRequests() {
|
|
504
|
-
const cutoff = Date.now() - this.rateLimitWindow;
|
|
505
|
-
this.requestHistory = this.requestHistory.filter(r => r.timestamp > cutoff);
|
|
506
|
-
return this.requestHistory.reduce((sum, r) => sum + r.calls, 0);
|
|
507
|
-
}
|
|
508
|
-
|
|
509
|
-
calculateWaitTime() {
|
|
510
|
-
if (this.requestHistory.length === 0) return 0;
|
|
511
|
-
|
|
512
|
-
const oldestRequest = this.requestHistory[0];
|
|
513
|
-
const timeUntilExpiry = (oldestRequest.timestamp + this.rateLimitWindow) - Date.now();
|
|
514
|
-
|
|
515
|
-
return Math.max(timeUntilExpiry, 60000); // At least 1 minute
|
|
516
|
-
}
|
|
517
|
-
|
|
518
|
-
estimateAPICalls(testFile) {
|
|
519
|
-
// Parse test file for API call estimates
|
|
520
|
-
// Could use static analysis or embedded metadata
|
|
521
|
-
// For now, use conservative estimates based on test type
|
|
522
|
-
|
|
523
|
-
if (testFile.includes('delta')) return 15;
|
|
524
|
-
if (testFile.includes('reconciliation')) return 25;
|
|
525
|
-
if (testFile.includes('transaction')) return 12;
|
|
526
|
-
if (testFile.includes('budget')) return 8;
|
|
527
|
-
|
|
528
|
-
return 10; // default estimate
|
|
529
|
-
}
|
|
530
|
-
|
|
531
|
-
async runTest(testFile) {
|
|
532
|
-
return new Promise((resolve, reject) => {
|
|
533
|
-
const proc = spawn('npx', ['vitest', 'run', testFile], {
|
|
534
|
-
stdio: 'inherit',
|
|
535
|
-
env: { ...process.env },
|
|
536
|
-
});
|
|
537
|
-
|
|
538
|
-
proc.on('close', (code) => {
|
|
539
|
-
if (code === 0) resolve();
|
|
540
|
-
else reject(new Error(`Test failed with code ${code}`));
|
|
541
|
-
});
|
|
542
|
-
});
|
|
543
|
-
}
|
|
544
|
-
|
|
545
|
-
sleep(ms) {
|
|
546
|
-
return new Promise(resolve => setTimeout(resolve, ms));
|
|
547
|
-
}
|
|
548
|
-
|
|
549
|
-
getIntegrationTests() {
|
|
550
|
-
// Use glob or similar to find all *.integration.test.ts files
|
|
551
|
-
// Return sorted array of test file paths
|
|
552
|
-
}
|
|
553
|
-
}
|
|
554
|
-
|
|
555
|
-
// Run it
|
|
556
|
-
const runner = new ThrottledTestRunner();
|
|
557
|
-
runner.run().catch(console.error);
|
|
558
|
-
```
|
|
559
|
-
|
|
560
|
-
---
|
|
561
|
-
|
|
562
|
-
## Migration Path
|
|
563
|
-
|
|
564
|
-
### Current State (v0.10.0)
|
|
565
|
-
- ✅ 21 integration test files exist
|
|
566
|
-
- ⚠️ All tests hit real API without throttling
|
|
567
|
-
- ⚠️ No tier organization
|
|
568
|
-
- ⚠️ Rate limits cause test failures
|
|
569
|
-
|
|
570
|
-
### Target State
|
|
571
|
-
- ✅ Integration tests organized into 3 tiers
|
|
572
|
-
- ✅ Core tests run frequently (CI, pre-commit)
|
|
573
|
-
- ✅ Domain tests run selectively (feature work)
|
|
574
|
-
- ✅ Full suite runs scheduled/on-demand with throttling
|
|
575
|
-
- ✅ CI runs only unit + core tests
|
|
576
|
-
- ✅ Never hit rate limits
|
|
577
|
-
|
|
578
|
-
### Migration Steps
|
|
579
|
-
|
|
580
|
-
1. **Tag Existing Tests** (1-2 hours)
|
|
581
|
-
- Add `meta: { tier, domain }` to all integration tests
|
|
582
|
-
- Identify 8-10 tests for core tier
|
|
583
|
-
- Organize remaining tests by domain
|
|
584
|
-
|
|
585
|
-
2. **Update Vitest Config** (30 minutes)
|
|
586
|
-
- Create `integration:core`, `integration:domain`, `integration:full` projects
|
|
587
|
-
- Configure test filtering by tier
|
|
588
|
-
- Set appropriate timeouts per tier
|
|
589
|
-
|
|
590
|
-
3. **Create NPM Scripts** (15 minutes)
|
|
591
|
-
- Add tier-specific test scripts
|
|
592
|
-
- Add domain-specific test scripts
|
|
593
|
-
- Update default `test:integration` to run core tier
|
|
594
|
-
|
|
595
|
-
4. **Implement Throttled Runner** (2-3 hours)
|
|
596
|
-
- Create `scripts/run-throttled-integration-tests.js`
|
|
597
|
-
- Implement rate limit tracking and scheduling
|
|
598
|
-
- Add API call estimation logic
|
|
599
|
-
- Test with subset of integration tests
|
|
600
|
-
|
|
601
|
-
5. **Update CI Configuration** (30 minutes)
|
|
602
|
-
- Configure CI to run unit + core only on PRs
|
|
603
|
-
- Set up scheduled full suite runs (nightly/weekly)
|
|
604
|
-
- Add manual trigger for full suite
|
|
605
|
-
|
|
606
|
-
6. **Document Workflow** (1 hour)
|
|
607
|
-
- Update developer documentation
|
|
608
|
-
- Create examples for each tier
|
|
609
|
-
- Document when to run which tier
|
|
610
|
-
|
|
611
|
-
7. **Validate** (2-3 hours)
|
|
612
|
-
- Run core tier locally (should be fast)
|
|
613
|
-
- Run domain tiers selectively
|
|
614
|
-
- Run full suite with throttling (validate no rate limit hits)
|
|
615
|
-
- Verify CI runs successfully
|
|
616
|
-
|
|
617
|
-
**Total Effort**: ~8-12 hours
|
|
618
|
-
|
|
619
|
-
---
|
|
620
|
-
|
|
621
|
-
## Best Practices
|
|
622
|
-
|
|
623
|
-
### 1. Test Data Management
|
|
624
|
-
- **Use Dedicated Test Budget**: Create a YNAB budget specifically for testing
|
|
625
|
-
- **Consistent Test Data**: Use predictable account/category names for easier test writing
|
|
626
|
-
- **Clean Up After Tests**: Delete created transactions/payees to keep budget clean
|
|
627
|
-
- **Idempotent Tests**: Tests should be repeatable without manual cleanup
|
|
628
|
-
|
|
629
|
-
### 2. Rate Limit Awareness
|
|
630
|
-
- **Monitor Headers**: Always log rate limit headers during full suite runs
|
|
631
|
-
- **Buffer Safety**: Never use more than 180 calls in full suite (leave 20 call buffer)
|
|
632
|
-
- **Graceful Degradation**: If rate limit hit, pause and resume (don't fail)
|
|
633
|
-
- **Estimate Conservatively**: Overestimate API calls per test for safer throttling
|
|
634
|
-
|
|
635
|
-
### 3. Test Organization
|
|
636
|
-
- **Core = Critical Path**: Only include must-have functionality in core tier
|
|
637
|
-
- **Domain = Feature Work**: Group tests by domain for selective running
|
|
638
|
-
- **Full = Scheduled**: Accept that full suite is slow, run overnight/weekly
|
|
639
|
-
- **Tag Consistently**: Use consistent tier/domain tags for filtering
|
|
640
|
-
|
|
641
|
-
### 4. CI/CD Strategy
|
|
642
|
-
- **Fast Feedback**: Unit + core tests should complete in <5 minutes
|
|
643
|
-
- **PR Validation**: Core tests catch regressions without burning rate limits
|
|
644
|
-
- **Scheduled Deep Tests**: Full suite runs when you're not waiting for results
|
|
645
|
-
- **Manual Override**: Allow manual full suite runs for pre-release validation
|
|
646
|
-
|
|
647
|
-
---
|
|
648
|
-
|
|
649
|
-
## Troubleshooting
|
|
650
|
-
|
|
651
|
-
### Rate Limit Exceeded During Tests
|
|
652
|
-
|
|
653
|
-
**Symptoms**: Tests fail with 429 responses or HTML error pages
|
|
654
|
-
|
|
655
|
-
**Solutions**:
|
|
656
|
-
1. Check recent API usage: `grep "X-Rate-Limit-Remaining" test-output.log`
|
|
657
|
-
2. Wait for rate limit window to reset (check `X-Rate-Limit-Reset` header)
|
|
658
|
-
3. Reduce test scope: Run domain tests instead of full suite
|
|
659
|
-
4. Increase throttling: Adjust `INTEGRATION_THROTTLE_MS` environment variable
|
|
660
|
-
|
|
661
|
-
### Tests Failing Inconsistently
|
|
662
|
-
|
|
663
|
-
**Symptoms**: Same test passes/fails on different runs
|
|
664
|
-
|
|
665
|
-
**Possible Causes**:
|
|
666
|
-
- Network issues (timeouts, latency)
|
|
667
|
-
- YNAB API throttling (not rate limit, but request pacing)
|
|
668
|
-
- State pollution (previous test didn't clean up)
|
|
669
|
-
- Delta state issues (server_knowledge out of sync)
|
|
670
|
-
|
|
671
|
-
**Solutions**:
|
|
672
|
-
- Increase test timeouts: `testTimeout: 60000`
|
|
673
|
-
- Add retry logic for network errors
|
|
674
|
-
- Ensure tests clean up created resources
|
|
675
|
-
- Reset delta state between tests
|
|
676
|
-
|
|
677
|
-
### Full Suite Takes Too Long
|
|
678
|
-
|
|
679
|
-
**Symptoms**: Full suite exceeds expected 2-3 hour window
|
|
680
|
-
|
|
681
|
-
**Solutions**:
|
|
682
|
-
- Reduce throttling buffer: Lower `RATE_LIMIT_BUFFER` (but risk hitting limits)
|
|
683
|
-
- Optimize tests: Combine multiple assertions into single API call
|
|
684
|
-
- Remove redundant tests: Eliminate duplicate coverage
|
|
685
|
-
- Use delta requests: Prefer incremental fetches over full data
|
|
686
|
-
|
|
687
|
-
### CI Running Out of Rate Limits
|
|
688
|
-
|
|
689
|
-
**Symptoms**: CI fails due to rate limits on every PR
|
|
690
|
-
|
|
691
|
-
**Solutions**:
|
|
692
|
-
- Verify CI only runs core tier: Check workflow uses `test:integration:core`
|
|
693
|
-
- Reduce core tier tests: Move less critical tests to domain tier
|
|
694
|
-
- Use separate YNAB token for CI: Isolate CI rate limits from dev
|
|
695
|
-
- Implement CI-level throttling: Add delays between workflow runs
|
|
696
|
-
|
|
697
|
-
---
|
|
698
|
-
|
|
699
|
-
## Resources
|
|
700
|
-
|
|
701
|
-
### YNAB API
|
|
702
|
-
- **API Documentation**: https://api.youneedabudget.com/
|
|
703
|
-
- **Rate Limits**: 200 requests/hour per token (rolling window)
|
|
704
|
-
- **SDK**: https://github.com/ynab/ynab-sdk-js
|
|
705
|
-
- **Support**: https://support.youneedabudget.com/
|
|
706
|
-
|
|
707
|
-
### Testing Tools
|
|
708
|
-
- **Vitest**: https://vitest.dev/
|
|
709
|
-
- **Vitest Projects**: https://vitest.dev/guide/workspace
|
|
710
|
-
- **GitHub Actions**: https://docs.github.com/en/actions
|
|
711
|
-
|
|
712
|
-
### Related Documentation
|
|
713
|
-
- [Testing Guide](./TESTING.md) - Comprehensive testing documentation
|
|
714
|
-
- [Build Guide](../development/BUILD.md) - Build and development workflow
|
|
715
|
-
- [Deployment Guide](./DEPLOYMENT.md) - Deployment and packaging
|
|
716
|
-
|
|
717
|
-
---
|
|
718
|
-
|
|
719
|
-
## Conclusion
|
|
720
|
-
|
|
721
|
-
This tiered integration testing strategy provides:
|
|
722
|
-
|
|
723
|
-
✅ **Fast Feedback**: Core tests run in <3 minutes for quick validation
|
|
724
|
-
✅ **Selective Testing**: Domain tests focus on areas you're actively developing
|
|
725
|
-
✅ **Comprehensive Coverage**: Full suite validates entire system before releases
|
|
726
|
-
✅ **Rate Limit Safety**: Intelligent throttling prevents API limit exhaustion
|
|
727
|
-
✅ **CI/CD Friendly**: Lightweight core tests don't burn rate limits on every PR
|
|
728
|
-
✅ **Real API Testing**: No mocks - tests validate actual YNAB API behavior
|
|
729
|
-
|
|
730
|
-
**The key insight**: Not all integration tests need to run all the time. Organize tests by purpose and run them at the right frequency to balance speed, coverage, and rate limits.
|