@isdk/proxy 0.1.1 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.cn.md +191 -9
- package/README.md +191 -7
- package/dist/index.d.mts +272 -30
- package/dist/index.d.ts +272 -30
- package/dist/index.js +1 -1
- package/dist/index.mjs +1 -1
- package/docs/README.md +191 -7
- package/docs/classes/SmartCache.md +59 -11
- package/docs/functions/createCachedFetch.md +1 -1
- package/docs/functions/createFetchWithCache.md +1 -1
- package/docs/functions/extractData.md +34 -5
- package/docs/functions/fetchWithCache.md +11 -1
- package/docs/functions/generateCacheKey.md +34 -4
- package/docs/functions/getSiteConfig.md +39 -0
- package/docs/functions/isAllowed.md +35 -8
- package/docs/functions/isGlob.md +23 -0
- package/docs/functions/isMatch.md +44 -0
- package/docs/globals.md +5 -0
- package/docs/interfaces/BodyFilterConfig.md +77 -0
- package/docs/interfaces/CacheEntry.md +9 -9
- package/docs/interfaces/CacheMetadata.md +8 -8
- package/docs/interfaces/CacheRule.md +80 -0
- package/docs/interfaces/FetchWithCacheContext.md +43 -13
- package/docs/interfaces/FetchWithCacheOptions.md +40 -10
- package/docs/interfaces/KeyFilterConfig.md +11 -7
- package/docs/interfaces/ProxyConfig.md +4 -4
- package/docs/interfaces/SiteCacheConfig.md +46 -11
- package/docs/interfaces/SmartCacheOptions.md +32 -6
- package/package.json +4 -2
package/docs/README.md
CHANGED
|
@@ -19,6 +19,8 @@ In high-concurrency environments—like **API Proxies**, **Web Scrapers**, or **
|
|
|
19
19
|
## Key Features
|
|
20
20
|
|
|
21
21
|
- **🚀 Hybrid Multi-tier Cache**: Extreme speed with L1 (LRU Memory) and persistence with L2 (Content Addressable Disk via `cacache`).
|
|
22
|
+
- **📥 HTTP POST & Method Support**: Full support for caching POST, PUT, and other methods with intelligent request body fingerprinting.
|
|
23
|
+
- **🎯 Precision Filtering**: Fine-grained `cacheRules` to intercept specific paths or query parameters.
|
|
22
24
|
- **🌊 Streaming Native**: Fully stream-based internal pipeline natively prevents Out-Of-Memory (OOM) issues when proxying large files.
|
|
23
25
|
- **🧠 Intelligent Meta-Residency**: Metadata (Headers, Status, Policy) stays in memory regardless of body size, ensuring nanosecond cache policy evaluations.
|
|
24
26
|
- **🔄 Stale-While-Revalidate (SWR)**: Serve stale content instantly while updating the cache silently in the background.
|
|
@@ -37,6 +39,8 @@ pnpm add @isdk/proxy
|
|
|
37
39
|
|
|
38
40
|
The primary way to use `@isdk/proxy` is via the `fetchWithCache` function, which can wrap any HTTP request logic.
|
|
39
41
|
|
|
42
|
+
### Basic Usage (GET)
|
|
43
|
+
|
|
40
44
|
```typescript
|
|
41
45
|
import { SmartCache, createCachedFetch } from '@isdk/proxy';
|
|
42
46
|
|
|
@@ -46,28 +50,98 @@ const cache = new SmartCache({
|
|
|
46
50
|
maxMemorySize: 1024 * 1024 // 1MB threshold
|
|
47
51
|
});
|
|
48
52
|
|
|
49
|
-
// 2. Create a pre-configured cached fetcher
|
|
53
|
+
// 2. Create a pre-configured cached fetcher
|
|
50
54
|
const myFetch = createCachedFetch({
|
|
51
55
|
cache,
|
|
52
56
|
config: {
|
|
53
57
|
staleIfError: true,
|
|
54
|
-
forceCache: false // Set to true to cache everything (ignore no-store) for offline-first apps
|
|
55
58
|
},
|
|
56
59
|
backgroundUpdate: true // Enable SWR
|
|
57
60
|
});
|
|
58
61
|
|
|
59
|
-
// 3. Use it
|
|
60
|
-
const
|
|
61
|
-
|
|
62
|
+
// 3. Use it!
|
|
63
|
+
const response = await myFetch(new Request('https://api.example.com/data'), (req) => fetch(req));
|
|
64
|
+
console.log(response.headers.get('x-proxy-cache'));
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
### Advanced Usage: Caching POST Requests
|
|
62
68
|
|
|
63
|
-
|
|
64
|
-
|
|
69
|
+
You can cache POST/PUT requests by enabling methods and defining body filters to ignore dynamic fields (like timestamps) in the request body.
|
|
70
|
+
|
|
71
|
+
```typescript
|
|
72
|
+
const myPostFetch = createCachedFetch({
|
|
73
|
+
cache,
|
|
74
|
+
config: {
|
|
75
|
+
methods: ['GET', 'POST'], // Enable POST caching
|
|
76
|
+
body: {
|
|
77
|
+
exclude: ['timestamp', 'nonce'] // Ignore these fields when generating cache keys
|
|
78
|
+
},
|
|
79
|
+
cacheRules: [
|
|
80
|
+
{ method: 'POST', path: '/api/v1/query' } // Only cache specific POST endpoints
|
|
81
|
+
],
|
|
82
|
+
forceCache: true // Often needed for POST if backend doesn't send Cache-Control
|
|
83
|
+
}
|
|
84
|
+
});
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
## Configuration: `SiteCacheConfig`
|
|
88
|
+
|
|
89
|
+
| Field | Type | Description |
|
|
90
|
+
| :--- | :--- | :--- |
|
|
91
|
+
| `methods` | `string[]` | List of allowed HTTP methods. Default: `['GET', 'HEAD']`. |
|
|
92
|
+
| `cacheRules` | `CacheRule[]` | Fine-grained rules. If set, a request must match at least one rule to be cached. |
|
|
93
|
+
| `query` | `KeyFilterConfig` | Filters for URL search parameters (`include`/`exclude`). |
|
|
94
|
+
| `headers` | `KeyFilterConfig` | Filters for request headers. |
|
|
95
|
+
| `cookies` | `KeyFilterConfig` | Filters for cookies. |
|
|
96
|
+
| `body` | `KeyFilterConfig` | Filters for JSON request body fields. |
|
|
97
|
+
| `staleIfError`| `boolean` | Serve stale cache on network failure. |
|
|
98
|
+
| `forceCache` | `boolean` | Ignore `no-store` and force caching (useful for offline support). |
|
|
99
|
+
|
|
100
|
+
### `CacheRule` Object
|
|
101
|
+
|
|
102
|
+
- `method`: HTTP method to match.
|
|
103
|
+
- `path`: URL pathname matching (supports **RegExp**, **Glob**, **Array**, or **prefix match**).
|
|
104
|
+
- `query`: Key-value pairs. Values can be `string` (exact/Glob match), `true` (must exist), `false` (must not exist), or `RegExp`.
|
|
105
|
+
- `body`: Body content matching (supports **RegExp**, **Glob**, or **Array**).
|
|
106
|
+
|
|
107
|
+
### Pattern Matching
|
|
108
|
+
|
|
109
|
+
`@isdk/proxy` provides powerful pattern matching for all configurable fields:
|
|
110
|
+
|
|
111
|
+
| Pattern Type | Example | Description |
|
|
112
|
+
| :--- | :--- | :--- |
|
|
113
|
+
| **RegExp** | `/api/v[12]/.*/i` | JavaScript RegExp (in JSON, use string like `"/api/v[12]/.*/i"`) |
|
|
114
|
+
| **Glob** | `/**/*.json` | File path style wildcard matching |
|
|
115
|
+
| **Negation** | `['!/api/private/**', '/api/**']` | Exclude patterns (prefixed with `!`, checked first) |
|
|
116
|
+
| **Array** | `['/api/v1/*', '/api/v2/*']` | Multiple patterns (OR logic, negative takes precedence) |
|
|
117
|
+
| **Boolean** | `true` / `false` | For query params: must/must not exist |
|
|
118
|
+
|
|
119
|
+
**Example with advanced pattern matching:**
|
|
120
|
+
|
|
121
|
+
```typescript
|
|
122
|
+
const myFetch = createCachedFetch({
|
|
123
|
+
cache,
|
|
124
|
+
config: {
|
|
125
|
+
cacheRules: [
|
|
126
|
+
{
|
|
127
|
+
path: ['/api/v1/items/*', '!/api/v1/items/private/*'], // v1 items, exclude private
|
|
128
|
+
query: {
|
|
129
|
+
format: '/^(json|xml)$/', // Regex for format param
|
|
130
|
+
'page*': true // Glob: any param starting with 'page' must exist
|
|
131
|
+
},
|
|
132
|
+
body: /\"action\"\s*:\s*\"query\"/ // Regex body match
|
|
133
|
+
}
|
|
134
|
+
]
|
|
135
|
+
}
|
|
136
|
+
});
|
|
65
137
|
```
|
|
66
138
|
|
|
67
139
|
## Adapters
|
|
68
140
|
|
|
69
141
|
`@isdk/proxy` is designed to be framework-agnostic. While the core library is pure, you can find (or build) adapters for specific environments:
|
|
70
142
|
|
|
143
|
+
- **HTTP Caching Proxy Server (Node.js)**: See [@isdk/proxy-server](https://www.npmjs.com/package/@isdk/proxy-server) (separate package) for running a standalone HTTP forward proxy.
|
|
144
|
+
- **Crawlee Adapter**: See [@isdk/proxy-crawlee](https://www.npmjs.com/package/@isdk/proxy-crawlee) (separate package) for integrating with Crawlee web scraping lifecycle.
|
|
71
145
|
- **MSW Adapter**: See `@isdk/proxy-msw` (separate package) to use this caching engine as an MSW interceptor.
|
|
72
146
|
- **Axios Adapter**: Easily implemented by converting Axios config to Web `Request`.
|
|
73
147
|
|
|
@@ -120,9 +194,119 @@ The hybrid multi-tier storage engine.
|
|
|
120
194
|
- **`options.maxMemorySize`**: Threshold (in bytes) for offloading bodies to disk (default `1048576`, i.e., 1MB).
|
|
121
195
|
- **`options.storagePath`**: Disk storage path for the `cacache` engine (defaults to a system temp folder).
|
|
122
196
|
|
|
197
|
+
### Utility Functions
|
|
198
|
+
|
|
199
|
+
Exported from `@isdk/proxy` for advanced usage:
|
|
200
|
+
|
|
201
|
+
#### `isMatch(pattern, value, usePrefix?)`
|
|
202
|
+
|
|
203
|
+
Universal pattern matching function. Supports RegExp, Glob, array patterns (with negation), and string prefix/exact matching.
|
|
204
|
+
|
|
205
|
+
- **`pattern`**: `string | RegExp | (string | RegExp)[]`
|
|
206
|
+
- **`value`**: The string to test against
|
|
207
|
+
- **`usePrefix`**: For plain strings, use prefix match instead of exact match (default: `false`)
|
|
208
|
+
- **Returns**: `boolean`
|
|
209
|
+
|
|
210
|
+
```typescript
|
|
211
|
+
import { isMatch } from '@isdk/proxy';
|
|
212
|
+
|
|
213
|
+
isMatch('/api/v[12]/.*', '/api/v1/users'); // RegExp
|
|
214
|
+
isMatch('/api/**/*.json', '/api/v1/data.json'); // Glob
|
|
215
|
+
isMatch(['!/private/**', '/api/**'], '/api/data'); // Negation
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
#### `isGlob(pattern)`
|
|
219
|
+
|
|
220
|
+
Check if a pattern is Glob syntax.
|
|
221
|
+
|
|
222
|
+
- **`pattern`**: `string`
|
|
223
|
+
- **Returns**: `boolean`
|
|
224
|
+
|
|
225
|
+
#### `getSiteConfig(urlString, proxyConfig)`
|
|
226
|
+
|
|
227
|
+
Get the site-specific cache configuration for a given URL.
|
|
228
|
+
|
|
229
|
+
- **`urlString`**: Full URL to match
|
|
230
|
+
- **`proxyConfig`**: `ProxyConfig` object with `sites` and `default` config
|
|
231
|
+
- **Returns**: `SiteCacheConfig`
|
|
232
|
+
|
|
233
|
+
```typescript
|
|
234
|
+
import { getSiteConfig } from '@isdk/proxy';
|
|
235
|
+
|
|
236
|
+
const config = getSiteConfig('https://api.example.com/data', {
|
|
237
|
+
default: { methods: ['GET'] },
|
|
238
|
+
sites: {
|
|
239
|
+
'api.example.com': { methods: ['GET', 'POST'], forceCache: true },
|
|
240
|
+
'/internal/': { staleIfError: true } // prefix match
|
|
241
|
+
}
|
|
242
|
+
});
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
#### `isAllowed(key, config, defaultAllowed?)`
|
|
246
|
+
|
|
247
|
+
Check if a key is allowed to participate in cache key fingerprinting.
|
|
248
|
+
|
|
249
|
+
- **`key`**: The key name to check
|
|
250
|
+
- **`config`**: `KeyFilterConfig` with `include` (whitelist) or `exclude` (blacklist)
|
|
251
|
+
- **`defaultAllowed`**: Optional. Default value when no config or no match
|
|
252
|
+
- **Returns**: `boolean | undefined`
|
|
253
|
+
|
|
254
|
+
**Priority Logic**:
|
|
255
|
+
|
|
256
|
+
1. `exclude` hit → returns `false` (highest priority)
|
|
257
|
+
2. `include` exists and hits → returns `true`
|
|
258
|
+
3. `include` exists but no hit → returns `false`
|
|
259
|
+
4. No config → uses `defaultAllowed` (returns `undefined` if not provided)
|
|
260
|
+
|
|
261
|
+
```typescript
|
|
262
|
+
import { isAllowed } from '@isdk/proxy';
|
|
263
|
+
|
|
264
|
+
// No config
|
|
265
|
+
isAllowed('key'); // undefined (falsy)
|
|
266
|
+
|
|
267
|
+
// Whitelist
|
|
268
|
+
isAllowed('id', { include: ['id', 'name'] }); // true
|
|
269
|
+
isAllowed('email', { include: ['id', 'name'] }); // false
|
|
270
|
+
|
|
271
|
+
// Blacklist
|
|
272
|
+
isAllowed('password', { exclude: ['password'] }); // false
|
|
273
|
+
isAllowed('name', { exclude: ['password'] }); // undefined (falsy)
|
|
274
|
+
|
|
275
|
+
// Need defaultAllowed to set default
|
|
276
|
+
isAllowed('name', { exclude: ['password'] }, true); // true
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
#### `extractData(source, config, defaultAllowed?)`
|
|
280
|
+
|
|
281
|
+
Extract and normalize data from a source object based on filter config. Used for generating cache fingerprints.
|
|
282
|
+
|
|
283
|
+
- **`source`**: Original data object (Query, Headers, Cookies, etc.)
|
|
284
|
+
- **`config`**: `KeyFilterConfig` object
|
|
285
|
+
- **`defaultAllowed`**: Optional. Whether to allow extraction when no config or no match (default `false`)
|
|
286
|
+
- **Returns**: `Record<string, string[]>` normalized data with lowercase keys and sorted array values
|
|
287
|
+
|
|
288
|
+
```typescript
|
|
289
|
+
import { extractData } from '@isdk/proxy';
|
|
290
|
+
|
|
291
|
+
const headers = { 'Content-Type': 'application/json', 'X-Request-Id': '123' };
|
|
292
|
+
|
|
293
|
+
// No extraction by default
|
|
294
|
+
extractData(headers); // {}
|
|
295
|
+
|
|
296
|
+
// Extract all keys
|
|
297
|
+
extractData(headers, undefined, true); // { 'content-type': ['application/json'], 'x-request-id': ['123'] }
|
|
298
|
+
|
|
299
|
+
// Whitelist
|
|
300
|
+
extractData(headers, { include: ['content-type'] }); // { 'content-type': ['application/json'] }
|
|
301
|
+
|
|
302
|
+
// Blacklist
|
|
303
|
+
extractData(headers, { include: ['*'], exclude: ['x-request-id'] }, true); // { 'content-type': ['application/json'] }
|
|
304
|
+
```
|
|
305
|
+
|
|
123
306
|
### Cache Status Headers
|
|
124
307
|
|
|
125
308
|
Every response processed by `@isdk/proxy` will include an `x-proxy-cache` header indicating its lifecycle:
|
|
309
|
+
|
|
126
310
|
- `HIT`: Served entirely from L1 or L2 cache.
|
|
127
311
|
- `MISS`: Bypassed cache and fetched from the origin server.
|
|
128
312
|
- `STALE`: Served from stale cache while a background update was initiated (SWR).
|
|
@@ -6,9 +6,19 @@
|
|
|
6
6
|
|
|
7
7
|
# Class: SmartCache
|
|
8
8
|
|
|
9
|
-
Defined in: [core/SmartCache.ts:
|
|
9
|
+
Defined in: [core/SmartCache.ts:39](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L39)
|
|
10
10
|
|
|
11
|
-
智能混合缓存类 (Hybrid Cache)
|
|
11
|
+
智能混合缓存类 (Hybrid Multi-tier Cache)
|
|
12
|
+
|
|
13
|
+
该类实现了 L1 (内存) 和 L2 (磁盘) 的双层混合存储架构,旨在提供高性能且大容量的缓存能力。
|
|
14
|
+
|
|
15
|
+
### 核心特性:
|
|
16
|
+
- **双层架构**: L1 使用 LRU 内存缓存(基于 `secondary-cache` 的 LRUCache),L2 使用持久化磁盘缓存(基于 `cacache`)。
|
|
17
|
+
- **大小感知存储**: 自动识别响应体大小。小于阈值的文件同时存于内存和磁盘;超过阈值的文件仅存于磁盘,但其元数据仍保留在内存中。
|
|
18
|
+
- **元数据驻留 (Meta-Residency)**: 无论 Body 多大,Headers、Status、Policy 等信息始终优先从内存读取,确保缓存判定性能。
|
|
19
|
+
- **流式支持**: 支持通过 `setStream` 和 `getStream` 直接操作大数据流,防止 OOM。
|
|
20
|
+
- **一致性保障**: 在并发写入时自动清理内存,确保后续读取不会拿到被污染的旧数据。
|
|
21
|
+
- **内存限制**: 通过 `maxTotalMemorySize` 控制 L1 缓存的总内存占用。
|
|
12
22
|
|
|
13
23
|
## Constructors
|
|
14
24
|
|
|
@@ -16,7 +26,7 @@ Defined in: [core/SmartCache.ts:22](https://github.com/isdk/proxy.js/blob/bed37f
|
|
|
16
26
|
|
|
17
27
|
> **new SmartCache**(`options`): `SmartCache`
|
|
18
28
|
|
|
19
|
-
Defined in: [core/SmartCache.ts:
|
|
29
|
+
Defined in: [core/SmartCache.ts:44](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L44)
|
|
20
30
|
|
|
21
31
|
#### Parameters
|
|
22
32
|
|
|
@@ -34,7 +44,7 @@ Defined in: [core/SmartCache.ts:27](https://github.com/isdk/proxy.js/blob/bed37f
|
|
|
34
44
|
|
|
35
45
|
> **clear**(): `Promise`\<`void`\>
|
|
36
46
|
|
|
37
|
-
Defined in: [core/SmartCache.ts:
|
|
47
|
+
Defined in: [core/SmartCache.ts:193](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L193)
|
|
38
48
|
|
|
39
49
|
#### Returns
|
|
40
50
|
|
|
@@ -46,7 +56,7 @@ Defined in: [core/SmartCache.ts:122](https://github.com/isdk/proxy.js/blob/bed37
|
|
|
46
56
|
|
|
47
57
|
> **delete**(`key`): `Promise`\<`void`\>
|
|
48
58
|
|
|
49
|
-
Defined in: [core/SmartCache.ts:
|
|
59
|
+
Defined in: [core/SmartCache.ts:188](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L188)
|
|
50
60
|
|
|
51
61
|
#### Parameters
|
|
52
62
|
|
|
@@ -64,10 +74,15 @@ Defined in: [core/SmartCache.ts:117](https://github.com/isdk/proxy.js/blob/bed37
|
|
|
64
74
|
|
|
65
75
|
> **get**(`key`): `Promise`\<[`CacheEntry`](../interfaces/CacheEntry.md) \| `null`\>
|
|
66
76
|
|
|
67
|
-
Defined in: [core/SmartCache.ts:
|
|
77
|
+
Defined in: [core/SmartCache.ts:79](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L79)
|
|
68
78
|
|
|
69
79
|
获取缓存条目
|
|
70
|
-
|
|
80
|
+
|
|
81
|
+
逻辑:
|
|
82
|
+
1. 首先尝试从 L1 内存获取。
|
|
83
|
+
2. 如果内存中有 Body,直接返回(Buffer 类型)。
|
|
84
|
+
3. 如果内存中只有 Meta(大文件),则从 L2 磁盘创建并返回 ReadStream。
|
|
85
|
+
4. 如果内存完全未命中,从磁盘 L2 检索,并根据大小决定是否回填 L1。
|
|
71
86
|
|
|
72
87
|
#### Parameters
|
|
73
88
|
|
|
@@ -75,17 +90,25 @@ Defined in: [core/SmartCache.ts:41](https://github.com/isdk/proxy.js/blob/bed37f
|
|
|
75
90
|
|
|
76
91
|
`string`
|
|
77
92
|
|
|
93
|
+
缓存指纹键
|
|
94
|
+
|
|
78
95
|
#### Returns
|
|
79
96
|
|
|
80
97
|
`Promise`\<[`CacheEntry`](../interfaces/CacheEntry.md) \| `null`\>
|
|
81
98
|
|
|
99
|
+
完整的缓存条目(带 Buffer 或 Stream 的 Body),未命中返回 null
|
|
100
|
+
|
|
82
101
|
***
|
|
83
102
|
|
|
84
103
|
### getStream()
|
|
85
104
|
|
|
86
105
|
> **getStream**(`key`): `ReadableStream`
|
|
87
106
|
|
|
88
|
-
Defined in: [core/SmartCache.ts:
|
|
107
|
+
Defined in: [core/SmartCache.ts:159](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L159)
|
|
108
|
+
|
|
109
|
+
获取磁盘读取流
|
|
110
|
+
|
|
111
|
+
允许直接从 L2 磁盘层以流的形式读取数据,适用于大文件代理。
|
|
89
112
|
|
|
90
113
|
#### Parameters
|
|
91
114
|
|
|
@@ -93,19 +116,25 @@ Defined in: [core/SmartCache.ts:107](https://github.com/isdk/proxy.js/blob/bed37
|
|
|
93
116
|
|
|
94
117
|
`string`
|
|
95
118
|
|
|
119
|
+
缓存指纹键
|
|
120
|
+
|
|
96
121
|
#### Returns
|
|
97
122
|
|
|
98
123
|
`ReadableStream`
|
|
99
124
|
|
|
125
|
+
Node.js 可读流
|
|
126
|
+
|
|
100
127
|
***
|
|
101
128
|
|
|
102
129
|
### set()
|
|
103
130
|
|
|
104
131
|
> **set**(`key`, `body`, `metadata`): `Promise`\<`void`\>
|
|
105
132
|
|
|
106
|
-
Defined in: [core/SmartCache.ts:
|
|
133
|
+
Defined in: [core/SmartCache.ts:129](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L129)
|
|
107
134
|
|
|
108
|
-
|
|
135
|
+
写入缓存条目 (原子写入)
|
|
136
|
+
|
|
137
|
+
适用于已知长度的小型数据块。该操作会同时写入磁盘并回填内存(如果大小未超标)。
|
|
109
138
|
|
|
110
139
|
#### Parameters
|
|
111
140
|
|
|
@@ -113,14 +142,20 @@ Defined in: [core/SmartCache.ts:85](https://github.com/isdk/proxy.js/blob/bed37f
|
|
|
113
142
|
|
|
114
143
|
`string`
|
|
115
144
|
|
|
145
|
+
缓存指纹键
|
|
146
|
+
|
|
116
147
|
##### body
|
|
117
148
|
|
|
118
149
|
`Buffer`
|
|
119
150
|
|
|
151
|
+
响应体数据 Buffer
|
|
152
|
+
|
|
120
153
|
##### metadata
|
|
121
154
|
|
|
122
155
|
`Omit`\<[`CacheMetadata`](../interfaces/CacheMetadata.md), `"size"`\>
|
|
123
156
|
|
|
157
|
+
响应元数据(不含 size,由本方法自动计算)
|
|
158
|
+
|
|
124
159
|
#### Returns
|
|
125
160
|
|
|
126
161
|
`Promise`\<`void`\>
|
|
@@ -131,7 +166,14 @@ Defined in: [core/SmartCache.ts:85](https://github.com/isdk/proxy.js/blob/bed37f
|
|
|
131
166
|
|
|
132
167
|
> **setStream**(`key`, `metadata`): `WritableStream`
|
|
133
168
|
|
|
134
|
-
Defined in: [core/SmartCache.ts:
|
|
169
|
+
Defined in: [core/SmartCache.ts:175](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/SmartCache.ts#L175)
|
|
170
|
+
|
|
171
|
+
获取磁盘写入流 (流式缓存)
|
|
172
|
+
|
|
173
|
+
该方法用于支持真正的流式代理。它会执行以下一致性操作:
|
|
174
|
+
1. 立即清除 L1 内存中的对应键,防止读到旧数据。
|
|
175
|
+
2. 返回一个可写流,数据将直接流入磁盘。
|
|
176
|
+
3. **一致性修复**: 在流写入完成(finish)时再次清理内存,防止写入期间的并发读取将旧数据再次回填进内存。
|
|
135
177
|
|
|
136
178
|
#### Parameters
|
|
137
179
|
|
|
@@ -139,10 +181,16 @@ Defined in: [core/SmartCache.ts:111](https://github.com/isdk/proxy.js/blob/bed37
|
|
|
139
181
|
|
|
140
182
|
`string`
|
|
141
183
|
|
|
184
|
+
缓存指纹键
|
|
185
|
+
|
|
142
186
|
##### metadata
|
|
143
187
|
|
|
144
188
|
`Omit`\<[`CacheMetadata`](../interfaces/CacheMetadata.md), `"size"`\>
|
|
145
189
|
|
|
190
|
+
响应元数据
|
|
191
|
+
|
|
146
192
|
#### Returns
|
|
147
193
|
|
|
148
194
|
`WritableStream`
|
|
195
|
+
|
|
196
|
+
Node.js 可写流
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **createCachedFetch**(`defaultOptions`): (`request`, `fetcher`, `overrideOptions?`) => `Promise`\<`Response`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [core/createCachedFetch.ts:17](https://github.com/isdk/proxy.js/blob/
|
|
11
|
+
Defined in: [core/createCachedFetch.ts:17](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/createCachedFetch.ts#L17)
|
|
12
12
|
|
|
13
13
|
缓存请求工厂函数 (针对终端用户的顶层高阶 API)
|
|
14
14
|
|
|
@@ -8,7 +8,7 @@
|
|
|
8
8
|
|
|
9
9
|
> **createFetchWithCache**(`activeCacheWrites?`): (`request`, `fetcher`, `options`) => `Promise`\<`Response`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [core/createFetchWithCache.ts:16](https://github.com/isdk/proxy.js/blob/
|
|
11
|
+
Defined in: [core/createFetchWithCache.ts:16](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/createFetchWithCache.ts#L16)
|
|
12
12
|
|
|
13
13
|
单一职责高阶函数:专门用于封装和隔离 activeCacheWrites 并发追踪器。
|
|
14
14
|
|
|
@@ -6,18 +6,23 @@
|
|
|
6
6
|
|
|
7
7
|
# Function: extractData()
|
|
8
8
|
|
|
9
|
-
> **extractData**(`source`, `config?`): `Record`\<`string`, `string`[]\>
|
|
9
|
+
> **extractData**(`source`, `config?`, `defaultAllowed?`): `Record`\<`string`, `string`[]\>
|
|
10
10
|
|
|
11
|
-
Defined in: [utils/extractData.ts:
|
|
11
|
+
Defined in: [utils/extractData.ts:40](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/utils/extractData.ts#L40)
|
|
12
12
|
|
|
13
13
|
从源对象中根据过滤配置提取数据并标准化。
|
|
14
14
|
|
|
15
15
|
此函数主要用于生成缓存指纹。它会:
|
|
16
|
-
1. 根据 `config` (include/exclude)
|
|
16
|
+
1. 根据 `config` (include/exclude) 过滤键,调用 `isAllowed` 判断每个键是否允许。
|
|
17
17
|
2. 对键进行排序以保证指纹的一致性。
|
|
18
18
|
3. 将所有键转换为小写。
|
|
19
19
|
4. 将值统一包装为数组并进行排序,消除数组项顺序差异。
|
|
20
20
|
|
|
21
|
+
**关于 `defaultAllowed` 参数**:
|
|
22
|
+
- 只有当没有配置 `include` 和 `exclude` 时,`defaultAllowed` 才会生效。
|
|
23
|
+
- 如果配置了 `include`(即使为空数组),`defaultAllowed` 也不会生效。
|
|
24
|
+
- 详见 `isAllowed` 函数的优先级逻辑。
|
|
25
|
+
|
|
21
26
|
## Parameters
|
|
22
27
|
|
|
23
28
|
### source
|
|
@@ -30,10 +35,34 @@ Defined in: [utils/extractData.ts:17](https://github.com/isdk/proxy.js/blob/bed3
|
|
|
30
35
|
|
|
31
36
|
[`KeyFilterConfig`](../interfaces/KeyFilterConfig.md)
|
|
32
37
|
|
|
33
|
-
|
|
38
|
+
过滤配置,支持 `include`(白名单)和 `exclude`(黑名单)
|
|
39
|
+
|
|
40
|
+
### defaultAllowed?
|
|
41
|
+
|
|
42
|
+
`boolean`
|
|
43
|
+
|
|
44
|
+
当没有配置时的默认值(默认 `false`,即不提取任何键)
|
|
34
45
|
|
|
35
46
|
## Returns
|
|
36
47
|
|
|
37
48
|
`Record`\<`string`, `string`[]\>
|
|
38
49
|
|
|
39
|
-
标准化后的数据 Map
|
|
50
|
+
标准化后的数据 Map,键为小写,值为排序后的字符串数组
|
|
51
|
+
|
|
52
|
+
## Example
|
|
53
|
+
|
|
54
|
+
```typescript
|
|
55
|
+
const headers = { 'Content-Type': 'application/json', 'X-Request-Id': '123' };
|
|
56
|
+
|
|
57
|
+
// 默认不提取任何键
|
|
58
|
+
extractData(headers); // {}
|
|
59
|
+
|
|
60
|
+
// 提取所有键
|
|
61
|
+
extractData(headers, undefined, true); // { 'content-type': ['application/json'], 'x-request-id': ['123'] }
|
|
62
|
+
|
|
63
|
+
// 白名单
|
|
64
|
+
extractData(headers, { include: ['content-type'] }); // { 'content-type': ['application/json'] }
|
|
65
|
+
|
|
66
|
+
// 黑名单(需要 include 或 defaultAllowed)
|
|
67
|
+
extractData(headers, { include: ['*'], exclude: ['x-request-id'] }, true); // { 'content-type': ['application/json'] }
|
|
68
|
+
```
|
|
@@ -8,11 +8,13 @@
|
|
|
8
8
|
|
|
9
9
|
> **fetchWithCache**(`request`, `fetcher`, `options`): `Promise`\<`Response`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [core/fetchWithCache.ts:
|
|
11
|
+
Defined in: [core/fetchWithCache.ts:370](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/fetchWithCache.ts#L370)
|
|
12
12
|
|
|
13
13
|
核心协调函数 (Fetcher Orchestrator)
|
|
14
14
|
|
|
15
15
|
实现了基于流的混合缓存代理核心逻辑,主要机制包括:
|
|
16
|
+
- **多方法支持与过滤**:支持通过 `allowedMethods` 配置可缓存的方法(如 POST, PUT),并通过 `cacheRules` 进行精细化的路径与参数匹配拦截。
|
|
17
|
+
- **异步 Request Body 处理**:当缓存 POST/PUT 请求时,会自动读取 Body 并计算唯一指纹(支持 JSON 字段过滤)。
|
|
16
18
|
- **大文件流式处理**:底层完全通过 Streams 实现,代理大文件时自动写入磁盘且防 OOM。
|
|
17
19
|
- **SWR (Stale-While-Revalidate)**:后台静默更新机制。
|
|
18
20
|
- **并发防击穿 (Request Coalescing)**:利用 `activeCacheWrites` 将并发请求合并。
|
|
@@ -26,14 +28,22 @@ Defined in: [core/fetchWithCache.ts:244](https://github.com/isdk/proxy.js/blob/b
|
|
|
26
28
|
|
|
27
29
|
`Request`
|
|
28
30
|
|
|
31
|
+
原始 Web 标准 Request 对象
|
|
32
|
+
|
|
29
33
|
### fetcher
|
|
30
34
|
|
|
31
35
|
(`req`) => `Promise`\<`Response`\>
|
|
32
36
|
|
|
37
|
+
实际执行网络请求的函数
|
|
38
|
+
|
|
33
39
|
### options
|
|
34
40
|
|
|
35
41
|
[`FetchWithCacheOptions`](../interfaces/FetchWithCacheOptions.md)
|
|
36
42
|
|
|
43
|
+
缓存配置选项
|
|
44
|
+
|
|
37
45
|
## Returns
|
|
38
46
|
|
|
39
47
|
`Promise`\<`Response`\>
|
|
48
|
+
|
|
49
|
+
带有缓存标识头和流式 Body 的 Response 对象
|
|
@@ -6,11 +6,26 @@
|
|
|
6
6
|
|
|
7
7
|
# Function: generateCacheKey()
|
|
8
8
|
|
|
9
|
-
> **generateCacheKey**(`req`, `config`): `string
|
|
9
|
+
> **generateCacheKey**(`req`, `config`): `Promise`\<`string`\>
|
|
10
10
|
|
|
11
|
-
Defined in: [core/generateCacheKey.ts:
|
|
11
|
+
Defined in: [core/generateCacheKey.ts:36](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/core/generateCacheKey.ts#L36)
|
|
12
12
|
|
|
13
|
-
根据 Request
|
|
13
|
+
根据 Request 对象和站点配置生成唯一的缓存指纹 (异步)
|
|
14
|
+
|
|
15
|
+
该函数是缓存系统的核心组件,用于将复杂的 HTTP 请求对象转换为唯一的 SHA-256 字符串。
|
|
16
|
+
它实现了高度可定制的提取逻辑,允许通过配置排除掉请求中不稳定的因素(如时间戳、Nonce 等)。
|
|
17
|
+
|
|
18
|
+
### 生成指纹包含的要素:
|
|
19
|
+
1. **Method**: 请求方法(统一转为大写)。
|
|
20
|
+
2. **Host & Path**: 请求的域名和路径。
|
|
21
|
+
3. **Query Params**: URL 查询参数,受 `config.query` 过滤影响。
|
|
22
|
+
4. **Headers**: 请求头信息,受 `config.headers` 过滤影响。默认排除 `cookie` 头。
|
|
23
|
+
5. **Cookies**: 特别提取的 Cookie 字段,受 `config.cookies` 过滤影响。
|
|
24
|
+
6. **Request Body**:
|
|
25
|
+
- 对于 `POST`, `PUT`, `PATCH` 请求,会自动尝试读取 Body。
|
|
26
|
+
- **JSON 类型**: 如果 `Content-Type` 包含 `application/json`,则解析为对象并应用 `config.body` 过滤。
|
|
27
|
+
- **非 JSON/流类型**: 回退到对原始 Body 字节流进行 SHA-256 哈希计算。
|
|
28
|
+
- **安全性**: 使用 `req.clone()` 读取 Body,确保不影响后续真实的 Fetch 请求流消费。
|
|
14
29
|
|
|
15
30
|
## Parameters
|
|
16
31
|
|
|
@@ -18,10 +33,25 @@ Defined in: [core/generateCacheKey.ts:8](https://github.com/isdk/proxy.js/blob/b
|
|
|
18
33
|
|
|
19
34
|
`Request`
|
|
20
35
|
|
|
36
|
+
原始 Web 标准 Request 对象。
|
|
37
|
+
|
|
21
38
|
### config
|
|
22
39
|
|
|
23
40
|
[`SiteCacheConfig`](../interfaces/SiteCacheConfig.md)
|
|
24
41
|
|
|
42
|
+
站点级缓存配置,决定了哪些字段参与指纹计算。
|
|
43
|
+
|
|
25
44
|
## Returns
|
|
26
45
|
|
|
27
|
-
`string
|
|
46
|
+
`Promise`\<`string`\>
|
|
47
|
+
|
|
48
|
+
返回一个 64 位十六进制的 SHA-256 哈希字符串作为缓存键。
|
|
49
|
+
|
|
50
|
+
## Example
|
|
51
|
+
|
|
52
|
+
```typescript
|
|
53
|
+
const cacheKey = await generateCacheKey(request, {
|
|
54
|
+
query: { exclude: ['timestamp'] },
|
|
55
|
+
body: { include: ['id', 'action'] }
|
|
56
|
+
});
|
|
57
|
+
```
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
[**@isdk/proxy**](../README.md)
|
|
2
|
+
|
|
3
|
+
***
|
|
4
|
+
|
|
5
|
+
[@isdk/proxy](../globals.md) / getSiteConfig
|
|
6
|
+
|
|
7
|
+
# Function: getSiteConfig()
|
|
8
|
+
|
|
9
|
+
> **getSiteConfig**(`urlString`, `proxyConfig`): [`SiteCacheConfig`](../interfaces/SiteCacheConfig.md)
|
|
10
|
+
|
|
11
|
+
Defined in: [utils/getSiteConfig.ts:17](https://github.com/isdk/proxy.js/blob/76fee3a101f98e5bf29599fe7ea02ab06479cf70/src/utils/getSiteConfig.ts#L17)
|
|
12
|
+
|
|
13
|
+
根据 URL 获取对应的站点缓存配置
|
|
14
|
+
|
|
15
|
+
匹配逻辑:
|
|
16
|
+
1. 遍历 sites 中的所有 key。
|
|
17
|
+
2. 如果 key 是正则或 Glob 格式字符串,则对完整 URL 进行匹配。
|
|
18
|
+
3. 如果 key 是普通字符串,则作为 URL 前缀进行匹配。
|
|
19
|
+
4. 返回第一个匹配到的配置;若均未匹配,则返回 defaultConfig。
|
|
20
|
+
|
|
21
|
+
## Parameters
|
|
22
|
+
|
|
23
|
+
### urlString
|
|
24
|
+
|
|
25
|
+
`string`
|
|
26
|
+
|
|
27
|
+
请求的完整 URL
|
|
28
|
+
|
|
29
|
+
### proxyConfig
|
|
30
|
+
|
|
31
|
+
[`ProxyConfig`](../interfaces/ProxyConfig.md)
|
|
32
|
+
|
|
33
|
+
全局代理配置
|
|
34
|
+
|
|
35
|
+
## Returns
|
|
36
|
+
|
|
37
|
+
[`SiteCacheConfig`](../interfaces/SiteCacheConfig.md)
|
|
38
|
+
|
|
39
|
+
匹配到的站点配置
|