@zukhruf/relevance-filtering 0.5.1 → 0.6.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (42) hide show
  1. package/README.md +66 -63
  2. package/dist/decision-scorer.d.ts +2 -2
  3. package/dist/decision-scorer.d.ts.map +1 -1
  4. package/dist/decision-scorer.js +13 -8
  5. package/dist/decision-scorer.js.map +1 -1
  6. package/dist/errors.d.ts +0 -7
  7. package/dist/errors.d.ts.map +1 -1
  8. package/dist/errors.js +2 -19
  9. package/dist/errors.js.map +1 -1
  10. package/dist/index.d.ts +3 -2
  11. package/dist/index.d.ts.map +1 -1
  12. package/dist/index.js +3 -2
  13. package/dist/index.js.map +1 -1
  14. package/dist/keyword-scorer.d.ts +14 -0
  15. package/dist/keyword-scorer.d.ts.map +1 -0
  16. package/dist/keyword-scorer.js +21 -0
  17. package/dist/keyword-scorer.js.map +1 -0
  18. package/dist/scorer.d.ts +2 -4
  19. package/dist/scorer.d.ts.map +1 -1
  20. package/dist/scores.d.ts +1 -6
  21. package/dist/scores.d.ts.map +1 -1
  22. package/dist/scores.js +15 -23
  23. package/dist/scores.js.map +1 -1
  24. package/dist/similar.d.ts +13 -0
  25. package/dist/similar.d.ts.map +1 -0
  26. package/dist/similar.js +40 -0
  27. package/dist/similar.js.map +1 -0
  28. package/dist/similarity-scorer.d.ts +9 -9
  29. package/dist/similarity-scorer.d.ts.map +1 -1
  30. package/dist/similarity-scorer.js +50 -88
  31. package/dist/similarity-scorer.js.map +1 -1
  32. package/dist/taught-answers.d.ts +15 -0
  33. package/dist/taught-answers.d.ts.map +1 -0
  34. package/dist/taught-answers.js +25 -0
  35. package/dist/taught-answers.js.map +1 -0
  36. package/dist/threshold.d.ts +1 -2
  37. package/dist/threshold.d.ts.map +1 -1
  38. package/package.json +2 -2
  39. package/dist/profile.d.ts +0 -18
  40. package/dist/profile.d.ts.map +0 -1
  41. package/dist/profile.js +0 -35
  42. package/dist/profile.js.map +0 -1
package/README.md CHANGED
@@ -1,11 +1,12 @@
1
1
  # @zukhruf/relevance-filtering
2
2
 
3
- A reader wants the new articles of a feed that are about an interest, and not the others. This package ranks the articles of a stream for a profile of the reader, the best first, so a reader can show the best N articles of each day. A profile has a name, one sentence that describes the interest, optional keywords, and the yes and no answers of the reader. A scorer gives each article a probability and ranks the stream by it. This kind of program is also known as routing, as in TREC. A threshold that the host chooses can turn the probabilities into a yes or no for an action, for example a notification.
3
+ A reader wants the new articles of a feed that are about an interest, and not the others. This package ranks the articles of a stream for one interest of the reader, the best first, so a reader can show the best N articles of each day. A scorer, made for one interest, gives each article a probability and ranks the stream by it. This kind of program is also known as routing, as in TREC. A threshold that the host chooses can turn the probabilities into a yes or no for an action, for example a notification.
4
4
 
5
- The package has two scorers:
5
+ The package has three scorers:
6
6
 
7
- - **`DecisionScorer`** asks a decision model, through the AI SDK, if the main subject of each article is the description. It ranked best in research rounds 13 and 14.
8
- - **`SimilarityScorer`** compares each article with the description by embedding similarity, makes each similarity a calibrated probability from the reader's answers, and adds the evidence of the keywords. It needs no decision model.
7
+ - **`DecisionScorer`** asks a decision model, through the AI SDK, if the main subject of each article is the description of the interest. It ranked best in research rounds 13 and 14, and it needs nothing but the description.
8
+ - **`SimilarityScorer`** compares each article with the description by embedding similarity, makes each similarity a calibrated probability from the reader's yes and no answers, and adds the evidence of the keywords. It needs at least one yes answer.
9
+ - **`KeywordScorer`** is the cold start of the similarity design: the articles that match a keyword come first. It needs at least one keyword.
9
10
 
10
11
  The words in these documents have one meaning each. See the glossary in [CONTEXT.md](./CONTEXT.md).
11
12
 
@@ -19,31 +20,23 @@ npm install @zukhruf/relevance-filtering ai
19
20
 
20
21
  ### With a decision model
21
22
 
22
- Give the scorer an AI SDK decision model, for example Liquid AI's d1-2B on a llama.cpp server through `@ai-sdk/typesafe-ai`.
23
+ Give the scorer an AI SDK decision model, for example Liquid AI's d1-2B on a llama.cpp server through `@ai-sdk/typesafe-ai`, and the description of the interest.
23
24
 
24
25
  ```ts
25
26
  import { createTypeSafeAi } from '@ai-sdk/typesafe-ai';
26
27
 
27
- import {
28
- DecisionScorer,
29
- Profile,
30
- UtilityThreshold,
31
- } from '@zukhruf/relevance-filtering';
28
+ import { DecisionScorer, UtilityThreshold } from '@zukhruf/relevance-filtering';
32
29
 
33
30
  const scorer = new DecisionScorer({
34
31
  model: createTypeSafeAi({
35
32
  baseURL: 'http://127.0.0.1:9931/v1',
36
33
  apiKey: 'none',
37
34
  }).decisionModel('LiquidAI/d1-2B-GGUF:Q8_0'),
38
- });
39
- const profile = new Profile({
40
- name: 'Security',
41
35
  description: 'Cybersecurity: attacks, breaches and the tools that stop them.',
42
- relevant: [],
43
- notRelevant: [],
44
36
  });
45
37
 
46
- const scores = await scorer.score(profile, newArticles); // [{ id, publishedAt, text: `${title} ${lead}` }, …]
38
+ // The new articles by id: { publishedAt, text: `${title} ${lead}` }.
39
+ const scores = await scorer.score(newArticles);
47
40
 
48
41
  // The best 5 articles of one day, the best first.
49
42
  scores.ranked({ from: dayStart, to: dayEnd }).slice(0, 5);
@@ -54,7 +47,7 @@ scores.accepted(new UtilityThreshold({ falseAlarm: 1, miss: 3 }));
54
47
 
55
48
  ### With embeddings
56
49
 
57
- Give the scorer a vectors port that holds the vectors of the articles, for example a `Collection` of [`@zukhruf/embeddings`](../../embeddings), and a function that gives the vector of a description from the same model. The design comes from research round 6 of #3195, and it was frozen before its exam ([ADR 0002](./docs/adr/0002-the-design-is-frozen-by-research-round-6.md)). It was measured with EmbeddingGemma 2, with the document prompt for the articles and the query prompt for the description.
50
+ Give the scorer a vectors port that holds the vectors of the articles, for example a `Collection` of [`@zukhruf/embeddings`](../../embeddings), and the vector of the description from the same model. The design comes from research round 6 of #3195, and it was frozen before its exam ([ADR 0002](./docs/adr/0002-the-design-is-frozen-by-research-round-6.md)). It was measured with EmbeddingGemma 2, with the document prompt for the articles and the query prompt for the description.
58
51
 
59
52
  ```ts
60
53
  import { DatabaseSync } from 'node:sqlite';
@@ -63,7 +56,11 @@ import { createOpenAICompatible } from '@ai-sdk/openai-compatible';
63
56
  import { embed } from 'ai';
64
57
 
65
58
  import { Collection } from '@zukhruf/embeddings';
66
- import { Profile, SimilarityScorer } from '@zukhruf/relevance-filtering';
59
+ import {
60
+ KeywordScorer,
61
+ SimilarityScorer,
62
+ TaughtAnswers,
63
+ } from '@zukhruf/relevance-filtering';
67
64
 
68
65
  const model = createOpenAICompatible({
69
66
  name: 'llama.cpp',
@@ -74,47 +71,54 @@ const collection = new Collection({ database, model });
74
71
  // The host upserts each article with the document prompt of the model:
75
72
  // title: <title> | text: <title> <lead>.
76
73
 
77
- const scorer = new SimilarityScorer({
78
- vectors: collection,
79
- embedDescription: async (description) =>
80
- (
81
- await embed({
82
- model,
83
- value: `task: search result | query: ${description}`,
84
- })
85
- ).embedding,
86
- taughtText: (id) => textOf(id),
87
- });
88
- const profile = new Profile({
89
- name: 'Security',
90
- description: 'Cybersecurity: attacks, breaches and the tools that stop them.',
91
- keywords: ['secur', 'cyber'],
92
- relevant: ['ransomware-hospital'],
93
- notRelevant: ['chip-plant'],
74
+ const description =
75
+ 'Cybersecurity: attacks, breaches and the tools that stop them.';
76
+ const { embedding: descriptionVector } = await embed({
77
+ model,
78
+ value: `task: search result | query: ${description}`,
94
79
  });
95
80
 
96
- const scores = await scorer.score(profile, newArticles);
81
+ // The reader's answers by article id, each with the text of its article.
82
+ const answers = TaughtAnswers.parse(
83
+ new Map([
84
+ ['ransomware-hospital', { relevant: true, text: 'Ransomware hits …' }],
85
+ ['chip-plant', { relevant: false, text: 'A chip plant opens …' }],
86
+ ]),
87
+ );
88
+ const scorer =
89
+ answers === undefined
90
+ ? new KeywordScorer({
91
+ vectors: collection,
92
+ descriptionVector,
93
+ keywords: ['secur', 'cyber'],
94
+ })
95
+ : new SimilarityScorer({
96
+ vectors: collection,
97
+ descriptionVector,
98
+ answers,
99
+ keywords: ['secur', 'cyber'],
100
+ });
101
+
102
+ const scores = await scorer.score(newArticles);
97
103
  scores.ranked({ from: dayStart, to: dayEnd }).slice(0, 5);
98
104
  ```
99
105
 
100
106
  ## API
101
107
 
102
- - `new Profile({ name, description, keywords?, relevant, notRelevant })` makes a profile. `relevant` and `notRelevant` are the ids of the articles that the reader answered yes and no for. An id in both lists is a `ConflictingAnswerError`. A profile does no I/O ([ADR 0001](./docs/adr/0001-a-profile-is-a-value-and-train-decides-on-one-stream.md)).
103
- - `profile.with(changes)` gives a copy with the changed fields, for example a new answer. The profile does not change.
104
- - `keywords` holds strings and regular expressions. A string matches at the start of a word, in any case: `secur` matches "Security" and "securing", but not "cybersecurity". A regular expression matches as it is ([ADR 0003](./docs/adr/0003-a-keyword-is-a-word-start-or-a-regular-expression.md)).
105
- - A `Scorer` has `score(profile, stream)`, which gives a `Promise` of `Scores` ([ADR 0005](./docs/adr/0005-a-scorer-and-a-threshold-are-ports.md)). `stream` is a list of one or more articles: `{ id, publishedAt, text }`. `text` is, for example, the title and the lead.
106
- - `new DecisionScorer({ model })` takes an AI SDK decision model. For each article, it asks the model "Is the main subject of this article: <description>?" about `text`, one article at a time. It uses no keyword and no answer.
107
- - `new SimilarityScorer({ vectors, embedDescription, taughtText })` is the frozen design.
108
- - `vectors` is the port to the vectors of the articles: `query(vector, { from, to })` and `fetch(ids)`. The `Collection` of `@zukhruf/embeddings` has this shape. The stream and the taught articles must be in it. `publishedAt` must be the date that the vectors store for the article.
109
- - `embedDescription(description)` gives the vector of a description, from the model of the stored vectors. Make it with the prompt that the model uses for a query.
110
- - `taughtText(id)` gives the text of a taught relevant article, which the keywords are matched in. The scorer calls it only when the profile has keywords.
111
- - A profile with at least one relevant answer gets calibrated probabilities, ranked by probability, then by similarity, then by id. A profile with no relevant answer, or a profile whose binormal fit cannot hold and that has no not-relevant answer, ranks the articles that match a keyword first, the newest first, and then the other articles by similarity. Its probability is 1 for a match and 0 for the others.
112
- - `Scores` gives `ranked(range?)`: the ids of its stream, the best first. With `{ from, to }`, only the articles published in that range, in the same order. The best N of a day are `ranked({ from, to }).slice(0, n)` ([ADR 0004](./docs/adr/0004-the-reader-ranks-and-does-not-decide-yes-or-no.md)).
113
- - `Scores` gives `probability(id)` for each article of its stream, and `accepted(threshold)`: the articles that the threshold accepts, in the order of `ranked()`.
114
- - A `Threshold` has `cut(scores, profile)`, which gives the lowest probability to accept (`Infinity` accepts nothing).
115
- - `new UtilityThreshold({ falseAlarm, miss })` accepts when p >= falseAlarm / (falseAlarm + miss), the Bayes decision rule for calibrated probabilities. The two costs are of 0 or more, and not both 0.
108
+ - A `Scorer` has `score(stream)`, which gives a `Promise` of `Scores`. A scorer is made for one interest ([ADR 0006](./docs/adr/0006-a-scorer-is-made-for-one-interest-and-its-inputs-rule-out-illegal-states.md)). The `stream` is a map from article id to `{ publishedAt, text }`; `text` is, for example, the title and the lead. It can be empty.
109
+ - `new DecisionScorer({ model, description })` takes an AI SDK decision model and the description. For each article, it asks the model "Is the main subject of this article: <description>?" about `text`, one article at a time.
110
+ - `new SimilarityScorer({ vectors, descriptionVector, answers, keywords })` is the frozen design for an interest with at least one yes answer.
111
+ - `vectors` is the port to the vectors of the articles: `query(vector, { from, to })` and `fetch(ids)`. The `Collection` of `@zukhruf/embeddings` has this shape. The stream and the taught articles must be in it, and `publishedAt` must be the date that the vectors store for the article.
112
+ - `descriptionVector` is the vector of the description from the model of the stored vectors, made with the prompt that the model uses for a query.
113
+ - `answers` is a `TaughtAnswers`. `keywords` holds strings and regular expressions, and can be empty. A string matches at the start of a word, in any case: `secur` matches "Security" and "securing", but not "cybersecurity". A regular expression matches as it is ([ADR 0003](./docs/adr/0003-a-keyword-is-a-word-start-or-a-regular-expression.md)).
114
+ - It ranks by the calibrated probability, then by similarity, then by id. When neither the binormal fit nor Platt's method can hold, it ranks the keyword matches first, the newest first, and then the other articles by similarity, with the probability 1 for a match and 0 for the others.
115
+ - `TaughtAnswers.parse(answers)` takes the reader's answers as a map from article id to `{ relevant, text }`, and gives `undefined` when no answer is a yes.
116
+ - `new KeywordScorer({ vectors, descriptionVector, keywords })` is the cold start: `keywords` has at least one keyword. It ranks the matches first, the newest first, and then the other articles by similarity. Its probability is 1 for a match and 0 for the others, and it is not calibrated.
117
+ - `Scores` gives `ranked(range?)`: the ids of the stream, the best first. With `{ from, to }`, only the articles published in that range, in the same order. The best N of a day are `ranked({ from, to }).slice(0, n)` ([ADR 0004](./docs/adr/0004-the-reader-ranks-and-does-not-decide-yes-or-no.md)). It also gives `probability(id)`, and `accepted(threshold)`: the articles that the threshold accepts, in the order of `ranked()`.
118
+ - A `Threshold` has `cut(scores)`, which gives the lowest probability to accept (`Infinity` accepts nothing).
119
+ - `new UtilityThreshold({ falseAlarm, miss })` accepts when p >= falseAlarm / (falseAlarm + miss), the Bayes decision rule for calibrated probabilities. The two costs are finite, 0 or more, and not both 0.
116
120
  - `new ExpectedFThreshold({ beta })` accepts the m most probable articles, where m maximises the expected F-beta of the stream. With beta 0.5 it is the threshold of the frozen design.
117
- - To add a method, implement `Scorer` or `Threshold`. A scorer gives `new Scores({ profile, ranked })`, with `ranked` the stream in its order, each article with its probability.
121
+ - To add a method, implement `Scorer` or `Threshold`. A scorer gives `new Scores(ranked)`, with `ranked` a map from article id to `{ publishedAt, probability }` in its order.
118
122
 
119
123
  ## Rank, then decide for each action
120
124
 
@@ -124,23 +128,21 @@ Show the ranking, and decide yes or no only for an action that needs it. Researc
124
128
 
125
129
  1. **Similarity.** The similarity of each article is the cosine of its vector and the description vector. The answers do not change the similarity.
126
130
  2. **Binormal model.** The scorer standardises the similarities of the stream by their mean and standard deviation. It fits a relevant normal distribution and an other normal distribution by semi-supervised EM. The taught similarities stay in the component of their answer. The probability of an article is its posterior of the relevant component.
127
- 3. **Platt.** When the stream similarities are all equal, or the relevant mean falls to or below the other mean, the binormal fit cannot hold. Then Platt's method on the taught similarities gives the probabilities.
128
- 4. **Keyword evidence.** When the profile has keywords, naive Bayes adds the log likelihood ratio of a match, or of no match, to the log odds of each article.
131
+ 3. **Platt.** When the stream similarities are all equal, or the relevant mean falls to or below the other mean, the binormal fit cannot hold. Then Platt's method on the taught similarities gives the probabilities, when at least one answer is a no.
132
+ 4. **Keyword evidence.** When there are keywords, naive Bayes adds the log likelihood ratio of a match, or of no match, to the log odds of each article.
129
133
  5. **Order.** The scorer ranks the stream by the probability, then by the similarity, then by id.
130
134
 
131
135
  ## Errors
132
136
 
133
- - `ConflictingAnswerError`: one id is a relevant and a not-relevant answer.
134
- - `NoEvidenceError`: the similarity scorer got a profile with no relevant answer that it can learn from and no keyword.
135
137
  - `MissingVectorError`: the vectors have no article of the stream with its publish date.
136
138
  - `UnknownArticleError`: `probability` got an id that is not in the stream.
137
- - `RangeError`: the stream has one id two times, or the costs of a `UtilityThreshold` give no threshold.
139
+ - `RangeError`: the costs of a `UtilityThreshold` give no threshold.
138
140
  - The error of the vectors port, for example the `UnknownDocumentError` of `@zukhruf/embeddings`: a taught article is not in the vectors.
139
141
  - The errors of the AI SDK, for example `Experimental_DecisionRefusalError` when the decision model refuses a question. The decision scorer gives no probability in place of a refused or failed call.
140
142
 
141
143
  ## Result
142
144
 
143
- Rounds 13 and 14 ranked January to March 2025 (1,370 articles) for each profile. The score is MAP over the profiles with at least 20 strict relevant articles:
145
+ Rounds 13 and 14 ranked January to March 2025 (1,370 articles) for each research profile. The score is MAP over the profiles with at least 20 strict relevant articles:
144
146
 
145
147
  | Answer key | Decision scorer (d1-2B) | Similarity scorer | The reader's keyword list |
146
148
  | ---------------------------------------- | ----------------------- | ----------------- | ------------------------- |
@@ -161,28 +163,29 @@ A keyword list puts the matches first, the newest first. The 95% interval of (si
161
163
  ## Limits
162
164
 
163
165
  - The decision scorer was measured only with d1-2B (Q8_0), the question of round 13, and the title and the lead as the text. Another model or another question needs a new research round.
164
- - The decision scorer calls the model once for each article and each profile. The host decides when to score and if it keeps the scores.
166
+ - The decision scorer calls the model once for each article of the stream. The host decides when to score and if it keeps the scores.
165
167
  - The answer keys of rounds 13 and 14 come from language models. No person labelled that slice (limitation 16 of the research record).
166
168
  - The similarity design is valid only with EmbeddingGemma 2, the vector text of event detection (`title: <title> | text: <title>\n<lead>`) for the articles and the query prompt for the description. Another model gives another similarity scale.
167
169
  - The binormal model is fitted on the stream. A stream of a few articles gives a weak fit. The research streams held 1,600 to 1,700 articles.
168
170
  - When the answers are only the most similar articles of a large pool, the taught relevant similarities are far above the relevant articles of the stream. Then the probabilities of the similarity scorer are too low, and a threshold accepts too few articles. The order stays good.
169
- - With one relevant answer, the relevant standard deviation sits at its floor, and the keyword evidence sets most of the order.
171
+ - With one yes answer, the relevant standard deviation sits at its floor, and the keyword evidence sets most of the order.
170
172
  - A keyword string matches at the start of a word only. Type each part that you want: `cyber` and `secur` for "cybersecurity".
171
- - The similarity scorer reads the vectors of the whole publish range of the stream. A stream that spans a long time reads many vectors.
173
+ - The similarity and keyword scorers read the vectors of the whole publish range of the stream. A stream that spans a long time reads many vectors.
172
174
 
173
175
  ## Decisions
174
176
 
175
177
  Before you change a step of the similarity design, read ADR 0002, and run `nx run relevance-filtering:eval` before and after the change.
176
178
 
177
- - [ADR 0001: A profile is a value, and train decides on one stream](./docs/adr/0001-a-profile-is-a-value-and-train-decides-on-one-stream.md)
179
+ - [ADR 0001: A profile is a value, and train decides on one stream](./docs/adr/0001-a-profile-is-a-value-and-train-decides-on-one-stream.md) (superseded)
178
180
  - [ADR 0002: The design is frozen by research round 6](./docs/adr/0002-the-design-is-frozen-by-research-round-6.md)
179
181
  - [ADR 0003: A keyword is a word start or a regular expression](./docs/adr/0003-a-keyword-is-a-word-start-or-a-regular-expression.md)
180
182
  - [ADR 0004: The reader ranks, and does not decide yes or no](./docs/adr/0004-the-reader-ranks-and-does-not-decide-yes-or-no.md)
181
183
  - [ADR 0005: A scorer and a threshold are ports](./docs/adr/0005-a-scorer-and-a-threshold-are-ports.md)
184
+ - [ADR 0006: A scorer is made for one interest, and its inputs rule out illegal states](./docs/adr/0006-a-scorer-is-made-for-one-interest-and-its-inputs-rule-out-illegal-states.md)
182
185
  - [Code copied from the event-detection eval](./docs/copied-code.md)
183
186
 
184
187
  ## Recipes
185
188
 
186
189
  - [Rank a feed with a decision model](./docs/recipes/rank-a-feed-with-a-decision-model.md)
187
- - [Show the best articles of each day with a taught profile](./docs/recipes/filter-a-feed-with-a-taught-profile.md)
188
- - [Start a profile with keywords, then teach it](./docs/recipes/start-with-keywords-then-teach.md)
190
+ - [Show the best articles of each day with taught answers](./docs/recipes/filter-a-feed-with-a-taught-profile.md)
191
+ - [Start an interest with keywords, then teach it](./docs/recipes/start-with-keywords-then-teach.md)
@@ -1,12 +1,12 @@
1
1
  import { type Experimental_DecisionModel } from 'ai';
2
- import type { Profile } from './profile.ts';
3
2
  import type { Scorer, Stream } from './scorer.ts';
4
3
  import { Scores } from './scores.ts';
5
4
  export declare class DecisionScorer implements Scorer {
6
5
  #private;
7
6
  constructor(options: {
8
7
  readonly model: Experimental_DecisionModel;
8
+ readonly description: string;
9
9
  });
10
- score(profile: Profile, stream: Stream): Promise<Scores>;
10
+ score(stream: Stream): Promise<Scores>;
11
11
  }
12
12
  //# sourceMappingURL=decision-scorer.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"decision-scorer.d.ts","sourceRoot":"","sources":["../src/decision-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,0BAA0B,EAAuB,MAAM,IAAI,CAAC;AAE1E,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,MAAM,EAAc,MAAM,aAAa,CAAC;AAOjD,qBAAa,cAAe,YAAW,MAAM;;gBAG/B,OAAO,EAAE;QAAE,QAAQ,CAAC,KAAK,EAAE,0BAA0B,CAAA;KAAE;IAI7D,KAAK,CAAC,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;CAoB/D"}
1
+ {"version":3,"file":"decision-scorer.d.ts","sourceRoot":"","sources":["../src/decision-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,KAAK,0BAA0B,EAAuB,MAAM,IAAI,CAAC;AAE1E,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,MAAM,EAAc,MAAM,aAAa,CAAC;AAOjD,qBAAa,cAAe,YAAW,MAAM;;gBAI/B,OAAO,EAAE;QACnB,QAAQ,CAAC,KAAK,EAAE,0BAA0B,CAAC;QAE3C,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;KAC9B;IAKK,KAAK,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;CA2B7C"}
@@ -7,23 +7,28 @@ import { Scores, compareIds } from "./scores.js";
7
7
  // The question is the one that round 13 asked, word for word.
8
8
  export class DecisionScorer {
9
9
  #model;
10
+ #instructions;
10
11
  constructor(options) {
11
12
  this.#model = options.model;
13
+ this.#instructions = `Is the main subject of this article: ${options.description}?`;
12
14
  }
13
- async score(profile, stream) {
14
- const instructions = `Is the main subject of this article: ${profile.description}?`;
15
- const scored = await Array.fromAsync(stream, async ({ id, publishedAt, text }) => {
15
+ async score(stream) {
16
+ const scored = await Array.fromAsync(stream, async ([id, { publishedAt, text }]) => {
16
17
  const { answers } = await experimental_decide({
17
18
  model: this.#model,
18
19
  state: text,
19
- questions: { relevant: { type: 'boolean', instructions } },
20
+ questions: {
21
+ relevant: { type: 'boolean', instructions: this.#instructions },
22
+ },
20
23
  });
21
24
  return { id, publishedAt, probability: answers.relevant.probability };
22
25
  });
23
- return new Scores({
24
- profile,
25
- ranked: scored.toSorted((a, b) => b.probability - a.probability || compareIds(a.id, b.id)),
26
- });
26
+ return new Scores(new Map(scored
27
+ .toSorted((a, b) => b.probability - a.probability || compareIds(a.id, b.id))
28
+ .map(({ id, publishedAt, probability }) => [
29
+ id,
30
+ { publishedAt, probability },
31
+ ])));
27
32
  }
28
33
  }
29
34
  //# sourceMappingURL=decision-scorer.js.map
@@ -1 +1 @@
1
- {"version":3,"file":"decision-scorer.js","sourceRoot":"","sources":["../src/decision-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAmC,mBAAmB,EAAE,MAAM,IAAI,CAAC;AAI1E,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAEjD,2EAA2E;AAC3E,wEAAwE;AACxE,kEAAkE;AAClE,2EAA2E;AAC3E,8DAA8D;AAC9D,MAAM,OAAO,cAAc;IAChB,MAAM,CAA6B;IAE5C,YAAY,OAAuD;QACjE,IAAI,CAAC,MAAM,GAAG,OAAO,CAAC,KAAK,CAAC;IAC9B,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,OAAgB,EAAE,MAAc;QAC1C,MAAM,YAAY,GAAG,wCAAwC,OAAO,CAAC,WAAW,GAAG,CAAC;QACpF,MAAM,MAAM,GAAG,MAAM,KAAK,CAAC,SAAS,CAClC,MAAM,EACN,KAAK,EAAE,EAAE,EAAE,EAAE,WAAW,EAAE,IAAI,EAAE,EAAE,EAAE;YAClC,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,mBAAmB,CAAC;gBAC5C,KAAK,EAAE,IAAI,CAAC,MAAM;gBAClB,KAAK,EAAE,IAAI;gBACX,SAAS,EAAE,EAAE,QAAQ,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,YAAY,EAAE,EAAE;aAC3D,CAAC,CAAC;YACH,OAAO,EAAE,EAAE,EAAE,WAAW,EAAE,WAAW,EAAE,OAAO,CAAC,QAAQ,CAAC,WAAW,EAAE,CAAC;QACxE,CAAC,CACF,CAAC;QACF,OAAO,IAAI,MAAM,CAAC;YAChB,OAAO;YACP,MAAM,EAAE,MAAM,CAAC,QAAQ,CACrB,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,WAAW,GAAG,CAAC,CAAC,WAAW,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CAClE;SACF,CAAC,CAAC;IACL,CAAC;CACF"}
1
+ {"version":3,"file":"decision-scorer.js","sourceRoot":"","sources":["../src/decision-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAmC,mBAAmB,EAAE,MAAM,IAAI,CAAC;AAG1E,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAEjD,2EAA2E;AAC3E,wEAAwE;AACxE,kEAAkE;AAClE,2EAA2E;AAC3E,8DAA8D;AAC9D,MAAM,OAAO,cAAc;IAChB,MAAM,CAA6B;IACnC,aAAa,CAAS;IAE/B,YAAY,OAIX;QACC,IAAI,CAAC,MAAM,GAAG,OAAO,CAAC,KAAK,CAAC;QAC5B,IAAI,CAAC,aAAa,GAAG,wCAAwC,OAAO,CAAC,WAAW,GAAG,CAAC;IACtF,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,MAAc;QACxB,MAAM,MAAM,GAAG,MAAM,KAAK,CAAC,SAAS,CAClC,MAAM,EACN,KAAK,EAAE,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,IAAI,EAAE,CAAC,EAAE,EAAE;YACpC,MAAM,EAAE,OAAO,EAAE,GAAG,MAAM,mBAAmB,CAAC;gBAC5C,KAAK,EAAE,IAAI,CAAC,MAAM;gBAClB,KAAK,EAAE,IAAI;gBACX,SAAS,EAAE;oBACT,QAAQ,EAAE,EAAE,IAAI,EAAE,SAAS,EAAE,YAAY,EAAE,IAAI,CAAC,aAAa,EAAE;iBAChE;aACF,CAAC,CAAC;YACH,OAAO,EAAE,EAAE,EAAE,WAAW,EAAE,WAAW,EAAE,OAAO,CAAC,QAAQ,CAAC,WAAW,EAAE,CAAC;QACxE,CAAC,CACF,CAAC;QACF,OAAO,IAAI,MAAM,CACf,IAAI,GAAG,CACL,MAAM;aACH,QAAQ,CACP,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,WAAW,GAAG,CAAC,CAAC,WAAW,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CAClE;aACA,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC;YACzC,EAAE;YACF,EAAE,WAAW,EAAE,WAAW,EAAE;SAC7B,CAAC,CACL,CACF,CAAC;IACJ,CAAC;CACF"}
package/dist/errors.d.ts CHANGED
@@ -1,10 +1,3 @@
1
- export declare class ConflictingAnswerError extends Error {
2
- readonly id: string;
3
- constructor(id: string);
4
- }
5
- export declare class NoEvidenceError extends Error {
6
- constructor(profile: string);
7
- }
8
1
  export declare class MissingVectorError extends Error {
9
2
  readonly id: string;
10
3
  constructor(id: string);
@@ -1 +1 @@
1
- {"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AACA,qBAAa,sBAAuB,SAAQ,KAAK;IAC/C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAOvB;AAID,qBAAa,eAAgB,SAAQ,KAAK;gBAC5B,OAAO,EAAE,MAAM;CAM5B;AAID,qBAAa,kBAAmB,SAAQ,KAAK;IAC3C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAOvB;AAGD,qBAAa,mBAAoB,SAAQ,KAAK;IAC5C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAKvB"}
1
+ {"version":3,"file":"errors.d.ts","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAEA,qBAAa,kBAAmB,SAAQ,KAAK;IAC3C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAOvB;AAGD,qBAAa,mBAAoB,SAAQ,KAAK;IAC5C,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;gBAER,EAAE,EAAE,MAAM;CAKvB"}
package/dist/errors.js CHANGED
@@ -1,20 +1,3 @@
1
- // One article is both a relevant and a not-relevant answer of a profile.
2
- export class ConflictingAnswerError extends Error {
3
- id;
4
- constructor(id) {
5
- super(`The article ${id} is both a relevant and a not-relevant answer. Give it one answer.`);
6
- this.name = 'ConflictingAnswerError';
7
- this.id = id;
8
- }
9
- }
10
- // The profile has no relevant answer that the filter can learn from and no
11
- // keyword, so it has nothing to decide with.
12
- export class NoEvidenceError extends Error {
13
- constructor(profile) {
14
- super(`The profile ${profile} has no keyword and no relevant answer that the filter can learn from. Teach it a relevant article, or give it a keyword.`);
15
- this.name = 'NoEvidenceError';
16
- }
17
- }
18
1
  // The vectors gave no similarity for an article of the stream: the article
19
2
  // is not stored, or it is stored with another publish date.
20
3
  export class MissingVectorError extends Error {
@@ -25,11 +8,11 @@ export class MissingVectorError extends Error {
25
8
  this.id = id;
26
9
  }
27
10
  }
28
- // The filter was asked about an article that is not in its stream.
11
+ // Scores were asked about an article that is not in their stream.
29
12
  export class UnknownArticleError extends Error {
30
13
  id;
31
14
  constructor(id) {
32
- super(`The article ${id} is not in the stream of this filter.`);
15
+ super(`The article ${id} is not in the stream of these scores.`);
33
16
  this.name = 'UnknownArticleError';
34
17
  this.id = id;
35
18
  }
@@ -1 +1 @@
1
- {"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,yEAAyE;AACzE,MAAM,OAAO,sBAAuB,SAAQ,KAAK;IACtC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CACH,eAAe,EAAE,oEAAoE,CACtF,CAAC;QACF,IAAI,CAAC,IAAI,GAAG,wBAAwB,CAAC;QACrC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF;AAED,2EAA2E;AAC3E,6CAA6C;AAC7C,MAAM,OAAO,eAAgB,SAAQ,KAAK;IACxC,YAAY,OAAe;QACzB,KAAK,CACH,eAAe,OAAO,2HAA2H,CAClJ,CAAC;QACF,IAAI,CAAC,IAAI,GAAG,iBAAiB,CAAC;IAChC,CAAC;CACF;AAED,2EAA2E;AAC3E,4DAA4D;AAC5D,MAAM,OAAO,kBAAmB,SAAQ,KAAK;IAClC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CACH,+BAA+B,EAAE,uFAAuF,CACzH,CAAC;QACF,IAAI,CAAC,IAAI,GAAG,oBAAoB,CAAC;QACjC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF;AAED,mEAAmE;AACnE,MAAM,OAAO,mBAAoB,SAAQ,KAAK;IACnC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CAAC,eAAe,EAAE,uCAAuC,CAAC,CAAC;QAChE,IAAI,CAAC,IAAI,GAAG,qBAAqB,CAAC;QAClC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF"}
1
+ {"version":3,"file":"errors.js","sourceRoot":"","sources":["../src/errors.ts"],"names":[],"mappings":"AAAA,2EAA2E;AAC3E,4DAA4D;AAC5D,MAAM,OAAO,kBAAmB,SAAQ,KAAK;IAClC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CACH,+BAA+B,EAAE,uFAAuF,CACzH,CAAC;QACF,IAAI,CAAC,IAAI,GAAG,oBAAoB,CAAC;QACjC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF;AAED,kEAAkE;AAClE,MAAM,OAAO,mBAAoB,SAAQ,KAAK;IACnC,EAAE,CAAS;IAEpB,YAAY,EAAU;QACpB,KAAK,CAAC,eAAe,EAAE,wCAAwC,CAAC,CAAC;QACjE,IAAI,CAAC,IAAI,GAAG,qBAAqB,CAAC;QAClC,IAAI,CAAC,EAAE,GAAG,EAAE,CAAC;IACf,CAAC;CACF"}
package/dist/index.d.ts CHANGED
@@ -1,11 +1,12 @@
1
1
  export { DecisionScorer } from './decision-scorer.ts';
2
- export { ConflictingAnswerError, MissingVectorError, NoEvidenceError, UnknownArticleError, } from './errors.ts';
2
+ export { MissingVectorError, UnknownArticleError } from './errors.ts';
3
3
  export { ExpectedFThreshold } from './expected-f-threshold.ts';
4
4
  export type { Keyword } from './keyword.ts';
5
- export { Profile, type ProfileFields } from './profile.ts';
5
+ export { KeywordScorer } from './keyword-scorer.ts';
6
6
  export type { Article, Scorer, Stream } from './scorer.ts';
7
7
  export { type PublishRange, type ScoredArticle, Scores } from './scores.ts';
8
8
  export { SimilarityScorer } from './similarity-scorer.ts';
9
+ export { type Answer, type Taught, TaughtAnswers } from './taught-answers.ts';
9
10
  export type { Threshold } from './threshold.ts';
10
11
  export { UtilityThreshold } from './utility-threshold.ts';
11
12
  export type { Similarity, Vectors } from './vectors.ts';
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EACL,sBAAsB,EACtB,kBAAkB,EAClB,eAAe,EACf,mBAAmB,GACpB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAC/D,YAAY,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,OAAO,EAAE,KAAK,aAAa,EAAE,MAAM,cAAc,CAAC;AAC3D,YAAY,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAC3D,OAAO,EAAE,KAAK,YAAY,EAAE,KAAK,aAAa,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAC5E,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,YAAY,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,YAAY,EAAE,UAAU,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AACtE,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAC/D,YAAY,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACpD,YAAY,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAC3D,OAAO,EAAE,KAAK,YAAY,EAAE,KAAK,aAAa,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAC5E,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,OAAO,EAAE,KAAK,MAAM,EAAE,KAAK,MAAM,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AAC9E,YAAY,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAChD,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,YAAY,EAAE,UAAU,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC"}
package/dist/index.js CHANGED
@@ -1,8 +1,9 @@
1
1
  export { DecisionScorer } from "./decision-scorer.js";
2
- export { ConflictingAnswerError, MissingVectorError, NoEvidenceError, UnknownArticleError, } from "./errors.js";
2
+ export { MissingVectorError, UnknownArticleError } from "./errors.js";
3
3
  export { ExpectedFThreshold } from "./expected-f-threshold.js";
4
- export { Profile } from "./profile.js";
4
+ export { KeywordScorer } from "./keyword-scorer.js";
5
5
  export { Scores } from "./scores.js";
6
6
  export { SimilarityScorer } from "./similarity-scorer.js";
7
+ export { TaughtAnswers } from "./taught-answers.js";
7
8
  export { UtilityThreshold } from "./utility-threshold.js";
8
9
  //# sourceMappingURL=index.js.map
package/dist/index.js.map CHANGED
@@ -1 +1 @@
1
- {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EACL,sBAAsB,EACtB,kBAAkB,EAClB,eAAe,EACf,mBAAmB,GACpB,MAAM,aAAa,CAAC;AACrB,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAE/D,OAAO,EAAE,OAAO,EAAsB,MAAM,cAAc,CAAC;AAE3D,OAAO,EAAyC,MAAM,EAAE,MAAM,aAAa,CAAC;AAC5E,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAE1D,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC"}
1
+ {"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAAE,MAAM,sBAAsB,CAAC;AACtD,OAAO,EAAE,kBAAkB,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AACtE,OAAO,EAAE,kBAAkB,EAAE,MAAM,2BAA2B,CAAC;AAE/D,OAAO,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AAEpD,OAAO,EAAyC,MAAM,EAAE,MAAM,aAAa,CAAC;AAC5E,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC;AAC1D,OAAO,EAA4B,aAAa,EAAE,MAAM,qBAAqB,CAAC;AAE9E,OAAO,EAAE,gBAAgB,EAAE,MAAM,wBAAwB,CAAC"}
@@ -0,0 +1,14 @@
1
+ import type { Keyword } from './keyword.ts';
2
+ import type { Scorer, Stream } from './scorer.ts';
3
+ import { Scores } from './scores.ts';
4
+ import type { Vectors } from './vectors.ts';
5
+ export declare class KeywordScorer implements Scorer {
6
+ #private;
7
+ constructor(options: {
8
+ readonly vectors: Vectors;
9
+ readonly descriptionVector: readonly number[];
10
+ readonly keywords: readonly [Keyword, ...Keyword[]];
11
+ });
12
+ score(stream: Stream): Promise<Scores>;
13
+ }
14
+ //# sourceMappingURL=keyword-scorer.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"keyword-scorer.d.ts","sourceRoot":"","sources":["../src/keyword-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAErC,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAK5C,qBAAa,aAAc,YAAW,MAAM;;gBAK9B,OAAO,EAAE;QAEnB,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;QAG1B,QAAQ,CAAC,iBAAiB,EAAE,SAAS,MAAM,EAAE,CAAC;QAC9C,QAAQ,CAAC,QAAQ,EAAE,SAAS,CAAC,OAAO,EAAE,GAAG,OAAO,EAAE,CAAC,CAAC;KACrD;IAMK,KAAK,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;CAW7C"}
@@ -0,0 +1,21 @@
1
+ import { Scores } from "./scores.js";
2
+ import { keywordRanking, similarities } from "./similar.js";
3
+ // The cold start of the frozen design (docs/adr/0002) for one interest that
4
+ // has no yes answer yet: the articles that match a keyword come first, the
5
+ // newest first, and then the other articles by similarity.
6
+ export class KeywordScorer {
7
+ #vectors;
8
+ #descriptionVector;
9
+ #keywords;
10
+ constructor(options) {
11
+ this.#vectors = options.vectors;
12
+ this.#descriptionVector = options.descriptionVector;
13
+ this.#keywords = options.keywords;
14
+ }
15
+ async score(stream) {
16
+ if (stream.size === 0)
17
+ return new Scores(new Map());
18
+ return keywordRanking(similarities(this.#vectors, this.#descriptionVector, stream, this.#keywords));
19
+ }
20
+ }
21
+ //# sourceMappingURL=keyword-scorer.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"keyword-scorer.js","sourceRoot":"","sources":["../src/keyword-scorer.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAG5D,4EAA4E;AAC5E,2EAA2E;AAC3E,2DAA2D;AAC3D,MAAM,OAAO,aAAa;IACf,QAAQ,CAAU;IAClB,kBAAkB,CAAoB;IACtC,SAAS,CAAmC;IAErD,YAAY,OAOX;QACC,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,OAAO,CAAC;QAChC,IAAI,CAAC,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC;QACpD,IAAI,CAAC,SAAS,GAAG,OAAO,CAAC,QAAQ,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,MAAc;QACxB,IAAI,MAAM,CAAC,IAAI,KAAK,CAAC;YAAE,OAAO,IAAI,MAAM,CAAC,IAAI,GAAG,EAAE,CAAC,CAAC;QACpD,OAAO,cAAc,CACnB,YAAY,CACV,IAAI,CAAC,QAAQ,EACb,IAAI,CAAC,kBAAkB,EACvB,MAAM,EACN,IAAI,CAAC,SAAS,CACf,CACF,CAAC;IACJ,CAAC;CACF"}
package/dist/scorer.d.ts CHANGED
@@ -1,12 +1,10 @@
1
- import type { Profile } from './profile.ts';
2
1
  import type { Scores } from './scores.ts';
3
2
  export type Article = {
4
- readonly id: string;
5
3
  readonly publishedAt: Date;
6
4
  readonly text: string;
7
5
  };
8
- export type Stream = readonly [Article, ...Article[]];
6
+ export type Stream = ReadonlyMap<string, Article>;
9
7
  export interface Scorer {
10
- score(profile: Profile, stream: Stream): Promise<Scores>;
8
+ score(stream: Stream): Promise<Scores>;
11
9
  }
12
10
  //# sourceMappingURL=scorer.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"scorer.d.ts","sourceRoot":"","sources":["../src/scorer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAG1C,MAAM,MAAM,OAAO,GAAG;IACpB,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IAGpB,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAG3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;CACvB,CAAC;AAGF,MAAM,MAAM,MAAM,GAAG,SAAS,CAAC,OAAO,EAAE,GAAG,OAAO,EAAE,CAAC,CAAC;AAGtD,MAAM,WAAW,MAAM;IACrB,KAAK,CAAC,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;CAC1D"}
1
+ {"version":3,"file":"scorer.d.ts","sourceRoot":"","sources":["../src/scorer.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAG1C,MAAM,MAAM,OAAO,GAAG;IAGpB,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAG3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;CACvB,CAAC;AAGF,MAAM,MAAM,MAAM,GAAG,WAAW,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;AAIlD,MAAM,WAAW,MAAM;IACrB,KAAK,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC,CAAC;CACxC"}
package/dist/scores.d.ts CHANGED
@@ -1,20 +1,15 @@
1
- import type { Profile } from './profile.ts';
2
1
  import type { Threshold } from './threshold.ts';
3
2
  export type PublishRange = {
4
3
  readonly from: Date;
5
4
  readonly to: Date;
6
5
  };
7
6
  export type ScoredArticle = {
8
- readonly id: string;
9
7
  readonly publishedAt: Date;
10
8
  readonly probability: number;
11
9
  };
12
10
  export declare class Scores {
13
11
  #private;
14
- constructor(options: {
15
- readonly profile: Profile;
16
- readonly ranked: readonly ScoredArticle[];
17
- });
12
+ constructor(ranked: ReadonlyMap<string, ScoredArticle>);
18
13
  probability(id: string): number;
19
14
  ranked(range?: PublishRange): string[];
20
15
  accepted(threshold: Threshold): string[];
@@ -1 +1 @@
1
- {"version":3,"file":"scores.d.ts","sourceRoot":"","sources":["../src/scores.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAGhD,MAAM,MAAM,YAAY,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,IAAI,CAAC;IAAC,QAAQ,CAAC,EAAE,EAAE,IAAI,CAAA;CAAE,CAAC;AAGtE,MAAM,MAAM,aAAa,GAAG;IAC1B,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAC3B,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;CAC9B,CAAC;AAIF,qBAAa,MAAM;;gBAML,OAAO,EAAE;QACnB,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;QAC1B,QAAQ,CAAC,MAAM,EAAE,SAAS,aAAa,EAAE,CAAC;KAC3C;IAaD,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,MAAM;IAS/B,MAAM,CAAC,KAAK,CAAC,EAAE,YAAY,GAAG,MAAM,EAAE;IAYtC,QAAQ,CAAC,SAAS,EAAE,SAAS,GAAG,MAAM,EAAE;CAMzC;AAID,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAI9D"}
1
+ {"version":3,"file":"scores.d.ts","sourceRoot":"","sources":["../src/scores.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,SAAS,EAAE,MAAM,gBAAgB,CAAC;AAGhD,MAAM,MAAM,YAAY,GAAG;IAAE,QAAQ,CAAC,IAAI,EAAE,IAAI,CAAC;IAAC,QAAQ,CAAC,EAAE,EAAE,IAAI,CAAA;CAAE,CAAC;AAGtE,MAAM,MAAM,aAAa,GAAG;IAC1B,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAC3B,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;CAC9B,CAAC;AAIF,qBAAa,MAAM;;gBAIL,MAAM,EAAE,WAAW,CAAC,MAAM,EAAE,aAAa,CAAC;IAKtD,WAAW,CAAC,EAAE,EAAE,MAAM,GAAG,MAAM;IAS/B,MAAM,CAAC,KAAK,CAAC,EAAE,YAAY,GAAG,MAAM,EAAE;IAWtC,QAAQ,CAAC,SAAS,EAAE,SAAS,GAAG,MAAM,EAAE;CAMzC;AAID,wBAAgB,UAAU,CAAC,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,GAAG,MAAM,CAI9D"}
package/dist/scores.js CHANGED
@@ -1,42 +1,34 @@
1
1
  import { UnknownArticleError } from "./errors.js";
2
- // What a scorer gives for one profile and one stream: the articles, the best
3
- // first, each with its probability (docs/adr/0005).
2
+ // What a scorer gives for one stream: the articles by id, the best first,
3
+ // each with its probability (docs/adr/0005).
4
4
  export class Scores {
5
- #profile;
6
5
  #ranked;
7
- #probabilities;
8
6
  // `ranked` is the stream in the order of the scorer, the best first.
9
- constructor(options) {
10
- const { profile, ranked } = options;
11
- const probabilities = new Map(ranked.map(({ id, probability }) => [id, probability]));
12
- if (probabilities.size !== ranked.length)
13
- throw new RangeError('Each article of the stream must have its own id.');
14
- this.#profile = profile;
7
+ constructor(ranked) {
15
8
  this.#ranked = ranked;
16
- this.#probabilities = probabilities;
17
9
  }
18
- // The probability that the article is relevant to the profile.
10
+ // The probability that the article is relevant to the interest.
19
11
  probability(id) {
20
- const probability = this.#probabilities.get(id);
21
- if (probability === undefined)
12
+ const article = this.#ranked.get(id);
13
+ if (article === undefined)
22
14
  throw new UnknownArticleError(id);
23
- return probability;
15
+ return article.probability;
24
16
  }
25
17
  // The ids of the stream, the best first (docs/adr/0004). With a range, only
26
18
  // the articles published in it, in the same order: the best N of a day are
27
19
  // ranked({ from, to }).slice(0, n).
28
20
  ranked(range) {
29
- const inRange = range === undefined
30
- ? this.#ranked
31
- : this.#ranked.filter(({ publishedAt }) => publishedAt >= range.from && publishedAt <= range.to);
32
- return inRange.map(({ id }) => id);
21
+ return [...this.#ranked]
22
+ .filter(([, { publishedAt }]) => range === undefined ||
23
+ (publishedAt >= range.from && publishedAt <= range.to))
24
+ .map(([id]) => id);
33
25
  }
34
26
  // The articles that the threshold accepts, in the order of ranked().
35
27
  accepted(threshold) {
36
- const cut = threshold.cut(this, this.#profile);
37
- return this.#ranked
38
- .filter(({ probability }) => probability >= cut)
39
- .map(({ id }) => id);
28
+ const cut = threshold.cut(this);
29
+ return [...this.#ranked]
30
+ .filter(([, { probability }]) => probability >= cut)
31
+ .map(([id]) => id);
40
32
  }
41
33
  }
42
34
  // Compares by code unit, not by locale, so the order is the same on every
@@ -1 +1 @@
1
- {"version":3,"file":"scores.js","sourceRoot":"","sources":["../src/scores.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAclD,6EAA6E;AAC7E,oDAAoD;AACpD,MAAM,OAAO,MAAM;IACR,QAAQ,CAAU;IAClB,OAAO,CAA2B;IAClC,cAAc,CAA8B;IAErD,qEAAqE;IACrE,YAAY,OAGX;QACC,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,OAAO,CAAC;QACpC,MAAM,aAAa,GAAG,IAAI,GAAG,CAC3B,MAAM,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,EAAE,WAAW,CAAC,CAAC,CACvD,CAAC;QACF,IAAI,aAAa,CAAC,IAAI,KAAK,MAAM,CAAC,MAAM;YACtC,MAAM,IAAI,UAAU,CAAC,kDAAkD,CAAC,CAAC;QAC3E,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;QACxB,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC;QACtB,IAAI,CAAC,cAAc,GAAG,aAAa,CAAC;IACtC,CAAC;IAED,+DAA+D;IAC/D,WAAW,CAAC,EAAU;QACpB,MAAM,WAAW,GAAG,IAAI,CAAC,cAAc,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QAChD,IAAI,WAAW,KAAK,SAAS;YAAE,MAAM,IAAI,mBAAmB,CAAC,EAAE,CAAC,CAAC;QACjE,OAAO,WAAW,CAAC;IACrB,CAAC;IAED,4EAA4E;IAC5E,2EAA2E;IAC3E,oCAAoC;IACpC,MAAM,CAAC,KAAoB;QACzB,MAAM,OAAO,GACX,KAAK,KAAK,SAAS;YACjB,CAAC,CAAC,IAAI,CAAC,OAAO;YACd,CAAC,CAAC,IAAI,CAAC,OAAO,CAAC,MAAM,CACjB,CAAC,EAAE,WAAW,EAAE,EAAE,EAAE,CAClB,WAAW,IAAI,KAAK,CAAC,IAAI,IAAI,WAAW,IAAI,KAAK,CAAC,EAAE,CACvD,CAAC;QACR,OAAO,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACrC,CAAC;IAED,qEAAqE;IACrE,QAAQ,CAAC,SAAoB;QAC3B,MAAM,GAAG,GAAG,SAAS,CAAC,GAAG,CAAC,IAAI,EAAE,IAAI,CAAC,QAAQ,CAAC,CAAC;QAC/C,OAAO,IAAI,CAAC,OAAO;aAChB,MAAM,CAAC,CAAC,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC,WAAW,IAAI,GAAG,CAAC;aAC/C,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACzB,CAAC;CACF;AAED,0EAA0E;AAC1E,WAAW;AACX,MAAM,UAAU,UAAU,CAAC,IAAY,EAAE,KAAa;IACpD,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC,CAAC;IAC5B,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC;IAC3B,OAAO,CAAC,CAAC;AACX,CAAC"}
1
+ {"version":3,"file":"scores.js","sourceRoot":"","sources":["../src/scores.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAYlD,0EAA0E;AAC1E,6CAA6C;AAC7C,MAAM,OAAO,MAAM;IACR,OAAO,CAAqC;IAErD,qEAAqE;IACrE,YAAY,MAA0C;QACpD,IAAI,CAAC,OAAO,GAAG,MAAM,CAAC;IACxB,CAAC;IAED,gEAAgE;IAChE,WAAW,CAAC,EAAU;QACpB,MAAM,OAAO,GAAG,IAAI,CAAC,OAAO,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACrC,IAAI,OAAO,KAAK,SAAS;YAAE,MAAM,IAAI,mBAAmB,CAAC,EAAE,CAAC,CAAC;QAC7D,OAAO,OAAO,CAAC,WAAW,CAAC;IAC7B,CAAC;IAED,4EAA4E;IAC5E,2EAA2E;IAC3E,oCAAoC;IACpC,MAAM,CAAC,KAAoB;QACzB,OAAO,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC;aACrB,MAAM,CACL,CAAC,CAAC,EAAE,EAAE,WAAW,EAAE,CAAC,EAAE,EAAE,CACtB,KAAK,KAAK,SAAS;YACnB,CAAC,WAAW,IAAI,KAAK,CAAC,IAAI,IAAI,WAAW,IAAI,KAAK,CAAC,EAAE,CAAC,CACzD;aACA,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACvB,CAAC;IAED,qEAAqE;IACrE,QAAQ,CAAC,SAAoB;QAC3B,MAAM,GAAG,GAAG,SAAS,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;QAChC,OAAO,CAAC,GAAG,IAAI,CAAC,OAAO,CAAC;aACrB,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,WAAW,EAAE,CAAC,EAAE,EAAE,CAAC,WAAW,IAAI,GAAG,CAAC;aACnD,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,EAAE,CAAC,CAAC;IACvB,CAAC;CACF;AAED,0EAA0E;AAC1E,WAAW;AACX,MAAM,UAAU,UAAU,CAAC,IAAY,EAAE,KAAa;IACpD,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC,CAAC;IAC5B,IAAI,IAAI,GAAG,KAAK;QAAE,OAAO,CAAC,CAAC;IAC3B,OAAO,CAAC,CAAC;AACX,CAAC"}
@@ -0,0 +1,13 @@
1
+ import { type Keyword } from './keyword.ts';
2
+ import type { Stream } from './scorer.ts';
3
+ import { Scores } from './scores.ts';
4
+ import type { Vectors } from './vectors.ts';
5
+ export type Similar = {
6
+ readonly id: string;
7
+ readonly publishedAt: Date;
8
+ readonly similarity: number;
9
+ readonly match: boolean;
10
+ };
11
+ export declare function similarities(vectors: Vectors, descriptionVector: readonly number[], stream: Stream, keywords: readonly Keyword[]): Similar[];
12
+ export declare function keywordRanking(similar: readonly Similar[]): Scores;
13
+ //# sourceMappingURL=similar.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"similar.d.ts","sourceRoot":"","sources":["../src/similar.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,KAAK,OAAO,EAAc,MAAM,cAAc,CAAC;AACxD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAC1C,OAAO,EAAE,MAAM,EAAc,MAAM,aAAa,CAAC;AACjD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAI5C,MAAM,MAAM,OAAO,GAAG;IACpB,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,WAAW,EAAE,IAAI,CAAC;IAC3B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;IAC5B,QAAQ,CAAC,KAAK,EAAE,OAAO,CAAC;CACzB,CAAC;AAKF,wBAAgB,YAAY,CAC1B,OAAO,EAAE,OAAO,EAChB,iBAAiB,EAAE,SAAS,MAAM,EAAE,EACpC,MAAM,EAAE,MAAM,EACd,QAAQ,EAAE,SAAS,OAAO,EAAE,GAC3B,OAAO,EAAE,CAiBX;AAKD,wBAAgB,cAAc,CAAC,OAAO,EAAE,SAAS,OAAO,EAAE,GAAG,MAAM,CAiBlE"}
@@ -0,0 +1,40 @@
1
+ import { MissingVectorError } from "./errors.js";
2
+ import { matchesAny } from "./keyword.js";
3
+ import { Scores, compareIds } from "./scores.js";
4
+ // Each article of a non-empty stream, in the order of the stream, with its
5
+ // similarity to the description from one query over the publish dates of the
6
+ // stream, and its keyword match.
7
+ export function similarities(vectors, descriptionVector, stream, keywords) {
8
+ const dates = [...stream.values()].map(({ publishedAt }) => publishedAt.getTime());
9
+ const found = new Map(vectors
10
+ .query(descriptionVector, {
11
+ from: new Date(dates.reduce((a, b) => Math.min(a, b))),
12
+ to: new Date(dates.reduce((a, b) => Math.max(a, b))),
13
+ })
14
+ .map(({ id, similarity }) => [id, similarity]));
15
+ return [...stream].map(([id, { publishedAt, text }]) => {
16
+ const similarity = found.get(id);
17
+ if (similarity === undefined)
18
+ throw new MissingVectorError(id);
19
+ return { id, publishedAt, similarity, match: matchesAny(keywords, text) };
20
+ });
21
+ }
22
+ // The keyword order: the matches first, the newest first, and then the other
23
+ // articles by similarity. The probability is 1 for a match and 0 for the
24
+ // others, and it is not calibrated.
25
+ export function keywordRanking(similar) {
26
+ return new Scores(new Map(similar
27
+ .toSorted((a, b) => {
28
+ if (a.match !== b.match)
29
+ return a.match ? -1 : 1;
30
+ const key = a.match
31
+ ? b.publishedAt.getTime() - a.publishedAt.getTime()
32
+ : b.similarity - a.similarity;
33
+ return key || compareIds(a.id, b.id);
34
+ })
35
+ .map(({ id, publishedAt, match }) => [
36
+ id,
37
+ { publishedAt, probability: match ? 1 : 0 },
38
+ ])));
39
+ }
40
+ //# sourceMappingURL=similar.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"similar.js","sourceRoot":"","sources":["../src/similar.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,kBAAkB,EAAE,MAAM,aAAa,CAAC;AACjD,OAAO,EAAgB,UAAU,EAAE,MAAM,cAAc,CAAC;AAExD,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAYjD,2EAA2E;AAC3E,6EAA6E;AAC7E,iCAAiC;AACjC,MAAM,UAAU,YAAY,CAC1B,OAAgB,EAChB,iBAAoC,EACpC,MAAc,EACd,QAA4B;IAE5B,MAAM,KAAK,GAAG,CAAC,GAAG,MAAM,CAAC,MAAM,EAAE,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,WAAW,EAAE,EAAE,EAAE,CACzD,WAAW,CAAC,OAAO,EAAE,CACtB,CAAC;IACF,MAAM,KAAK,GAAG,IAAI,GAAG,CACnB,OAAO;SACJ,KAAK,CAAC,iBAAiB,EAAE;QACxB,IAAI,EAAE,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QACtD,EAAE,EAAE,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;KACrD,CAAC;SACD,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,EAAE,UAAU,CAAC,CAAC,CACjD,CAAC;IACF,OAAO,CAAC,GAAG,MAAM,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,IAAI,EAAE,CAAC,EAAE,EAAE;QACrD,MAAM,UAAU,GAAG,KAAK,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACjC,IAAI,UAAU,KAAK,SAAS;YAAE,MAAM,IAAI,kBAAkB,CAAC,EAAE,CAAC,CAAC;QAC/D,OAAO,EAAE,EAAE,EAAE,WAAW,EAAE,UAAU,EAAE,KAAK,EAAE,UAAU,CAAC,QAAQ,EAAE,IAAI,CAAC,EAAE,CAAC;IAC5E,CAAC,CAAC,CAAC;AACL,CAAC;AAED,6EAA6E;AAC7E,yEAAyE;AACzE,oCAAoC;AACpC,MAAM,UAAU,cAAc,CAAC,OAA2B;IACxD,OAAO,IAAI,MAAM,CACf,IAAI,GAAG,CACL,OAAO;SACJ,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;QACjB,IAAI,CAAC,CAAC,KAAK,KAAK,CAAC,CAAC,KAAK;YAAE,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;QACjD,MAAM,GAAG,GAAG,CAAC,CAAC,KAAK;YACjB,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC,WAAW,CAAC,OAAO,EAAE;YACnD,CAAC,CAAC,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,UAAU,CAAC;QAChC,OAAO,GAAG,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;IACvC,CAAC,CAAC;SACD,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC;QACnC,EAAE;QACF,EAAE,WAAW,EAAE,WAAW,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE;KAC5C,CAAC,CACL,CACF,CAAC;AACJ,CAAC"}
@@ -1,16 +1,16 @@
1
- import type { Profile } from './profile.ts';
1
+ import { type Keyword } from './keyword.ts';
2
2
  import type { Scorer, Stream } from './scorer.ts';
3
3
  import { Scores } from './scores.ts';
4
+ import type { TaughtAnswers } from './taught-answers.ts';
4
5
  import type { Vectors } from './vectors.ts';
5
- type SimilarityScorerOptions = {
6
- readonly vectors: Vectors;
7
- readonly embedDescription: (description: string) => Promise<readonly number[]>;
8
- readonly taughtText: (id: string) => string;
9
- };
10
6
  export declare class SimilarityScorer implements Scorer {
11
7
  #private;
12
- constructor(options: SimilarityScorerOptions);
13
- score(profile: Profile, stream: Stream): Promise<Scores>;
8
+ constructor(options: {
9
+ readonly vectors: Vectors;
10
+ readonly descriptionVector: readonly number[];
11
+ readonly answers: TaughtAnswers;
12
+ readonly keywords: readonly Keyword[];
13
+ });
14
+ score(stream: Stream): Promise<Scores>;
14
15
  }
15
- export {};
16
16
  //# sourceMappingURL=similarity-scorer.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"similarity-scorer.d.ts","sourceRoot":"","sources":["../src/similarity-scorer.ts"],"names":[],"mappings":"AAKA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,MAAM,EAAc,MAAM,aAAa,CAAC;AACjD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAE5C,KAAK,uBAAuB,GAAG;IAE7B,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;IAG1B,QAAQ,CAAC,gBAAgB,EAAE,CACzB,WAAW,EAAE,MAAM,KAChB,OAAO,CAAC,SAAS,MAAM,EAAE,CAAC,CAAC;IAGhC,QAAQ,CAAC,UAAU,EAAE,CAAC,EAAE,EAAE,MAAM,KAAK,MAAM,CAAC;CAC7C,CAAC;AAgBF,qBAAa,gBAAiB,YAAW,MAAM;;gBAGjC,OAAO,EAAE,uBAAuB;IAItC,KAAK,CAAC,OAAO,EAAE,OAAO,EAAE,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;CAmD/D"}
1
+ {"version":3,"file":"similarity-scorer.d.ts","sourceRoot":"","sources":["../src/similarity-scorer.ts"],"names":[],"mappings":"AAEA,OAAO,EAAE,KAAK,OAAO,EAAc,MAAM,cAAc,CAAC;AAExD,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAClD,OAAO,EAAE,MAAM,EAAc,MAAM,aAAa,CAAC;AAEjD,OAAO,KAAK,EAAE,aAAa,EAAE,MAAM,qBAAqB,CAAC;AACzD,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAO5C,qBAAa,gBAAiB,YAAW,MAAM;;gBAMjC,OAAO,EAAE;QAEnB,QAAQ,CAAC,OAAO,EAAE,OAAO,CAAC;QAG1B,QAAQ,CAAC,iBAAiB,EAAE,SAAS,MAAM,EAAE,CAAC;QAC9C,QAAQ,CAAC,OAAO,EAAE,aAAa,CAAC;QAChC,QAAQ,CAAC,QAAQ,EAAE,SAAS,OAAO,EAAE,CAAC;KACvC;IAOK,KAAK,CAAC,MAAM,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,CAAC;CAkE7C"}
@@ -1,109 +1,71 @@
1
1
  import { binormal } from "./binormal.js";
2
- import { MissingVectorError, NoEvidenceError } from "./errors.js";
3
2
  import { withKeywordEvidence } from "./evidence.js";
4
3
  import { matchesAny } from "./keyword.js";
5
4
  import { platt, plattProbability } from "./platt.js";
6
5
  import { Scores, compareIds } from "./scores.js";
7
- // The frozen design of research round 6 (docs/adr/0002): the similarity of
8
- // each article to the description, calibrated by a binormal model on the
9
- // stream and the answers, or by Platt's method on the answers, with the
10
- // evidence of the keywords. A profile with no relevant answer ranks the
11
- // keyword matches first.
6
+ import { keywordRanking, similarities } from "./similar.js";
7
+ // The frozen design of research round 6 (docs/adr/0002) for one interest:
8
+ // the similarity of each article to the description, calibrated by a
9
+ // binormal model on the stream and the answers, or by Platt's method on the
10
+ // answers, with the evidence of the keywords. When neither fit can hold, the
11
+ // keyword matches come first.
12
12
  export class SimilarityScorer {
13
- #options;
13
+ #vectors;
14
+ #descriptionVector;
15
+ #answers;
16
+ #keywords;
14
17
  constructor(options) {
15
- this.#options = options;
18
+ this.#vectors = options.vectors;
19
+ this.#descriptionVector = options.descriptionVector;
20
+ this.#answers = options.answers;
21
+ this.#keywords = options.keywords;
16
22
  }
17
- async score(profile, stream) {
18
- const descriptionVector = await this.#options.embedDescription(profile.description);
19
- const similar = streamScores(this.#options.vectors, descriptionVector, stream, profile);
20
- if (profile.relevant.length === 0)
21
- return keywordScores(profile, similar);
23
+ async score(stream) {
24
+ if (stream.size === 0)
25
+ return new Scores(new Map());
26
+ const similar = similarities(this.#vectors, this.#descriptionVector, stream, this.#keywords);
22
27
  const scores = similar.map(({ similarity }) => similarity);
23
- const taught = taughtScores(this.#options.vectors, descriptionVector, profile);
28
+ const taught = this.#taughtScores();
24
29
  let probabilities = binormal(scores, taught.relevant, taught.notRelevant);
25
- if (probabilities === undefined && profile.notRelevant.length > 0) {
30
+ if (probabilities === undefined && this.#answers.notRelevant.length > 0) {
26
31
  const fit = platt(taught.relevant, taught.notRelevant);
27
32
  probabilities = scores.map((score) => plattProbability(fit, score));
28
33
  }
29
34
  if (probabilities === undefined)
30
- return keywordScores(profile, similar);
31
- if (profile.keywords.length > 0)
32
- probabilities = withKeywordEvidence(probabilities, similar.map(({ match }) => match), profile.relevant.map((id) => matchesAny(profile.keywords, this.#options.taughtText(id))));
35
+ return keywordRanking(similar);
36
+ if (this.#keywords.length > 0)
37
+ probabilities = withKeywordEvidence(probabilities, similar.map(({ match }) => match), this.#answers.relevant.map(({ text }) => matchesAny(this.#keywords, text)));
33
38
  const scored = similar.map((article, index) => ({
34
39
  ...article,
35
40
  probability: probabilities[index] ?? 0,
36
41
  }));
37
- return new Scores({
38
- profile,
39
- ranked: scored
40
- .toSorted((a, b) => b.probability - a.probability ||
41
- b.similarity - a.similarity ||
42
- compareIds(a.id, b.id))
43
- .map(({ id, publishedAt, probability }) => ({
44
- id,
45
- publishedAt,
46
- probability,
47
- })),
48
- });
49
- }
50
- }
51
- // A profile with no relevant answer to learn from: the probability is 1 for
52
- // a keyword match and 0 for the others, and it is not calibrated. The
53
- // matches come first, the newest first, and then the other articles by
54
- // similarity.
55
- function keywordScores(profile, similar) {
56
- if (profile.keywords.length === 0)
57
- throw new NoEvidenceError(profile.name);
58
- return new Scores({
59
- profile,
60
- ranked: similar
61
- .toSorted((a, b) => {
62
- if (a.match !== b.match)
63
- return a.match ? -1 : 1;
64
- const key = a.match
65
- ? b.publishedAt.getTime() - a.publishedAt.getTime()
66
- : b.similarity - a.similarity;
67
- return key || compareIds(a.id, b.id);
68
- })
69
- .map(({ id, publishedAt, match }) => ({
42
+ return new Scores(new Map(scored
43
+ .toSorted((a, b) => b.probability - a.probability ||
44
+ b.similarity - a.similarity ||
45
+ compareIds(a.id, b.id))
46
+ .map(({ id, publishedAt, probability }) => [
70
47
  id,
71
- publishedAt,
72
- probability: match ? 1 : 0,
73
- })),
74
- });
75
- }
76
- // Each article of the stream, in the order of the stream, with its
77
- // similarity to the description from one query over the publish dates of
78
- // the stream, and its keyword match.
79
- function streamScores(vectors, descriptionVector, stream, { keywords }) {
80
- const dates = stream.map(({ publishedAt }) => publishedAt.getTime());
81
- const similarities = new Map(vectors
82
- .query(descriptionVector, {
83
- from: new Date(dates.reduce((a, b) => Math.min(a, b))),
84
- to: new Date(dates.reduce((a, b) => Math.max(a, b))),
85
- })
86
- .map(({ id, similarity }) => [id, similarity]));
87
- return stream.map(({ id, publishedAt, text }) => {
88
- const similarity = similarities.get(id);
89
- if (similarity === undefined)
90
- throw new MissingVectorError(id);
91
- return { id, publishedAt, similarity, match: matchesAny(keywords, text) };
92
- });
93
- }
94
- // The cosine of the description with each taught article: the dot product
95
- // of the stored unit vector and the description made a unit vector.
96
- function taughtScores(vectors, descriptionVector, { relevant, notRelevant }) {
97
- const length = Math.hypot(...descriptionVector);
98
- const stored = vectors.fetch([...relevant, ...notRelevant]);
99
- const score = (id) => {
100
- const vector = stored.get(id) ?? [];
101
- let total = 0;
102
- for (let index = 0; index < descriptionVector.length; index++)
103
- total +=
104
- (vector[index] ?? 0) * ((descriptionVector[index] ?? 0) / length);
105
- return total;
106
- };
107
- return { relevant: relevant.map(score), notRelevant: notRelevant.map(score) };
48
+ { publishedAt, probability },
49
+ ])));
50
+ }
51
+ // The cosine of the description with each taught article: the dot product
52
+ // of the stored unit vector and the description made a unit vector.
53
+ #taughtScores() {
54
+ const { relevant, notRelevant } = this.#answers;
55
+ const description = this.#descriptionVector;
56
+ const length = Math.hypot(...description);
57
+ const stored = this.#vectors.fetch([...relevant, ...notRelevant].map(({ id }) => id));
58
+ const score = ({ id }) => {
59
+ const vector = stored.get(id) ?? [];
60
+ let total = 0;
61
+ for (let index = 0; index < description.length; index++)
62
+ total += (vector[index] ?? 0) * ((description[index] ?? 0) / length);
63
+ return total;
64
+ };
65
+ return {
66
+ relevant: relevant.map(score),
67
+ notRelevant: notRelevant.map(score),
68
+ };
69
+ }
108
70
  }
109
71
  //# sourceMappingURL=similarity-scorer.js.map
@@ -1 +1 @@
1
- {"version":3,"file":"similarity-scorer.js","sourceRoot":"","sources":["../src/similarity-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,eAAe,CAAC;AACzC,OAAO,EAAE,kBAAkB,EAAE,eAAe,EAAE,MAAM,aAAa,CAAC;AAClE,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAE,UAAU,EAAE,MAAM,cAAc,CAAC;AAC1C,OAAO,EAAE,KAAK,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAGrD,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAyBjD,2EAA2E;AAC3E,yEAAyE;AACzE,wEAAwE;AACxE,wEAAwE;AACxE,yBAAyB;AACzB,MAAM,OAAO,gBAAgB;IAClB,QAAQ,CAA0B;IAE3C,YAAY,OAAgC;QAC1C,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC;IAC1B,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,OAAgB,EAAE,MAAc;QAC1C,MAAM,iBAAiB,GAAG,MAAM,IAAI,CAAC,QAAQ,CAAC,gBAAgB,CAC5D,OAAO,CAAC,WAAW,CACpB,CAAC;QACF,MAAM,OAAO,GAAG,YAAY,CAC1B,IAAI,CAAC,QAAQ,CAAC,OAAO,EACrB,iBAAiB,EACjB,MAAM,EACN,OAAO,CACR,CAAC;QACF,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,KAAK,CAAC;YAAE,OAAO,aAAa,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QAC1E,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC;QAC3D,MAAM,MAAM,GAAG,YAAY,CACzB,IAAI,CAAC,QAAQ,CAAC,OAAO,EACrB,iBAAiB,EACjB,OAAO,CACR,CAAC;QACF,IAAI,aAAa,GAAG,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,CAAC;QAC1E,IAAI,aAAa,KAAK,SAAS,IAAI,OAAO,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YAClE,MAAM,GAAG,GAAG,KAAK,CAAC,MAAM,CAAC,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,CAAC;YACvD,aAAa,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,gBAAgB,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC,CAAC;QACtE,CAAC;QACD,IAAI,aAAa,KAAK,SAAS;YAAE,OAAO,aAAa,CAAC,OAAO,EAAE,OAAO,CAAC,CAAC;QACxE,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,GAAG,CAAC;YAC7B,aAAa,GAAG,mBAAmB,CACjC,aAAa,EACb,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,KAAK,CAAC,EACjC,OAAO,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,CAC1B,UAAU,CAAC,OAAO,CAAC,QAAQ,EAAE,IAAI,CAAC,QAAQ,CAAC,UAAU,CAAC,EAAE,CAAC,CAAC,CAC3D,CACF,CAAC;QACJ,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC;YAC9C,GAAG,OAAO;YACV,WAAW,EAAE,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC;SACvC,CAAC,CAAC,CAAC;QACJ,OAAO,IAAI,MAAM,CAAC;YAChB,OAAO;YACP,MAAM,EAAE,MAAM;iBACX,QAAQ,CACP,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CACP,CAAC,CAAC,WAAW,GAAG,CAAC,CAAC,WAAW;gBAC7B,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,UAAU;gBAC3B,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CACzB;iBACA,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC,CAAC;gBAC1C,EAAE;gBACF,WAAW;gBACX,WAAW;aACZ,CAAC,CAAC;SACN,CAAC,CAAC;IACL,CAAC;CACF;AAED,4EAA4E;AAC5E,sEAAsE;AACtE,uEAAuE;AACvE,cAAc;AACd,SAAS,aAAa,CAAC,OAAgB,EAAE,OAA2B;IAClE,IAAI,OAAO,CAAC,QAAQ,CAAC,MAAM,KAAK,CAAC;QAAE,MAAM,IAAI,eAAe,CAAC,OAAO,CAAC,IAAI,CAAC,CAAC;IAC3E,OAAO,IAAI,MAAM,CAAC;QAChB,OAAO;QACP,MAAM,EAAE,OAAO;aACZ,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE;YACjB,IAAI,CAAC,CAAC,KAAK,KAAK,CAAC,CAAC,KAAK;gBAAE,OAAO,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;YACjD,MAAM,GAAG,GAAG,CAAC,CAAC,KAAK;gBACjB,CAAC,CAAC,CAAC,CAAC,WAAW,CAAC,OAAO,EAAE,GAAG,CAAC,CAAC,WAAW,CAAC,OAAO,EAAE;gBACnD,CAAC,CAAC,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,UAAU,CAAC;YAChC,OAAO,GAAG,IAAI,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CAAC;QACvC,CAAC,CAAC;aACD,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,CAAC;YACpC,EAAE;YACF,WAAW;YACX,WAAW,EAAE,KAAK,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;SAC3B,CAAC,CAAC;KACN,CAAC,CAAC;AACL,CAAC;AAED,mEAAmE;AACnE,yEAAyE;AACzE,qCAAqC;AACrC,SAAS,YAAY,CACnB,OAAgB,EAChB,iBAAoC,EACpC,MAAc,EACd,EAAE,QAAQ,EAAW;IAErB,MAAM,KAAK,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC,WAAW,CAAC,OAAO,EAAE,CAAC,CAAC;IACrE,MAAM,YAAY,GAAG,IAAI,GAAG,CAC1B,OAAO;SACJ,KAAK,CAAC,iBAAiB,EAAE;QACxB,IAAI,EAAE,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;QACtD,EAAE,EAAE,IAAI,IAAI,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,CAAC;KACrD,CAAC;SACD,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,CAAC,EAAE,EAAE,UAAU,CAAC,CAAC,CACjD,CAAC;IACF,OAAO,MAAM,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,IAAI,EAAE,EAAE,EAAE;QAC9C,MAAM,UAAU,GAAG,YAAY,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC;QACxC,IAAI,UAAU,KAAK,SAAS;YAAE,MAAM,IAAI,kBAAkB,CAAC,EAAE,CAAC,CAAC;QAC/D,OAAO,EAAE,EAAE,EAAE,WAAW,EAAE,UAAU,EAAE,KAAK,EAAE,UAAU,CAAC,QAAQ,EAAE,IAAI,CAAC,EAAE,CAAC;IAC5E,CAAC,CAAC,CAAC;AACL,CAAC;AAED,0EAA0E;AAC1E,oEAAoE;AACpE,SAAS,YAAY,CACnB,OAAgB,EAChB,iBAAoC,EACpC,EAAE,QAAQ,EAAE,WAAW,EAAW;IAElC,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,iBAAiB,CAAC,CAAC;IAChD,MAAM,MAAM,GAAG,OAAO,CAAC,KAAK,CAAC,CAAC,GAAG,QAAQ,EAAE,GAAG,WAAW,CAAC,CAAC,CAAC;IAC5D,MAAM,KAAK,GAAG,CAAC,EAAU,EAAE,EAAE;QAC3B,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,CAAC;QACpC,IAAI,KAAK,GAAG,CAAC,CAAC;QACd,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,iBAAiB,CAAC,MAAM,EAAE,KAAK,EAAE;YAC3D,KAAK;gBACH,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,iBAAiB,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC;QACtE,OAAO,KAAK,CAAC;IACf,CAAC,CAAC;IACF,OAAO,EAAE,QAAQ,EAAE,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,WAAW,EAAE,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC,EAAE,CAAC;AAChF,CAAC"}
1
+ {"version":3,"file":"similarity-scorer.js","sourceRoot":"","sources":["../src/similarity-scorer.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,QAAQ,EAAE,MAAM,eAAe,CAAC;AACzC,OAAO,EAAE,mBAAmB,EAAE,MAAM,eAAe,CAAC;AACpD,OAAO,EAAgB,UAAU,EAAE,MAAM,cAAc,CAAC;AACxD,OAAO,EAAE,KAAK,EAAE,gBAAgB,EAAE,MAAM,YAAY,CAAC;AAErD,OAAO,EAAE,MAAM,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AACjD,OAAO,EAAE,cAAc,EAAE,YAAY,EAAE,MAAM,cAAc,CAAC;AAI5D,0EAA0E;AAC1E,qEAAqE;AACrE,4EAA4E;AAC5E,6EAA6E;AAC7E,8BAA8B;AAC9B,MAAM,OAAO,gBAAgB;IAClB,QAAQ,CAAU;IAClB,kBAAkB,CAAoB;IACtC,QAAQ,CAAgB;IACxB,SAAS,CAAqB;IAEvC,YAAY,OAQX;QACC,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,OAAO,CAAC;QAChC,IAAI,CAAC,kBAAkB,GAAG,OAAO,CAAC,iBAAiB,CAAC;QACpD,IAAI,CAAC,QAAQ,GAAG,OAAO,CAAC,OAAO,CAAC;QAChC,IAAI,CAAC,SAAS,GAAG,OAAO,CAAC,QAAQ,CAAC;IACpC,CAAC;IAED,KAAK,CAAC,KAAK,CAAC,MAAc;QACxB,IAAI,MAAM,CAAC,IAAI,KAAK,CAAC;YAAE,OAAO,IAAI,MAAM,CAAC,IAAI,GAAG,EAAE,CAAC,CAAC;QACpD,MAAM,OAAO,GAAG,YAAY,CAC1B,IAAI,CAAC,QAAQ,EACb,IAAI,CAAC,kBAAkB,EACvB,MAAM,EACN,IAAI,CAAC,SAAS,CACf,CAAC;QACF,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,UAAU,EAAE,EAAE,EAAE,CAAC,UAAU,CAAC,CAAC;QAC3D,MAAM,MAAM,GAAG,IAAI,CAAC,aAAa,EAAE,CAAC;QACpC,IAAI,aAAa,GAAG,QAAQ,CAAC,MAAM,EAAE,MAAM,CAAC,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,CAAC;QAC1E,IAAI,aAAa,KAAK,SAAS,IAAI,IAAI,CAAC,QAAQ,CAAC,WAAW,CAAC,MAAM,GAAG,CAAC,EAAE,CAAC;YACxE,MAAM,GAAG,GAAG,KAAK,CAAC,MAAM,CAAC,QAAQ,EAAE,MAAM,CAAC,WAAW,CAAC,CAAC;YACvD,aAAa,GAAG,MAAM,CAAC,GAAG,CAAC,CAAC,KAAK,EAAE,EAAE,CAAC,gBAAgB,CAAC,GAAG,EAAE,KAAK,CAAC,CAAC,CAAC;QACtE,CAAC;QACD,IAAI,aAAa,KAAK,SAAS;YAAE,OAAO,cAAc,CAAC,OAAO,CAAC,CAAC;QAChE,IAAI,IAAI,CAAC,SAAS,CAAC,MAAM,GAAG,CAAC;YAC3B,aAAa,GAAG,mBAAmB,CACjC,aAAa,EACb,OAAO,CAAC,GAAG,CAAC,CAAC,EAAE,KAAK,EAAE,EAAE,EAAE,CAAC,KAAK,CAAC,EACjC,IAAI,CAAC,QAAQ,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,EAAE,IAAI,EAAE,EAAE,EAAE,CACtC,UAAU,CAAC,IAAI,CAAC,SAAS,EAAE,IAAI,CAAC,CACjC,CACF,CAAC;QACJ,MAAM,MAAM,GAAG,OAAO,CAAC,GAAG,CAAC,CAAC,OAAO,EAAE,KAAK,EAAE,EAAE,CAAC,CAAC;YAC9C,GAAG,OAAO;YACV,WAAW,EAAE,aAAa,CAAC,KAAK,CAAC,IAAI,CAAC;SACvC,CAAC,CAAC,CAAC;QACJ,OAAO,IAAI,MAAM,CACf,IAAI,GAAG,CACL,MAAM;aACH,QAAQ,CACP,CAAC,CAAC,EAAE,CAAC,EAAE,EAAE,CACP,CAAC,CAAC,WAAW,GAAG,CAAC,CAAC,WAAW;YAC7B,CAAC,CAAC,UAAU,GAAG,CAAC,CAAC,UAAU;YAC3B,UAAU,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,EAAE,CAAC,CACzB;aACA,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,WAAW,EAAE,WAAW,EAAE,EAAE,EAAE,CAAC;YACzC,EAAE;YACF,EAAE,WAAW,EAAE,WAAW,EAAE;SAC7B,CAAC,CACL,CACF,CAAC;IACJ,CAAC;IAED,0EAA0E;IAC1E,oEAAoE;IACpE,aAAa;QACX,MAAM,EAAE,QAAQ,EAAE,WAAW,EAAE,GAAG,IAAI,CAAC,QAAQ,CAAC;QAChD,MAAM,WAAW,GAAG,IAAI,CAAC,kBAAkB,CAAC;QAC5C,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,GAAG,WAAW,CAAC,CAAC;QAC1C,MAAM,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAC,KAAK,CAChC,CAAC,GAAG,QAAQ,EAAE,GAAG,WAAW,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,EAAE,EAAE,EAAE,EAAE,CAAC,EAAE,CAAC,CAClD,CAAC;QACF,MAAM,KAAK,GAAG,CAAC,EAAE,EAAE,EAA2B,EAAE,EAAE;YAChD,MAAM,MAAM,GAAG,MAAM,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,EAAE,CAAC;YACpC,IAAI,KAAK,GAAG,CAAC,CAAC;YACd,KAAK,IAAI,KAAK,GAAG,CAAC,EAAE,KAAK,GAAG,WAAW,CAAC,MAAM,EAAE,KAAK,EAAE;gBACrD,KAAK,IAAI,CAAC,MAAM,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,WAAW,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,GAAG,MAAM,CAAC,CAAC;YACvE,OAAO,KAAK,CAAC;QACf,CAAC,CAAC;QACF,OAAO;YACL,QAAQ,EAAE,QAAQ,CAAC,GAAG,CAAC,KAAK,CAAC;YAC7B,WAAW,EAAE,WAAW,CAAC,GAAG,CAAC,KAAK,CAAC;SACpC,CAAC;IACJ,CAAC;CACF"}
@@ -0,0 +1,15 @@
1
+ export type Answer = {
2
+ readonly relevant: boolean;
3
+ readonly text: string;
4
+ };
5
+ export type Taught = {
6
+ readonly id: string;
7
+ readonly text: string;
8
+ };
9
+ export declare class TaughtAnswers {
10
+ readonly relevant: readonly [Taught, ...Taught[]];
11
+ readonly notRelevant: readonly Taught[];
12
+ private constructor();
13
+ static parse(answers: ReadonlyMap<string, Answer>): TaughtAnswers | undefined;
14
+ }
15
+ //# sourceMappingURL=taught-answers.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"taught-answers.d.ts","sourceRoot":"","sources":["../src/taught-answers.ts"],"names":[],"mappings":"AACA,MAAM,MAAM,MAAM,GAAG;IAEnB,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC;IAE3B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;CACvB,CAAC;AAGF,MAAM,MAAM,MAAM,GAAG;IAAE,QAAQ,CAAC,EAAE,EAAE,MAAM,CAAC;IAAC,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAAC;AAIpE,qBAAa,aAAa;IACxB,QAAQ,CAAC,QAAQ,EAAE,SAAS,CAAC,MAAM,EAAE,GAAG,MAAM,EAAE,CAAC,CAAC;IAClD,QAAQ,CAAC,WAAW,EAAE,SAAS,MAAM,EAAE,CAAC;IAExC,OAAO;IAUP,MAAM,CAAC,KAAK,CACV,OAAO,EAAE,WAAW,CAAC,MAAM,EAAE,MAAM,CAAC,GACnC,aAAa,GAAG,SAAS;CAc7B"}
@@ -0,0 +1,25 @@
1
+ // The answers that a similarity scorer learns from: at least one yes, and one
2
+ // answer for each article (docs/adr/0005).
3
+ export class TaughtAnswers {
4
+ relevant;
5
+ notRelevant;
6
+ constructor(relevant, notRelevant) {
7
+ this.relevant = relevant;
8
+ this.notRelevant = notRelevant;
9
+ }
10
+ // The answers by article id, in the order of the map. Undefined when no
11
+ // answer is a yes: then the reader has taught nothing to learn from.
12
+ static parse(answers) {
13
+ const taught = [...answers].map(([id, { relevant, text }]) => ({
14
+ relevant,
15
+ article: { id, text },
16
+ }));
17
+ const [first, ...rest] = taught
18
+ .filter(({ relevant }) => relevant)
19
+ .map(({ article }) => article);
20
+ if (first === undefined)
21
+ return undefined;
22
+ return new TaughtAnswers([first, ...rest], taught.filter(({ relevant }) => !relevant).map(({ article }) => article));
23
+ }
24
+ }
25
+ //# sourceMappingURL=taught-answers.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"taught-answers.js","sourceRoot":"","sources":["../src/taught-answers.ts"],"names":[],"mappings":"AAWA,8EAA8E;AAC9E,2CAA2C;AAC3C,MAAM,OAAO,aAAa;IACf,QAAQ,CAAiC;IACzC,WAAW,CAAoB;IAExC,YACE,QAAwC,EACxC,WAA8B;QAE9B,IAAI,CAAC,QAAQ,GAAG,QAAQ,CAAC;QACzB,IAAI,CAAC,WAAW,GAAG,WAAW,CAAC;IACjC,CAAC;IAED,wEAAwE;IACxE,qEAAqE;IACrE,MAAM,CAAC,KAAK,CACV,OAAoC;QAEpC,MAAM,MAAM,GAAG,CAAC,GAAG,OAAO,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE,CAAC,EAAE,EAAE,CAAC,CAAC;YAC7D,QAAQ;YACR,OAAO,EAAE,EAAE,EAAE,EAAE,IAAI,EAAE;SACtB,CAAC,CAAC,CAAC;QACJ,MAAM,CAAC,KAAK,EAAE,GAAG,IAAI,CAAC,GAAG,MAAM;aAC5B,MAAM,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,QAAQ,CAAC;aAClC,GAAG,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,CAAC;QACjC,IAAI,KAAK,KAAK,SAAS;YAAE,OAAO,SAAS,CAAC;QAC1C,OAAO,IAAI,aAAa,CACtB,CAAC,KAAK,EAAE,GAAG,IAAI,CAAC,EAChB,MAAM,CAAC,MAAM,CAAC,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE,CAAC,CAAC,QAAQ,CAAC,CAAC,GAAG,CAAC,CAAC,EAAE,OAAO,EAAE,EAAE,EAAE,CAAC,OAAO,CAAC,CACzE,CAAC;IACJ,CAAC;CACF"}
@@ -1,6 +1,5 @@
1
- import type { Profile } from './profile.ts';
2
1
  import type { Scores } from './scores.ts';
3
2
  export interface Threshold {
4
- cut(scores: Scores, profile: Profile): number;
3
+ cut(scores: Scores): number;
5
4
  }
6
5
  //# sourceMappingURL=threshold.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../src/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAC5C,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAM1C,MAAM,WAAW,SAAS;IAExB,GAAG,CAAC,MAAM,EAAE,MAAM,EAAE,OAAO,EAAE,OAAO,GAAG,MAAM,CAAC;CAC/C"}
1
+ {"version":3,"file":"threshold.d.ts","sourceRoot":"","sources":["../src/threshold.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AAI1C,MAAM,WAAW,SAAS;IAExB,GAAG,CAAC,MAAM,EAAE,MAAM,GAAG,MAAM,CAAC;CAC7B"}
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@zukhruf/relevance-filtering",
3
- "version": "0.5.1",
3
+ "version": "0.6.0",
4
4
  "description": "Ranks a stream of articles for a reader's profile through a scorer port (a decision model, or embedding similarity calibrated on the reader's answers), with thresholds that turn the probabilities into yes or no.",
5
5
  "license": "MIT",
6
6
  "repository": {
@@ -44,6 +44,6 @@
44
44
  },
45
45
  "devDependencies": {
46
46
  "@stdlib/blas-ext-base-gsumpw": "^0.3.1",
47
- "@zukhruf/embeddings": "^0.5.1"
47
+ "@zukhruf/embeddings": "^0.6.0"
48
48
  }
49
49
  }
package/dist/profile.d.ts DELETED
@@ -1,18 +0,0 @@
1
- import type { Keyword } from './keyword.ts';
2
- export type ProfileFields = {
3
- readonly name: string;
4
- readonly description: string;
5
- readonly keywords?: readonly Keyword[];
6
- readonly relevant: readonly string[];
7
- readonly notRelevant: readonly string[];
8
- };
9
- export declare class Profile {
10
- readonly name: string;
11
- readonly description: string;
12
- readonly keywords: readonly Keyword[];
13
- readonly relevant: readonly string[];
14
- readonly notRelevant: readonly string[];
15
- constructor(fields: ProfileFields);
16
- with(changes: Partial<ProfileFields>): Profile;
17
- }
18
- //# sourceMappingURL=profile.d.ts.map
@@ -1 +0,0 @@
1
- {"version":3,"file":"profile.d.ts","sourceRoot":"","sources":["../src/profile.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AAE5C,MAAM,MAAM,aAAa,GAAG;IAE1B,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IAEtB,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,QAAQ,CAAC,EAAE,SAAS,OAAO,EAAE,CAAC;IAEvC,QAAQ,CAAC,QAAQ,EAAE,SAAS,MAAM,EAAE,CAAC;IACrC,QAAQ,CAAC,WAAW,EAAE,SAAS,MAAM,EAAE,CAAC;CACzC,CAAC;AAMF,qBAAa,OAAO;IAClB,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB,QAAQ,CAAC,WAAW,EAAE,MAAM,CAAC;IAC7B,QAAQ,CAAC,QAAQ,EAAE,SAAS,OAAO,EAAE,CAAC;IACtC,QAAQ,CAAC,QAAQ,EAAE,SAAS,MAAM,EAAE,CAAC;IACrC,QAAQ,CAAC,WAAW,EAAE,SAAS,MAAM,EAAE,CAAC;gBAE5B,MAAM,EAAE,aAAa;IAYjC,IAAI,CAAC,OAAO,EAAE,OAAO,CAAC,aAAa,CAAC,GAAG,OAAO;CAU/C"}
package/dist/profile.js DELETED
@@ -1,35 +0,0 @@
1
- import { ConflictingAnswerError } from "./errors.js";
2
- // A reader's interest: a name, a description, optional keywords and the
3
- // answers that taught it. It is a value: it does no I/O, and a correction
4
- // gives a new profile (docs/adr/0001). A scorer scores a stream for it
5
- // (docs/adr/0005).
6
- export class Profile {
7
- name;
8
- description;
9
- keywords;
10
- relevant;
11
- notRelevant;
12
- constructor(fields) {
13
- const relevant = new Set(fields.relevant);
14
- const conflict = fields.notRelevant.find((id) => relevant.has(id));
15
- if (conflict !== undefined)
16
- throw new ConflictingAnswerError(conflict);
17
- this.name = fields.name;
18
- this.description = fields.description;
19
- this.keywords = Object.freeze([...(fields.keywords ?? [])]);
20
- this.relevant = Object.freeze([...fields.relevant]);
21
- this.notRelevant = Object.freeze([...fields.notRelevant]);
22
- }
23
- // A copy of the profile with the given fields changed.
24
- with(changes) {
25
- return new Profile({
26
- name: this.name,
27
- description: this.description,
28
- keywords: this.keywords,
29
- relevant: this.relevant,
30
- notRelevant: this.notRelevant,
31
- ...changes,
32
- });
33
- }
34
- }
35
- //# sourceMappingURL=profile.js.map
@@ -1 +0,0 @@
1
- {"version":3,"file":"profile.js","sourceRoot":"","sources":["../src/profile.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,sBAAsB,EAAE,MAAM,aAAa,CAAC;AAcrD,wEAAwE;AACxE,0EAA0E;AAC1E,uEAAuE;AACvE,mBAAmB;AACnB,MAAM,OAAO,OAAO;IACT,IAAI,CAAS;IACb,WAAW,CAAS;IACpB,QAAQ,CAAqB;IAC7B,QAAQ,CAAoB;IAC5B,WAAW,CAAoB;IAExC,YAAY,MAAqB;QAC/B,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC;QAC1C,MAAM,QAAQ,GAAG,MAAM,CAAC,WAAW,CAAC,IAAI,CAAC,CAAC,EAAE,EAAE,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,CAAC,CAAC,CAAC;QACnE,IAAI,QAAQ,KAAK,SAAS;YAAE,MAAM,IAAI,sBAAsB,CAAC,QAAQ,CAAC,CAAC;QACvE,IAAI,CAAC,IAAI,GAAG,MAAM,CAAC,IAAI,CAAC;QACxB,IAAI,CAAC,WAAW,GAAG,MAAM,CAAC,WAAW,CAAC;QACtC,IAAI,CAAC,QAAQ,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,CAAC,MAAM,CAAC,QAAQ,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;QAC5D,IAAI,CAAC,QAAQ,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC;QACpD,IAAI,CAAC,WAAW,GAAG,MAAM,CAAC,MAAM,CAAC,CAAC,GAAG,MAAM,CAAC,WAAW,CAAC,CAAC,CAAC;IAC5D,CAAC;IAED,uDAAuD;IACvD,IAAI,CAAC,OAA+B;QAClC,OAAO,IAAI,OAAO,CAAC;YACjB,IAAI,EAAE,IAAI,CAAC,IAAI;YACf,WAAW,EAAE,IAAI,CAAC,WAAW;YAC7B,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,QAAQ,EAAE,IAAI,CAAC,QAAQ;YACvB,WAAW,EAAE,IAAI,CAAC,WAAW;YAC7B,GAAG,OAAO;SACX,CAAC,CAAC;IACL,CAAC;CACF"}