volunteer-contributions-fetch 2.1.1 → 2.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -9,9 +9,13 @@ This library fetches contributions from several different sources and saves them
9
9
  Currently the following sources are supported. Feel free to contribute a PR if you'd like to support another source!
10
10
 
11
11
  * GitHub
12
- * Issues
12
+ * Issues created
13
+ * Comments on issues
13
14
  * PRs
14
15
  * PR Reviews
16
+ * Bugzilla
17
+ * Bug reported
18
+ * Bug commented on
15
19
  * MediaWiki
16
20
  * Page edited
17
21
  * Discourse
@@ -27,18 +31,13 @@ Each of the sources can be enabled separately. See the configuration section bel
27
31
 
28
32
  The following sources are planned:
29
33
 
30
- * GitHub
31
- * Comments
32
- * Bugzilla
33
- * Bug reported
34
- * Bug commented on
35
34
  * Phabricator
36
35
  * Revision created
37
36
  * Revision updated
38
37
 
39
38
  ### Possible future sources
40
39
 
41
- The following sources would be interesting for (at least) Mozilla contributors:
40
+ The following sources would be interesting for (at least) Mozilla contributors, however they do not expose an activity API endpoint to easily gather information for a certain user.
42
41
 
43
42
  * support.mozilla.org
44
43
  * Pontoon
@@ -80,11 +79,24 @@ This section documents all the possible configuration values. For examples see t
80
79
  | Field | Data Type | Required | Default | Description |
81
80
  |-------------------|-----------|----------|---------|----------------------------------------------------|
82
81
  | `outputFile` | string | Yes | - | Path to contributions output file |
82
+ | `bugzilla` | Object | No | - | Bugzilla configuration options (see below) |
83
83
  | `communityPortal` | Object | No | - | Community Portal configuration options (see below) |
84
84
  | `discourse` | Object | No | - | Discourse configuration options (see below) |
85
85
  | `github` | Object | No | - | GitHub configuration options (see below) |
86
86
  | `mediaWiki` | Object | No | - | MediaWiki configuration options (see below) |
87
87
 
88
+ #### Bugzilla
89
+
90
+ | Field | Data Type | Required | Default | Description |
91
+ |-----------------------|-----------|----------|---------------------------|----------------------------------------------------------------------------------------------------------------------------|
92
+ | `enabled` | boolean | Yes | - | Indicates whether this data source is enabled. Alternatively you can also leave off the `mediaWiki` object entirely. |
93
+ | `baseUrl` | string | Yes | - | API base url without trailing slash (for example: `https://bugzilla.mozilla.org`) |
94
+ | `username` | string | Yes | - | Username on MediaWiki |
95
+ | `stopDate` | string | Yes | - | `new Date()` parseable string containing a date of possibly earliest contribution to track |
96
+ | `types` | Object | No | - | Object containing the different labels below |
97
+ | `types.createdType` | string | No | Created a Bug Report | Label for entries related to having report a bug/enhancement. |
98
+ | `types.commentedType` | string | No | Commented on a Bug Report | Label for entries related to having commented on a bug/enhancement. |
99
+
88
100
  #### Community Portal
89
101
 
90
102
  | Field | Data Type | Required | Default | Description |
@@ -111,6 +123,7 @@ This section documents all the possible configuration values. For examples see t
111
123
  |----------------------------|-----------|----------|---------|----------------------------------------------------------------------------------------------------------------------------|
112
124
  | `enabled` | boolean | Yes | - | Indicates whether this data source is enabled. Alternatively you can also leave off the `github` object entirely. |
113
125
  | `allowPrivate` | boolean | No | false | Indicates whether fetching from private repositories is allowed. If not set, the script will throw if you pass a `GITHUB_TOKEN` with private repo scope. In most cases you will not want to have a token with private scope as you do not want to leak any private information in your contribution overview. Please think twice before enabling this config. |
126
+ | `commentsEnabled` | boolean | No | true | Indicates whether fetching comments is enabled. |
114
127
  | `commitsEnabled` | boolean | No | true | Indicates whether fetching commits is enabled. |
115
128
  | `issuesEnabled` | boolean | No | true | Indicates whether fetching issues is enabled. |
116
129
  | `reviewsEnabled` | boolean | No | true | Indicates whether fetching reviews is enabled. |
@@ -122,11 +135,12 @@ This section documents all the possible configuration values. For examples see t
122
135
  | `types.privateCommit` | string | No | Commit in private repository | Label for entries related to private GitHub commits |
123
136
  | `types.createdPR` | string | No | Created PR | Label for entries related to opening GitHub PRs |
124
137
  | `types.createdIssue` | string | No | Created Issue Report | Label for entries related to creating GitHub issues |
138
+ | `types.commented` | string | No | Commented on an Issue | Label for entries related to commenting on an issue |
125
139
  | `types.commentedPR` | string | No | Commented on a Pull Request | Label for entries related to commenting on a GitHub PR (review) |
126
140
  | `types.approvedPR` | string | No | Approved a Pull Request | Label for entries related to approving a GitHub PR |
127
141
  | `types.changesRequestedPR` | string | No | Requested changes on a Pull Request | Label for entries related to requesting changes on a GitHub PR |
128
142
  | `types.reviewedPR` | string | No | Reviewed a Pull Request | Label for entries related to reviewing a GitHub PR (fallback) |
129
- | `delayMsPerRequest` | number | No | 1000 | Delay between each request to GitHub. You might hit rate limiting if you set this lower than default. |
143
+ | `delayMsPerRequest` | number | No | 2000 | Delay between each request to GitHub. You might hit rate limiting if you set this lower than default. |
130
144
 
131
145
  #### MediaWiki
132
146
 
@@ -143,6 +157,12 @@ This section documents all the possible configuration values. For examples see t
143
157
  ```js
144
158
  const config = {
145
159
  outputFile: `${__dirname}/_data/contributions.json`,
160
+ bugzilla: {
161
+ enabled: true,
162
+ baseUrl: 'https://bugzilla.mozilla.org',
163
+ username: 'me@michaelkohler.info',
164
+ stopDate: '2008-01-01',
165
+ },
146
166
  communityPortal: {
147
167
  enabled: true,
148
168
  baseUrl: 'https://community.mozilla.org/people',
@@ -0,0 +1,141 @@
1
+ const debug = require('debug')('contributions:bugzilla');
2
+ const axios = require('axios');
3
+
4
+ const { getPadded } = require('./utils');
5
+
6
+ const BUGZILLA_CREATED_CATEGORY = 'bugzilla-created';
7
+ const BUGZILLA_COMMENTED_CATEGORY = 'bugzilla-comments';
8
+
9
+ const TYPE_CREATED = 'Created a Bug Report';
10
+ const TYPE_COMMENTED = 'Commented on a Bug Report';
11
+
12
+ const RESULTS_PER_PAGE = 500;
13
+ const FETCH_TIMEOUT = 120000;
14
+
15
+ module.exports = {
16
+ gather,
17
+ validate,
18
+ };
19
+
20
+ function validate(config) {
21
+ debug('Validating Bugzilla config');
22
+
23
+ if (!config) {
24
+ return;
25
+ }
26
+
27
+ if (config && config.enabled && !config.baseUrl) {
28
+ throw new Error('Bugzilla: baseUrl is required');
29
+ }
30
+
31
+ if (config && config.enabled && !config.username) {
32
+ throw new Error('Bugzilla: username is required');
33
+ }
34
+
35
+ if (config && config.enabled && !config.stopDate) {
36
+ throw new Error('Bugzilla: stopDate is required');
37
+ }
38
+ }
39
+
40
+ async function gather({ bugzilla }, existingContributions = []) {
41
+ if (!bugzilla || (bugzilla && !bugzilla.enabled)) {
42
+ debug('Bugzilla source not enabled, skipping');
43
+ return [];
44
+ }
45
+
46
+ const latest = existingContributions.find((contribution) => {
47
+ return contribution.source === BUGZILLA_COMMENTED_CATEGORY
48
+ || contribution.source === BUGZILLA_CREATED_CATEGORY;
49
+ }) || { createdAt: bugzilla.stopDate };
50
+
51
+ const commentedBugs = [];
52
+ const bugsList = bugPageGenerator(bugzilla, latest.createdAt);
53
+ for await (const bugList of bugsList) {
54
+ commentedBugs.push(bugList.map((bug) => ({
55
+ id: bug.id,
56
+ summary: bug.summary,
57
+ })));
58
+ }
59
+ const flatCommentedBugs = commentedBugs.flat();
60
+
61
+ debug(`Got ${flatCommentedBugs.length} bugs this user has commented on.`);
62
+
63
+ const titleMap = new Map();
64
+ flatCommentedBugs.forEach((bug) => titleMap.set(bug.id, bug.summary));
65
+
66
+ // Now that we have all bugs we commented on, we can fetch the comments
67
+ // for each bug..
68
+ let contributions = [];
69
+ const commentsList = getComments(bugzilla, flatCommentedBugs, titleMap);
70
+ for await (const comments of commentsList) {
71
+ contributions = contributions.concat(comments);
72
+ }
73
+
74
+ return contributions;
75
+ }
76
+
77
+ async function* bugPageGenerator(bugzilla, stopDate) {
78
+ let hasMoreToFetch = true;
79
+ let offset = 0;
80
+ const date = new Date(stopDate);
81
+ const stop = `${date.getFullYear()}-${getPadded(date.getMonth() + 1)}-${getPadded(date.getDate())}`;
82
+ while (hasMoreToFetch) {
83
+ const query = `quicksearch=ALL%20commenter:${bugzilla.username}&limit=${RESULTS_PER_PAGE}&offset=${offset}&last_change_time=${stop}`;
84
+ const bugsUrl = `${bugzilla.baseUrl}/rest/bug?${query}`;
85
+ debug(`Fetching from ${bugsUrl}`);
86
+ // eslint-disable-next-line no-await-in-loop
87
+ const { data } = await axios.request({
88
+ method: 'GET',
89
+ url: bugsUrl,
90
+ timeout: FETCH_TIMEOUT,
91
+ });
92
+ debug(`Got ${data.bugs.length} bugs results`);
93
+ // If we have less results than the max result, we're at the end
94
+ if (data.bugs.length < RESULTS_PER_PAGE) {
95
+ hasMoreToFetch = false;
96
+ yield data.bugs;
97
+ return;
98
+ }
99
+
100
+ offset += RESULTS_PER_PAGE;
101
+ yield data.bugs;
102
+ }
103
+ }
104
+
105
+ async function* getComments(bugzilla, bugs, titleMap) {
106
+ for await (const bug of bugs) {
107
+ const commentsUrl = `${bugzilla.baseUrl}/rest/bug/${bug.id}/comment`;
108
+ debug(`Fetching from ${commentsUrl}`);
109
+ // eslint-disable-next-line no-await-in-loop
110
+ const { data } = await axios.request({
111
+ method: 'GET',
112
+ url: commentsUrl,
113
+ timeout: FETCH_TIMEOUT,
114
+ });
115
+
116
+ const ownComments = data.bugs[bug.id].comments
117
+ .filter((comment) => comment.creator === bugzilla.username);
118
+
119
+ yield processEntities(bugzilla, ownComments, titleMap);
120
+ }
121
+
122
+ return [];
123
+ }
124
+
125
+ function processEntities({ types }, entities, titleMap) {
126
+ return entities
127
+ .map((entity) => format(types || {}, entity, titleMap));
128
+ }
129
+
130
+ function format(types, entity, titleMap) {
131
+ const created = types.createdType || TYPE_CREATED;
132
+ const commented = types.commentedType || TYPE_COMMENTED;
133
+
134
+ return {
135
+ createdAt: new Date(entity.creation_time),
136
+ description: titleMap.get(entity.bug_id),
137
+ link: `https://bugzilla.mozilla.org/show_bug.cgi?id=${entity.bug_id}#c${entity.count}`,
138
+ type: entity.count === 0 ? created : commented,
139
+ source: entity.count === 0 ? BUGZILLA_CREATED_CATEGORY : BUGZILLA_COMMENTED_CATEGORY,
140
+ };
141
+ }
@@ -0,0 +1,170 @@
1
+ const test = require('ava');
2
+ const axios = require('axios');
3
+ const sinon = require('sinon');
4
+
5
+ const bugzilla = require('./bugzilla');
6
+
7
+ const config = {
8
+ bugzilla: {
9
+ enabled: true,
10
+ baseUrl: 'https://bugzilla.mozilla.org',
11
+ username: 'user',
12
+ stopDate: new Date('2021-11-01'),
13
+ }
14
+ };
15
+
16
+ test.beforeEach((t) => {
17
+ // eslint-disable-next-line no-param-reassign
18
+ t.context.sandbox = sinon.createSandbox();
19
+
20
+ const firstBugsPage = {
21
+ bugs: new Array(500).fill(0).map(() => ({
22
+ created_at: new Date('2020-04-13'),
23
+ summary: 'Summary',
24
+ // This would be the bug ID which would be different, but enough for testing purposes
25
+ id: 1,
26
+ })),
27
+ };
28
+
29
+ const secondBugsPage = {
30
+ bugs: [{
31
+ created_at: new Date('2020-04-13'),
32
+ summary: 'SecondSummary',
33
+ id: 1,
34
+ }]
35
+ };
36
+
37
+ const comments = {
38
+ bugs: {
39
+ // This would be the bug ID which would be different, but enough for testing purposes
40
+ 1: {
41
+ comments: [{
42
+ creation_time: new Date('2020-04-13'),
43
+ creator: 'user',
44
+ bug_id: 1,
45
+ count: 0,
46
+ }, {
47
+ creation_time: new Date('2020-04-13'),
48
+ creator: 'user',
49
+ bug_id: 1,
50
+ count: 1,
51
+ }, {
52
+ creation_time: new Date('2020-04-13'),
53
+ creator: 'WILL_BE_FILTERED_OUT',
54
+ bug_id: 1,
55
+ count: 2,
56
+ }],
57
+ },
58
+ },
59
+ };
60
+
61
+ t.context.sandbox.stub(axios, 'request');
62
+ axios.request.resolves({ data: comments });
63
+ axios.request.onCall(0)
64
+ .resolves({ data: firstBugsPage });
65
+ axios.request.onCall(1)
66
+ .resolves({ data: secondBugsPage });
67
+ });
68
+
69
+ test.afterEach.always((t) => {
70
+ t.context.sandbox.restore();
71
+ });
72
+
73
+ test.serial('should validate config - undefined', (t) => {
74
+ t.notThrows(() => bugzilla.validate(undefined));
75
+ });
76
+
77
+ test.serial('should validate config - missing completely', (t) => {
78
+ const emptyConfig = {};
79
+ t.notThrows(() => bugzilla.validate(emptyConfig));
80
+ });
81
+
82
+ test.serial('should validate config - missing baseUrl', (t) => {
83
+ const failConfig = {
84
+ enabled: true,
85
+ username: 'A',
86
+ stopDate: new Date(),
87
+ };
88
+
89
+ t.throws(() => bugzilla.validate(failConfig));
90
+ });
91
+
92
+ test.serial('should validate config - missing username', (t) => {
93
+ const failConfig = {
94
+ enabled: true,
95
+ baseUrl: 'A',
96
+ stopDate: new Date(),
97
+ };
98
+
99
+ t.throws(() => bugzilla.validate(failConfig));
100
+ });
101
+
102
+ test.serial('should validate config - missing stopDate', (t) => {
103
+ const failConfig = {
104
+ enabled: true,
105
+ baseUrl: 'A',
106
+ username: 'user',
107
+ };
108
+
109
+ t.throws(() => bugzilla.validate(failConfig));
110
+ });
111
+
112
+ test.serial('should fetch', async (t) => {
113
+ const contributions = await bugzilla.gather(config);
114
+ t.is(contributions.length, (500 + 1) * 2);
115
+ });
116
+
117
+ test.serial('should not fetch if disabled - by enable property', async (t) => {
118
+ const specificTypeConfig = {
119
+ bugzilla: {
120
+ ...config.bugzilla,
121
+ enabled: false,
122
+ },
123
+ };
124
+
125
+ const contributions = await bugzilla.gather(specificTypeConfig);
126
+
127
+ t.is(contributions.length, 0);
128
+ });
129
+
130
+ test.serial('should not fetch if disabled - by leaving off config value', async (t) => {
131
+ const contributions = await bugzilla.gather({});
132
+
133
+ t.is(contributions.length, 0);
134
+ });
135
+
136
+ test.serial('should format', async (t) => {
137
+ const contributions = await bugzilla.gather(config);
138
+
139
+ t.deepEqual(contributions[0], {
140
+ createdAt: new Date('2020-04-13'),
141
+ description: 'SecondSummary',
142
+ link: 'https://bugzilla.mozilla.org/show_bug.cgi?id=1#c0',
143
+ type: 'Created a Bug Report',
144
+ source: 'bugzilla-created',
145
+ });
146
+ t.deepEqual(contributions[1], {
147
+ createdAt: new Date('2020-04-13'),
148
+ description: 'SecondSummary',
149
+ link: 'https://bugzilla.mozilla.org/show_bug.cgi?id=1#c1',
150
+ type: 'Commented on a Bug Report',
151
+ source: 'bugzilla-comments',
152
+ });
153
+ });
154
+
155
+ test.serial('should format with custom types', async (t) => {
156
+ const specificTypeConfig = {
157
+ bugzilla: {
158
+ ...config.bugzilla,
159
+ types: {
160
+ createdType: 'CREATED!',
161
+ commentedType: 'COMMENTED!',
162
+ },
163
+ },
164
+ };
165
+
166
+ const contributions = await bugzilla.gather(specificTypeConfig);
167
+
168
+ t.is(contributions[0].type, specificTypeConfig.bugzilla.types.createdType);
169
+ t.is(contributions[1].type, specificTypeConfig.bugzilla.types.commentedType);
170
+ });
@@ -24,6 +24,10 @@ test.afterEach.always((t) => {
24
24
  t.context.sandbox.restore();
25
25
  });
26
26
 
27
+ test.serial('should validate config - undefined', (t) => {
28
+ t.notThrows(() => communityPortal.validate(undefined));
29
+ });
30
+
27
31
  test.serial('should validate config - missing completely', (t) => {
28
32
  const emptyConfig = {};
29
33
  t.notThrows(() => communityPortal.validate(emptyConfig));
package/lib/fetch.js CHANGED
@@ -1,6 +1,7 @@
1
1
  const debug = require('debug')('contributions:gather');
2
2
  const fs = require('fs/promises');
3
3
 
4
+ const bugzilla = require('./bugzilla');
4
5
  const communityPortal = require('./community-portal');
5
6
  const discourse = require('./discourse');
6
7
  const github = require('./github');
@@ -13,6 +14,7 @@ async function fetchAll(config) {
13
14
  debug('fetching...');
14
15
  const existingContributions = JSON.parse(await fs.readFile(config.outputFile));
15
16
 
17
+ const bugzillaResult = await bugzilla.gather(config, existingContributions);
16
18
  const githubResult = await github.gather(config, existingContributions);
17
19
  const wikiResult = await wiki.gather(config);
18
20
  const communityPortalResult = await communityPortal.gather(config);
@@ -34,6 +36,7 @@ async function fetchAll(config) {
34
36
  });
35
37
  debug('ALREADY_EXISTING_CONTRIBUTIONS', existingContributionsToSave.length);
36
38
 
39
+ debug('NEW_BUGZILLA_CONTRIBUTIONS', bugzillaResult);
37
40
  debug('NEW_GITHUB_CONTRIBUTIONS', githubResult);
38
41
  debug('COMMUNITY_PORTAL_CONTRIBUTIONS', communityPortalResult.length);
39
42
  debug('DISCOURSE_CONTRIBUTIONS', discourseResult.length);
@@ -45,6 +48,7 @@ async function fetchAll(config) {
45
48
  ...wikiResult, // always a new fetch (previous entries removed above)
46
49
  ...communityPortalResult, // always a new fetch (previous entries removed above)
47
50
  ...githubResult,
51
+ ...bugzillaResult,
48
52
  ]));
49
53
 
50
54
  if (uniqueContributions.length === existingContributions.length) {
@@ -8,6 +8,7 @@ module.exports = {
8
8
  searchCommits,
9
9
  searchIssues,
10
10
  listReviews,
11
+ listComments,
11
12
  throwIfTokenHasPrivateRepoScope,
12
13
  };
13
14
 
@@ -62,6 +63,16 @@ async function listReviews(options) {
62
63
  return response.data;
63
64
  }
64
65
 
66
+ async function listComments(options) {
67
+ const octokit = getOctokit();
68
+ const octokitOptions = {
69
+ ...options,
70
+ per_page: RESULTS_PER_PAGE,
71
+ };
72
+ const response = await octokit.issues.listComments(octokitOptions);
73
+ return response.data;
74
+ }
75
+
65
76
  async function throwIfTokenHasPrivateRepoScope() {
66
77
  const octokit = getOctokit();
67
78
  const response = await octokit.rest.meta.root();
package/lib/github.js CHANGED
@@ -1,12 +1,14 @@
1
1
  const debug = require('debug')('contributions:github');
2
2
  const fetcher = require('./github-fetcher');
3
+ const { getPadded } = require('./utils');
3
4
 
4
5
  const ONE_YEAR_IN_MS = 1000 * 60 * 60 * 24 * 365;
5
- const DELAY_FETCHING_MS = 1000;
6
+ const DELAY_FETCHING_MS = 2000;
6
7
  const GITHUB_COMMIT_CATEGORY = 'github-commit';
7
8
  const GITHUB_ISSUES_CREATED_CATEGORY = 'github-issues-created';
8
9
  const GITHUB_PR_CREATED_CATEGORY = 'github-pr-created';
9
10
  const GITHUB_REVIEW_CATEGORY = 'github-reviews';
11
+ const GITHUB_COMMENT_CATEGORY = 'github-comments';
10
12
 
11
13
  const TYPE_COMMIT = 'GitHub Commit';
12
14
  const TYPE_PRIVATE_COMMIT = 'Commit in private repository';
@@ -16,6 +18,7 @@ const TYPE_COMMENTED_PR = 'Commented on a Pull Request';
16
18
  const TYPE_APPROVED_PR = 'Approved a Pull Request';
17
19
  const TYPE_CHANGES_REQUESTED_PR = 'Requested changes on a Pull Request';
18
20
  const TYPE_REVIEWED_PR = 'Reviewed a Pull Request';
21
+ const TYPE_COMMENTED = 'Commented on an Issue';
19
22
 
20
23
  module.exports = {
21
24
  gather,
@@ -58,6 +61,9 @@ async function gather({ github }, existingContributions = []) {
58
61
  const latestReview = existingContributions.find((contribution) => {
59
62
  return contribution.source === GITHUB_REVIEW_CATEGORY;
60
63
  }) || { createdAt: github.stopDate };
64
+ const latestComment = existingContributions.find((contribution) => {
65
+ return contribution.source === GITHUB_COMMENT_CATEGORY;
66
+ }) || { createdAt: github.stopDate };
61
67
 
62
68
  try {
63
69
  await fetcher.throwIfTokenHasPrivateRepoScope();
@@ -74,10 +80,12 @@ async function gather({ github }, existingContributions = []) {
74
80
  const commits = await processCommits(github, latestCommit.createdAt);
75
81
  const issues = await processIssues(github, latestIssue.createdAt);
76
82
  const reviews = await processReviews(github, latestReview.createdAt);
83
+ const comments = await processComments(github, latestComment.createdAt);
77
84
  const contributions = [
78
85
  ...commits,
79
86
  ...issues,
80
87
  ...reviews,
88
+ ...comments,
81
89
  ];
82
90
 
83
91
  debug('Finished gathering contributions');
@@ -99,7 +107,7 @@ async function processIssues(github, latestCreationDate) {
99
107
  const currentPage = pageGenerator(
100
108
  github,
101
109
  periodChunk,
102
- (period, page) => getIssuePage(github, period, page)
110
+ (period, page) => getIssuePage(github, 'author', period, page)
103
111
  );
104
112
  for await (const issues of currentPage) {
105
113
  debug(`Got ${issues.length} issues`);
@@ -187,6 +195,57 @@ async function processReviews(github, latestCreationDate) {
187
195
  });
188
196
  }
189
197
 
198
+ async function processComments(github, latestCreationDate) {
199
+ if (typeof github.commentsEnabled !== 'undefined' && github.commentsEnabled === false) {
200
+ debug('Fetching comments is disabled, skipping!');
201
+ return [];
202
+ }
203
+
204
+ debug('Getting comments from Github');
205
+ const allCommentedIssues = [];
206
+
207
+ // We can only search for commented issues and then get the comments from there. As we
208
+ // can't sort by comment date, we need to fetch a certain number of issues.
209
+ // There is a possibility that an issue was created before `latestCreationDate`
210
+ // but has later been commented on. Therefore we fetch the commented issues for one year
211
+ // and then later on filter out already stored comments.
212
+ const lowerBoundDate = new Date(latestCreationDate) - ONE_YEAR_IN_MS;
213
+ const generatedPeriod = periodGenerator(github.stopDate, lowerBoundDate);
214
+ for await (const periodChunk of generatedPeriod) {
215
+ const currentPage = pageGenerator(
216
+ github,
217
+ periodChunk,
218
+ (period, page) => getIssuePage(github, 'commenter', period, page),
219
+ { skipProcessing: true }
220
+ );
221
+ for await (const commentIssues of currentPage) {
222
+ debug(`Got ${commentIssues.length} commented issues`);
223
+ allCommentedIssues.push(commentIssues);
224
+ }
225
+ }
226
+
227
+ const commentTitleMap = new Map();
228
+ allCommentedIssues.flat()
229
+ .forEach((commentIssue) => commentTitleMap.set(commentIssue.url, commentIssue.title));
230
+
231
+ // Now that we have all the commented issues, we need to fetch the actual comments
232
+ const allComments = [];
233
+
234
+ const commentsList = getComments(
235
+ allCommentedIssues.flat(),
236
+ github.username,
237
+ github.delayMsPerRequest
238
+ );
239
+ for await (const comments of commentsList) {
240
+ allComments.push(comments);
241
+ }
242
+
243
+ const processedComments = processEntities(github, allComments.flat(), commentTitleMap);
244
+ return processedComments.filter((comment) => {
245
+ return new Date(comment.createdAt) > new Date(latestCreationDate);
246
+ });
247
+ }
248
+
190
249
  function getNextPeriod(stopDate, period, lowerBoundDate) {
191
250
  const [before, after] = period;
192
251
 
@@ -213,14 +272,6 @@ function getNextPeriod(stopDate, period, lowerBoundDate) {
213
272
  return [newBeforeDate, newAfterDate];
214
273
  }
215
274
 
216
- function getPadded(number) {
217
- if (number < 10) {
218
- return `0${number}`;
219
- }
220
-
221
- return `${number}`;
222
- }
223
-
224
275
  async function* periodGenerator(stopDate, lowerBoundDate) {
225
276
  const date = new Date();
226
277
  const firstOfThisMonth = `${date.getFullYear()}-${getPadded(date.getMonth() + 1)}-01`;
@@ -278,19 +329,24 @@ async function getCommitPage({ username, delayMsPerRequest }, commitsPeriod, pag
278
329
  }
279
330
  }
280
331
 
281
- async function getIssuePage({ username, delayMsPerRequest }, issuesPeriod, page) {
332
+ async function getIssuePage(github, relationship, issuesPeriod, page) {
282
333
  const [before, after] = issuesPeriod;
283
334
 
284
335
  debug(`Getting issues page ${page} between ${after} and ${before}`);
285
- await waitFor(delayMsPerRequest ?? DELAY_FETCHING_MS);
336
+ await waitFor(github.delayMsPerRequest ?? DELAY_FETCHING_MS);
286
337
  try {
338
+ const additionalFields = relationship === 'commenter' ? '+is:issue' : '';
287
339
  const items = await fetcher.searchIssues({
288
- q: `author:${username}+created:${after}..${before}`,
340
+ q: `${relationship}:${github.username}+created:${after}..${before}${additionalFields}`,
289
341
  sort: 'created',
290
342
  page,
291
343
  });
292
344
 
293
- return items;
345
+ // We already want to filter here, as otherwise we will fetch many comments
346
+ // that will later be discarded anyway.
347
+ const filteredItems = items.filter((item) => filter(github.filter, item));
348
+
349
+ return filteredItems;
294
350
  } catch (error) {
295
351
  debug('Error fetching issues', error);
296
352
  return [];
@@ -347,6 +403,32 @@ async function* getPRReviews(prList, delay) {
347
403
  return [];
348
404
  }
349
405
 
406
+ async function* getComments(issueList, commenterUsername, delay) {
407
+ debug('Getting comments');
408
+ for await (const issue of issueList) {
409
+ try {
410
+ const [, owner, repo] = /github\.com\/repos\/(.+)\/(.*)/.exec(issue.repository_url);
411
+ // We're waiting for a bit here, as we've previously hit a secondary GitHub rate limit...
412
+ await waitFor(delay ?? DELAY_FETCHING_MS);
413
+ debug(`Fetching comments for ${owner}/${repo} issue ${issue.number}`);
414
+ const items = await fetcher.listComments({
415
+ owner,
416
+ repo,
417
+ issue_number: issue.number,
418
+ });
419
+
420
+ const onlyMyComments = items.filter((comment) => comment.user.login === commenterUsername);
421
+
422
+ yield onlyMyComments;
423
+ } catch (error) {
424
+ debug('Error fetching comments', error);
425
+ return [];
426
+ }
427
+ }
428
+
429
+ return [];
430
+ }
431
+
350
432
  async function waitFor(time) {
351
433
  return new Promise((resolve) => {
352
434
  setTimeout(resolve, time);
@@ -375,6 +457,10 @@ function format(types, entity, titleMap) {
375
457
  return formatPRReview(types, entity, titleMap);
376
458
  }
377
459
 
460
+ if (entity.issue_url) {
461
+ return formatComment(types, entity, titleMap);
462
+ }
463
+
378
464
  return formatIssue(types, entity);
379
465
  }
380
466
 
@@ -411,6 +497,21 @@ function formatIssue(types, entity) {
411
497
  };
412
498
  }
413
499
 
500
+ function formatComment(types, entity, titleMap) {
501
+ let description = '';
502
+ if (titleMap) {
503
+ description = titleMap.get(entity.issue_url);
504
+ }
505
+
506
+ return {
507
+ createdAt: new Date(entity.created_at),
508
+ description,
509
+ link: entity.html_url,
510
+ type: types?.commented || TYPE_COMMENTED,
511
+ source: GITHUB_COMMENT_CATEGORY,
512
+ };
513
+ }
514
+
414
515
  function formatPRReview(types, entity, titleMap) {
415
516
  const type = (() => {
416
517
  switch (entity.state) {
@@ -26,6 +26,7 @@ test.beforeEach((t) => {
26
26
  t.context.sandbox.stub(fetcher, 'searchCommits');
27
27
  t.context.sandbox.stub(fetcher, 'searchIssues');
28
28
  t.context.sandbox.stub(fetcher, 'listReviews');
29
+ t.context.sandbox.stub(fetcher, 'listComments');
29
30
  t.context.sandbox.stub(fetcher, 'throwIfTokenHasPrivateRepoScope');
30
31
  });
31
32
 
@@ -820,3 +821,248 @@ test.serial('should ignore reviews not matching filter', async (t) => {
820
821
  const result = await github.gather(commitConfig);
821
822
  t.is(result.length, 0);
822
823
  });
824
+
825
+ test.serial('should fetch comments until stop date', async (t) => {
826
+ const commitConfig = {
827
+ github: {
828
+ ...config.github,
829
+ commitsEnabled: false,
830
+ issuesEnabled: false,
831
+ reviewsEnabled: false,
832
+ },
833
+ };
834
+
835
+ // We want to fetch 2 pages for the first period, and one page for the subsequent periods.
836
+ // We expect 3 periods to be queried based on the passed stopDate and the faked clock.
837
+ fetcher.searchIssues.resolves([]);
838
+ fetcher.searchIssues.onCall(0).resolves(new Array(100).fill(0).map(() => ({
839
+ createdAt: new Date('2022-03-01'),
840
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
841
+ number: 1111, // This would be dynamic, but fine enough for this test
842
+ title: 'IssueTitle',
843
+ })));
844
+ fetcher.searchIssues.onCall(1).resolves(new Array(5).fill(0).map(() => ({
845
+ createdAt: new Date('2022-02-01'),
846
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
847
+ number: 1111,
848
+ title: 'IssueTitle',
849
+ })));
850
+ fetcher.searchIssues.onCall(2).resolves(new Array(5).fill(0).map(() => ({
851
+ createdAt: new Date('2022-01-01'),
852
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
853
+ number: 1111,
854
+ title: 'IssueTitle',
855
+ })));
856
+
857
+ fetcher.listComments.resolves([]);
858
+ fetcher.listComments.onCall(0).resolves([{
859
+ createdAt: new Date('2022-03-01'),
860
+ user: {
861
+ login: config.github.username,
862
+ },
863
+ }]);
864
+
865
+ await github.gather(commitConfig);
866
+ t.deepEqual(fetcher.searchIssues.getCall(0).args[0], {
867
+ q: `commenter:${config.github.username}+created:2022-03-01..2022-03-31+is:issue`,
868
+ sort: 'created',
869
+ page: 1,
870
+ });
871
+ t.deepEqual(fetcher.searchIssues.getCall(1).args[0], {
872
+ q: `commenter:${config.github.username}+created:2022-03-01..2022-03-31+is:issue`,
873
+ sort: 'created',
874
+ page: 2,
875
+ });
876
+ t.deepEqual(fetcher.searchIssues.getCall(2).args[0], {
877
+ q: `commenter:${config.github.username}+created:2022-02-01..2022-02-28+is:issue`,
878
+ sort: 'created',
879
+ page: 1,
880
+ });
881
+ t.deepEqual(fetcher.searchIssues.getCall(3).args[0], {
882
+ q: `commenter:${config.github.username}+created:2022-01-01..2022-01-31+is:issue`,
883
+ sort: 'created',
884
+ page: 1,
885
+ });
886
+ t.is(fetcher.listComments.callCount, 110);
887
+ t.deepEqual(fetcher.listComments.getCall(0).args[0], {
888
+ owner: 'contribution-test',
889
+ repo: 'foo',
890
+ issue_number: 1111,
891
+ });
892
+ });
893
+
894
+ test.serial('should not fetch comments if disabled', async (t) => {
895
+ const commitConfig = {
896
+ github: {
897
+ ...config.github,
898
+ commitsEnabled: false,
899
+ issuesEnabled: false,
900
+ reviewsEnabled: false,
901
+ commentsEnabled: false,
902
+ filter: 'contribution-test',
903
+ stopDate: new Date('2022-03-01'),
904
+ },
905
+ };
906
+
907
+ await github.gather(commitConfig);
908
+ t.is(fetcher.searchIssues.callCount, 0);
909
+ });
910
+
911
+ test.serial('should format comments', async (t) => {
912
+ const commitConfig = {
913
+ github: {
914
+ ...config.github,
915
+ issuesEnabled: false,
916
+ commitsEnabled: false,
917
+ reviewsEnabled: false,
918
+ filter: 'contribution-test',
919
+ stopDate: new Date('2022-03-01'),
920
+ },
921
+ };
922
+
923
+ fetcher.searchIssues.onCall(0).resolves([{
924
+ createdAt: new Date('2022-03-02'),
925
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
926
+ url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
927
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
928
+ title: 'Some Issue',
929
+ number: 1111,
930
+ }]);
931
+
932
+ fetcher.listComments.onCall(0).resolves([{
933
+ created_at: new Date('2022-03-02'),
934
+ issue_url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
935
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
936
+ user: {
937
+ login: config.github.username,
938
+ },
939
+ }]);
940
+
941
+ const result = await github.gather(commitConfig);
942
+ t.is(result.length, 1);
943
+ t.deepEqual(result[0], {
944
+ createdAt: new Date('2022-03-02'),
945
+ description: 'Some Issue',
946
+ link: 'https://github.com/contribution-test/foo/issues/1111',
947
+ source: 'github-comments',
948
+ type: 'Commented on an Issue',
949
+ });
950
+ });
951
+
952
+ test.serial('should custom format comments', async (t) => {
953
+ const commitConfig = {
954
+ github: {
955
+ ...config.github,
956
+ issuesEnabled: false,
957
+ commitsEnabled: false,
958
+ reviewsEnabled: false,
959
+ filter: 'contribution-test',
960
+ stopDate: new Date('2022-03-01'),
961
+ types: {
962
+ commented: 'COMMENTED!',
963
+ },
964
+ },
965
+ };
966
+
967
+ fetcher.searchIssues.onCall(0).resolves([{
968
+ createdAt: new Date('2022-03-02'),
969
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
970
+ url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
971
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
972
+ title: 'Some Issue',
973
+ number: 1111,
974
+ }]);
975
+
976
+ fetcher.listComments.onCall(0).resolves([{
977
+ created_at: new Date('2022-03-02'),
978
+ issue_url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
979
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
980
+ user: {
981
+ login: config.github.username,
982
+ },
983
+ }]);
984
+
985
+ const result = await github.gather(commitConfig);
986
+ t.is(result.length, 1);
987
+ t.deepEqual(result[0].type, 'COMMENTED!');
988
+ });
989
+
990
+ test.serial('should ignore comments from before stopDate', async (t) => {
991
+ const commitConfig = {
992
+ github: {
993
+ ...config.github,
994
+ issuesEnabled: false,
995
+ commitsEnabled: false,
996
+ reviewsEnabled: false,
997
+ filter: 'contribution-test',
998
+ stopDate: new Date('2022-03-01'),
999
+ },
1000
+ };
1001
+
1002
+ fetcher.searchIssues.onCall(0).resolves([{
1003
+ createdAt: new Date('2022-03-02'),
1004
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
1005
+ url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
1006
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
1007
+ title: 'Some Issue',
1008
+ number: 1111,
1009
+ }]);
1010
+
1011
+ fetcher.listComments.onCall(0).resolves([{
1012
+ created_at: new Date('1999-01-01'),
1013
+ issue_url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
1014
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
1015
+ user: {
1016
+ login: config.github.username,
1017
+ },
1018
+ }]);
1019
+
1020
+ const result = await github.gather(commitConfig);
1021
+ t.is(result.length, 0);
1022
+ });
1023
+
1024
+ test.serial('should ignore comments not matching filter', async (t) => {
1025
+ const commitConfig = {
1026
+ github: {
1027
+ ...config.github,
1028
+ issuesEnabled: false,
1029
+ commitsEnabled: false,
1030
+ reviewsEnabled: false,
1031
+ filter: 'contribution-test',
1032
+ stopDate: new Date('2022-03-01'),
1033
+ },
1034
+ };
1035
+
1036
+ fetcher.searchIssues.onCall(0).resolves([{
1037
+ createdAt: new Date('2022-03-02'),
1038
+ repository_url: 'https://api.github.com/repos/not-matching-filter/foo',
1039
+ url: 'https://api.github.com/repos/not-matching-filter/foo/issues/1111',
1040
+ html_url: 'https://github.com/not-matching-filter/foo/issues/1111',
1041
+ title: 'Some Issue',
1042
+ number: 1111,
1043
+ }, {
1044
+ createdAt: new Date('2022-03-02'),
1045
+ repository_url: 'https://api.github.com/repos/contribution-test/foo',
1046
+ url: 'https://api.github.com/repos/contribution-test/foo/issues/1111',
1047
+ html_url: 'https://github.com/contribution-test/foo/issues/1111',
1048
+ title: 'Some Issue',
1049
+ number: 1111,
1050
+ }]);
1051
+
1052
+ // While this is the first call, it will be the call for the second issue
1053
+ // as we filter out the first issue from above due to not matching,.
1054
+ fetcher.listComments.onCall(0).resolves([{
1055
+ created_at: new Date('2022-03-02'),
1056
+ issue_url: 'https://api.github.com/repos/not-matching-filter/foo/issues/1111',
1057
+ html_url: 'https://github.com/not-matching-filter/foo/issues/1111',
1058
+ user: {
1059
+ login: config.github.username,
1060
+ },
1061
+ }]);
1062
+
1063
+ const result = await github.gather(commitConfig);
1064
+ t.is(result.length, 0);
1065
+ // Details should only have been fetched for one issue as the other one is
1066
+ // not matching the filter..
1067
+ t.is(fetcher.listComments.callCount, 1);
1068
+ });
package/lib/utils.js ADDED
@@ -0,0 +1,11 @@
1
+ module.exports = {
2
+ getPadded,
3
+ };
4
+
5
+ function getPadded(number) {
6
+ if (number < 10) {
7
+ return `0${number}`;
8
+ }
9
+
10
+ return `${number}`;
11
+ }
@@ -0,0 +1,18 @@
1
+ const test = require('ava');
2
+
3
+ const utils = require('./utils');
4
+
5
+ test.serial('should pad number below 10', (t) => {
6
+ const padded = utils.getPadded(8);
7
+ t.is(padded, '08');
8
+ });
9
+
10
+ test.serial('should not pad 10', (t) => {
11
+ const padded = utils.getPadded(10);
12
+ t.is(padded, '10');
13
+ });
14
+
15
+ test.serial('should not pad 11', (t) => {
16
+ const padded = utils.getPadded(11);
17
+ t.is(padded, '11');
18
+ });
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "volunteer-contributions-fetch",
3
- "version": "2.1.1",
3
+ "version": "2.3.0",
4
4
  "description": "Library to fetch volunteer contributions from several sources such as GitHub",
5
5
  "author": "Michael Kohler",
6
6
  "license": "MIT",