open-research 1.1.1 → 1.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/builtin-skills/draft-paper/references/formats/imrad.md +150 -0
- package/builtin-skills/draft-paper/references/formats/position-paper.md +170 -0
- package/builtin-skills/draft-paper/references/formats/systematic-review.md +176 -0
- package/builtin-skills/draft-paper/references/venues/acl.md +146 -0
- package/builtin-skills/draft-paper/references/venues/chi.md +131 -0
- package/builtin-skills/draft-paper/references/venues/ieee-transactions.md +173 -0
- package/builtin-skills/draft-paper/references/venues/nature.md +167 -0
- package/builtin-skills/draft-paper/references/venues/neurips.md +115 -0
- package/dist/chunk-32Q63SUH.js +6440 -0
- package/dist/chunk-CRSHN3PQ.js +98 -0
- package/dist/chunk-EW27OWCA.js +272 -0
- package/dist/{chunk-HRVDYJEC.js → chunk-KOBMIIQM.js} +6 -1
- package/dist/chunk-W3MWVV2O.js +533 -0
- package/dist/{chunk-TJA4CAZE.js → chunk-XOSPMXYH.js} +43 -101
- package/dist/cli.js +351 -6499
- package/dist/context-manager-2FA5ANQN.js +28 -0
- package/dist/finish-subagent-NCDLMSBT.js +22 -0
- package/dist/server/serve.js +20 -0
- package/dist/server-PLHMTHCG.js +16 -0
- package/dist/{sessions-KL4LUGD7.js → sessions-SDED5HVO.js} +1 -1
- package/dist/traverse-citations-CPKPE33Y.js +90 -0
- package/dist/{web-search-IBZ6UAXL.js → web-search-OBNKKXV7.js} +87 -14
- package/package.json +3 -1
- package/dist/{manager-queue-FBAUCAGI.js → manager-queue-NK5B47A4.js} +4 -4
- package/dist/{query-agent-WM6UNZ37.js → query-agent-HINWAVC5.js} +3 -3
|
@@ -0,0 +1,150 @@
|
|
|
1
|
+
# IMRAD Paper Structure Guide
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
**IMRAD** = Introduction, Methods, Results, and Discussion
|
|
6
|
+
|
|
7
|
+
The standard structure for original empirical research papers across sciences, medicine, engineering, and social sciences. Adopted universally by the 1980s. Reflects the scientific method and how researchers consume literature (by browsing specific sections).
|
|
8
|
+
|
|
9
|
+
## When to Use IMRAD
|
|
10
|
+
|
|
11
|
+
✅ Original empirical research (experimental, observational, clinical)
|
|
12
|
+
✅ Quantitative studies across all STEM fields
|
|
13
|
+
✅ Any study reporting collected data with defined methodology
|
|
14
|
+
|
|
15
|
+
❌ Literature reviews or systematic reviews → use review-specific structure
|
|
16
|
+
❌ Theoretical/conceptual papers → use argument-based structure
|
|
17
|
+
❌ Editorials, opinion pieces → use intro-body-conclusion
|
|
18
|
+
❌ Case reports → use patient/case description format
|
|
19
|
+
|
|
20
|
+
## Section-by-Section Guide
|
|
21
|
+
|
|
22
|
+
### INTRODUCTION (~15% of total length, 3-4 paragraphs)
|
|
23
|
+
|
|
24
|
+
**Purpose**: Establish context, demonstrate knowledge, identify the gap, state the research question.
|
|
25
|
+
|
|
26
|
+
**What to include:**
|
|
27
|
+
- What is known about the topic (background)
|
|
28
|
+
- What remains unknown or contested (the gap)
|
|
29
|
+
- Why closing this gap matters (significance)
|
|
30
|
+
- The specific research question, objective, or hypothesis
|
|
31
|
+
- Brief note on approach (without full methodology)
|
|
32
|
+
- Roadmap sentence (optional in shorter papers)
|
|
33
|
+
|
|
34
|
+
**Tense**: Present tense for established facts; past tense for prior studies.
|
|
35
|
+
|
|
36
|
+
**Common mistakes:**
|
|
37
|
+
- Opening too broadly ("Since the beginning of time...")
|
|
38
|
+
- Not identifying a specific gap or research question
|
|
39
|
+
- Burying the hypothesis at the very end without signposting
|
|
40
|
+
- Providing methodology details that belong in Methods
|
|
41
|
+
|
|
42
|
+
### METHODS (~30% of total length, 6-9 paragraphs)
|
|
43
|
+
|
|
44
|
+
**Purpose**: Enable replication. Any competent researcher should be able to reproduce your study from this section alone.
|
|
45
|
+
|
|
46
|
+
**What to include:**
|
|
47
|
+
- Study design and rationale for design choice
|
|
48
|
+
- Population or sample (inclusion/exclusion criteria, sampling, sample size)
|
|
49
|
+
- Materials, instruments, equipment (with specifications)
|
|
50
|
+
- Procedures performed in sequential order
|
|
51
|
+
- Timeline of data collection
|
|
52
|
+
- Software used for analysis
|
|
53
|
+
- Statistical methods and significance thresholds
|
|
54
|
+
- Ethical approvals and informed consent (where required)
|
|
55
|
+
- Justification for non-obvious methodological choices ("We use X because Y")
|
|
56
|
+
|
|
57
|
+
**Tense**: Past tense ("samples were collected" or "we collected samples").
|
|
58
|
+
|
|
59
|
+
**Common mistakes:**
|
|
60
|
+
- Too vague to enable replication
|
|
61
|
+
- Failing to justify non-standard design choices
|
|
62
|
+
- Omitting statistical methods or significance threshold
|
|
63
|
+
- Including results or observations here
|
|
64
|
+
- Missing ethical approvals in human/animal research
|
|
65
|
+
|
|
66
|
+
### RESULTS (~26% of total length, 4-9 paragraphs)
|
|
67
|
+
|
|
68
|
+
**Purpose**: Present findings objectively, without interpretation.
|
|
69
|
+
|
|
70
|
+
**What to include:**
|
|
71
|
+
- Participant flow or sample characteristics (often first)
|
|
72
|
+
- Primary outcomes/findings — most important first
|
|
73
|
+
- Secondary outcomes
|
|
74
|
+
- Tables and figures with descriptive prose guiding the reader
|
|
75
|
+
- Statistical results (effect sizes, confidence intervals, p-values)
|
|
76
|
+
- Unexpected findings reported neutrally
|
|
77
|
+
|
|
78
|
+
**Tense**: Past tense.
|
|
79
|
+
|
|
80
|
+
**What does NOT belong here:**
|
|
81
|
+
- Interpretation or explanation of why results occurred
|
|
82
|
+
- Comparison with other studies
|
|
83
|
+
- Speculation
|
|
84
|
+
|
|
85
|
+
**Common mistakes:**
|
|
86
|
+
- Including interpretation (belongs in Discussion)
|
|
87
|
+
- Presenting secondary results before primary results
|
|
88
|
+
- Omitting negative or null findings
|
|
89
|
+
- Over-relying on tables without contextualizing prose
|
|
90
|
+
- Inconsistency between values in text vs. tables
|
|
91
|
+
|
|
92
|
+
### DISCUSSION (~29% of total length, 8-10 paragraphs)
|
|
93
|
+
|
|
94
|
+
**Purpose**: Interpret findings in context, assess implications, acknowledge limitations.
|
|
95
|
+
|
|
96
|
+
**Recommended sequence:**
|
|
97
|
+
1. Restate research question and give direct answer based on findings
|
|
98
|
+
2. Interpret meaning of primary findings (NOT re-state results)
|
|
99
|
+
3. Compare with related prior studies (agree, disagree, extend)
|
|
100
|
+
4. Explain unexpected findings
|
|
101
|
+
5. Address study limitations honestly
|
|
102
|
+
6. State broader significance or implications
|
|
103
|
+
7. Suggest specific future research directions
|
|
104
|
+
8. Concluding statement (if no separate Conclusion section)
|
|
105
|
+
|
|
106
|
+
**Tense**: Present tense for interpretations; past tense for specific prior studies.
|
|
107
|
+
|
|
108
|
+
**Common mistakes (most frequently flagged by reviewers):**
|
|
109
|
+
- Restating results verbatim instead of interpreting them (the #1 mistake)
|
|
110
|
+
- Introducing new data not reported in Results
|
|
111
|
+
- Over-interpreting beyond what evidence supports
|
|
112
|
+
- Under-acknowledging limitations
|
|
113
|
+
- Not directly answering the research question from Introduction
|
|
114
|
+
|
|
115
|
+
## Additional Components
|
|
116
|
+
|
|
117
|
+
### ABSTRACT
|
|
118
|
+
- Structured (Background, Objective, Methods, Results, Conclusions) or unstructured per journal
|
|
119
|
+
- Written LAST, 150-250 words typical
|
|
120
|
+
- Should mirror paper structure
|
|
121
|
+
|
|
122
|
+
### CONCLUSION
|
|
123
|
+
- Not part of the IMRAD acronym but present in most papers
|
|
124
|
+
- Summarizes principal findings, restates significance
|
|
125
|
+
- Notes limitations briefly, suggests future work
|
|
126
|
+
- Does NOT introduce new information
|
|
127
|
+
|
|
128
|
+
### TITLE
|
|
129
|
+
- Should convey the finding, not just the topic
|
|
130
|
+
- Informative, specific, searchable
|
|
131
|
+
|
|
132
|
+
## Common Structural Variations
|
|
133
|
+
|
|
134
|
+
| Variation | When Used |
|
|
135
|
+
|-----------|----------|
|
|
136
|
+
| IMRAD (standard) | Most empirical science papers |
|
|
137
|
+
| Combined Results + Discussion | Shorter papers, qualitative studies |
|
|
138
|
+
| + Literature Review section | Dissertations, education/social science |
|
|
139
|
+
| Methods Last | Some biology journals (Nature, Cell) — same logic, different placement |
|
|
140
|
+
|
|
141
|
+
## Section Length Proportions
|
|
142
|
+
|
|
143
|
+
| Section | % of Total | Paragraphs |
|
|
144
|
+
|---------|-----------|------------|
|
|
145
|
+
| Introduction | ~15% | 1-4 |
|
|
146
|
+
| Methods | ~30% | 6-9 |
|
|
147
|
+
| Results | ~26% | 4-9 |
|
|
148
|
+
| Discussion | ~29% | 8-10 |
|
|
149
|
+
|
|
150
|
+
Note: In professional articles, Methods is often the longest section. In student theses, Introduction dominates (~50%), which inverts professional norms.
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
# Position / Opinion Paper Structure
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
A position paper presents and defends a specific stance on a debatable issue. It does NOT report original empirical data. Instead, it synthesizes existing evidence, applies expert reasoning, and argues for a particular conclusion, recommendation, or interpretation.
|
|
6
|
+
|
|
7
|
+
**Essential requirement**: There must be a legitimate opposing position. If there is no defensible other side, the paper is not a position paper — it is a factual report.
|
|
8
|
+
|
|
9
|
+
## When to Use This Structure
|
|
10
|
+
|
|
11
|
+
✅ A genuine debate exists and a position needs defending
|
|
12
|
+
✅ Policy or practice implications require expert argument
|
|
13
|
+
✅ A field needs direction-setting synthesis from senior researchers
|
|
14
|
+
✅ A new theoretical framework needs advocacy before empirical testing
|
|
15
|
+
✅ Response to another published paper is warranted
|
|
16
|
+
|
|
17
|
+
❌ When empirical data is available and should be collected → do the study
|
|
18
|
+
❌ When there is no legitimately defensible other side → not a debate
|
|
19
|
+
❌ When a systematic review would better synthesize the evidence
|
|
20
|
+
❌ In journals that exclusively publish empirical primary research
|
|
21
|
+
|
|
22
|
+
## How Position Papers Differ from Empirical Papers
|
|
23
|
+
|
|
24
|
+
| Dimension | Empirical (IMRAD) | Position Paper |
|
|
25
|
+
|-----------|-------------------|----------------|
|
|
26
|
+
| Primary goal | Report new data | Argue a position using existing evidence |
|
|
27
|
+
| Structure | IMRAD (rigid) | Intro-argument-counterargument-conclusion (flexible) |
|
|
28
|
+
| Evidence | Original collected data | Synthesized published literature |
|
|
29
|
+
| New data | Required | Not included |
|
|
30
|
+
| Voice | Typically passive | Active, first person acceptable |
|
|
31
|
+
| Length | 3,000-8,000 words | 1,500-4,000 words |
|
|
32
|
+
|
|
33
|
+
## Section Structure
|
|
34
|
+
|
|
35
|
+
### 1. TITLE
|
|
36
|
+
Should signal opinion/perspective nature. Some journals require article type labeling.
|
|
37
|
+
|
|
38
|
+
### 2. ABSTRACT (~150 words, if required)
|
|
39
|
+
States the issue, the position taken, primary supporting arguments, and conclusion. Not all venues require abstracts for opinion pieces.
|
|
40
|
+
|
|
41
|
+
### 3. INTRODUCTION
|
|
42
|
+
- Engage reader immediately with a compelling hook or problem statement
|
|
43
|
+
- Provide necessary background context
|
|
44
|
+
- Define key terms and scope of argument
|
|
45
|
+
- **State the thesis clearly** — the exact position the paper defends — near the end of the introduction
|
|
46
|
+
- Preview argument structure (in longer papers)
|
|
47
|
+
|
|
48
|
+
**The thesis must be specific, debatable, and actionable.**
|
|
49
|
+
- ❌ "AI has implications for medicine"
|
|
50
|
+
- ✅ "AI diagnostic tools should not replace physician judgment in primary care without mandatory human review requirements"
|
|
51
|
+
|
|
52
|
+
### 4. BACKGROUND / LITERATURE CONTEXT (Optional but Strengthening)
|
|
53
|
+
Brief review of the existing debate. Identifies key positions in the literature and where the author's position fits within or against them. Demonstrates the author knows the existing debate.
|
|
54
|
+
|
|
55
|
+
### 5. SUPPORTING ARGUMENTS (Multiple Sections)
|
|
56
|
+
Each argument gets its own section or substantial paragraph:
|
|
57
|
+
|
|
58
|
+
1. State the claim clearly
|
|
59
|
+
2. Provide supporting evidence (citations, data, expert testimony, logic)
|
|
60
|
+
3. Explain how this evidence supports the thesis
|
|
61
|
+
4. Transition to the next argument
|
|
62
|
+
|
|
63
|
+
**Acceptable evidence:**
|
|
64
|
+
- Published empirical studies, systematic reviews, meta-analyses
|
|
65
|
+
- Statistical data from credible sources
|
|
66
|
+
- Theoretical frameworks, expert consensus statements
|
|
67
|
+
- Logical deduction from established principles
|
|
68
|
+
|
|
69
|
+
**NOT acceptable:**
|
|
70
|
+
- Unpublished original data (unless venue explicitly permits)
|
|
71
|
+
- Personal anecdotes as primary evidence
|
|
72
|
+
- Unverified claims or non-credible sources
|
|
73
|
+
|
|
74
|
+
### 6. COUNTERARGUMENT SECTION (Required for Rigorous Academic Papers)
|
|
75
|
+
This distinguishes a sophisticated position paper from advocacy.
|
|
76
|
+
|
|
77
|
+
**Required approach:**
|
|
78
|
+
1. Identify the STRONGEST opposing argument(s) — not the weakest
|
|
79
|
+
2. Present counterargument fairly and accurately (steelmanning)
|
|
80
|
+
3. Refute using fact-based reasoning
|
|
81
|
+
4. Demonstrate why the original position still holds even granting the counterargument some validity
|
|
82
|
+
|
|
83
|
+
**What to avoid:**
|
|
84
|
+
- Personal attacks on opposing scholars
|
|
85
|
+
- Dismissive treatment of opposing views
|
|
86
|
+
- Selecting only the weakest version of opposing arguments (straw-manning)
|
|
87
|
+
|
|
88
|
+
**Placement**: Near end of argument sequence before Conclusion, OR woven throughout (each argument followed by its counterargument).
|
|
89
|
+
|
|
90
|
+
### 7. CONCLUSION
|
|
91
|
+
- Restate thesis (synthesized, not word-for-word)
|
|
92
|
+
- Summarize key arguments briefly
|
|
93
|
+
- State real-world implications of accepting the position
|
|
94
|
+
- Identify remaining questions or future research areas
|
|
95
|
+
- End with a strong, forward-looking statement
|
|
96
|
+
- Do NOT introduce new arguments
|
|
97
|
+
|
|
98
|
+
## Handling Evidence vs Opinion (Critical)
|
|
99
|
+
|
|
100
|
+
Every claim must be clearly signaled as one of:
|
|
101
|
+
|
|
102
|
+
| Type | Language | Citation? |
|
|
103
|
+
|------|----------|-----------|
|
|
104
|
+
| Established fact | "Studies demonstrate..." "Evidence shows..." | Required |
|
|
105
|
+
| Author's interpretation | "These findings suggest that..." "This indicates..." | Cite underlying evidence |
|
|
106
|
+
| Author's position | "We argue..." "We contend..." "We propose..." | Cite supporting evidence |
|
|
107
|
+
| Speculation | "May..." "Might..." "Could..." "Future research may reveal..." | Optional |
|
|
108
|
+
|
|
109
|
+
**Rules:**
|
|
110
|
+
- Every factual claim requires citation
|
|
111
|
+
- Every interpretive claim must be transparently labeled
|
|
112
|
+
- Use confident language for your own claims ("We argue" not "maybe")
|
|
113
|
+
- Use careful epistemic language for genuinely uncertain claims
|
|
114
|
+
|
|
115
|
+
## Common Mistakes
|
|
116
|
+
|
|
117
|
+
**Structural:**
|
|
118
|
+
- Burying the thesis (must appear in Introduction, not Conclusion)
|
|
119
|
+
- Writing a descriptive literature summary instead of arguing a position
|
|
120
|
+
- Treating counterarguments as optional or perfunctory
|
|
121
|
+
- Steelmanning only the weakest opposing view
|
|
122
|
+
- Ending with vague or repetitive conclusion instead of implications
|
|
123
|
+
|
|
124
|
+
**Evidence and argument:**
|
|
125
|
+
- Relying on anecdote as primary evidence
|
|
126
|
+
- Using only sources that support the position (confirmation bias)
|
|
127
|
+
- Making factual claims without citations
|
|
128
|
+
- Failing to distinguish consensus from author interpretation
|
|
129
|
+
- Over-claiming beyond what evidence supports
|
|
130
|
+
|
|
131
|
+
**Writing:**
|
|
132
|
+
- Excessive hedging weakens the argument (position papers require conviction)
|
|
133
|
+
- Passive voice throughout obscures the author's voice
|
|
134
|
+
- Emotionally charged language instead of rigorous argumentation
|
|
135
|
+
|
|
136
|
+
**Submission:**
|
|
137
|
+
- Submitting to journals that only accept opinion pieces by invitation without pre-inquiry
|
|
138
|
+
- Exceeding word limits (opinion pieces have strict limits, typically 1,500-4,000 words)
|
|
139
|
+
|
|
140
|
+
## Publication Venue Types
|
|
141
|
+
|
|
142
|
+
| Type | Word Count | New Data? | Peer Reviewed? |
|
|
143
|
+
|------|-----------|-----------|----------------|
|
|
144
|
+
| Perspective article | 2,000-4,000 | Sometimes | Usually yes |
|
|
145
|
+
| Opinion article | 2,000-2,500 | No | Usually yes |
|
|
146
|
+
| Commentary | 1,000-1,500 | No | Usually yes |
|
|
147
|
+
| Viewpoint | 1,500-3,000 | No | Usually yes |
|
|
148
|
+
| Forum / Essay | 3,000-5,000 | Sometimes | Yes |
|
|
149
|
+
|
|
150
|
+
**Many high-impact journals publish perspectives only by invitation.** For unsolicited submissions, send a pre-submission inquiry to the editor.
|
|
151
|
+
|
|
152
|
+
## LaTeX Template
|
|
153
|
+
|
|
154
|
+
Position papers typically use the target journal's standard template. If writing a working paper or preprint:
|
|
155
|
+
|
|
156
|
+
```latex
|
|
157
|
+
\documentclass[12pt, a4paper]{article}
|
|
158
|
+
\usepackage[T1]{fontenc}
|
|
159
|
+
\usepackage[utf8]{inputenc}
|
|
160
|
+
\usepackage{lmodern}
|
|
161
|
+
\usepackage[margin=1in]{geometry}
|
|
162
|
+
\usepackage{natbib}
|
|
163
|
+
\usepackage{amsmath}
|
|
164
|
+
\usepackage{graphicx}
|
|
165
|
+
\usepackage{booktabs}
|
|
166
|
+
\usepackage{hyperref}
|
|
167
|
+
\usepackage{microtype}
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
Use the author-year citation style (\citet, \citep) common in social sciences and humanities venues that publish position papers.
|
|
@@ -0,0 +1,176 @@
|
|
|
1
|
+
# Systematic Review Structure (PRISMA 2020)
|
|
2
|
+
|
|
3
|
+
## Overview
|
|
4
|
+
|
|
5
|
+
A systematic review uses pre-specified, reproducible methods to identify, appraise, and synthesize ALL relevant evidence on a defined question. PRISMA 2020 (Preferred Reporting Items for Systematic reviews and Meta-Analyses) is the current reporting standard with a 27-item checklist.
|
|
6
|
+
|
|
7
|
+
## When to Use This Structure
|
|
8
|
+
|
|
9
|
+
✅ Synthesizing existing evidence on a defined research question
|
|
10
|
+
✅ Informing clinical practice, policy, or future research directions
|
|
11
|
+
✅ When a rigorous, reproducible evidence synthesis is needed
|
|
12
|
+
|
|
13
|
+
❌ When original empirical data is being reported → use IMRAD
|
|
14
|
+
❌ When a quick narrative overview suffices → use narrative review
|
|
15
|
+
❌ When arguing a position → use position paper structure
|
|
16
|
+
|
|
17
|
+
## Pre-Review Requirements
|
|
18
|
+
|
|
19
|
+
### Protocol Registration (PROSPERO)
|
|
20
|
+
- Register BEFORE data extraction begins (post-hoc registration reduces credibility)
|
|
21
|
+
- Include: title, eligibility criteria, search strategy overview, synthesis methods, conflicts
|
|
22
|
+
- Upload full detailed protocol as PDF
|
|
23
|
+
- Report PROSPERO registration number in abstract AND methods
|
|
24
|
+
|
|
25
|
+
### Research Question Framework (PICO/PICOS)
|
|
26
|
+
- **P**opulation, **I**ntervention, **C**omparison, **O**utcome, [**S**tudy design]
|
|
27
|
+
- For observational: **PECO** (Exposure instead of Intervention)
|
|
28
|
+
- Each PICO element becomes an inclusion/exclusion criterion
|
|
29
|
+
|
|
30
|
+
## Required Sections
|
|
31
|
+
|
|
32
|
+
### ABSTRACT (12-item mini-checklist)
|
|
33
|
+
Background, objectives, eligibility criteria, information sources, risk of bias method, synthesis methods, included studies count, results summary, limitations, conclusions, funding, registration number
|
|
34
|
+
|
|
35
|
+
### INTRODUCTION
|
|
36
|
+
- **Rationale** (PRISMA Item 3): Why was this review needed?
|
|
37
|
+
- **Objectives** (Item 4): Explicit research questions or hypotheses
|
|
38
|
+
|
|
39
|
+
### METHODS (Most Detailed Section — Items 5-16)
|
|
40
|
+
|
|
41
|
+
1. **Eligibility Criteria** (Item 5)
|
|
42
|
+
- All inclusion/exclusion criteria specified in advance
|
|
43
|
+
- Cover: population, intervention/exposure, comparator, outcomes, study designs, settings, languages, publication years, grey literature policy
|
|
44
|
+
|
|
45
|
+
2. **Information Sources** (Item 6)
|
|
46
|
+
- Every database searched (with platform name)
|
|
47
|
+
- Date of last search
|
|
48
|
+
- Grey literature sources, trial registries, reference list checking, citation searching, expert consultation
|
|
49
|
+
|
|
50
|
+
3. **Search Strategy** (Item 7)
|
|
51
|
+
- Full reproducible search for at least one database (ideally all)
|
|
52
|
+
- Include: search terms, Boolean operators, field codes, date/language filters
|
|
53
|
+
- Full strategy in appendix or supplementary material
|
|
54
|
+
|
|
55
|
+
4. **Selection Process** (Item 8)
|
|
56
|
+
- Number of reviewers, whether independent and in duplicate
|
|
57
|
+
- Disagreement resolution method
|
|
58
|
+
- Automation tools used (Covidence, Rayyan, etc.)
|
|
59
|
+
|
|
60
|
+
5. **Data Collection Process** (Item 9)
|
|
61
|
+
- Data extraction form design
|
|
62
|
+
- Number of extractors, dual extraction, discrepancy resolution
|
|
63
|
+
- Whether study authors were contacted for missing data
|
|
64
|
+
|
|
65
|
+
6. **Data Items** (Item 10)
|
|
66
|
+
- Every variable extracted with rationale
|
|
67
|
+
|
|
68
|
+
7. **Risk of Bias Assessment** (Item 11)
|
|
69
|
+
- Named tool and version (see tools table below)
|
|
70
|
+
- Domains assessed, number of reviewers, independence, disagreement resolution
|
|
71
|
+
|
|
72
|
+
8. **Effect Measures** (Item 12)
|
|
73
|
+
- For meta-analyses: statistical measure (OR, RR, MD, SMD, etc.)
|
|
74
|
+
|
|
75
|
+
9. **Synthesis Methods** (Items 13a-13f — 6 sub-items)
|
|
76
|
+
- What studies were combined and how
|
|
77
|
+
- Tabulation and visualization methods
|
|
78
|
+
- Meta-analysis parameters OR narrative synthesis approach
|
|
79
|
+
- Heterogeneity exploration (I², Cochran Q, subgroup analyses)
|
|
80
|
+
- Sensitivity analyses
|
|
81
|
+
- Reporting bias assessment (funnel plots, Egger's test if ≥10 studies)
|
|
82
|
+
|
|
83
|
+
10. **Certainty Assessment** (Item 15)
|
|
84
|
+
- System used to assess certainty of evidence (typically GRADE)
|
|
85
|
+
|
|
86
|
+
### RESULTS (Items 16-23)
|
|
87
|
+
|
|
88
|
+
1. **Study Selection** — with PRISMA flow diagram (mandatory)
|
|
89
|
+
2. **Study Characteristics** — summary table of all included studies
|
|
90
|
+
3. **Risk of Bias Results** — per study and overall
|
|
91
|
+
4. **Synthesis Results** — per outcome
|
|
92
|
+
5. **Reporting Bias Results**
|
|
93
|
+
6. **Certainty of Evidence Results**
|
|
94
|
+
|
|
95
|
+
### DISCUSSION (Items 24-26)
|
|
96
|
+
- Interpretation in context of prior reviews and clinical/policy context
|
|
97
|
+
- Limitations of the review itself (search limitations, methodological limitations)
|
|
98
|
+
- Conclusions with practical implications
|
|
99
|
+
|
|
100
|
+
### OTHER INFORMATION (Item 27)
|
|
101
|
+
- Registration info (PROSPERO number)
|
|
102
|
+
- Protocol accessibility
|
|
103
|
+
- Protocol amendments (when and why)
|
|
104
|
+
- Funding, conflicts, data availability
|
|
105
|
+
- Automation tools used
|
|
106
|
+
|
|
107
|
+
## PRISMA Flow Diagram (Mandatory)
|
|
108
|
+
|
|
109
|
+
Four phases tracked in TWO columns (2020 update):
|
|
110
|
+
|
|
111
|
+
**Column 1: Database Records**
|
|
112
|
+
- Records identified from databases (by database name)
|
|
113
|
+
- Records after deduplication
|
|
114
|
+
- Records screened → excluded (with reasons)
|
|
115
|
+
- Full-text reports assessed → excluded (with specific reasons per exclusion)
|
|
116
|
+
|
|
117
|
+
**Column 2: Other Sources**
|
|
118
|
+
- Records from citation searching, grey literature, expert consultation
|
|
119
|
+
- Reports sought → not retrieved → assessed → excluded
|
|
120
|
+
|
|
121
|
+
**Merge point**: Studies included in review
|
|
122
|
+
- Report: number of studies AND number of reports (one study may have multiple reports)
|
|
123
|
+
|
|
124
|
+
## Risk of Bias Tools
|
|
125
|
+
|
|
126
|
+
| Study Design | Tool | Notes |
|
|
127
|
+
|-------------|------|-------|
|
|
128
|
+
| Randomized Controlled Trials | RoB 2 (Cochrane, 2019) | 5 domains; low/some concerns/high |
|
|
129
|
+
| Non-randomized interventions | ROBINS-I | 7 domains; requires epidemiological expertise |
|
|
130
|
+
| Observational (cohort, case-control) | Newcastle-Ottawa Scale (NOS) | Faster but less granular |
|
|
131
|
+
| Diagnostic accuracy studies | QUADAS-2 | |
|
|
132
|
+
| Systematic reviews being appraised | AMSTAR-2 | |
|
|
133
|
+
| Qualitative studies | CASP Qualitative Checklist | |
|
|
134
|
+
|
|
135
|
+
## Meta-Analysis vs Narrative Synthesis
|
|
136
|
+
|
|
137
|
+
### Use meta-analysis when:
|
|
138
|
+
- Studies address same PICO elements
|
|
139
|
+
- Compatible measurement scales and time points
|
|
140
|
+
- Adequate methodological quality
|
|
141
|
+
- Statistical heterogeneity acceptably low (I² generally <50-60%)
|
|
142
|
+
|
|
143
|
+
### Use narrative synthesis when:
|
|
144
|
+
- Significant clinical or methodological heterogeneity
|
|
145
|
+
- Incompatible outcome measures
|
|
146
|
+
- Quantitative pooling would be misleading
|
|
147
|
+
- Question is about mechanisms, context, or complexity
|
|
148
|
+
- Only 2-3 studies exist (pooling can mislead)
|
|
149
|
+
|
|
150
|
+
### Often use both:
|
|
151
|
+
Meta-analysis for comparable outcomes; narrative synthesis where pooling is inappropriate.
|
|
152
|
+
|
|
153
|
+
## Common Mistakes
|
|
154
|
+
|
|
155
|
+
**Methodological:**
|
|
156
|
+
- Beginning without a registered protocol
|
|
157
|
+
- Searching too few databases (minimum 2-3 required)
|
|
158
|
+
- Omitting grey literature (unpublished studies, conference abstracts)
|
|
159
|
+
- Single reviewer for screening or extraction (dual independent is standard)
|
|
160
|
+
- Not pre-specifying outcomes (outcome switching is a serious validity threat)
|
|
161
|
+
- Wrong risk of bias tool for study design
|
|
162
|
+
- Performing meta-analysis despite high heterogeneity without investigation
|
|
163
|
+
|
|
164
|
+
**Reporting:**
|
|
165
|
+
- Failing to report full search strategy
|
|
166
|
+
- Missing PRISMA flow diagram
|
|
167
|
+
- No certainty of evidence assessment (GRADE)
|
|
168
|
+
- Not reporting protocol amendments with dates
|
|
169
|
+
- Omitting automation tools used (Covidence, Rayyan, ML screening)
|
|
170
|
+
|
|
171
|
+
**Reviewer expectations:**
|
|
172
|
+
- Full, reproducible search strategy in appendix
|
|
173
|
+
- Completed PRISMA checklist submitted alongside manuscript
|
|
174
|
+
- PROSPERO registration number in abstract and methods
|
|
175
|
+
- Risk of bias summary figure AND table
|
|
176
|
+
- Transparent reporting of both study limitations AND review limitations
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
# ACL (Association for Computational Linguistics) Submission Guidelines
|
|
2
|
+
|
|
3
|
+
## LaTeX Template
|
|
4
|
+
|
|
5
|
+
Use the official ACL style files. Replace the generic preamble with:
|
|
6
|
+
|
|
7
|
+
```latex
|
|
8
|
+
\documentclass[11pt]{article}
|
|
9
|
+
\usepackage[hyperref]{acl}
|
|
10
|
+
% For camera-ready: \usepackage[hyperref,accepted]{acl}
|
|
11
|
+
|
|
12
|
+
\usepackage{times}
|
|
13
|
+
\usepackage{latexsym}
|
|
14
|
+
\usepackage[T1]{fontenc}
|
|
15
|
+
\usepackage[utf8]{inputenc}
|
|
16
|
+
\usepackage{microtype}
|
|
17
|
+
\usepackage{amsmath}
|
|
18
|
+
\usepackage{graphicx}
|
|
19
|
+
\usepackage{booktabs}
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Style files: github.com/acl-org/acl-style-files. Do NOT modify the style file.
|
|
23
|
+
|
|
24
|
+
## Format Constraints
|
|
25
|
+
|
|
26
|
+
- **Paper size**: A4 (NOT US Letter — unlike NeurIPS/IEEE)
|
|
27
|
+
- **Layout**: Two-column, single-spaced
|
|
28
|
+
- **Font**: Times Roman, 11pt body text
|
|
29
|
+
- **Column width**: 7.7 cm, gap 0.6 cm
|
|
30
|
+
- **Margins**: 2.5 cm on all sides
|
|
31
|
+
- **Line numbers**: Required in review version (ruler in left/right margins)
|
|
32
|
+
- **All fonts embedded** in PDF — verify with `pdffonts mypaper.pdf`
|
|
33
|
+
|
|
34
|
+
### Page Limits
|
|
35
|
+
| Type | Submission | Camera-Ready |
|
|
36
|
+
|------|-----------|-------------|
|
|
37
|
+
| Long paper | 8 pages | 9 pages |
|
|
38
|
+
| Short paper | 4 pages | 5 pages |
|
|
39
|
+
|
|
40
|
+
References, Limitations, Ethics section, and appendices are UNLIMITED and do NOT count toward page limits.
|
|
41
|
+
|
|
42
|
+
## Required Sections (Mandatory)
|
|
43
|
+
|
|
44
|
+
### 1. Abstract
|
|
45
|
+
- **Max 200 words**, 10pt, centered, indented 0.6cm each side
|
|
46
|
+
- Also entered separately in OpenReview submission form
|
|
47
|
+
|
|
48
|
+
### 2. Main Body (within page limit)
|
|
49
|
+
Introduction → [Literature Review] → Method → Results → Conclusion
|
|
50
|
+
|
|
51
|
+
### 3. Limitations Section (MANDATORY — desk reject if missing)
|
|
52
|
+
- Title must be exactly "Limitations"
|
|
53
|
+
- Placed after Conclusion, before References, NO page break between them
|
|
54
|
+
- Does NOT count toward page limit
|
|
55
|
+
- Content: assumptions, scope (datasets, languages, domains), computational constraints, generalizability, bias definitions
|
|
56
|
+
- Discussion only — do NOT introduce new experiments or results here
|
|
57
|
+
|
|
58
|
+
### 4. References
|
|
59
|
+
- Alphabetically arranged, unnumbered heading
|
|
60
|
+
- Unlimited, outside page limit
|
|
61
|
+
|
|
62
|
+
### Optional But Strongly Encouraged
|
|
63
|
+
- **Ethical Considerations / Broader Impact** — after Conclusion alongside Limitations, before References. Functionally required for papers with sensitive data or potentially harmful applications. Papers that should address ethics but don't will not be accepted.
|
|
64
|
+
- **Appendices** — after References, labeled A, B, C. MUST use double-column format (desk rejection since July 2025 if not). Single-column permitted only for math-heavy sections.
|
|
65
|
+
|
|
66
|
+
## Anonymization
|
|
67
|
+
|
|
68
|
+
- Remove author names and affiliations from paper
|
|
69
|
+
- Write about your own work in third person
|
|
70
|
+
- Exclude Acknowledgments entirely from review version
|
|
71
|
+
- Anonymize supplementary materials and code links (use anonymous.4open.science)
|
|
72
|
+
- No anonymity period restriction — preprints allowed at any time before/during/after review
|
|
73
|
+
|
|
74
|
+
## Review Criteria (ARR Three-Score System)
|
|
75
|
+
|
|
76
|
+
### Soundness (1-5, half-point)
|
|
77
|
+
Technical correctness, thoroughness, adequate support for claims.
|
|
78
|
+
- 5: One of the most thorough studies of its type
|
|
79
|
+
- 3: Sufficient support for main claims
|
|
80
|
+
- 1: Major correctness or rigor problems
|
|
81
|
+
|
|
82
|
+
### Excitement (1-5, half-point)
|
|
83
|
+
Subjective interest and potential impact. Orthogonal to soundness.
|
|
84
|
+
- 5: Would strongly recommend to others
|
|
85
|
+
- 3: Worth publishing at the venue
|
|
86
|
+
- 1: Would not prioritize
|
|
87
|
+
|
|
88
|
+
### Overall Assessment (1-5, half-point)
|
|
89
|
+
Composite recommendation:
|
|
90
|
+
- 5: Consider for Award
|
|
91
|
+
- 4: Conference (main track acceptance)
|
|
92
|
+
- 3: Findings (ACL Findings acceptance)
|
|
93
|
+
- 2: Resubmit
|
|
94
|
+
- 1: Do not resubmit
|
|
95
|
+
|
|
96
|
+
### Problem Codes Reviewers Flag
|
|
97
|
+
**Methodology**: LLM-only evaluation without reliability validation (M1), insufficient reproducibility (M2), unmotivated model selection (M4)
|
|
98
|
+
**Results**: Statistical errors/p-hacking (R1), overclaiming beyond evaluated scope (R2, R4), speculation as fact (R3)
|
|
99
|
+
**General**: Unclear research question (G1), misrepresented related work (G3), misleading citations (G5)
|
|
100
|
+
|
|
101
|
+
## Responsible NLP Checklist (Mandatory)
|
|
102
|
+
|
|
103
|
+
Completed in OpenReview submission form. Incorrect/incomplete/misleading answers = desk rejection (enforced since Dec 2024). Published as appendix for accepted papers.
|
|
104
|
+
|
|
105
|
+
Key sections:
|
|
106
|
+
- **A**: Limitations discussion, potential risks
|
|
107
|
+
- **B**: Existing artifacts (citations, licenses, intended use, personal data screening)
|
|
108
|
+
- **C**: Computational experiments (budget, setup, error bars, packages)
|
|
109
|
+
- **D**: Human annotators (instructions, payment ≥ minimum wage, consent, IRB)
|
|
110
|
+
- **E**: AI assistant disclosure (ChatGPT, Copilot usage)
|
|
111
|
+
|
|
112
|
+
## Common Rejection Reasons
|
|
113
|
+
|
|
114
|
+
**Automatic desk rejection:**
|
|
115
|
+
- Missing Limitations section
|
|
116
|
+
- Page limit exceeded or wrong paper size (must be A4)
|
|
117
|
+
- Wrong template or modified style files
|
|
118
|
+
- Anonymization breach
|
|
119
|
+
- Checklist violations
|
|
120
|
+
- Non-double-column appendices (since July 2025)
|
|
121
|
+
|
|
122
|
+
**Substantive rejection:**
|
|
123
|
+
- Claims not supported by evidence
|
|
124
|
+
- LLM-only evaluation without validation
|
|
125
|
+
- Overclaiming from benchmark results
|
|
126
|
+
- Missing significance tests or effect sizes
|
|
127
|
+
- Key related work missing or misrepresented
|
|
128
|
+
- Insufficient reproducibility details
|
|
129
|
+
|
|
130
|
+
## ACL Rolling Review (ARR) Process
|
|
131
|
+
|
|
132
|
+
- Centralized review via OpenReview; monthly submission cycles
|
|
133
|
+
- Papers receive reviews once, then authors "commit" reviews to a venue (ACL, EMNLP, NAACL, etc.)
|
|
134
|
+
- Minimum 3 independent reviewers per paper
|
|
135
|
+
- After reviews: author response → reviewer discussion → meta-review
|
|
136
|
+
- Revised resubmissions must include revision notes and link to prior version
|
|
137
|
+
|
|
138
|
+
## Key Formatting Details
|
|
139
|
+
|
|
140
|
+
- **Title**: 15pt bold, centered
|
|
141
|
+
- **Section headings**: 12pt bold
|
|
142
|
+
- **Subsection**: 11pt bold
|
|
143
|
+
- **Captions**: 10pt
|
|
144
|
+
- **Footnotes**: 9pt
|
|
145
|
+
- **References**: 10pt
|
|
146
|
+
- **Acknowledgments**: Excluded from review version, included in camera-ready only
|