apex-ruby 1.0.17 → 1.0.20

data/ext/apex_ext/apex_src/docs/CITATIONS_IMPLEMENTATION_PLAN.md

@@ -0,0 +1,749 @@
+# Apex Citations and Bibliography Implementation Plan
+
+## Executive Summary
+
+This document provides a comprehensive plan for implementing
+citations and bibliographies in Apex, based on analysis of
+how mmark, MultiMarkdown, and Pandoc handle citations. It
+addresses key questions about bibliography formats, external
+files, metadata integration, and syntax modes.
+
+## Analysis of Existing Tools
+
+### Pandoc Output Analysis
+
+**Citation Format:**
+
+Uses `<span class="citation" data-cites="key1 key2">(Formatted Citation)</span>`
+
+Supports multiple citations in one span: `data-cites="doe99 smith2000 smith2004"`
+
+- Author-date style: `(Doe 1999; Jane Smith 2000, 2004)`
+- Author-in-text: `John Smith (2004)` (no parentheses)
+- Author-suppressed: `(2004)` (no author name)
+
+**Bibliography Format:**
+
+Uses `<div id="refs" class="references csl-bib-body hanging-indent">`
+
+- Each entry: `<div id="ref-key" class="csl-entry">`
+- Formatted with CSL (Citation Style Language)
+- Includes proper typography (em dashes, quotes, italics)
+
+**Key Observations:**
+
+Pandoc uses CSL (Citation Style Language) for all formatting
+
+- Citations are processed through citeproc
+- Bibliography is automatically generated from cited items
+- Supports BibTeX, CSL JSON, and other formats as input
+
+### mmark Output Analysis
+
+**Citation Format:**
+
+- Uses XML2RFC format: `<xref target="RFC2535"></xref>`
+
+For HTML output, converts to simple links or formatted text
+
+- Special handling for RFC/BCP/STD/I-D/W3C documents
+- Auto-generates references for these document types
+
+**Bibliography Format:**
+
+- Requires `{backmatter}` marker to generate bibliography
+- Uses XML2RFC `<reference>` elements
+- Auto-fetches RFC/I-D metadata from citation index
+
+**Key Observations:**
+
+- mmark is specialized for RFC documents
+- Uses external citation index for RFC/I-D metadata
+- Bibliography only appears if `{backmatter}` is present
+- Less flexible than Pandoc for general academic citations
+
+### MultiMarkdown Output Analysis
+
+**Citation Format:**
+
+Uses numeric references: `<a href="#Doe:2006" id="Doe:2006:1">[1]</a>`
+
+- Simple numbered citations
+- Links to bibliography entries
+
+**Bibliography Format:**
+
+Uses `<div id="bibliography">` with entries like `<div id="Doe:2006">`
+
+- Simple format, no CSL styling
+- References defined inline or in separate section
+
+**Key Observations:**
+
+- Simpler than Pandoc
+- No CSL support
+- Numeric citations only
+- Good for basic use cases
+
+## Recommended Approach for Apex
+
+### Bibliography Format: CSL JSON (Primary)
+
+**Why CSL JSON?**
+
+1. **Native Format**: CSL JSON is the native format for
+
+   citation processing
+
+2. **Flexibility**: Supports all citation styles via CSL
+
+   stylesheets
+
+**Standard**: Widely used (Pandoc, Zotero, Mendeley, etc.)
+
+4. **Extensibility**: Easy to add new fields and entry types
+5. **Processing**: Can be directly consumed by citeproc
+
+   processors
+
+**Format Priority:**
+
+1. **CSL JSON** (`.json`) - Primary format, native support
+2. **BibTeX** (`.bib`) - Most common, convert to CSL JSON
+
+   internally
+
+**CSL YAML** (`.yaml`) - Human-readable, convert to CSL JSON
+
+4. **RIS** (`.ris`) - Lower priority, can add later
+
+### Bibliography Storage: External Files (Recommended)
+
+**Yes, external files should be supported** for the
+following reasons:
+
+1. **Reusability**: Same bibliography can be used across
+
+   multiple documents
+
+2. **Management**: Reference managers (Zotero, Mendeley)
+
+   export to `.bib` or `.json`
+
+3. **Standard Practice**: Pandoc, Quarto, and other tools
+
+   use external files
+
+4. **Scalability**: Large bibliographies are easier to
+
+   manage in separate files
+
+**Implementation Options:**
+
+#### Option 1: External Files Only (Recommended)
+- Specify via metadata: `bibliography: references.bib`
+- Specify via CLI: `--bibliography references.bib`
+
+Support multiple files: `bibliography: [refs1.bib, refs2.bib]`
+
+- Auto-detect format from extension
+
+#### Option 2: Hybrid (External + Inline)
+- Support external files (primary)
+- Support inline YAML `references:` field (for small
+
+  bibliographies)
+
+- Support MultiMarkdown reference definitions (for
+
+  compatibility)
+
+- Merge all sources (inline takes precedence)
+
+**Recommendation**: Implement Option 2 (Hybrid) for maximum
+flexibility.
+
+### Metadata Integration
+
+**YAML Front Matter:**
+```yaml
+---
+bibliography: references.bib
+# or multiple files:
+bibliography:
+  - refs1.bib
+  - refs2.bib
+
+csl: apa.csl  # Citation style file
+suppress-bibliography: false
+link-citations: true
+show-tooltips: false
+nocite: |
+  @item1, @item2
+# or include all:
+nocite: '@*'
+
+references:  # Inline references (CSL JSON format)
+  - id: doe99
+    type: article-journal
+    author:
+      - family: Doe
+        given: John
+    issued:
+      date-parts: [[1999]]
+    title: Article Title
+    container-title: Journal Name
+    volume: 1
+    page: 1-10
+---
+
+```
+
+**MultiMarkdown Metadata:**
+```
+Bibliography: references.bib
+CSL: apa.csl
+Suppress Bibliography: false
+Link Citations: true
+
+```
+
+**Pandoc Title Block:**
+```
+% Bibliography: references.bib
+% CSL: apa.csl
+
+```
+
+### Command-Line Flags
+
+```bash
+# Single bibliography file
+apex input.md --bibliography refs.bib
+
+# Multiple bibliography files
+apex input.md --bibliography refs1.bib --bibliography refs2.bib
+
+# Citation style
+apex input.md --bibliography refs.bib --csl apa.csl
+
+# Suppress bibliography
+apex input.md --bibliography refs.bib --no-bibliography
+
+# Link citations to bibliography
+apex input.md --bibliography refs.bib --link-citations
+
+# Show tooltips on citations
+apex input.md --bibliography refs.bib --show-tooltips
+
+```
+
+**Precedence Order:**
+
+1. Command-line flags (highest priority)
+2. Document metadata
+3. Default values (lowest priority)
+
+### Syntax Modes
+
+#### Mode-Specific Behavior
+
+**CommonMark Mode:**
+
+- Citations: **Disabled** (pure CommonMark, no extensions)
+- Syntax: None
+
+**GFM Mode:**
+
+- Citations: **Disabled** (GFM doesn't support citations)
+- Syntax: None
+
+**MultiMarkdown Mode:**
+
+- Citations: **Enabled**
+- Syntax: MultiMarkdown syntax only (`[#key]`)
+- Bibliography: From reference definitions or external file
+- Output: Numeric citations (like MultiMarkdown)
+
+**Kramdown Mode:**
+
+- Citations: **Disabled** (Kramdown uses different citation
+
+  system)
+
+- Syntax: None
+- *Note: Could add Kramdown citation syntax in future*
+
+**Unified Mode:**
+
+- Citations: **Enabled**
+- Syntax: **All syntaxes** (Pandoc, MultiMarkdown, mmark)
+- Auto-detection: Flexible parsing
+- Bibliography: All sources (external files, inline YAML,
+
+  MMD definitions)
+
+#### Syntax Detection in Unified Mode
+
+**Strategy: Pattern-Based Detection**
+
+1. **Pandoc Syntax** (`[@key]` or `@key`):
+   - Pattern: `[@` followed by alphanumeric key
+   - Most common, try first
+
+Supports: prefixes, locators, suffixes, author suppression
+
+2. **MultiMarkdown Syntax** (`[#key]`):
+   - Pattern: `[#` followed by citation key
+   - Distinctive `#` marker
+   - Supports: locators before citation
+
+3. **mmark Syntax** (`[@RFC1234]`):
+   - Pattern: `[@` followed by RFC/BCP/STD/I-D/W3C prefix
+   - Special case: auto-generate references
+
+Supports: modifiers (`!`, `?`, `-`), combined references
+
+**Detection Algorithm:**
+```
+For each potential citation:
+  1. Check for mmark pattern: [@RFC...] or [@BCP...] etc.
+     → If match: Process as mmark citation
+  2. Check for MultiMarkdown pattern: [#key]
+     → If match: Process as MMD citation
+  3. Check for Pandoc pattern: [@key] or @key
+     → If match: Process as Pandoc citation
+  4. If no match: Leave as-is (not a citation)
+
+```
+
+**Edge Cases:**
+
+- `[@RFC1234]` could be Pandoc or mmark → Prefer mmark if
+
+  matches RFC pattern
+
+- `[#key]` is unambiguous → Always MultiMarkdown
+- `[@key]` without RFC prefix → Pandoc syntax
+
+## Implementation Architecture
+
+### Processing Pipeline
+
+```
+1. Preprocessing Phase
+   ├─ Extract metadata (YAML, MMD, Pandoc)
+   ├─ Load bibliography files (from metadata or CLI)
+   ├─ Parse bibliography (BibTeX → CSL JSON, CSL JSON, CSL YAML)
+   └─ Build citation index (map keys to entries)
+
+2. Parsing Phase
+   ├─ Scan for citations (mode-aware)
+   ├─ Extract citation data (key, locator, prefix, suffix)
+   ├─ Match citations to bibliography entries
+   └─ Store citation registry
+
+3. Rendering Phase
+   ├─ Process citations through CSL (or simple formatter)
+   ├─ Generate citation HTML
+   ├─ Collect cited items for bibliography
+   └─ Format bibliography entries
+
+4. Post-processing Phase
+   ├─ Insert bibliography at marker or end
+   ├─ Add links between citations and bibliography
+   ├─ Add tooltips (if enabled)
+   └─ Final HTML output
+
+```
+
+### File Structure
+
+```
+src/extensions/
+├── citations.h              # Citation parsing header
+├── citations.c              # Citation parsing implementation
+├── bibliography.h            # Bibliography handling header
+├── bibliography.c           # Bibliography parsing/formatting
+├── bibtex_parser.h          # BibTeX parser header
+├── bibtex_parser.c          # BibTeX parser implementation
+└── csl_formatter.h          # CSL formatter header (if implementing minimal CSL)
+└── csl_formatter.c          # CSL formatter implementation (if implementing minimal CSL)
+
+```
+
+### Core Data Structures
+
+```c
+/* Citation structure */
+typedef struct {
+    char *key;                    // Citation key (e.g., "doe99")
+    char *prefix;                 // Prefix text (e.g., "see")
+    char *locator;                // Locator (e.g., "pp. 33-35")
+    char *suffix;                 // Suffix text (e.g., "and passim")
+    bool author_suppressed;       // -@key syntax
+    bool author_in_text;          // @key syntax (no brackets)
+    int syntax_type;              // PANDOC, MMD, MMARK
+    int position;                 // Position in document
+} apex_citation;
+
+/* Bibliography entry (CSL JSON format) */
+typedef struct {
+    char *id;                     // Citation key
+    char *type;                   // Entry type (article-journal, book, etc.)
+    cJSON *data;                  // Full CSL JSON object (using cJSON)
+} apex_bibliography_entry;
+
+/* Citation registry */
+typedef struct {
+    apex_citation **citations;   // Array of citations
+    size_t count;                 // Number of citations
+    apex_bibliography_entry **entries;  // Bibliography entries
+    size_t entry_count;           // Number of entries
+    cJSON *csl_style;             // CSL style (if using)
+} apex_citation_registry;
+
+```
+
+## Bibliography Format Details
+
+### CSL JSON Structure
+
+```json
+[
+  {
+    "id": "doe99",
+    "type": "article-journal",
+    "author": [
+      {
+        "family": "Doe",
+        "given": "John"
+      }
+    ],
+    "issued": {
+      "date-parts": [[1999]]
+    },
+    "title": "Article Title",
+    "container-title": "Journal Name",
+    "volume": "1",
+    "page": "1-10"
+  }
+]
+
+```
+
+### BibTeX to CSL JSON Conversion
+
+**Common BibTeX Entry Types:**
+
+- `@article` → `article-journal`
+- `@book` → `book`
+- `@inbook` → `chapter`
+- `@incollection` → `chapter`
+- `@inproceedings` → `paper-conference`
+- `@phdthesis` → `thesis`
+- `@mastersthesis` → `thesis`
+- `@techreport` → `report`
+
+**Field Mapping:**
+
+- `author` → `author` (parse names)
+- `title` → `title`
+- `journal` → `container-title`
+- `year` → `issued.date-parts`
+- `pages` → `page`
+- `publisher` → `publisher`
+- `volume` → `volume`
+- `number` → `issue`
+
+### CSL YAML Structure
+
+```yaml
+- id: doe99
+  type: article-journal
+  author:
+    - family: Doe
+      given: John
+  issued:
+    date-parts:
+      - - 1999
+  title: Article Title
+  container-title: Journal Name
+  volume: 1
+  page: 1-10
+
+```
+
+## Output Format
+
+### Citation HTML (Pandoc-style)
+
+**Author-Date Style:**
+```html
+<span class="citation" data-cites="doe99">(Doe 1999)</span>
+
+```
+
+**Multiple Citations:**
+```html
+<span class="citation" data-cites="doe99 smith2000 smith2004">
+  (Doe 1999; Jane Smith 2000, 2004)
+</span>
+
+```
+
+**With Locator:**
+```html
+<span class="citation" data-cites="doe99">Doe (1999, 33–35)</span>
+
+```
+
+**Author-Suppressed:**
+```html
+<span class="citation" data-cites="smith04">(2004)</span>
+
+```
+
+**With Links:**
+```html
+<a href="#ref-doe99" class="citation" data-cites="doe99">
+  (Doe 1999)
+</a>
+
+```
+
+**With Tooltips:**
+```html
+<a href="#ref-doe99" class="citation" data-cites="doe99"
+   title="Doe, John. 1999. 'Article Title.' Journal Name 1: 1-10.">
+  (Doe 1999)
+</a>
+
+```
+
+### Bibliography HTML
+
+```html
+<div id="refs" class="references csl-bib-body hanging-indent" role="doc-bibliography">
+  <div id="ref-doe99" class="csl-entry" role="doc-biblioentry">
+    Doe, John. 1999. <span>"Article Title."</span> <em>Journal Name</em> 1: 1–10.
+  </div>
+  <div id="ref-smith2000" class="csl-entry" role="doc-biblioentry">
+    Smith, Jane. 2000. <em>Book Title</em>. Publisher.
+  </div>
+</div>
+
+```
+
+### Bibliography Insertion Points
+
+1. **`<!-- REFERENCES -->` marker** (if present)
+2. **`{backmatter}` marker** (mmark style, if present)
+3. **`<div id="refs">`** (if present)
+4. **End of document** (default)
+
+## Dependencies
+
+### Required Libraries
+
+1. **JSON Parser**: `cJSON` (lightweight C JSON library)
+   - For parsing CSL JSON and BibTeX (after conversion)
+   - License: MIT
+   - Small footprint
+
+2. **BibTeX Parser**: Implement minimal parser in C
+   - Parse BibTeX entries
+   - Convert to CSL JSON
+   - Handle common entry types and fields
+
+3. **YAML Parser**: Reuse existing Apex YAML parser
+   - Already used for metadata
+   - Can parse CSL YAML format
+
+### CSL Processing Options
+
+**Option 1: Minimal CSL Implementation (Recommended for
+MVP)**
+
+- Implement basic author-date formatting
+- Support common entry types (article, book, chapter)
+- Simple formatting rules
+- **Pros**: No external dependencies, full control
+- **Cons**: Limited style support, more work
+
+**Option 2: Use citeproc-rs via FFI**
+
+- Rust library with C bindings
+- Full CSL support
+- **Pros**: Complete CSL support, well-tested
+- **Cons**: Requires Rust toolchain, FFI complexity
+
+**Option 3: Use citeproc-js via Node.js**
+
+- JavaScript library
+- **Pros**: Full CSL support
+- **Cons**: Requires Node.js, complex integration
+
+**Recommendation**: Start with Option 1 (minimal
+implementation) for MVP, consider Option 2 for full CSL
+support later.
+
+## Implementation Phases
+
+### Phase 1: Basic Citation Parsing (MVP)
+- [ ] Parse Pandoc syntax `[@key]` (simple keys only)
+- [ ] Parse MultiMarkdown syntax `[#key]`
+- [ ] Parse mmark syntax `[@RFC1234]` (basic)
+- [ ] Store citations in registry
+- [ ] Basic HTML output (show key or simple format)
+- [ ] Mode-aware syntax detection
+
+### Phase 2: Bibliography Loading
+- [ ] Load CSL JSON files
+- [ ] Parse CSL JSON to internal format
+- [ ] Load BibTeX files
+- [ ] Convert BibTeX to CSL JSON
+- [ ] Load CSL YAML files
+- [ ] Parse MultiMarkdown reference definitions
+- [ ] Merge multiple bibliography sources
+- [ ] Match citations to bibliography entries
+
+### Phase 3: Basic Citation Formatting
+- [ ] Implement minimal author-date formatter
+- [ ] Format citations with author names and years
+- [ ] Handle multiple citations
+- [ ] Generate proper HTML output
+- [ ] Add CSS classes
+
+### Phase 4: Bibliography Generation
+- [ ] Collect cited items
+- [ ] Sort bibliography entries
+- [ ] Format bibliography entries
+- [ ] Generate bibliography HTML
+- [ ] Insert bibliography at marker or end
+
+### Phase 5: Advanced Features
+- [ ] Support locators (pages, chapters, etc.)
+- [ ] Support prefixes and suffixes
+- [ ] Support author suppression (`-@key`)
+- [ ] Support author-in-text (`@key`)
+- [ ] Add citation links
+- [ ] Add tooltips
+- [ ] Support `nocite` metadata
+
+### Phase 6: Full CSL Support (Future)
+- [ ] Integrate full CSL processor
+- [ ] Support multiple citation styles
+- [ ] Load CSL style files
+- [ ] Support numeric and note styles
+
+## Testing Strategy
+
+### Unit Tests
+
+1. **Citation Parsing**:
+   - Test each syntax (Pandoc, MMD, mmark)
+   - Test edge cases (nested brackets, special characters)
+   - Test mode detection in unified mode
+
+2. **Bibliography Parsing**:
+   - Test each format (BibTeX, CSL JSON, CSL YAML)
+   - Test multiple files
+   - Test inline references
+   - Test error handling (missing files, invalid format)
+
+3. **Citation Formatting**:
+   - Test author-date formatting
+   - Test multiple citations
+   - Test with/without links
+   - Test with/without tooltips
+
+4. **Bibliography Generation**:
+   - Test sorting
+   - Test formatting
+   - Test insertion points
+   - Test suppression
+
+### Integration Tests
+
+1. **End-to-end**:
+   - Full document with citations
+   - Multiple bibliography sources
+   - Different citation styles
+   - Different output formats
+
+2. **Mode Compatibility**:
+
+Test in each mode (commonmark, gfm, mmd, kramdown, unified)
+
+   - Test syntax detection in unified mode
+   - Test feature flags
+
+## Configuration
+
+### API Options
+
+```c
+typedef struct {
+    // ... existing options ...
+
+    // Citation options
+    bool enable_citations;
+    char **bibliography_files;      // NULL-terminated array
+    const char *csl_file;            // CSL style file
+    bool suppress_bibliography;
+    bool link_citations;
+    bool show_tooltips;
+    const char *nocite;              // Comma-separated keys, or "*" for all
+} apex_options;
+
+```
+
+## Open Questions
+
+1. **CSL Processor**: Start with minimal implementation or
+
+   use existing library?
+
+   - **Recommendation**: Start minimal, add full CSL later
+
+2. **BibTeX Parser**: Implement or use library?
+   - **Recommendation**: Implement minimal parser
+
+3. **Performance**: How to handle large bibliographies
+
+   (1000+ entries)?
+
+   - **Recommendation**: Use hash table for lookup, cache
+
+     parsed bibliographies
+
+4. **Validation**: How strict should citation validation be?
+   - **Recommendation**: Warn on missing citations, don't
+
+     error (like Pandoc)
+
+5. **Error Handling**: How to handle missing citations?
+
+**Recommendation**: Show citation key in output with warning
+
+6. **Backwards Compatibility**: How to ensure existing
+
+   documents still work?
+
+   - **Recommendation**: Citations only processed if
+
+     bibliography is provided
+
+## References
+
+[Pandoc Citations](https://pandoc.org/demo/example33/8.20-citation-syntax.html)
+
+[MultiMarkdown Citations](https://fletcher.github.io/MultiMarkdown-6/Documentation/bibliography.html)
+
+- [mmark Citations](https://mmark.miek.nl/post/citations/)
+- [CSL Project](https://citationstyles.org/)
+
+[Citation Style Language](https://docs.citationstyles.org/en/stable/)
+
+- [cJSON Library](https://github.com/DaveGamble/cJSON)