apex-ruby 1.0.18 → 1.0.20

data/ext/apex_ext/apex_src/docs/CITATIONS_PLAN.md

@@ -0,0 +1,744 @@
+# Citations and Bibliography Implementation Plan for Apex
+
+## Overview
+
+This document outlines a plan for implementing citation and
+bibliography support in Apex, supporting multiple syntaxes
+(Pandoc, MultiMarkdown, mmark) and bibliography formats
+(BibTeX, BibLaTeX, CSL JSON, CSL YAML).
+
+## Goals
+
+1. Support multiple citation syntaxes (Pandoc `[@key]`,
+
+   MultiMarkdown `[#key]`, mmark `[@RFC1234]`)
+
+2. Support multiple bibliography formats (BibTeX, BibLaTeX,
+
+   CSL JSON, CSL YAML)
+
+3. Generate formatted bibliographies in HTML output
+4. Support citation styles via CSL (Citation Style Language)
+5. Integrate with existing metadata system
+6. Mode-aware syntax detection (unified mode supports all
+
+   syntaxes)
+
+## Citation Syntaxes
+
+### 1. Pandoc Syntax (Primary)
+
+**Format**: `[@key]` or `@key` (author-in-text)
+
+**Features**:
+
+- Multiple citations: `[@doe99; @smith2000]`
+- Prefixes: `[see @doe99, pp. 33-35]`
+- Locators: `[@doe99, p. 23]`
+- Suffixes: `[@doe99, and *passim*]`
+- Author suppression: `[-@smith04]` (when author mentioned
+
+  in text)
+
+- Author-in-text: `@smith04 says blah` (renders author name)
+- Complex locators: `[@smith{ii, A, D-Z}]`
+
+**Key Rules**:
+
+Keys must start with letter/digit/`_` and contain alphanumerics + internal punctuation (`:.#$%&-+?<>~/`)
+
+Keys with special characters must be wrapped: `[@{https://example.com/bib?name=foobar}]`
+
+- Locator terms: page, chapter, section, etc. (from CSL
+
+  locale files)
+
+### 2. MultiMarkdown Syntax
+
+**Format**: `[locator][#key]` or `[][#key]` (no locator)
+
+**Features**:
+
+- Locator before citation: `[p. 23][#Doe:2006]`
+- Empty locator: `[][#Doe:2006]`
+
+Reference definitions at end: `[#Doe:2006]: John Doe. *Some Big Fancy Book*. Vanity Press, 2006.`
+
+Inline citations: `As per Doe.[#John Doe. *A Totally Fake Book 1*. Vanity Press, 2006.]`
+
+Uncited references: `[Not cited][#citekey]` (case insensitive)
+
+- LaTeX output: `[#citekey]` → `\citep{citekey}`,
+
+  `[#citekey;]` → `\citet{citekey}`
+
+**Key Rules**:
+
+- Citation key format is flexible (e.g., `Doe:2006`)
+- Must be preceded by `#` (like footnotes use `^`)
+- Reference descriptions can contain Markdown
+
+### 3. mmark Syntax (RFC/I-D focused)
+
+**Format**: `[@RFC2535]` or `[@!RFC2535]` (normative) or
+`[@-RFC1000]` (suppressed)
+
+**Features**:
+
+- Automatic reference generation for RFC, BCP, STD, I-D, W3C
+
+  documents
+
+Modifiers: `?` (informative, default), `!` (normative), `-` (suppressed)
+
+- Multiple citations: `[@RFC1034;@RFC1035]`
+- Combined references: `[@RFCxxx@STDyy]` (RFC of STD)
+- I-D sequence numbers: `[@?I-D.blah#06]`
+- Bibliography section created automatically if
+
+  `{backmatter}` present
+
+**Key Rules**:
+
+- First seen modifier determines type (suppressed,
+
+  normative, informative)
+
+- Auto-detection for RFC/BCP/STD/I-D/W3C prefixes
+- No validation of RFC-STD relationships
+
+## Bibliography Formats
+
+### Supported Formats
+
+1. **BibTeX** (`.bib`, `.bibtex`)
+   - Most common format
+   - LaTeX markup in fields
+   - Requires parser library
+
+2. **BibLaTeX** (`.bib`)
+   - Extended BibTeX
+   - More entry types
+   - Same parser as BibTeX
+
+3. **CSL JSON** (`.json`)
+   - Citation Style Language JSON
+
+HTML-like markup: `<i>`, `<b>`, `<sub>`, `<sup>`, `<sc>`
+
+   - Native format for citeproc
+
+4. **CSL YAML** (`.yaml`, `.yml`)
+   - CSL in YAML format
+   - Supports Markdown in fields
+   - Easy to read/write
+
+5. **RIS** (`.ris`)
+   - Research Information Systems format
+   - Common in reference managers
+   - Requires parser
+
+### Format Priority
+
+For Apex, we should prioritize:
+
+1. **CSL JSON** - Native format, easiest to process
+2. **BibTeX/BibLaTeX** - Most common, requires parser
+3. **CSL YAML** - Human-readable, good for inline references
+4. **RIS** - Lower priority, can add later
+
+## Bibliography Storage
+
+### Option 1: External Files (Recommended)
+
+**Pros**:
+
+- Standard approach (Pandoc, Quarto, etc.)
+- Reusable across documents
+- Can be managed by reference managers
+- Supports multiple formats
+
+**Cons**:
+
+- Requires file I/O
+- Path resolution needed
+
+**Implementation**:
+
+- Metadata field: `bibliography: references.bib`
+- Command-line flag: `--bibliography references.bib`
+
+Support multiple files: `bibliography: [refs1.bib, refs2.bib]`
+
+- Auto-detect format from extension
+
+### Option 2: Inline References (YAML Metadata)
+
+**Pros**:
+
+- Self-contained documents
+- No external file needed
+- Easy for simple documents
+
+**Cons**:
+
+- Not reusable
+- Verbose for large bibliographies
+- Harder to manage
+
+**Implementation**:
+
+- YAML metadata `references:` field
+- Array of CSL JSON objects
+- Can combine with external files (inline takes precedence)
+
+### Option 3: MultiMarkdown Reference Definitions
+
+**Pros**:
+
+- Native to MultiMarkdown syntax
+- Inline with document
+- Simple format
+
+**Cons**:
+
+- Limited formatting
+- Not standardized
+- Hard to convert to other formats
+
+**Implementation**:
+
+- Parse `[#key]: ...` definitions
+- Convert to internal format
+- Generate bibliography from definitions
+
+## Citation Processing Architecture
+
+### Phase 1: Parsing
+
+1. **Preprocessing**: Extract citations from markdown
+   - Scan for citation patterns (mode-aware)
+   - Extract citation keys, locators, prefixes, suffixes
+   - Store in citation registry
+
+2. **Syntax Detection** (unified mode):
+   - Try Pandoc syntax first (`[@key]`)
+   - Fall back to MultiMarkdown (`[#key]`)
+   - Fall back to mmark (`[@RFC1234]`)
+   - Or be explicit: detect based on `@` vs `#` prefix
+
+### Phase 2: Bibliography Loading
+
+1. **Load bibliography files**:
+   - From metadata `bibliography` field
+   - From command-line `--bibliography` flag
+   - From inline YAML `references` field
+   - From MultiMarkdown reference definitions
+
+2. **Parse bibliography format**:
+   - Auto-detect from extension (`.bib`, `.json`, `.yaml`,
+
+     `.ris`)
+
+   - Parse to internal CSL JSON format
+   - Merge multiple sources (inline takes precedence)
+
+3. **Build citation index**:
+   - Map citation keys to bibliography entries
+   - Handle missing citations (warn or ignore)
+
+### Phase 3: Citation Rendering
+
+1. **Citation Style Processing**:
+   - Load CSL style file (default: Chicago author-date)
+   - Process citations through citeproc engine
+   - Generate formatted citation text
+
+2. **HTML Generation**:
+   - Replace citation markers with HTML
+   - Add links to bibliography (if `link-citations: true`)
+   - Add tooltips (if `show-tooltips: true`)
+   - Apply CSS classes
+
+### Phase 4: Bibliography Generation
+
+1. **Collect cited items**:
+   - Gather all unique citations from document
+   - Include `nocite` items (uncited but in bibliography)
+   - Sort by citation order or style rules
+
+2. **Format bibliography**:
+   - Apply CSL style formatting
+   - Generate HTML for each entry
+   - Add section heading (if not suppressed)
+
+3. **Insert bibliography**:
+   - At `<!-- REFERENCES -->` marker (if present)
+   - At `{backmatter}` marker (mmark style)
+   - At end of document (default)
+   - In `{#refs}` div (if present)
+
+## Implementation Plan
+
+### Step 1: Core Infrastructure
+
+**Files to create**:
+
+- `src/extensions/citations.h` - Header file
+- `src/extensions/citations.c` - Main implementation
+- `src/extensions/bibliography.h` - Bibliography handling
+
+`src/extensions/bibliography.c` - Bibliography parsing/formatting
+
+**Dependencies**:
+
+- CSL processor (citeproc-js equivalent in C, or use
+
+  existing library)
+
+- BibTeX parser (or use existing library like bibtexparser)
+- JSON parser (cJSON or similar, may already be available)
+
+**Core structures**:
+```c
+typedef struct {
+    char *key;
+    char *prefix;
+    char *locator;
+    char *suffix;
+    bool author_suppressed;
+    bool author_in_text;
+    int syntax_type;  // PANDOC, MMD, MMARK
+} apex_citation;
+
+typedef struct {
+    char *id;
+    char *type;  // article, book, etc.
+    // ... CSL JSON fields
+    char *raw_json;  // Original CSL JSON
+} apex_bibliography_entry;
+
+```
+
+### Step 2: Citation Parsing
+
+**Preprocessing function**:
+
+- `apex_parse_citations()` - Scan document for citations
+- Mode-aware: enable syntaxes based on mode
+- Unified mode: try all syntaxes, detect automatically
+
+**Citation extraction**:
+
+- Regex patterns for each syntax
+- Extract key, locator, prefix, suffix
+- Handle edge cases (nested brackets, special characters)
+
+### Step 3: Bibliography Loading
+
+**File loading**:
+
+- `apex_load_bibliography()` - Load from file(s)
+- Auto-detect format
+- Parse to internal format
+- Merge multiple sources
+
+**Format parsers**:
+
+- `apex_parse_bibtex()` - BibTeX parser
+- `apex_parse_csl_json()` - CSL JSON parser
+- `apex_parse_csl_yaml()` - CSL YAML parser
+- `apex_parse_ris()` - RIS parser (optional)
+
+**Metadata integration**:
+
+- Read `bibliography` from metadata
+- Read `references` from YAML metadata
+- Support command-line `--bibliography` flag
+
+### Step 4: Citation Processing
+
+**CSL integration**:
+
+- Use citeproc library (C port or bindings)
+- Load CSL style file
+- Process citations with style
+- Generate formatted output
+
+**Citation rendering**:
+
+- Replace citation markers with HTML
+- Add CSS classes for styling
+- Support link-citations option
+- Support tooltips option
+
+### Step 5: Bibliography Generation
+
+**Bibliography formatting**:
+
+- Apply CSL style to entries
+- Sort by citation order or style rules
+- Generate HTML for each entry
+
+**Bibliography insertion**:
+
+- Find insertion point (`<!-- REFERENCES -->`,
+
+  `{backmatter}`, `{#refs}`, or end)
+
+- Insert formatted bibliography
+- Add section heading if needed
+
+### Step 6: Integration
+
+**Extension registration**:
+
+- Add to `apex_register_extensions()` in `apex.c`
+- Enable based on mode (unified, mmd modes)
+- Add feature flag: `enable_citations`
+
+**CLI integration**:
+
+- Add `--bibliography` flag
+- Add `--csl` flag (citation style)
+- Add `--no-bibliography` flag (suppress)
+- Add `--link-citations` flag
+- Add `--show-tooltips` flag
+
+**Metadata integration**:
+
+- Support `bibliography` metadata field
+- Support `references` metadata field
+- Support `csl` metadata field
+- Support `nocite` metadata field
+- Support `suppress-bibliography` metadata field
+
+## Output Format
+
+### Citation HTML
+
+**Author-date style** (default):
+```html
+<span class="citation" data-cites="doe99">
+  (Doe 1999)
+</span>
+
+```
+
+**Numeric style**:
+```html
+<span class="citation" data-cites="doe99">
+  [1]
+</span>
+
+```
+
+**With links**:
+```html
+<a href="#ref-doe99" class="citation" data-cites="doe99">
+  (Doe 1999)
+</a>
+
+```
+
+**With tooltips**:
+```html
+<a href="#ref-doe99" class="citation" data-cites="doe99" title="Doe, John. 1999. 'Article Title.' Journal Name 1: 1-10.">
+  (Doe 1999)
+</a>
+
+```
+
+### Bibliography HTML
+
+```html
+<div id="refs" class="references csl-bib-body">
+  <div id="ref-doe99" class="csl-entry">
+    Doe, John. 1999. "Article Title." <i>Journal Name</i> 1: 1-10.
+  </div>
+  <div id="ref-smith2000" class="csl-entry">
+    Smith, Jane. 2000. <i>Book Title</i>. Publisher.
+  </div>
+</div>
+
+```
+
+## Dependencies
+
+### Required Libraries
+
+1. **CSL Processor**
+
+Option A: Use citeproc-js (via Node.js bindings) - complex
+
+Option B: Use citeproc-rs (Rust) via FFI - medium complexity
+
+   - Option C: Implement minimal CSL processor in C - high
+
+     complexity
+
+   - Option D: Use pandoc's citeproc (Haskell) via bindings
+     - complex
+   - **Recommendation**: Start with Option C (minimal), or
+
+     use existing C library if available
+
+2. **BibTeX Parser**
+   - Option A: Use bibtexparser (Python) - requires Python
+
+Option B: Use bibtex-rs (Rust) via FFI - medium complexity
+
+   - Option C: Implement minimal BibTeX parser in C - medium
+
+     complexity
+
+   - **Recommendation**: Option C (implement minimal parser)
+
+3. **JSON Parser**
+   - Check if cmark-gfm or existing code has JSON support
+   - If not, use cJSON (lightweight C JSON library)
+   - **Recommendation**: Use cJSON if needed
+
+4. **YAML Parser**
+   - Apex already has YAML parsing for metadata
+   - Reuse existing YAML parser if possible
+   - **Recommendation**: Reuse existing parser
+
+### Optional Libraries
+
+- **RIS Parser**: Lower priority, can implement later
+- **CSL Style Library**: Bundle common styles (APA, Chicago,
+
+  MLA, etc.)
+
+## Testing Strategy
+
+### Unit Tests
+
+1. **Citation parsing**:
+   - Test each syntax (Pandoc, MMD, mmark)
+   - Test edge cases (nested brackets, special characters)
+   - Test mode detection
+
+2. **Bibliography parsing**:
+   - Test each format (BibTeX, CSL JSON, CSL YAML)
+   - Test multiple files
+   - Test inline references
+   - Test error handling (missing files, invalid format)
+
+3. **Citation rendering**:
+   - Test different styles (author-date, numeric, note)
+   - Test with/without links
+   - Test with/without tooltips
+   - Test multiple citations
+
+4. **Bibliography generation**:
+   - Test sorting
+   - Test formatting
+   - Test insertion points
+   - Test suppression
+
+### Integration Tests
+
+1. **End-to-end**:
+   - Full document with citations
+   - Multiple bibliography sources
+   - Different citation styles
+   - Different output formats
+
+2. **Mode compatibility**:
+
+Test in each mode (commonmark, gfm, mmd, kramdown, unified)
+
+   - Test syntax detection in unified mode
+   - Test feature flags
+
+## Configuration Options
+
+### Metadata Fields
+
+```yaml
+---
+bibliography: references.bib
+# or
+bibliography:
+  - refs1.bib
+  - refs2.bib
+
+csl: apa.csl  # Citation style
+suppress-bibliography: false
+link-citations: true
+show-tooltips: false
+nocite: |
+  @item1, @item2
+# or
+nocite: '@*'  # All items
+
+references:  # Inline references (CSL JSON)
+  - id: doe99
+    type: article-journal
+    author:
+      - family: Doe
+        given: John
+    issued:
+      date-parts: [[1999]]
+    title: Article Title
+    container-title: Journal Name
+    volume: 1
+    page: 1-10
+---
+
+```
+
+### Command-Line Flags
+
+```bash
+apex input.md --bibliography refs.bib
+apex input.md --bibliography refs1.bib --bibliography refs2.bib
+apex input.md --csl apa.csl
+apex input.md --no-bibliography
+apex input.md --link-citations
+apex input.md --show-tooltips
+
+```
+
+### API Options
+
+```c
+typedef struct {
+    // ... existing options ...
+
+    // Citation options
+    bool enable_citations;
+    char **bibliography_files;  // NULL-terminated array
+    const char *csl_file;
+    bool suppress_bibliography;
+    bool link_citations;
+    bool show_tooltips;
+    const char *nocite;  // Comma-separated citation keys, or "*" for all
+} apex_options;
+
+```
+
+## Mode-Specific Behavior
+
+### CommonMark Mode
+- **Citations**: Disabled (pure CommonMark)
+- **Syntax**: None
+
+### GFM Mode
+- **Citations**: Disabled (GFM doesn't support citations)
+- **Syntax**: None
+
+### MultiMarkdown Mode
+- **Citations**: Enabled
+- **Syntax**: MultiMarkdown syntax only (`[#key]`)
+- **Bibliography**: From reference definitions or external
+
+  file
+
+### Kramdown Mode
+- **Citations**: Disabled (Kramdown uses different citation
+
+  system)
+
+- **Syntax**: None (or could add Kramdown citation syntax
+
+  later)
+
+### Unified Mode
+- **Citations**: Enabled
+- **Syntax**: All syntaxes (Pandoc, MultiMarkdown, mmark)
+- **Auto-detection**: Try Pandoc first, then MMD, then mmark
+- **Bibliography**: All sources (external files, inline
+
+  YAML, MMD definitions)
+
+## Future Enhancements
+
+1. **LaTeX Output**: Support `\citep` and `\citet` for LaTeX
+
+   rendering
+
+2. **More Formats**: RIS, EndNote, RefWorks
+3. **Citation Styles**: Bundle more CSL styles
+4. **Interactive Citations**: JavaScript for hover/click
+
+   interactions
+
+5. **Citation Validation**: Warn about missing or duplicate
+
+   citations
+
+6. **Citation Analytics**: Count citations, find most cited
+
+   items
+
+7. **Export**: Export bibliography to different formats
+
+## Implementation Phases
+
+### Phase 1: Basic Citation Parsing (MVP)
+- Parse Pandoc syntax `[@key]`
+
+Parse citation keys only (no locators/prefixes/suffixes yet)
+
+- Store citations in registry
+- Basic HTML output (just key as text)
+
+### Phase 2: Bibliography Loading
+- Load CSL JSON files
+- Parse to internal format
+- Match citations to bibliography entries
+- Generate basic bibliography HTML
+
+### Phase 3: Citation Formatting
+- Integrate CSL processor (or minimal implementation)
+- Format citations with style
+- Generate proper HTML output
+
+### Phase 4: Advanced Features
+- Support locators, prefixes, suffixes
+- Support multiple citation syntaxes
+- Support multiple bibliography formats
+- Support citation links and tooltips
+
+### Phase 5: Polish
+- Error handling
+- Performance optimization
+- Comprehensive testing
+- Documentation
+
+## Open Questions
+
+1. **CSL Processor**: Which library to use? Implement
+
+   minimal version or use existing?
+
+2. **BibTeX Parser**: Implement or use library?
+3. **Performance**: How to handle large bibliographies
+
+   (1000+ entries)?
+
+4. **Caching**: Cache parsed bibliographies?
+5. **Validation**: How strict should citation validation be?
+6. **Error Handling**: How to handle missing citations?
+
+   Warn, error, or ignore?
+
+7. **Backwards Compatibility**: How to ensure existing
+
+   documents still work?
+
+## References
+
+[Pandoc Citations](https://pandoc.org/demo/example33/8.20-citation-syntax.html)
+
+[MultiMarkdown Citations](https://fletcher.github.io/MultiMarkdown-6/Documentation/bibliography.html)
+
+- [mmark Citations](https://mmark.miek.nl/post/citations/)
+- [CSL Project](https://citationstyles.org/)
+
+[Citation Style Language](https://docs.citationstyles.org/en/stable/)
+
+[rehype-citation](https://unifiedjs.com/explore/package/rehype-citation) - JavaScript implementation reference