apex-ruby 1.0.30 → 1.0.31

data/ext/apex_ext/apex_src/docs/CITATIONS_PLAN.md

@@ -1,744 +0,0 @@
-# Citations and Bibliography Implementation Plan for Apex
-
-## Overview
-
-This document outlines a plan for implementing citation and
-bibliography support in Apex, supporting multiple syntaxes
-(Pandoc, MultiMarkdown, mmark) and bibliography formats
-(BibTeX, BibLaTeX, CSL JSON, CSL YAML).
-
-## Goals
-
-1. Support multiple citation syntaxes (Pandoc `[@key]`,
-
-   MultiMarkdown `[#key]`, mmark `[@RFC1234]`)
-
-2. Support multiple bibliography formats (BibTeX, BibLaTeX,
-
-   CSL JSON, CSL YAML)
-
-3. Generate formatted bibliographies in HTML output
-4. Support citation styles via CSL (Citation Style Language)
-5. Integrate with existing metadata system
-6. Mode-aware syntax detection (unified mode supports all
-
-   syntaxes)
-
-## Citation Syntaxes
-
-### 1. Pandoc Syntax (Primary)
-
-**Format**: `[@key]` or `@key` (author-in-text)
-
-**Features**:
-
-- Multiple citations: `[@doe99; @smith2000]`
-- Prefixes: `[see @doe99, pp. 33-35]`
-- Locators: `[@doe99, p. 23]`
-- Suffixes: `[@doe99, and *passim*]`
-- Author suppression: `[-@smith04]` (when author mentioned
-
-  in text)
-
-- Author-in-text: `@smith04 says blah` (renders author name)
-- Complex locators: `[@smith{ii, A, D-Z}]`
-
-**Key Rules**:
-
-Keys must start with letter/digit/`_` and contain alphanumerics + internal punctuation (`:.#$%&-+?<>~/`)
-
-Keys with special characters must be wrapped: `[@{https://example.com/bib?name=foobar}]`
-
-- Locator terms: page, chapter, section, etc. (from CSL
-
-  locale files)
-
-### 2. MultiMarkdown Syntax
-
-**Format**: `[locator][#key]` or `[][#key]` (no locator)
-
-**Features**:
-
-- Locator before citation: `[p. 23][#Doe:2006]`
-- Empty locator: `[][#Doe:2006]`
-
-Reference definitions at end: `[#Doe:2006]: John Doe. *Some Big Fancy Book*. Vanity Press, 2006.`
-
-Inline citations: `As per Doe.[#John Doe. *A Totally Fake Book 1*. Vanity Press, 2006.]`
-
-Uncited references: `[Not cited][#citekey]` (case insensitive)
-
-- LaTeX output: `[#citekey]` → `\citep{citekey}`,
-
-  `[#citekey;]` → `\citet{citekey}`
-
-**Key Rules**:
-
-- Citation key format is flexible (e.g., `Doe:2006`)
-- Must be preceded by `#` (like footnotes use `^`)
-- Reference descriptions can contain Markdown
-
-### 3. mmark Syntax (RFC/I-D focused)
-
-**Format**: `[@RFC2535]` or `[@!RFC2535]` (normative) or
-`[@-RFC1000]` (suppressed)
-
-**Features**:
-
-- Automatic reference generation for RFC, BCP, STD, I-D, W3C
-
-  documents
-
-Modifiers: `?` (informative, default), `!` (normative), `-` (suppressed)
-
-- Multiple citations: `[@RFC1034;@RFC1035]`
-- Combined references: `[@RFCxxx@STDyy]` (RFC of STD)
-- I-D sequence numbers: `[@?I-D.blah#06]`
-- Bibliography section created automatically if
-
-  `{backmatter}` present
-
-**Key Rules**:
-
-- First seen modifier determines type (suppressed,
-
-  normative, informative)
-
-- Auto-detection for RFC/BCP/STD/I-D/W3C prefixes
-- No validation of RFC-STD relationships
-
-## Bibliography Formats
-
-### Supported Formats
-
-1. **BibTeX** (`.bib`, `.bibtex`)
-   - Most common format
-   - LaTeX markup in fields
-   - Requires parser library
-
-2. **BibLaTeX** (`.bib`)
-   - Extended BibTeX
-   - More entry types
-   - Same parser as BibTeX
-
-3. **CSL JSON** (`.json`)
-   - Citation Style Language JSON
-
-HTML-like markup: `<i>`, `<b>`, `<sub>`, `<sup>`, `<sc>`
-
-   - Native format for citeproc
-
-4. **CSL YAML** (`.yaml`, `.yml`)
-   - CSL in YAML format
-   - Supports Markdown in fields
-   - Easy to read/write
-
-5. **RIS** (`.ris`)
-   - Research Information Systems format
-   - Common in reference managers
-   - Requires parser
-
-### Format Priority
-
-For Apex, we should prioritize:
-
-1. **CSL JSON** - Native format, easiest to process
-2. **BibTeX/BibLaTeX** - Most common, requires parser
-3. **CSL YAML** - Human-readable, good for inline references
-4. **RIS** - Lower priority, can add later
-
-## Bibliography Storage
-
-### Option 1: External Files (Recommended)
-
-**Pros**:
-
-- Standard approach (Pandoc, Quarto, etc.)
-- Reusable across documents
-- Can be managed by reference managers
-- Supports multiple formats
-
-**Cons**:
-
-- Requires file I/O
-- Path resolution needed
-
-**Implementation**:
-
-- Metadata field: `bibliography: references.bib`
-- Command-line flag: `--bibliography references.bib`
-
-Support multiple files: `bibliography: [refs1.bib, refs2.bib]`
-
-- Auto-detect format from extension
-
-### Option 2: Inline References (YAML Metadata)
-
-**Pros**:
-
-- Self-contained documents
-- No external file needed
-- Easy for simple documents
-
-**Cons**:
-
-- Not reusable
-- Verbose for large bibliographies
-- Harder to manage
-
-**Implementation**:
-
-- YAML metadata `references:` field
-- Array of CSL JSON objects
-- Can combine with external files (inline takes precedence)
-
-### Option 3: MultiMarkdown Reference Definitions
-
-**Pros**:
-
-- Native to MultiMarkdown syntax
-- Inline with document
-- Simple format
-
-**Cons**:
-
-- Limited formatting
-- Not standardized
-- Hard to convert to other formats
-
-**Implementation**:
-
-- Parse `[#key]: ...` definitions
-- Convert to internal format
-- Generate bibliography from definitions
-
-## Citation Processing Architecture
-
-### Phase 1: Parsing
-
-1. **Preprocessing**: Extract citations from markdown
-   - Scan for citation patterns (mode-aware)
-   - Extract citation keys, locators, prefixes, suffixes
-   - Store in citation registry
-
-2. **Syntax Detection** (unified mode):
-   - Try Pandoc syntax first (`[@key]`)
-   - Fall back to MultiMarkdown (`[#key]`)
-   - Fall back to mmark (`[@RFC1234]`)
-   - Or be explicit: detect based on `@` vs `#` prefix
-
-### Phase 2: Bibliography Loading
-
-1. **Load bibliography files**:
-   - From metadata `bibliography` field
-   - From command-line `--bibliography` flag
-   - From inline YAML `references` field
-   - From MultiMarkdown reference definitions
-
-2. **Parse bibliography format**:
-   - Auto-detect from extension (`.bib`, `.json`, `.yaml`,
-
-     `.ris`)
-
-   - Parse to internal CSL JSON format
-   - Merge multiple sources (inline takes precedence)
-
-3. **Build citation index**:
-   - Map citation keys to bibliography entries
-   - Handle missing citations (warn or ignore)
-
-### Phase 3: Citation Rendering
-
-1. **Citation Style Processing**:
-   - Load CSL style file (default: Chicago author-date)
-   - Process citations through citeproc engine
-   - Generate formatted citation text
-
-2. **HTML Generation**:
-   - Replace citation markers with HTML
-   - Add links to bibliography (if `link-citations: true`)
-   - Add tooltips (if `show-tooltips: true`)
-   - Apply CSS classes
-
-### Phase 4: Bibliography Generation
-
-1. **Collect cited items**:
-   - Gather all unique citations from document
-   - Include `nocite` items (uncited but in bibliography)
-   - Sort by citation order or style rules
-
-2. **Format bibliography**:
-   - Apply CSL style formatting
-   - Generate HTML for each entry
-   - Add section heading (if not suppressed)
-
-3. **Insert bibliography**:
-   - At `<!-- REFERENCES -->` marker (if present)
-   - At `{backmatter}` marker (mmark style)
-   - At end of document (default)
-   - In `{#refs}` div (if present)
-
-## Implementation Plan
-
-### Step 1: Core Infrastructure
-
-**Files to create**:
-
-- `src/extensions/citations.h` - Header file
-- `src/extensions/citations.c` - Main implementation
-- `src/extensions/bibliography.h` - Bibliography handling
-
-`src/extensions/bibliography.c` - Bibliography parsing/formatting
-
-**Dependencies**:
-
-- CSL processor (citeproc-js equivalent in C, or use
-
-  existing library)
-
-- BibTeX parser (or use existing library like bibtexparser)
-- JSON parser (cJSON or similar, may already be available)
-
-**Core structures**:
-```c
-typedef struct {
-    char *key;
-    char *prefix;
-    char *locator;
-    char *suffix;
-    bool author_suppressed;
-    bool author_in_text;
-    int syntax_type;  // PANDOC, MMD, MMARK
-} apex_citation;
-
-typedef struct {
-    char *id;
-    char *type;  // article, book, etc.
-    // ... CSL JSON fields
-    char *raw_json;  // Original CSL JSON
-} apex_bibliography_entry;
-
-```
-
-### Step 2: Citation Parsing
-
-**Preprocessing function**:
-
-- `apex_parse_citations()` - Scan document for citations
-- Mode-aware: enable syntaxes based on mode
-- Unified mode: try all syntaxes, detect automatically
-
-**Citation extraction**:
-
-- Regex patterns for each syntax
-- Extract key, locator, prefix, suffix
-- Handle edge cases (nested brackets, special characters)
-
-### Step 3: Bibliography Loading
-
-**File loading**:
-
-- `apex_load_bibliography()` - Load from file(s)
-- Auto-detect format
-- Parse to internal format
-- Merge multiple sources
-
-**Format parsers**:
-
-- `apex_parse_bibtex()` - BibTeX parser
-- `apex_parse_csl_json()` - CSL JSON parser
-- `apex_parse_csl_yaml()` - CSL YAML parser
-- `apex_parse_ris()` - RIS parser (optional)
-
-**Metadata integration**:
-
-- Read `bibliography` from metadata
-- Read `references` from YAML metadata
-- Support command-line `--bibliography` flag
-
-### Step 4: Citation Processing
-
-**CSL integration**:
-
-- Use citeproc library (C port or bindings)
-- Load CSL style file
-- Process citations with style
-- Generate formatted output
-
-**Citation rendering**:
-
-- Replace citation markers with HTML
-- Add CSS classes for styling
-- Support link-citations option
-- Support tooltips option
-
-### Step 5: Bibliography Generation
-
-**Bibliography formatting**:
-
-- Apply CSL style to entries
-- Sort by citation order or style rules
-- Generate HTML for each entry
-
-**Bibliography insertion**:
-
-- Find insertion point (`<!-- REFERENCES -->`,
-
-  `{backmatter}`, `{#refs}`, or end)
-
-- Insert formatted bibliography
-- Add section heading if needed
-
-### Step 6: Integration
-
-**Extension registration**:
-
-- Add to `apex_register_extensions()` in `apex.c`
-- Enable based on mode (unified, mmd modes)
-- Add feature flag: `enable_citations`
-
-**CLI integration**:
-
-- Add `--bibliography` flag
-- Add `--csl` flag (citation style)
-- Add `--no-bibliography` flag (suppress)
-- Add `--link-citations` flag
-- Add `--show-tooltips` flag
-
-**Metadata integration**:
-
-- Support `bibliography` metadata field
-- Support `references` metadata field
-- Support `csl` metadata field
-- Support `nocite` metadata field
-- Support `suppress-bibliography` metadata field
-
-## Output Format
-
-### Citation HTML
-
-**Author-date style** (default):
-```html
-<span class="citation" data-cites="doe99">
-  (Doe 1999)
-</span>
-
-```
-
-**Numeric style**:
-```html
-<span class="citation" data-cites="doe99">
-  [1]
-</span>
-
-```
-
-**With links**:
-```html
-<a href="#ref-doe99" class="citation" data-cites="doe99">
-  (Doe 1999)
-</a>
-
-```
-
-**With tooltips**:
-```html
-<a href="#ref-doe99" class="citation" data-cites="doe99" title="Doe, John. 1999. 'Article Title.' Journal Name 1: 1-10.">
-  (Doe 1999)
-</a>
-
-```
-
-### Bibliography HTML
-
-```html
-<div id="refs" class="references csl-bib-body">
-  <div id="ref-doe99" class="csl-entry">
-    Doe, John. 1999. "Article Title." <i>Journal Name</i> 1: 1-10.
-  </div>
-  <div id="ref-smith2000" class="csl-entry">
-    Smith, Jane. 2000. <i>Book Title</i>. Publisher.
-  </div>
-</div>
-
-```
-
-## Dependencies
-
-### Required Libraries
-
-1. **CSL Processor**
-
-Option A: Use citeproc-js (via Node.js bindings) - complex
-
-Option B: Use citeproc-rs (Rust) via FFI - medium complexity
-
-   - Option C: Implement minimal CSL processor in C - high
-
-     complexity
-
-   - Option D: Use pandoc's citeproc (Haskell) via bindings
-     - complex
-   - **Recommendation**: Start with Option C (minimal), or
-
-     use existing C library if available
-
-2. **BibTeX Parser**
-   - Option A: Use bibtexparser (Python) - requires Python
-
-Option B: Use bibtex-rs (Rust) via FFI - medium complexity
-
-   - Option C: Implement minimal BibTeX parser in C - medium
-
-     complexity
-
-   - **Recommendation**: Option C (implement minimal parser)
-
-3. **JSON Parser**
-   - Check if cmark-gfm or existing code has JSON support
-   - If not, use cJSON (lightweight C JSON library)
-   - **Recommendation**: Use cJSON if needed
-
-4. **YAML Parser**
-   - Apex already has YAML parsing for metadata
-   - Reuse existing YAML parser if possible
-   - **Recommendation**: Reuse existing parser
-
-### Optional Libraries
-
-- **RIS Parser**: Lower priority, can implement later
-- **CSL Style Library**: Bundle common styles (APA, Chicago,
-
-  MLA, etc.)
-
-## Testing Strategy
-
-### Unit Tests
-
-1. **Citation parsing**:
-   - Test each syntax (Pandoc, MMD, mmark)
-   - Test edge cases (nested brackets, special characters)
-   - Test mode detection
-
-2. **Bibliography parsing**:
-   - Test each format (BibTeX, CSL JSON, CSL YAML)
-   - Test multiple files
-   - Test inline references
-   - Test error handling (missing files, invalid format)
-
-3. **Citation rendering**:
-   - Test different styles (author-date, numeric, note)
-   - Test with/without links
-   - Test with/without tooltips
-   - Test multiple citations
-
-4. **Bibliography generation**:
-   - Test sorting
-   - Test formatting
-   - Test insertion points
-   - Test suppression
-
-### Integration Tests
-
-1. **End-to-end**:
-   - Full document with citations
-   - Multiple bibliography sources
-   - Different citation styles
-   - Different output formats
-
-2. **Mode compatibility**:
-
-Test in each mode (commonmark, gfm, mmd, kramdown, unified)
-
-   - Test syntax detection in unified mode
-   - Test feature flags
-
-## Configuration Options
-
-### Metadata Fields
-
-```yaml
----
-bibliography: references.bib
-# or
-bibliography:
-  - refs1.bib
-  - refs2.bib
-
-csl: apa.csl  # Citation style
-suppress-bibliography: false
-link-citations: true
-show-tooltips: false
-nocite: |
-  @item1, @item2
-# or
-nocite: '@*'  # All items
-
-references:  # Inline references (CSL JSON)
-  - id: doe99
-    type: article-journal
-    author:
-      - family: Doe
-        given: John
-    issued:
-      date-parts: [[1999]]
-    title: Article Title
-    container-title: Journal Name
-    volume: 1
-    page: 1-10
----
-
-```
-
-### Command-Line Flags
-
-```bash
-apex input.md --bibliography refs.bib
-apex input.md --bibliography refs1.bib --bibliography refs2.bib
-apex input.md --csl apa.csl
-apex input.md --no-bibliography
-apex input.md --link-citations
-apex input.md --show-tooltips
-
-```
-
-### API Options
-
-```c
-typedef struct {
-    // ... existing options ...
-
-    // Citation options
-    bool enable_citations;
-    char **bibliography_files;  // NULL-terminated array
-    const char *csl_file;
-    bool suppress_bibliography;
-    bool link_citations;
-    bool show_tooltips;
-    const char *nocite;  // Comma-separated citation keys, or "*" for all
-} apex_options;
-
-```
-
-## Mode-Specific Behavior
-
-### CommonMark Mode
-- **Citations**: Disabled (pure CommonMark)
-- **Syntax**: None
-
-### GFM Mode
-- **Citations**: Disabled (GFM doesn't support citations)
-- **Syntax**: None
-
-### MultiMarkdown Mode
-- **Citations**: Enabled
-- **Syntax**: MultiMarkdown syntax only (`[#key]`)
-- **Bibliography**: From reference definitions or external
-
-  file
-
-### Kramdown Mode
-- **Citations**: Disabled (Kramdown uses different citation
-
-  system)
-
-- **Syntax**: None (or could add Kramdown citation syntax
-
-  later)
-
-### Unified Mode
-- **Citations**: Enabled
-- **Syntax**: All syntaxes (Pandoc, MultiMarkdown, mmark)
-- **Auto-detection**: Try Pandoc first, then MMD, then mmark
-- **Bibliography**: All sources (external files, inline
-
-  YAML, MMD definitions)
-
-## Future Enhancements
-
-1. **LaTeX Output**: Support `\citep` and `\citet` for LaTeX
-
-   rendering
-
-2. **More Formats**: RIS, EndNote, RefWorks
-3. **Citation Styles**: Bundle more CSL styles
-4. **Interactive Citations**: JavaScript for hover/click
-
-   interactions
-
-5. **Citation Validation**: Warn about missing or duplicate
-
-   citations
-
-6. **Citation Analytics**: Count citations, find most cited
-
-   items
-
-7. **Export**: Export bibliography to different formats
-
-## Implementation Phases
-
-### Phase 1: Basic Citation Parsing (MVP)
-- Parse Pandoc syntax `[@key]`
-
-Parse citation keys only (no locators/prefixes/suffixes yet)
-
-- Store citations in registry
-- Basic HTML output (just key as text)
-
-### Phase 2: Bibliography Loading
-- Load CSL JSON files
-- Parse to internal format
-- Match citations to bibliography entries
-- Generate basic bibliography HTML
-
-### Phase 3: Citation Formatting
-- Integrate CSL processor (or minimal implementation)
-- Format citations with style
-- Generate proper HTML output
-
-### Phase 4: Advanced Features
-- Support locators, prefixes, suffixes
-- Support multiple citation syntaxes
-- Support multiple bibliography formats
-- Support citation links and tooltips
-
-### Phase 5: Polish
-- Error handling
-- Performance optimization
-- Comprehensive testing
-- Documentation
-
-## Open Questions
-
-1. **CSL Processor**: Which library to use? Implement
-
-   minimal version or use existing?
-
-2. **BibTeX Parser**: Implement or use library?
-3. **Performance**: How to handle large bibliographies
-
-   (1000+ entries)?
-
-4. **Caching**: Cache parsed bibliographies?
-5. **Validation**: How strict should citation validation be?
-6. **Error Handling**: How to handle missing citations?
-
-   Warn, error, or ignore?
-
-7. **Backwards Compatibility**: How to ensure existing
-
-   documents still work?
-
-## References
-
-[Pandoc Citations](https://pandoc.org/demo/example33/8.20-citation-syntax.html)
-
-[MultiMarkdown Citations](https://fletcher.github.io/MultiMarkdown-6/Documentation/bibliography.html)
-
-- [mmark Citations](https://mmark.miek.nl/post/citations/)
-- [CSL Project](https://citationstyles.org/)
-
-[Citation Style Language](https://docs.citationstyles.org/en/stable/)
-
-[rehype-citation](https://unifiedjs.com/explore/package/rehype-citation) - JavaScript implementation reference