apex-ruby 1.0.17 → 1.0.20

data/ext/apex_ext/apex_src/docs/INDEX_IMPLEMENTATION_ANALYSIS.md

@@ -0,0 +1,382 @@
+# Index Implementation Analysis
+
+This document analyzes the implementation of index syntax
+support in Apex, covering mmark syntax, TextIndex syntax,
+and implementation considerations.
+
+## mmark Index Syntax
+
+### Syntax Overview
+
+mmark uses parentheses-based syntax for index entries:
+
+- **Standard index entry**: `(!item)` - Creates an index
+
+  entry for "item"
+
+- **Sub-item entry**: `(!item, subitem)` - Creates a
+
+  sub-entry under "item"
+
+- **Primary entry**: `(!!item, subitem)` - Marks an entry as
+
+  primary (bold in output)
+
+- **Section-level index**: An index entry can apply to an
+
+  entire section by placing it as the first paragraph after
+  a section heading
+
+### Example Usage
+
+```markdown
+# Test Document
+
+This document contains protocols (!Protocol) and implementations (!Implementation).
+
+## HTTP
+
+HTTP (!HTTP) is a protocol (!Protocol) used for web communication.
+
+### HTTP/1.1
+
+HTTP/1.1 (!HTTP, HTTP/1.1) is a common version.
+
+### HTTP/2
+
+HTTP/2 (!HTTP, HTTP/2) introduced multiplexing.
+
+## Security
+
+Security (!Security) is important. We discuss encryption (!Encryption).
+
+### Encryption Methods
+
+Symmetric encryption (!Encryption, Symmetric) uses the same key.
+
+Asymmetric encryption (!Encryption, Asymmetric) uses different keys.
+
+## Primary Index Entry
+
+This section discusses the primary topic (!!Primary Topic, Sub Topic).
+
+## Section-Level Index
+
+(!Section Index Entry)
+
+This entire section should be indexed as "Section Index Entry".
+
+```
+
+### mmark Output
+
+When processed with `mmark -html`, the syntax generates:
+
+**Inline markers**: Each index entry is converted to a `<span class="index" id="idxref:N"></span>` marker in the HTML
+
+**Index section**: At the end of the document, mmark automatically generates an `<h1 id="index-section">Index</h1>` section with a structured index:
+
+   - Alphabetically organized by first letter
+   - Hierarchical structure for sub-items
+
+Links back to document locations using `idxref:N` anchors
+
+   - Primary entries are marked (typically bold)
+
+### Implementation Notes
+
+- The `-index=false` flag suppresses index generation
+- Index entries are processed during inline parsing
+- The index is generated as a post-processing step after the
+
+  document is parsed
+
+## TextIndex Syntax
+
+### Syntax Overview
+
+TextIndex uses curly braces with a caret for index entries:
+
+- **Basic syntax**: `{^}` - Marks the preceding word or
+
+  phrase for indexing
+
+- **With explicit term**: `[term]{^}` - Indexes "term"
+
+  explicitly
+
+- **With parameters**: `{^param1 param2}` - Supports various
+
+  parameters for sorting, cross-references, etc.
+
+### Example Usage
+
+```markdown
+# Test Document
+
+Most mechanical keyboard firmware{^} supports the use of [key combinations]{^}.
+
+## Protocols
+
+HTTP{^} is a protocol{^} used for web communication.
+
+### HTTP/1.1
+
+HTTP/1.1{^} is a common version of the protocol{^}.
+
+### HTTP/2
+
+HTTP/2{^} introduced multiplexing.
+
+## Security
+
+Security{^} is important. We discuss encryption{^} and authentication{^}.
+
+### Encryption Methods
+
+Symmetric encryption{^} uses the same key for encryption and decryption.
+
+Asymmetric encryption{^} uses different keys.
+
+```
+
+### TextIndex Features
+
+From the Python implementation, TextIndex supports:
+
+1. **Concordance files**: External files that automatically
+
+   mark terms for indexing
+
+2. **Cross-references**: Link related terms within the index
+3. **Aliases**: Provide alternative terms or synonyms
+4. **Plural handling**: Manage singular and plural forms
+5. **Section mode**: Index entire sections
+6. **Sorting options**: Control how entries are sorted
+7. **Emphasis handling**: Mark entries with emphasis
+
+### TextIndex Pattern
+
+The regex pattern used by TextIndex is:
+```regex
+(?:(?<!\\)\[([^\]<>]+)(?<!\\)\]|([^\s\[\]\{\}<>]++))*(?<!>)\{\^([^\}<\n]*)\}(?!<)
+
+```
+
+This matches:
+
+- Optional `[term]` or a word/phrase
+- Followed by `{^}` with optional parameters inside
+
+## Implementation Difficulty Assessment
+
+### Complexity: Medium
+
+Implementing index support in Apex would be **moderately
+complex**, similar to the citations extension but
+potentially simpler in some ways.
+
+### Similar Existing Features
+
+Apex already has several features that demonstrate similar
+patterns:
+
+1. **Citations Extension** (`src/extensions/citations.c`):
+
+Processes inline syntax patterns (`[@key]`, `[#key]`, etc.)
+
+   - Collects citations during parsing
+   - Generates bibliography at the end
+   - Similar two-phase approach: collect during parse,
+
+     generate during render
+
+2. **Table of Contents** (`src/extensions/toc.c`):
+   - Collects headers from AST
+   - Generates TOC HTML
+   - Inserts at `<!--TOC-->` markers or end of document
+   - Similar post-processing approach
+
+3. **Wiki Links** (`src/extensions/wiki_links.c`):
+   - Processes inline syntax `[[Page Name]]`
+   - Uses inline parser matching
+   - Converts to HTML links
+
+### Implementation Approach
+
+#### Phase 1: Inline Parsing
+
+1. **Create index extension** (`src/extensions/index.c`):
+   - Register as inline syntax extension
+   - Match patterns:
+
+mmark: `(!item)`, `(!item, subitem)`, `(!!item, subitem)`
+
+     - TextIndex: `{^}`, `[term]{^}`, `{^params}`
+
+Store index entries in a registry (similar to citations)
+
+2. **Index entry structure**:
+   ```c
+   typedef struct apex_index_entry {
+       char *item;              /* Main index term */
+       char *subitem;           /* Sub-item (optional) */
+       bool primary;            /* Primary entry flag (mmark) */
+       int position;            /* Position in document */
+       char *anchor_id;         /* Generated anchor ID */
+       struct apex_index_entry *next;
+   } apex_index_entry;
+
+   ```
+
+3. **Processing**:
+   - During inline parsing, detect index syntax
+   - Extract terms and parameters
+   - Create index entries
+   - Replace syntax with invisible markers or spans (like
+
+     mmark does)
+
+#### Phase 2: Index Generation
+
+1. **Collect all entries** from the registry
+2. **Sort alphabetically** by item, then subitem
+3. **Group by first letter** (optional, like mmark)
+4. **Generate HTML**:
+   - Create hierarchical structure (items with sub-items)
+   - Link to document positions using anchor IDs
+   - Mark primary entries appropriately
+5. **Insert at end** or at `<!--INDEX-->` marker
+
+### Code Structure
+
+```
+src/extensions/
+  index.c          - Main implementation
+  index.h          - Header with structures and functions
+
+```
+
+Key functions needed:
+
+- `create_index_extension()` - Register extension
+- `apex_process_index_entries()` - Parse and collect entries
+- `apex_generate_index_html()` - Generate index HTML
+- `apex_insert_index()` - Insert index into document
+
+### Challenges
+
+1. **Syntax conflicts**:
+
+`(!item)` could conflict with other parentheses-based syntax
+
+   - `{^}` could conflict with other brace syntax
+   - Need careful pattern matching to avoid false positives
+
+2. **Position tracking**:
+   - Need to track document positions for linking
+   - Similar to how citations track positions
+
+3. **Sorting and grouping**:
+   - Alphabetical sorting with proper handling of special
+
+     characters
+
+   - Grouping by first letter (optional feature)
+
+4. **Section-level indexing**:
+
+Detecting when an index entry applies to an entire section
+
+   - Requires understanding of document structure
+
+5. **Dual syntax support**:
+   - Supporting both mmark and TextIndex syntax
+   - May need configuration option to choose syntax or
+
+     support both
+
+### Estimated Effort
+
+- **Basic mmark syntax**: 2-3 days
+- **TextIndex syntax**: 1-2 days additional
+- **Section-level indexing**: 1 day
+- **Testing and refinement**: 1-2 days
+
+**Total**: ~5-8 days of development time
+
+## Other Markdown Flavors with Index Support
+
+### MultiMarkdown
+
+MultiMarkdown has native support for index creation using
+the same syntax as mmark:
+
+- `(!item)` for standard entries
+- `(!item, subitem)` for sub-entries
+- `(!!item, subitem)` for primary entries
+
+This makes sense since mmark is based on MultiMarkdown.
+
+### Pandoc
+
+Pandoc does **not** have native index support, but:
+
+- Supports extensive customization through filters
+- Can implement indexing using Lua filters
+- Requires additional scripting or external tools
+
+### Kramdown
+
+Kramdown does **not** provide built-in support for indexes.
+Users would need:
+
+- Custom solutions
+- External tools to generate indexes
+
+### Summary
+
+Among major Markdown processors:
+
+- **MultiMarkdown/mmark**: Native support with `(!item)`
+
+  syntax
+
+- **Pandoc**: No native support, but extensible via filters
+- **Kramdown**: No native support
+- **TextIndex**: Standalone tool with `{^}` syntax
+
+## Recommendations
+
+1. **Start with mmark syntax**: It's more established and
+
+   similar to MultiMarkdown (which Apex already supports)
+
+2. **Add TextIndex syntax as optional**: Can be enabled via
+
+   flag or mode
+
+3. **Follow citations pattern**: Use similar architecture to
+
+   citations extension
+
+**Make index generation optional**: Support `--no-index` flag (like mmark's `-index=false`)
+
+5. **Support index marker**: Allow `<!--INDEX-->` marker for
+
+   custom placement
+
+6. **Consider configuration options**:
+   - Alphabetical grouping (by first letter)
+   - Index placement (end vs. marker)
+   - Primary entry styling
+
+## Test Files
+
+Test documents have been created:
+
+- `test_index_mmark.md` - mmark syntax examples
+- `test_index_textindex.md` - TextIndex syntax examples
+
+These can be used for testing once implementation is
+complete.