jekyll-client-search 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: b07a5519cf45012ce0d198cedfe93e1e93e939ed9d9221386af87c10854f0cf4
+  data.tar.gz: 630a3dd0f345e4e263c8228ac3a565610b435723860dac1125e86bf780d072b1
+SHA512:
+  metadata.gz: 0d53f70fa2e1f37facfebe56ebf3384f3d5c376672e08f93cc5a3a761b03a294843b0cf236cbda9ffa7a7ace87261683a525f3a5e12e9da61592946cc878024a
+  data.tar.gz: 458dc19c9c0ad2b6c017057694b1af2103d918f16584205ec7c144a5e696fcb0d5609508fa414a7cb91facccf39a75dcd0c1c3e8c8a8e67bac8d1a09665419b5

data/CHANGELOG.md

@@ -0,0 +1,121 @@
+# Changelog
+
+## 0.1.0 — 2026-08-25
+
+- Add a pluggable search engine architecture with a base runtime and adapters.
+- Support `minisearch`, `elasticlunr`, and `semantic` engines via `engine` config.
+- Split the browser runtime into `client-search-base.js` (engine-agnostic shell
+  that owns the two-stage search strategy) and per-engine adapters under
+  `assets/adapters/` that act as pure translators.
+- Add ElasticLunr adapter that translates the uniform query model into
+  ElasticLunr's native API.
+- Add semantic adapter that ranks documents by cosine similarity against
+  pre-computed embeddings, supports async/typed-array query vectors, caches
+  query embeddings, and ignores stale asynchronous results.
+- Add packaged query embedders for semantic search: transformers.js running
+  entirely in the browser (default) and an Ollama-compatible remote HTTP API.
+  Generate browser embedder configuration from `_config.yml`, support fully
+  self-hosted library/model/WASM assets, and apply model-specific asymmetric
+  document/query prefixes for EmbeddingGemma and Nomic.
+- Pin transformers.js, run browser model inference in a Web Worker, add
+  loading progress, retry/timeout controls, query truncation, strict finite
+  vector/dimension validation, and main-thread fallback.
+- Add optional engine-aware live search with configurable minimum length,
+  lexical/semantic debounce, URL synchronization, stale-result protection,
+  and cancellation of obsolete remote API requests.
+- Add optional build-time related-article analysis combining shared tags,
+  exact/parent categories, and vector similarity above a configurable cutoff.
+  Write relations to a separate JSON artifact and provide a lightweight
+  browser renderer with relevance/newest sorting; search results can also be
+  sorted by relevance or publication date, and lexical indexes can omit
+  temporary analysis embeddings.
+- Add `{% related_articles %}` Liquid tag for one-line adoption in any post
+  layout. Supports `sort:date` and `no_scripts` parameters; renders nothing
+  when related is disabled.
+- Add `{% search_form %}` Liquid tag for config-driven search form + scripts.
+  Outputs the form HTML, status/results containers, and all runtime scripts
+  (engine library, runtime config, embedder config, query embedder, base
+  runtime, adapter) in the correct order based on `_config.yml`. Supports
+  `scripts_only` and `no_scripts` modes. Changing engines requires zero
+  template changes.
+- Enable live search by default for lexical engines (MiniSearch, ElasticLunr)
+  and disable by default for semantic (to avoid embedding every keystroke).
+  Form submission always remains available. Set `live_search.enabled: false`
+  to restore submit-only behavior.
+- Add `engine_url`, `engine_sri`, and `engine_crossorigin` config options
+  with per-engine CDN defaults (MiniSearch 7.2.0, ElasticLunr 0.9.5).
+  Override to self-host or pin a version; set `engine_sri` for Subresource
+  Integrity.
+- Add rake tasks `jekyll_client_search:reference_files` (list, diff, and
+  status of reference files) and `jekyll_client_search:install` (copy
+  reference layouts/includes into the consuming site, with overwrite
+  protection for modified copies).
+- Ship reference `_includes/related-articles.html` and
+  `_layouts/post-with-related.html` for copy-paste adoption.
+- Extend `client-search-related.js` with `renderItem` callback for custom
+  list item rendering, `filter` callback for narrowing visible relations,
+  richer default rendering (date, shared tags, shared categories, excerpt),
+  and `data-related-sort` attribute support for the Liquid tag.
+- Add fixture site demo page (`spec/fixtures/site/related-test.html`) and
+  post layout (`spec/fixtures/site/_layouts/post.html`) exercising all
+  related-articles adoption paths. Add system tests verifying relations file
+  structure, runtime asset copying, Liquid tag rendering, demo page content,
+  self-exclusion, and score sorting.
+- Add embedding generation at build time via a local Ollama server using
+  the `ollama-ruby` gem (optional dependency, lazy-loaded).
+- Add incremental indexing via an atomically written content-hash cache
+  (`.jekyll-client-search-cache.json`) — unchanged documents reuse cached
+  embeddings only when the provider, model, endpoint, and schema match.
+- Add configurable Ollama connection/read timeouts and fail semantic builds by
+  default when an embedding cannot be generated.
+- Report browser query-embedding failures as unavailable instead of presenting
+  them as valid zero-result searches.
+- Load only `cgi/escape` for Ruby 3.5 compatibility.
+- Add uniform JS test suite parameterized over all adapters — the same
+  assertions run against each engine.
+- Add Ruby system tests that build an 80-post fixture of 40 Wikipedia articles
+  and 40 unique arXiv papers per engine, plus JS system tests for baseline
+  results (AND, OR fallback, cross-domain) in jsdom.
+- Add Ollama integration tests that generate real embeddings via a local
+  Ollama server with `embeddinggemma:300m` and verify cache reuse.
+- Add RuboCop, ESLint, SimpleCov thresholds, and Ruby/JavaScript dependency
+  audits to CI.
+- Add reproducible performance history for cold, warm, incremental, index-size,
+  query-embedding, and engine-core search latency measurements.
+- Correct fixture documentation and make the Wikipedia/arXiv regeneration
+  scripts remove dated outputs correctly; arXiv regeneration now deduplicates
+  paper IDs across category queries and records the actual per-paper license.
+- Add `wikipedia_oldid` and permanent revision links to all Wikipedia fixture
+  posts so the exact downloaded version can be referenced as articles change.
+- Remove arXiv fixture posts from the repository due to mixed/restrictive
+  licenses (some papers use CC BY-NC, CC BY-NC-ND, or arXiv non-exclusive
+  licenses that do not clearly permit redistribution). Developers run
+  `download_arxiv.rb` locally to fetch them. The committed baseline JSON
+  and semantic embeddings remain as stable test artifacts. See
+  README.developer.md and NOTICE for details.
+- Add README.developer.md with fixture setup and LLM/vector testing
+  instructions, and a NOTICE file with source attribution.
+- Make JS system, gold, comparison, and meta tests consume required committed
+  baseline fixtures directly, eliminating order-dependent test setup and skips.
+- Preserve `#` as the safe link fallback for incomplete index records.
+- Document `jekyll-pagefind` as a recommended alternative for users who
+  want Pagefind's HTML-crawling indexing approach.
+- Add a safe Jekyll generator for MiniSearch JSON indexes.
+- Support posts and configurable custom collections.
+- Optionally include titled Jekyll pages.
+- Include searchable title, excerpt, content, categories, and tags fields.
+- Copy the browser runtime asset into the consuming site's assets directory.
+- Add RSpec coverage for configuration and document normalization.
+- Add Gundestrup.dk as the local integration test platform.
+- Validate configuration and reject unsafe output paths.
+- Deduplicate indexed documents and normalize searchable metadata.
+- Harden browser result rendering with DOM APIs and same-origin URLs.
+- Add runtime asset, generated page, page indexing, disabled mode, and edge-case tests.
+- Add browser runtime behavior and URL-safety tests with Node.js and jsdom.
+- Standardize development on Ruby 3.4.10 and test supported Ruby 3.2, 3.3,
+  and 3.4 versions in CI with Bundler 4.0.9.
+- License the gem under AGPL-3.0-or-later.
+- Rename gem from `jekyll-elasticlunr-search` to `jekyll-client-search`.
+- Replace Elasticlunr.js browser runtime with MiniSearch 7.2.0.
+- Implement two-stage search: exact AND with prefix, then fuzzy OR fallback.
+- Change configuration key from `elasticlunr_search` to `client_search`.

data/LICENSE

@@ -0,0 +1,22 @@
+GNU AFFERO GENERAL PUBLIC LICENSE
+Version 3, 19 November 2007
+
+Copyright (C) 2026 Svend Gundestrup
+
+This project is free software: you can redistribute it and/or modify it under
+the terms of the GNU Affero General Public License as published by the Free
+Software Foundation, either version 3 of the License, or (at your option) any
+later version.
+
+This project is distributed in the hope that it will be useful, but WITHOUT
+ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS
+FOR A PARTICULAR PURPOSE. See the GNU Affero General Public License for more
+details.
+
+You should have received a copy of the GNU Affero General Public License along
+with this project. If not, see <https://www.gnu.org/licenses/agpl-3.0.html>.
+
+The complete license text is available from the Free Software Foundation at:
+https://www.gnu.org/licenses/agpl-3.0.txt
+
+SPDX-License-Identifier: AGPL-3.0-or-later

data/NOTICE

@@ -0,0 +1,49 @@
+jekyll-client-search — Test Fixture Attribution
+================================================
+
+This repository includes test fixture posts derived from publicly available
+sources. Each fixture post records its source, license, and a permanent link
+in its YAML frontmatter.
+
+Wikipedia articles (40 posts)
+-----------------------------
+Location: spec/fixtures/site/_posts/*-wikipedia-*.md
+Source:   Wikipedia — The Free Encyclopedia
+License:  CC BY-SA 3.0 (https://creativecommons.org/licenses/by-sa/3.0/)
+Link:     Each post has a `wikipedia_oldid` field and a `source_url` pointing
+          to a permanent revision link (https://en.wikipedia.org/w/index.php?oldid=XXXX)
+          so the exact downloaded version can be referenced even as Wikipedia
+          articles change over time.
+
+The download script (spec/fixtures/download_wikipedia.rb) is kept for
+reference and regeneration but does not need to be re-run — the Wikipedia
+fixtures are committed under their CC BY-SA 3.0 license with attribution.
+
+arXiv papers (40 posts — NOT committed)
+---------------------------------------
+Location: spec/fixtures/site/_posts/*-arxiv-*.md (gitignored)
+Source:   arXiv.org
+License:  Varies per paper — arXiv non-exclusive license, CC BY 4.0,
+          CC BY-NC-SA 4.0, or CC BY-NC-ND 4.0. Each paper's actual license
+          is recorded in its frontmatter by the download script.
+
+arXiv fixture posts are NOT committed to this repository because some
+papers use licenses that restrict redistribution (non-commercial, no
+derivatives, or arXiv-only distribution rights). Developers who need the
+full 80-post fixture set must run the download script locally:
+
+    ruby spec/fixtures/download_arxiv.rb
+
+See README.developer.md for details.
+
+Committed test artifacts
+------------------------
+The following committed JSON files are generated test artifacts derived
+from the fixture posts. They contain indexed/search-processed data (titles,
+excerpts, tokenized content, embedding vectors), not the raw article text:
+
+  spec/fixtures/baseline/search-index-baseline.json
+  spec/fixtures/baseline/semantic-embeddings.json
+
+These are kept as stable gold-standard fixtures so that tests can run
+without requiring developers to download the arXiv papers or run Ollama.

data/README.developer.md

@@ -0,0 +1,204 @@
+# Developer Guide — jekyll-client-search
+
+This guide explains how to set up the full test fixture set for local
+development. Most tests run without any special setup, but a few require
+fixture data that is not committed to the repository for licensing reasons.
+See [README.md](README.md) for user-facing configuration and usage docs.
+
+## Quick start
+
+```bash
+rbenv install 3.4.10       # if not already installed
+rbenv local 3.4.10
+bundle install
+npm ci
+bundle exec rspec          # Ruby tests (176 examples; 7 opt-in Ollama examples pending)
+npm test                   # JavaScript tests (130 tests)
+```
+
+Both test suites pass with only the committed Wikipedia fixtures and the
+committed baseline JSON artifacts. No network access or external services
+are required for the default test run.
+
+## Build-time related articles
+
+The optional related-analysis pass writes `search-relations.json` separately
+from the normal search index. It combines exact shared tags/categories and
+hierarchical category parents with vector similarity when embeddings are
+available. Semantic relations use a configurable similarity cutoff; there is
+no default five-item limit. `client-search-related.js` renders the relations
+on an article page without loading an embedding model.
+
+For a MiniSearch site, set `embedding.include_in_index: false` (the default
+when the selected engine is not semantic) to use Ollama vectors during the
+build and omit them from the public search index.
+
+### Adoption paths
+
+Three adoption paths are documented in [README.md](README.md#related-articles):
+
+1. **`{% related_articles %}` Liquid tag** — one-line in any post layout
+2. **`{% include related-articles.html %}`** — reference include file shipped
+   at `assets/includes/related-articles.html`
+3. **Drop-in layout** — reference post layout shipped at
+   `assets/layouts/post-with-related.html`
+
+The JS helper (`assets/client-search-related.js`) also supports `renderItem`
+and `filter` callbacks for custom rendering and filtering.
+
+### Fixture site demo
+
+The fixture site (`spec/fixtures/site/`) has related articles enabled in its
+`_config.yml`. It includes:
+
+- `_layouts/post.html` — minimal post layout using `{% related_articles %}`
+- `related-test.html` — demo page at `/related-test/` exercising all five
+  variants: default Liquid tag, `sort:date` + `no_scripts`, custom
+  `renderItem`, `filter`, and raw JSON link
+
+System tests in `spec/system_spec.rb` verify the relations file structure,
+runtime asset copying, Liquid tag rendering in post pages, demo page
+content, self-exclusion, and score sorting.
+
+### Related test files
+
+| File | Description |
+| --- | --- |
+| `spec/related_analyzer_spec.rb` | Unit tests for the relation analyzer |
+| `spec/related_tag_spec.rb` | Unit tests for the `{% related_articles %}` Liquid tag |
+| `spec/search_tag_spec.rb` | Unit tests for the `{% search_form %}` Liquid tag |
+| `spec/tasks_spec.rb` | Unit tests for the rake tasks (reference_files, install) |
+| `spec/system_spec.rb` | System tests building the fixture site with related and search_form |
+| `test/related.test.js` | JS tests for the related renderer (default, renderItem, filter, sort) |
+
+## What is committed
+
+| Artifact | Committed? | Why |
+|---|---|---|
+| Wikipedia fixture posts (40) | Yes | CC BY-SA 3.0 allows redistribution with attribution |
+| arXiv fixture posts (40) | **No** | Mixed/restrictive licenses — see below |
+| Baseline search-index JSON | Yes | Generated test artifact (indexed data, not raw text) |
+| Semantic embeddings JSON | Yes | Generated test artifact (embedding vectors) |
+| Download scripts | Yes | Kept for reference and regeneration |
+
+## When you need the arXiv fixtures
+
+The arXiv fixture posts are needed only for:
+
+1. **Ruby system tests that build the fixture site** — without arXiv posts,
+   these tests run with 40 Wikipedia posts instead of 80 and skip the
+   baseline comparison (the baseline was generated from the full 80-post
+   set).
+
+2. **Ollama integration tests** (`OLLAMA_INTEGRATION=1`) — these build the
+   full 80-post index with real embeddings and require both Wikipedia and
+   arXiv posts.
+
+3. **Regenerating the baseline JSON** — `ruby spec/fixtures/generate_baseline.rb`
+   requires all 80 posts.
+
+### Downloading arXiv fixtures
+
+```bash
+ruby spec/fixtures/download_arxiv.rb
+```
+
+This script:
+- Fetches 40 recent arXiv papers from 5 CS/AI subfields
+- Downloads each PDF and extracts text with `pdftotext`
+- Records the actual per-paper license in each post's frontmatter
+- Writes posts to `spec/fixtures/site/_posts/`
+- Requires `pdftotext` (available via `brew install poppler` on macOS)
+
+The arXiv API returns versioned papers (e.g. `2608.23419v1`). Running the
+script at different times will return different papers, but a specific
+version is permanent — arXiv never changes a published version.
+
+### Downloading Wikipedia fixtures (optional)
+
+The Wikipedia fixtures are already committed. You only need this script if
+you want to regenerate them (e.g. to get newer article versions):
+
+```bash
+ruby spec/fixtures/download_wikipedia.rb
+```
+
+Each Wikipedia post includes a `wikipedia_oldid` field with a permanent
+link to the exact revision that was downloaded.
+
+## LLM / vector search testing
+
+### Without Ollama (default)
+
+Semantic search tests run without a local Ollama server. They use committed
+real-model embedding vectors (`spec/fixtures/baseline/semantic-embeddings.json`)
+injected into the committed baseline index. This covers the browser-side
+cosine similarity adapter and cross-engine comparison tests.
+
+### With Ollama (opt-in integration tests)
+
+To run the full integration tests that call a real Ollama server:
+
+1. Install [Ollama](https://ollama.ai/)
+
+2. Pull the required model:
+
+   ```bash
+   ollama pull embeddinggemma:300m
+   ```
+
+3. Run the integration tests:
+
+   ```bash
+   OLLAMA_INTEGRATION=1 bundle exec rspec spec/ollama_integration_spec.rb
+   OLLAMA_INTEGRATION=1 bundle exec rspec spec/llm_injection_spec.rb
+   ```
+
+These tests:
+- Build the full 80-post index with real Ollama embeddings
+- Verify embedding dimensions and validity
+- Test semantic search quality with concept queries
+- Generate the semantic index used by JS comparison tests
+- Require arXiv fixture posts to be downloaded first
+
+### Regenerating committed test artifacts
+
+If you need to regenerate the baseline or semantic gold fixtures (e.g.
+after changing the document builder or fixture content):
+
+```bash
+# 1. Ensure all 80 fixture posts are present
+ruby spec/fixtures/download_arxiv.rb
+
+# 2. Regenerate the baseline index (without embeddings)
+ruby spec/fixtures/generate_baseline.rb
+
+# 3. Run the Ollama integration test to generate the semantic index
+OLLAMA_INTEGRATION=1 bundle exec rspec spec/ollama_integration_spec.rb
+
+# 4. Regenerate the semantic gold fixture from the built index
+ruby spec/fixtures/generate_semantic_gold.rb
+```
+
+## Test categories
+
+| Tag | Description | Requires arXiv? | Requires Ollama? |
+|---|---|---|---|
+| `:unit` | Ruby unit tests | No | No |
+| `:system` | Ruby system tests (build fixture site) | Partially | No |
+| `:integration` | Ruby integration tests | No | No |
+| `:ollama_integration` | Real Ollama server tests | Yes | Yes |
+
+Run a specific category:
+
+```bash
+bundle exec rspec --tag unit
+bundle exec rspec --tag system
+bundle exec rspec --tag ollama_integration
+```
+
+## CI
+
+The CI pipeline runs `bundle exec rspec` and `npm test` without arXiv
+fixtures or Ollama. The committed baseline JSON and semantic embeddings
+ensure all tests pass in CI without external dependencies.