nimbi-cms 1.0.5 → 1.0.7

package/README.md

@@ -27,7 +27,8 @@ The code lives at [https://github.com/AbelVM/nimbiCMS](https://github.com/AbelVM
 - Client-side rendering of [GitHub‑flavored Markdown](https://github.github.com/gfm/) via [marked.js](https://marked.js.org/), no need for site compilation
 - No Jekyll, no Hugo, no CI at all
 - Perfect for static servers (GitHub Pages, S3, etc.)
-- Optional client-side search box built from headings and excerpts (enabled by default).
+- **FAST**: this site, the site of the project, contains more than 1200 Markdown documents (almost 3000 references) and it's indexed in a blink!
+- Client-side search box built from headings and excerpts (enabled by default).
 - Reading time estimation
 - Code is organized into small modules to ease maintenance and testing.
 - Sticky per-page, dynamically generated TOC.
@@ -38,6 +39,8 @@ The code lives at [https://github.com/AbelVM/nimbiCMS](https://github.com/AbelVM
 - Image preview modal with zoom controls, wheel zoom, drag pan, double-click to zoom, and touch pinch support.
 - Syntax highlighting using [highlight.js](https://highlightjs.org/) — languages are auto-registered when detected.
 - Simple theming (light/dark), Bulma and hightlight.js customization options.
+- Feeding support: [RSS 2.0](https://www.rssboard.org/rss-specification) at `/?rss` and [ATOM 1.0](https://www.rfc-editor.org/rfc/rfc4287) at  `/?atom`
+- Dynamic sitemap at `/?sitemap`
 - Simplified deliverables: regardless all the dynamic imports, the bundle is kept in one JS file and one CSS file
 - Bundle is compact size
   - JS file: 243.74 kB, gzipped 70.62 kB (for UMD bundle)
@@ -47,6 +50,17 @@ The code lives at [https://github.com/AbelVM/nimbiCMS](https://github.com/AbelVM
 - Fully typed and [documented](docs/README.md)
 - [MIT licensed](LICENSE.md)
 
+### Runtime sitemap & feeds
+
+nimbiCMS exposes client-side endpoints that generate sitemaps and feeds at runtime:
+
+- Endpoints: `/?sitemap` (also cosmetic `#/?sitemap` and path forms like `/sitemap` or `/sitemap.xml`) — returns an XML sitemap that uses canonical `?page=` URLs.
+- `/?rss` and `/?atom` (also `#/?rss`, `#/?atom`, and path forms `/rss`, `/rss.xml`, `/atom`, `/atom.xml`) — return RSS 2.0 and Atom 1.0 feeds respectively.
+
+Limitations & crawler advice
+- These endpoints are generated client-side; servers cannot set the HTTP `Content-Type` header for the generated XML. The implementation uses Blob/data URL fallbacks to present XML in browsers, but this is not guaranteed to satisfy all crawlers.
+- For reliable crawler indexing prefer a server-generated `sitemap.xml` (build-time) and list it in `robots.txt`. Use the runtime endpoints for development convenience or for crawlers that execute JavaScript.
+
 ## Installation
 
 ### From npm
@@ -154,27 +168,41 @@ Drop `.md` and/or `.html` files into your content directory. No build step is ne
 
 **Required files**
 
-- `_home.md` — required by default. You can override this with the `homePage` option to use a different `.md` or `.html` file as the home page.
-
-```javascript
-initCMS({ el: '#app', homePage: 'index.html' })
-```
-
 - `_navigation.md` — renders into the navbar; use Markdown links. Example nav markup:
 
 ```markdown
-[Home](_home.md)
+[Home](home.md)
 [Blog](blog.md)
 [About](about.md)
 ```
 
+-- `home page` — by default the library will use the first link found in `_navigation.md` when present. If no suitable link is found, the runtime will not automatically probe or fall back to `_home.md`; set the `homePage` option to explicitly select a page (for example `index.html`).
+
+```javascript
+initCMS({ el: '#app', homePage: 'index.html' })
+```
+
 - `_404.md` — optional fallback for 404 responses. When the server responds to a requested `.md` path with an HTML document (e.g., an SPA fallback serving `index.html`), the CMS treats that as a missing markdown page and will attempt to load `/_404.md` from the configured content base so a proper 404 page can be rendered instead of the site's index HTML.
 
 ```javascript
 initCMS({ el: '#app', notFoundPage: '_404.md' })
 ```
 
-Links are converted to hash‑based navigation (`?page=…`), preserving anchors and URL passed parameters
+### URL schemes — cosmetic vs canonical
+
+nimbiCMS supports two coexisting URL schemas so that user-facing navigation can be friendly while SEO remains stable:
+
+- Cosmetic (user-facing): `/#/slug[#anchor][?params]` — preferred for visible navigation, TOC links, and search results. Example: `https://example.com/#/blog/my-post#comments?lang=fr`
+- Canonical (SEO / internal): `/?page=slug[#anchor][&params]` — used for canonical `<link>` tags, internal fetches, and sitemaps so indexing stays consistent. Example: `https://example.com/?page=blog/my-post#comments&lang=fr`
+
+Behavior notes:
+
+- The UI prefers the cosmetic `/#/slug` form for links, but the library emits canonical URLs using the `?page=` form (for example in `<link rel="canonical">`) to keep SEO and indexing stable.
+- Anchors and query parameters are preserved and placed in the correct order: the anchor follows the slug, and parameters are appended after the anchor.
+- Internal fetches (including worker-based requests) use canonical `?page=` URLs so paths remain predictable on static hosts.
+- Because the cosmetic form relies on client-side routing, crawlers that do not execute JavaScript may require a server-side `sitemap.xml` or server-side routing for full indexing coverage.
+
+This behavior is covered by the test-suite (see the canonical-link and URL handling tests).
 
 ## Options
 
@@ -187,6 +215,7 @@ Links are converted to hash‑based navigation (`?page=…`), preserving anchors
 | `el` | `string` \ `Element` | required | CSS selector or DOM element used as the mount target. |
 | `contentPath` | `string` | `/content` | URL path to the content folder serving `.md`/`.html` files; normalized to a relative path with trailing slash. |
 | `allowUrlPathOverrides` | `boolean` | `false` | Opt-in: when `true`, `contentPath`, `homePage`, `notFoundPage`, and `navigationPage` may be overridden using URL params. |
+| `debugLevel` | `0 \|1 \| 2 \| 3` | `0` | Initial logging verbosity: `0`=disabled, `1`=errors, `2`=errors+warnings, `3`=all messages. |
 
 ### Indexing and Search
 
@@ -197,14 +226,16 @@ Links are converted to hash‑based navigation (`?page=…`), preserving anchors
 | `indexDepth` | `1 \| 2 \| 3` | `1` | How deep headings are indexed (H1, H2, H3). |
 | `noIndexing` | `string[]` | — | Paths (relative) to exclude from discovery and indexing. |
 | `skipRootReadme` | `boolean` | `false` | When `true`, skip link discovery inside a repository-root `README.md`. |
+| `manifest` | `Record<string,string>` | `null` | Optional in-memory content manifest mapping absolute paths (for example `/content/page.md`) to raw page text. When provided the runtime pre-seeds `allMarkdownPaths` and slug mappings and avoids network discovery; this is commonly used by build-time tooling that inlines page sources into a bundle. The init routine also honors a global `__NIMBI_CMS_MANIFEST__` if present. |
 
 ### Routing and Pages
 
 | Option | Type | Default | Description |
 |---|---:|:---:|---|
-| `homePage` | `string` | `'_home.md'` | Path for the site home page (`.md` or `.html`). |
-| `notFoundPage` | `string` | `'_404.md'` | Path for the not-found page (`.md` or `.html`). |
 | `navigationPage` | `string` | `'_navigation.md'` | Path for the navigation markdown used to build the navbar (`.md` or `.html`). |
+| `homePage` | `string` | `first link of navigationPage` | Path for the site home page (`.md` or `.html`). |
+| `notFoundPage` | `string\|null` | `null` | Path for the not-found page (`.md` or `.html`). When unset (`null`) the runtime renders a small inline "Not Found" message linking to the configured `homePage` instead of attempting to fetch `'_404.md'`. |
+| `exposeSitemap` | `boolean` | `true` | When `true` the client exposes a runtime sitemap at `/sitemap.xml` and `/sitemap.html`. The sitemap is generated at runtime (no build-time generation) and emits canonical `?page=` URLs (for example `/?page=blog/my-post`) while the UI preserves cosmetic `/#/slug` navigation. Requires SPA fallback (the server must serve `index.html` for `/sitemap*` requests). Disable with `initCMS({ exposeSitemap: false })`. For broad crawler coverage prefer generating a server-side `sitemap.xml` and listing it in `robots.txt`. You can also get an XML sitemap at `/?sitemap` endpoint. |
 
 > Note: All these files must be within `contentPath`
 
@@ -231,6 +262,8 @@ Links are converted to hash‑based navigation (`?page=…`), preserving anchors
 | `cacheTtlMinutes` | `number` | `5` | TTL for slug-resolution cache entries (minutes). Set `0` to disable expiration. |
 | `cacheMaxEntries` | `number` | — | Maximum entries in the router resolution cache. |
 | `crawlMaxQueue` | `number` | `1000` | Upper bound on directories queued during breadth-first crawl (0 disables the guard). |
+| `fetchConcurrency` | `number | 'auto'` | `'auto'` | Limits the number of simultaneous network fetches used for probing external HTML, building the search index, and slug resolution. Accepts a positive integer or `'auto'` (default). When `'auto'`, nimbiCMS derives a safe concurrency from `navigator.hardwareConcurrency` and caps it at `5`. Set to `1` to serialize requests. |
+| `negativeFetchCacheTTL` | `number` | `60000` | Milliseconds to keep negative (failed) fetch responses in cache to avoid repeated failing requests. Default `60000` (1 minute). Set to `0` to disable negative caching. |
 
 ### Advanced and Extensions
 
@@ -256,6 +289,15 @@ import { getVersion } from 'nimbi-cms'
 getVersion().then(v => console.log('nimbiCMS version', v))
 ```
 
+### Debugging
+
+- `setDebugLevel(level)` — adjust runtime logging verbosity programmatically. `level` is a number `0..3` where `0` disables messages, `1` enables errors, `2` enables errors and warnings, and `3` enables info/log messages. Example:
+
+```js
+import { setDebugLevel } from 'nimbi-cms'
+setDebugLevel(2)
+```
+
 ### Hooks (extension points)
 
 These helpers let you hook into internal rendering and navigation without forking the source.
@@ -479,6 +521,26 @@ initCMS({ el: '#app', allowUrlPathOverrides: true })
 
 > Note: enabling `allowUrlPathOverrides` still runs client-side validation; if an unsafe value is supplied the call to `initCMS()` will throw a `TypeError`. Prefer passing `contentPath`, `homePage`, and `notFoundPage` directly in the `options` object from secure script code rather than relying on URL query parameters.
 
+### Runtime sitemap (dynamic)
+
+When enabled (default `true`) nimbiCMS can expose a runtime-generated sitemap intended for crawlers and bots that execute JavaScript. The runtime sitemap is available at:
+
+- `/sitemap.xml` — XML sitemap using canonical `?page=` URLs
+- `/sitemap.html` — optional simple HTML listing (not a full UI)
+
+Key notes:
+
+- Runtime-only: the sitemap is built in the browser from discovered content paths; nothing is generated at build time.
+- Canonical URLs: the sitemap emits `?page=...` canonical URLs (for SEO and indexing) while the SPA UI keeps cosmetic `/#/slug` links for user navigation.
+- SPA fallback required: your static host must serve `index.html` for requests to `/sitemap.xml` or `/sitemap.html` so the client-side handler can render the sitemap.
+- Disabled when needed: opt out by passing `exposeSitemap: false` to `initCMS()`:
+
+```js
+initCMS({ el: '#app', exposeSitemap: false })
+```
+
+For widest crawler compatibility (search engines that don't execute JavaScript reliably) prefer generating and serving a server-side `sitemap.xml` and listing it in `robots.txt`. The runtime sitemap is a convenience for JS-aware crawlers and for environments where generating a sitemap at build time is not possible.
+
 ## Available Bulmaswatch themes
 
 The list of available Bulma themes is
@@ -492,7 +554,7 @@ Keep in mind that some themes do not play well with certain color schemas.
 
 ## Using with GitHub Pages and the GitHub file editor
 
-Nimbi CMS works well with GitHub Pages and the built-in GitHub web file editor. Minimal steps:
+nimbiCMS works well with GitHub Pages and the built-in GitHub web file editor. Minimal steps:
 
 - Enable GitHub Pages for the repository (Settings → Pages) and choose the branch/folder you want to publish (e.g., `gh-pages` or `main` / `/docs`).
 - If your content includes underscore-prefixed files (for example `_navigation.md` or `_home.md`), GitHub's Jekyll processor will ignore them by default. Add an empty `.nojekyll` file at the repository root to disable Jekyll so those files are served
@@ -504,7 +566,7 @@ Editing content via the GitHub web editor:
 1. Open the repository on [GitHub](https://github.com) and navigate to the `content/` folder (or your chosen `contentPath`).
 2. Click any `.md` file, then click the pencil icon to edit the file in the browser.
 3. Make changes and commit them directly to the branch. The published site will receive the updates on the next Pages build (or immediately if you host the `dist` on the same branch).
-4. Refresh the site to see the updated content. Nimbi CMS loads content at runtime, so browser refresh shows the latest files.
+4. Refresh the site to see the updated content. nimbiCMS loads content at runtime, so browser refresh shows the latest files.
 
 Tips:
 - Add or update `content/_navigation.md` to control the navigation bar; the nav is re-built when pages are crawled.
@@ -519,7 +581,6 @@ If you find a bug, have a feature request or a question, just [open an issue](ht
 ### Content not loading / 404 pages
 - Verify `contentPath` is correct and matches the directory containing your `.md` files.
 - Ensure your static server is serving those files (a 404 is often because the content folder isn’t published or the path is wrong).
-- If you’re using `homePage`/`notFoundPage`, confirm those files exist and are reachable (the default is `_home.md` and `_404.md`).
 
 ### Styles not appearing / Bulma missing
 - Ensure `dist/nimbi-cms.css` is loaded alongside `dist/nimbi-cms.js`.
@@ -527,4 +588,19 @@ If you find a bug, have a feature request or a question, just [open an issue](ht
 - If using a Bulmaswatch theme, verify the theme name is correct (see `Available Bulmaswatch themes`).
 
 ### Scripts failing / no mount element
-- Make sure the mount element exists (`<div id="app"></div>`) and `initCMS({ el: '#app' })` uses the correct selector.
+- Make sure the mount element exists (`<div id="app"></div>`) and `initCMS({ el: '#app' })` uses the correct selector.
+
+### I get a console error!
+
+One of the features of `nimbiCMS` is folder crawling for enhanced indexing
+performance, so it's going to try and fetch your `contentPath` folder just in
+case your server response is an HTML listing the files in the folder. If your
+server has this functionality disabled, which is quite common, it will send back a
+`404` response that will trigger an error in the dev console of your browser, but
+no worries, it's an expected one and breaks nothing. Your console might looks like
+this when this happens
+
+```shell
+nimbi-cms.js:71  GET https://example.com/contentfolder/ 404 (Not Found)
+nimbi-cms.js:71 [slugManager] crawlAllMarkdown: directory fetch non-ok {url: 'https://example.com/contentfolder/', status: 404}
+```