liferay-context-builder 0.6.2__py3-none-any.whl

liferay_context_builder-0.6.2.dist-info/METADATA

@@ -0,0 +1,354 @@
+Metadata-Version: 2.4
+Name: liferay-context-builder
+Version: 0.6.2
+Summary: Build a local, cited Liferay DXP context library for the liferay-expert Claude Code skill.
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: <3.14,>=3.10
+Requires-Dist: beautifulsoup4<5,>=4.12
+Requires-Dist: crawl4ai<1.0,>=0.9.0
+Description-Content-Type: text/markdown
+
+# liferay-context-builder
+
+Give Claude Code a local, source-backed Liferay DXP knowledge base it can
+actually read before answering.
+
+`liferay-context-builder` turns the public Liferay docs into a local context
+library and pairs it with the `liferay-expert` Claude Code skill. The result is
+simple: when someone on the team asks a Liferay question, the assistant can
+look up the relevant docs, cite the original URL, and avoid guessing from model
+memory.
+
+It is built for team use:
+
+- Answers stay tied to official `learn.liferay.com` sources.
+- Every project can share the same local docs folder.
+- There is no bundled Liferay content, vector database, or embedding service to
+  manage.
+- A doctor command checks whether the docs and skill are ready.
+
+![Demo of liferay-context-builder in Codex](docs/assets/liferay-doc-demo.gif)
+
+[Download the MP4 demo](docs/assets/liferay-doc-demo.mp4)
+
+[Project page](https://mordonez.github.io/liferay-context-builder/) ·
+[PyPI package](https://pypi.org/project/liferay-context-builder/) · Python
+3.10-3.13 · [MIT license](LICENSE)
+
+## Quickstart
+
+From zero to source-backed Liferay answers in Claude Code:
+
+```bash
+# 1. One-time browser setup for crawl4ai/Playwright
+uvx --from crawl4ai crawl4ai-setup
+
+# 2. Build the local Liferay DXP context library in ~/.liferay-docs
+uvx liferay-context-builder
+
+# 3. Install the Claude Code skill in your current project
+npx skills add mordonez/liferay-context-builder --skill liferay-expert -a claude-code
+
+# 4. Verify docs freshness and skill installation
+uvx --from liferay-context-builder liferay-context-builder-doctor
+```
+
+Then ask Claude Code something like:
+
+> How do I configure synonym sets in Liferay Search?
+
+The skill searches the local context library, reads the best matching pages,
+and cites the original `learn.liferay.com` URL.
+
+Keep `-a claude-code` in the install command. It avoids interactive installer
+edge cases where the skill can appear installed but not land in
+`.claude/skills/`.
+
+## Requirements
+
+- Python 3.10-3.13
+- [`uv`](https://docs.astral.sh/uv/)
+- Node/npm for `npx skills add`
+
+`crawl4ai` uses Playwright. Run the browser setup once per machine before the
+first scrape:
+
+```bash
+uvx --from crawl4ai crawl4ai-setup
+```
+
+## How It Works
+
+```mermaid
+flowchart LR
+  A[learn.liferay.com] --> B[crawl4ai BFS crawl]
+  B --> C[local Markdown in ~/.liferay-docs]
+  C --> D[search_index.jsonl and anomalies.jsonl]
+  C --> E[liferay-expert Claude Code skill]
+  D --> E
+  E --> F[cited Liferay answers]
+```
+
+The official context builder starts at
+`https://learn.liferay.com/w/dxp/index` and uses crawl4ai's BFS deep crawler to
+follow internal `/w/dxp/*` links. For each page, it extracts the article body,
+classifies the URL into a Liferay capability, and writes Markdown locally.
+
+The builder is intentionally boring:
+
+- It fetches from the live Liferay docs when you run it; this package does not
+  redistribute Liferay documentation text.
+- It writes to one shared docs directory, so every project can use the same
+  corpus.
+- It retries through crawl4ai, writes files atomically, and exits non-zero when
+  the crawl or page fetches fail.
+- It never starts a long scrape from inside the skill. If docs are missing, the
+  skill tells you which command to run.
+
+## Where Files Go
+
+By default, everything is written under:
+
+```text
+~/.liferay-docs
+```
+
+Use `LIFERAY_DOCS_DIR` when you want a repo-local or custom corpus:
+
+```bash
+export LIFERAY_DOCS_DIR="$PWD/.liferay-docs"
+uvx liferay-context-builder
+uvx --from liferay-context-builder liferay-context-builder-doctor
+```
+
+Layout:
+
+```text
+~/.liferay-docs/
+  raw/{capability}/*.md
+  raw/_navigation/{capability}/*.md
+  raw/_removed/{capability}/*.md
+  raw/community-howto/{capability}/*.md
+  raw/community-troubleshooting/{capability}/*.md
+  reports/filtered/
+    search_index.jsonl
+    anomalies.jsonl
+    summary.json
+    *_urls.txt
+```
+
+`raw/{capability}/*.md` is the main official-docs corpus the skill reads first.
+`raw/_navigation/` keeps table-of-contents/navigation pages out of normal
+answers while preserving them. `raw/_removed/` holds pages only after the
+scraper directly confirms their original URL is gone.
+
+## Refreshing Official Docs
+
+Run the scraper again whenever you want fresh docs:
+
+```bash
+uvx liferay-context-builder
+```
+
+A normal full run usually takes tens of minutes. For a smoke test:
+
+```bash
+uvx liferay-context-builder --max-pages 200
+```
+
+Useful options:
+
+```bash
+uvx liferay-context-builder --max-depth 12
+uvx liferay-context-builder --max-pages 3000
+```
+
+Each full run starts from the current site state. If a previously known page is
+not rediscovered by BFS, the scraper checks that page directly before moving it
+to `raw/_removed/`. If the page is still alive, it refreshes it directly and
+records the BFS coverage gap in the reports.
+
+## Community Articles
+
+Community articles are optional, larger, and lower-authority than the official
+DXP docs:
+
+```bash
+uvx --from liferay-context-builder liferay-context-builder-community
+```
+
+This fetches Liferay community How-To and Troubleshooting articles from
+`learn.liferay.com/kb-article/*`. They are stored separately:
+
+```text
+raw/community-howto/{capability}/*.md
+raw/community-troubleshooting/{capability}/*.md
+```
+
+Many community articles have no usable capability tag, so they go to
+`_uncategorized/`. The skill treats community content as secondary evidence and
+says so when citing it.
+
+Useful commands:
+
+```bash
+# Only How-To articles
+uvx --from liferay-context-builder liferay-context-builder-community --resource-type howto
+
+# Smaller test run per resource type
+uvx --from liferay-context-builder liferay-context-builder-community --limit 100
+```
+
+Community scraping can take much longer than the official-docs scrape because
+it fetches thousands of additional articles.
+
+## Installing The Skill
+
+Install `liferay-expert` into each Claude Code project where you want Liferay
+help:
+
+```bash
+npx skills add mordonez/liferay-context-builder --skill liferay-expert -a claude-code
+```
+
+Manual install also works: place the skill file at:
+
+```text
+.claude/skills/liferay-expert/SKILL.md
+```
+
+The skill resolves docs the same way the scraper does:
+
+1. `$LIFERAY_DOCS_DIR`, if set.
+2. `~/.liferay-docs`, otherwise.
+
+When answering, it searches `reports/filtered/search_index.jsonl` when present,
+falls back to normal file search under `raw/`, reads Markdown files directly,
+and cites the `url:` frontmatter. Official docs are preferred over community
+articles when both cover the same topic.
+
+## Doctor
+
+Use the doctor when something feels off:
+
+```bash
+uvx --from liferay-context-builder liferay-context-builder-doctor
+```
+
+It checks:
+
+- Which docs directory is active.
+- Whether official Markdown exists.
+- How many community Markdown files exist.
+- The official-docs freshness window.
+- Search index and anomaly report entry counts.
+- Whether `.claude/skills/liferay-expert/SKILL.md` exists in the current
+  project.
+
+To inspect a different project directory:
+
+```bash
+uvx --from liferay-context-builder liferay-context-builder-doctor --project-dir /path/to/project
+```
+
+The doctor does not scrape docs and does not install the skill. It only reports
+status and prints the next command to run.
+
+## Reports
+
+The scraper writes agent-facing reports under `reports/filtered/`.
+
+`search_index.jsonl` is a local retrieval index. Each JSON line includes title,
+source URL, source type, capability, file path, headings, and `fetched_at`. The
+skill uses it first because it is faster and cleaner than searching every
+Markdown file.
+
+`anomalies.jsonl` is an informational scrape-quality report. It flags signals
+like very short bodies, missing titles, known error markers, unusually large
+pages, and large body-size swings versus the previous local copy. It does not
+mean a page is unusable; it means the page may deserve a quick check before you
+trust or cite it heavily.
+
+`summary.json` records the latest run counts, crawl failures, direct refreshes,
+coverage gaps, and search index size.
+
+## Troubleshooting
+
+**`crawl4ai` or browser errors on the first run**
+
+Run the Playwright setup again:
+
+```bash
+uvx --from crawl4ai crawl4ai-setup
+```
+
+**Claude Code says the skill is missing**
+
+Run the install command from the project where you are using Claude Code:
+
+```bash
+npx skills add mordonez/liferay-context-builder --skill liferay-expert -a claude-code
+```
+
+Then verify:
+
+```bash
+uvx --from liferay-context-builder liferay-context-builder-doctor
+```
+
+**Claude Code says docs are missing**
+
+Check whether you are using a custom docs directory:
+
+```bash
+echo "$LIFERAY_DOCS_DIR"
+```
+
+If it is empty, the skill expects `~/.liferay-docs`. If it points somewhere
+else, run the scraper with that same environment variable.
+
+**Docs are stale**
+
+Refresh official docs:
+
+```bash
+uvx liferay-context-builder
+```
+
+The doctor warns when official docs are older than about seven days.
+
+**A scrape stops partway through**
+
+Rerun the same command. Already written Markdown remains usable, but a failed
+run exits non-zero and avoids treating untouched pages as removed.
+
+**Community answers feel weaker than official docs**
+
+That is expected. Community How-To and Troubleshooting articles are useful for
+practical cases and errors, but the skill should label them as community
+content and prefer official docs when official docs answer the question.
+
+## Development
+
+```bash
+uv sync --group dev
+uv run ruff check .
+uv run --with pytest python -m pytest
+uv build
+```
+
+Run `uv sync --group dev` once before local development so the project and dev
+tools are installed into uv's project environment. The pytest command uses
+`python -m pytest` with `--with pytest` because older or unsynced uv
+environments can fail to find the `pytest` console script even when Python can
+run the module.
+
+CI runs lint, tests, and package build on Python 3.10, 3.11, 3.12, and 3.13.
+It does not run a real scrape. Release publishing is documented in
+[`docs/release.md`](docs/release.md).
+
+## License
+
+[MIT](LICENSE) applies to this tool and skill only. Liferay documentation
+content remains Liferay's content and is fetched locally by each user.

liferay_context_builder-0.6.2.dist-info/RECORD

@@ -0,0 +1,12 @@
+liferay_docs_scraper/__init__.py,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+liferay_docs_scraper/classify_pages.py,sha256=yXo6eKLlsFpoM1CKzcZUNEWBN8RcQ1Nc_2qNgqnkLLs,1448
+liferay_docs_scraper/community.py,sha256=I0p6iZz-H66uwo4CPagnqiYbBi0LeexClNMoFdN1fo4,18860
+liferay_docs_scraper/doctor.py,sha256=A3f8jttJ-bw3a_JCouXpJyTp2pxLqEN17XLrcmpTNXY,6859
+liferay_docs_scraper/filter_urls.py,sha256=3qZCcGghJrjnfLNflj5OwwHzsKyKqmPBypVC0KrxaRQ,7403
+liferay_docs_scraper/index.py,sha256=xpdAewS-Edxe90_S0XJGGZT983LzAXVZNWnmxMTcjdQ,6442
+liferay_docs_scraper/pipeline.py,sha256=-JQZBKCmiQl6L62G9U3uPguZZAFiCdAqSTW_HGB2Lkk,24486
+liferay_context_builder-0.6.2.dist-info/METADATA,sha256=LmhbeWEHg6EH0lI83aEaxoDI_Eu-kQQXYXCyErg6OSk,10507
+liferay_context_builder-0.6.2.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+liferay_context_builder-0.6.2.dist-info/entry_points.txt,sha256=X5Cc5NdbdtYY4gdrunOYDupeQMZd-OAZYjrIa_bAC48,217
+liferay_context_builder-0.6.2.dist-info/licenses/LICENSE,sha256=li7C8a1CCl-Qhn8tmQ55RDdRnOziMgqa08pfVjJuSyw,1072
+liferay_context_builder-0.6.2.dist-info/RECORD,,

liferay_context_builder-0.6.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

liferay_context_builder-0.6.2.dist-info/entry_points.txt

@@ -0,0 +1,4 @@
+[console_scripts]
+liferay-context-builder = liferay_docs_scraper.pipeline:main
+liferay-context-builder-community = liferay_docs_scraper.community:main
+liferay-context-builder-doctor = liferay_docs_scraper.doctor:main

liferay_context_builder-0.6.2.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Miguel Ordoñez
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

liferay_docs_scraper/classify_pages.py

@@ -0,0 +1,41 @@
+#!/usr/bin/env python3
+"""Navigation-vs-content heuristic, shared with crawl4ai_pipeline.py.
+
+An "index"/navigation page is one whose body (frontmatter stripped) is
+short and consists mostly of links to subpages -- no substantial technical
+content of its own. Everything else is "content".
+
+Heuristic (no API calls, pure text analysis):
+  - Strip Markdown link syntax down to visible text (`[text](url)` -> `text`)
+    and strip heading/emphasis markup (`#`, `*`, `_`, backticks).
+  - total_words: word count of that visible text.
+  - link_ratio: fraction of those words that come from inside a Markdown
+    link's link-text span.
+  - "index" iff total_words < INDEX_MAX_WORDS and link_ratio >= INDEX_MIN_LINK_RATIO.
+"""
+
+import re
+
+INDEX_MAX_WORDS = 150
+INDEX_MIN_LINK_RATIO = 0.5
+
+LINK_RE = re.compile(r"\[([^\]]*)\]\(([^)]*)\)")
+MARKUP_RE = re.compile(r"[#*_`\\]")
+
+
+def analyze_body(body: str) -> tuple[int, float]:
+    links = LINK_RE.findall(body)
+    link_word_count = sum(len(text.split()) for text, _url in links)
+
+    visible = LINK_RE.sub(lambda m: m.group(1), body)
+    visible = MARKUP_RE.sub("", visible)
+    total_words = len(visible.split())
+
+    link_ratio = (link_word_count / total_words) if total_words else 0.0
+    return total_words, link_ratio
+
+
+def classify(total_words: int, link_ratio: float) -> str:
+    if total_words < INDEX_MAX_WORDS and link_ratio >= INDEX_MIN_LINK_RATIO:
+        return "index"
+    return "content"