ghimera 0.3.0__tar.gz

ghimera-0.3.0/.gitignore

@@ -0,0 +1,8 @@
+__pycache__/
+*.py[cod]
+.venv/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+dist/
+gate-work/

ghimera-0.3.0/CHANGELOG.md

@@ -0,0 +1,134 @@
+# Changelog
+
+## 0.3.0 — 2026-10-06
+
+- Consolidated project naming as `ghimera`: distribution, Python package,
+  command and GitHub repository. The primary configuration class is
+  `GhimeraConfig`. A narrow legacy `chimera` root facade and module command
+  reuse the same implementation; nested imports migrate to `ghimera.*`.
+  Existing `chimera.*` data schemas, prompt revisions and saved-result
+  identities are unchanged. The old `go-spider` releases remain unchanged.
+
+- Concrete live intent research acceptance joined real discovery, laptop
+  HTTP/HTML extraction, a pinned semantic encoder, self-hosted model judgment,
+  answer/review, incremental graph and sealed archive readback. This English,
+  same-model-review diagnostic does not establish representative accuracy or
+  close the remaining browser/publisher/document/runtime acceptance gates.
+
+- Explicit SearXNG ordinary-HTML search beside unchanged JSON mode. Version-2
+  search configuration selects the dialect; the Collector assembles the owning
+  adapter. The HTML adapter parses observed simple-theme results with pinned
+  Scrapling in the existing bounded passive worker, shares direct/Tor accounting,
+  and refuses unknown layouts or access barriers without format fallback.
+
+- Completed intent results now retain successful search responses, exact queries,
+  parsed hits and transport in `chimera.research-result/2`. Readback binds each
+  response to its accounted fetch and refuses missing, substituted or duplicate
+  discovery evidence. Citing-source candidates must be actual observed hits.
+  Legacy `/1` results remain readable without claiming raw-search retention;
+  snippets still cannot serve as answer citations.
+
+- Collection grading now explicitly evaluates retained evidence sufficiency,
+  not the presence of an answer draft, with a distinct recorded prompt revision.
+  A real self-hosted-model follow-up accepted sufficient native documentation
+  and rejected irrelevant, absent, and model-memory-only evidence. These bounded
+  controls do not establish representative accuracy or independent calibration.
+
+- Explicit `citation_format = "template_ids"` for assessment and drafting:
+  the model selects supplied context IDs and the client restores exact native
+  quotations, offsets and hashes. Unknown/out-of-context IDs refuse without
+  similarity repair; full-citation mode remains the compatibility default.
+  Document judges' second looks now expand native context to the configured
+  character ceiling while retaining the first span. A real self-hosted-model
+  trial confirms these paths; its original grader failure, subsequent repair,
+  and independent-evaluation gaps remain recorded rather than being described
+  as complete acceptance.
+
+- An explicit configuration-driven intent command (`python -m chimera`) with
+  bounded input reads, separately named environment credential bindings and
+  private no-overwrite complete-result archives. Originals, citations, graph,
+  ledger and model/extraction provenance survive revalidated checksum readback.
+  Partial outcomes and interrupted/unsealed archives never become answered runs.
+
+- A configuration-driven `Collector` facade assembles actual search, HTTP,
+  HTML/document, optional browser, embedding and completion adapters. It supports
+  intent research and seeded collection with fresh per-run state, retains the
+  effective search recipe, and offers an explicitly bounded TOML read. Graph,
+  journal and source-session settings use their existing owning contracts.
+  Controlled-server composition acceptance is not real-model accuracy.
+
+- Explicit intent-reference semantic scoring alongside unchanged pinned-shelf
+  scoring. The original intent's embedding spends the shared run budget once;
+  prepared vectors, call linkage, failure/cancellation and concurrent run
+  isolation are auditable through harvest and journal readers.
+
+- Configured owner-private per-run JSONL observations, fsync-before-ack storage,
+  hash-chain replay checks, completion summaries and read-only run inspection.
+  Interrupted runs remain explicitly unsealed; no automatic refetch or resume.
+
+- Configurable single generic HTML reparse over retained source bytes, without
+  a second fetch or renewed deadline. Success, refusal and cancellation attempts
+  retain typed source/configuration-bound provenance and round-trip ledger checks.
+
+- Persistent exact-publisher/profile locator health with a configured miss bar,
+  generic extraction recovery, restart-safe/concurrent state, read-only doctor
+  output and source/config-bound harvest ledger findings. No publisher-redesign
+  accuracy claim follows from the regression fixtures.
+
+- Explicit offline PDF layout/table/OCR recipe, pinned worker dependencies and
+  artifacts, configurable column-aware reading order and reproducible local PDF
+  acceptance with source-bound receipts. Representative-corpus and Marker
+  acceptance remain open.
+
+- Authorized source sessions with caller-supplied cookies/headers, exact origin
+  and path scope, protected browser resource support, non-secret audit metadata,
+  and separation from discovery/model-control credentials.
+- Configurable document-reference depth and citing-source discovery through the
+  injected search provider, using native parent context and the existing scorer.
+- Source-bound reference decisions and native Docling URL/hyperlink locators;
+  saved-harvest validation of source hashes, query/provider binding and budgets.
+- Shared reference/discovery host, parent, per-parent candidate and query limits
+  across follow-up rounds, with versioned configuration and an example.
+
+## 0.2.0 — 2026-10-06
+
+Rebuild of go-spider as a typed, configured standalone library. Distribution
+identity stays `go-spider`; the implementation package is `chimera`.
+
+### Added
+
+- Goal collection and intent-driven planning/discovery/coverage/answer-review
+  orchestration with exact native-text citations and explicit partial results.
+- Direct/Tor HTTP, native v3 onion support, bounded robots/redirect/retry/cache
+  policies and source-network separation from private model-control traffic.
+- Isolated Patchright rendering with parent-owned HTTP resource/redirect handling.
+- Native HTML fit-Markdown/adaptive locator extraction and language detection;
+  offline DOCX tables and native PDF text extraction with retained layout.
+- Self-hosted completion and embedding clients with declared model binding,
+  bounded native evidence, token/encoding observations and recorded failure.
+- Reference-vector relevance scoring, semantic/keyword link ranking, canonical
+  and near-duplicate grouping with independently retained source occurrences.
+- Configurable research graph, immutable typed harvest/ledger/receipt contracts,
+  conformance tests, deliberate mutation witnesses and full package gate.
+- Public README, package metadata, MIT licence text and migration guidance.
+
+### Breaking changes
+
+- Requires Python 3.11 or newer; v0.1.0 required 3.10.
+- `chimera` replaces the prototype `spider_core` imports. No compatibility shim
+  or legacy `spider` command is included.
+- No cloud LLM client, automatic model-server launch, VPN manager or silent
+  external-model/direct-network fallback.
+- Operational policy is explicit validated configuration and injected ports.
+
+### Not claimed complete
+
+Public publisher/locator and real-model quality acceptance; full PDF/OCR/Marker;
+Camoufox/nodriver; calibrated judge policy; one-hop references/cited-by expansion;
+production runtime/egress acceptance. Publication
+is not deployment or proof of completion of those planned features.
+
+## 0.1.0
+
+Original prototype release with `spider_core` and the `spider` CLI. Existing
+users may pin that version while migrating to the explicit v0.2.0 library API.

ghimera-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025–2026 Josh Gompert
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

ghimera-0.3.0/PKG-INFO

@@ -0,0 +1,314 @@
+Metadata-Version: 2.4
+Name: ghimera
+Version: 0.3.0
+Summary: Intent-driven web research with bounded collection and source-bound evidence
+Project-URL: Homepage, https://github.com/ginkorea/ghimera
+Project-URL: Repository, https://github.com/ginkorea/ghimera
+Project-URL: Issues, https://github.com/ginkorea/ghimera/issues
+Project-URL: Changelog, https://github.com/ginkorea/ghimera/blob/main/CHANGELOG.md
+Author: Josh Gompert
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: curl-cffi==0.16.3
+Requires-Dist: protego==0.7.0
+Requires-Dist: pydantic==2.13.5
+Provides-Extra: browser
+Requires-Dist: patchright==1.63.0; extra == 'browser'
+Provides-Extra: documents
+Requires-Dist: docling-core==2.99.0; extra == 'documents'
+Requires-Dist: docling-slim[convert-core,format-docx,format-pdf]==2.134.0; extra == 'documents'
+Requires-Dist: lingua-language-detector==2.1.1; extra == 'documents'
+Provides-Extra: html
+Requires-Dist: crawl4ai==0.9.4; extra == 'html'
+Requires-Dist: lingua-language-detector==2.1.1; extra == 'html'
+Requires-Dist: scrapling==0.4.2; extra == 'html'
+Description-Content-Type: text/markdown
+
+# ghimera
+
+Intent-driven web research: discover sources, collect native-language documents,
+follow evidence gaps, and return a source-cited answer—or an explicit partial
+result when the evidence or budget is insufficient.
+
+**v0.3.0 consolidates the repository, distribution and import as `ghimera`.**
+It succeeds the `go-spider` distribution and `chimera` implementation. It is not
+backward-compatible with v0.1.0's `spider_core` API or `spider` CLI. Python
+**3.11+** is required. Some planned browser/document adapters and public-corpus
+acceptance are still in progress; see the limitations below.
+
+ghimera is an independent library. Supply your own search provider,
+self-hosted model services, extraction policies and graph profile.
+
+## What is implemented
+
+- **Goal and intent loops.** Collect from configured seeds with `GoalLoop`, or
+  use `ResearchLoop` to plan questions, discover sources through an injected
+  search provider, assess gaps, and draft/review an evidence-cited answer.
+- **Local models first.** Configured, already-served self-hosted models supply
+  planning, judging, answer generation, review and embeddings. Compatible HTTP
+  interfaces are supported; no external LLM fallback, model weights or model
+  server are included.
+- **Bounded direct and Tor HTTP.** Native v3 onion collection and open-web
+  requests through Tor share the same route policy. Public-network validation,
+  pinned DNS for direct requests, per-hop redirect checks, robots policy,
+  concurrency/rate limits and retries are accounted for before returning data.
+  A failed Tor route never silently falls back to direct access.
+- **Isolated JavaScript rendering.** Patchright runs in a network-isolated
+  Linux worker; the parent fetch boundary handles its permitted HTTP resources,
+  redirects and accounting. Browser binaries are explicitly configured and
+  verified, not downloaded on import. Camoufox/nodriver adapters remain planned.
+- **Native extraction.** Configured HTML fit-Markdown, adaptive locator
+  profiles, language detection, DOCX tables and native PDF text preserve raw
+  bytes beside extracted native-language text. Full PDF/OCR and Marker
+  acceptance remain open.
+- **Relevance and deduplication.** An injected self-hosted encoder scores
+  native text/windows and observed links against pinned reference vectors.
+  Keyword/semantic ranking, encoding budgets, canonical URL handling, SHA-256
+  and configured near-duplicate grouping retain source-qualified evidence.
+  Similarity is not a calibrated probability or a substitute for a verdict.
+- **Graphs and audit records.** A configurable research graph starts with the
+  intent. Typed harvests, receipts and ledger rows retain configuration,
+  transport, model-call spend, omissions, verdicts, refusals and source hashes.
+  Operational discovery traces are distinct from evidence-supported claims.
+
+Every candidate reaching acceptance receives an accept/reject/hold verdict. A hold gets a
+second model pass. An intent is marked answered only after the coverage,
+citations, review and configured confidence checks pass; budget exhaustion is
+not silently presented as success.
+
+### Additions since go-spider 0.2.0
+
+The current working branch also has explicit authorized source sessions,
+configurable references/citing-source discovery, persistent publisher-locator
+drift detection with generic recovery and a doctor, and offline model-based PDF
+layout, tables, OCR and column-aware reading order, durable run journals, and
+intent-based semantic scoring without a prebuilt reference-vector file, and a
+configuration-driven `Collector` facade using actual adapters. These were not included
+in the old `go-spider==0.2.0` wheel and are included in `ghimera==0.3.0`.
+See [source sessions](docs/SOURCE_SESSIONS.md),
+[references](docs/C3_REFERENCES.md), [locator health](docs/C2_HTML.md), and
+[PDF configuration/acceptance](docs/C2_DOCUMENTS.md),
+[run journals](docs/RUN_JOURNAL.md) and
+[intent scoring](docs/C3_EMBEDDING_SCORING.md#intent-references-unreleased-source).
+For the assembled intent-only API and full non-active template, see
+[configured collector](docs/COLLECTOR.md) and `examples/collector.toml`.
+The package also supports explicitly configured ordinary HTML search
+alongside JSON, and complete research archives retain successful discovery
+responses with query/fetch bindings. See [HTML search](docs/C3_SEARCH_HTML.md)
+and [search evidence](docs/C3_SEARCH_EVIDENCE.md). Neither mode solves access
+challenges, and a refusal is not a successful research result.
+Representative-corpus accuracy and Marker acceptance remain open; passing a
+controlled document check is not a universal quality claim.
+
+## Installation
+
+Use a dedicated virtual environment:
+
+```bash
+python3.11 -m venv .venv
+. .venv/bin/activate
+python -m pip install 'ghimera==0.3.0'
+```
+
+Install the adapters you intend to configure:
+
+```bash
+python -m pip install 'ghimera[html,documents,browser]==0.3.0'
+```
+
+The base package contains the typed core, HTTP/Tor transport, research/search
+and self-hosted model/embedding clients. Extras add pinned HTML, document and
+Patchright dependencies. They do **not** install an inference server, browser
+binary, Tor daemon or PDF/OCR model artifacts. The isolated browser adapter
+requires Linux, a compatible explicitly supplied Chromium binary and Bubblewrap;
+other operating systems have not been accepted for that adapter.
+
+## Configuration and API
+
+Operational choices are typed, versioned configuration—not Python constants:
+scope, budgets, endpoints, model identities/revisions, thresholds, private
+worker directories, browser provenance and direct/Tor policy. Parse once with
+`GhimeraConfig.from_toml(Path(...))`; inject the matching collaborators.
+
+The [examples](https://github.com/ginkorea/ghimera/tree/v0.3.0/examples) are non-active templates. Replace invalid endpoints,
+contact information, private paths and model identifiers; reference-vector
+fixtures are **not** production relevance data. Adapter blocks belong in the
+main configuration under their named keys, not as unrelated root settings.
+Supply any model credential separately in memory, only to its authorized exact
+endpoint; configuration is not credential or destination approval.
+
+Download the starter configuration, or copy it from the repository:
+
+```bash
+curl --fail --proto '=https' --tlsv1.2 \
+  https://raw.githubusercontent.com/ginkorea/ghimera/v0.3.0/examples/chimera.toml \
+  --output chimera.toml
+```
+
+An entirely offline smoke example, using explicitly named test doubles:
+
+```python
+import asyncio
+from pathlib import Path
+
+from ghimera import GhimeraConfig, Goal, GoalLoop, Harvest, Scope
+from ghimera.doubles import FakeExtractor, FakeJudge, FakeRoute, KeywordScorer
+from ghimera.fetch import FetchLadder
+
+
+async def main() -> None:
+    config = GhimeraConfig.from_toml(Path("chimera.toml"))
+    collector = GoalLoop(
+        config=config,
+        fetcher=FetchLadder((FakeRoute(),)),
+        extractor=FakeExtractor(),
+        scorer=KeywordScorer(),
+        judge=FakeJudge(),
+    )
+    result = await collector.run(
+        Goal(text="ports", seeds=("https://example.org/start",)),
+        Scope(allowed_hosts=("example.org",), max_depth=1, content_types=("text/html",)),
+    )
+    # Reader revalidates retained sources, provenance and receipt accounting.
+    restored = Harvest.model_validate_json(result.model_dump_json())
+    print(restored.receipt.stop_reason)
+
+
+asyncio.run(main())
+```
+
+This example makes no network or real model calls and proves no research
+accuracy. For actual collection bind `CurlRoute`, configured extraction and
+scoring adapters, and the self-hosted judge through their ports. For intent-only
+research, inject those into `ResearchLoop` alongside `GroundedSearch`,
+`IntentPlanner`, `ResearchAnalyst` and `AnswerReviewer`, then call
+`run(ResearchRequest(intent="your research question"))`. `SearxSearch` is the
+implemented search adapter.
+
+### Configured intent API
+
+The configured collector assembles real adapters, so applications need not manually
+wire every port. Unlike the offline smoke above, this needs your configured
+services and the adapted `examples/collector.toml` template:
+
+```python
+import asyncio
+from pathlib import Path
+
+from ghimera import Collector
+
+
+async def main() -> None:
+    collector = Collector.from_toml(Path("collector.toml"), max_config_bytes=100_000)
+    result = await collector.run("Your research question")
+    print(result.status)
+
+
+asyncio.run(main())
+```
+
+Use the [collector guide](docs/COLLECTOR.md) to configure actual private models,
+source routing, extraction, optional graph/journal and separately supplied
+credentials. This API is included in `ghimera==0.3.0`, not the old `go-spider==0.2.0` wheel. The
+lower-level APIs remain supported for custom providers and composition.
+
+The package also supplies a configured intent command that retains the
+full result, original documents and citations in a private, checksum-sealed archive:
+
+```bash
+python -m ghimera --job /absolute/path/collector-command.toml --max-job-bytes 100000
+```
+
+See the [command guide](docs/COLLECTOR_COMMAND.md) and
+`examples/collector-command.toml`. Existing output identities are never overwritten;
+partial results remain partial. This command is not in the published 0.2.0 wheel.
+
+The following guides cover the existing lower-level wiring:
+
+| Area | Guide |
+|---|---|
+| Intent, discovery, coverage and answer review | [Intent research](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C3_RESEARCH.md) |
+| Model roles, credentials and native evidence context | [Self-hosted models](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C3_MODELS.md) |
+| Reference vectors, encoding and frontier ranking | [Embedding scoring](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C3_EMBEDDING_SCORING.md) |
+| Open web and native onion routing | [Tor policy](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/TOR.md) |
+| Browser isolation, resources and redirects | [Browser rendering](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C1_BROWSER.md) |
+| HTML extraction and adaptive locators | [HTML extraction](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C2_HTML.md) |
+| DOCX/native PDF and offline artifacts | [Document extraction](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C2_DOCUMENTS.md) |
+| Canonical and near-duplicate source evidence | [Deduplication](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C2_DEDUP.md) |
+| Configurable research graphs | [Research graph](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/RESEARCH_GRAPH.md) |
+| Architecture, contracts and completion tracker | [Specification](https://github.com/ginkorea/ghimera/blob/v0.3.0/docs/C0.md) |
+
+## Scope and limitations
+
+Robots are honored by default. An override requires a recorded, reasoned,
+exact-host configuration decision; it does not disable CAPTCHA, login, paywall
+or challenge refusal. There is no challenge solver or authenticated-site bypass.
+Tor routing is a transport capability, not a guarantee of anonymity or authority
+to access a source.
+
+Collection is not limited to anonymous access. Supply your own authorized
+cookies or headers through explicitly configured source sessions, with exact
+origin/path scope and no credential values in receipts. Browser resources use
+the same parent-owned session boundary. See [authorized sessions](docs/SOURCE_SESSIONS.md).
+
+Source/search requests run on the host executing the crawler. Model control is
+a separate private-service boundary. Your application owns authorization,
+deployment and storage; no external scheduler or registry is required to import
+or use the library.
+
+Configurable source expansion follows observed document URLs and discovers
+candidate citing sources through your search provider. Depth, host policy and
+budgets live in `[references]`, not in Python. Source hashes and native locators
+are retained; a citing-source search hit is not proof that a citation exists.
+See [reference expansion](docs/C3_REFERENCES.md).
+
+For durable observations, configure `[journal]` and pass a unique `run_id`.
+The collector persists JSONL events before acknowledgment and seals a completion
+summary only after receipt reconciliation. Interrupted prefixes remain inspectable
+without silently refetching sources. See [run journals](docs/RUN_JOURNAL.md).
+
+Still required for the complete planned spider: the remaining browser adapters,
+representative publisher/locator acceptance, full PDF/OCR and Marker validation,
+real reference/cited-by adequacy, real served-model quality/admission and
+calibrated decision policy and live runtime/egress acceptance.
+The repository's detailed tracker retains those requirements; this release
+does not erase them or describe fixture results as real-world model accuracy.
+
+## Development
+
+```bash
+git clone https://github.com/ginkorea/ghimera.git
+cd ghimera
+uv sync --locked --extra html --extra documents --extra browser --python 3.11
+# Explicit browser/isolation paths are required for the complete gate.
+export CHIMERA_TEST_BROWSER=/absolute/path/to/compatible/chrome
+export CHIMERA_TEST_ISOLATOR=/absolute/path/to/bwrap
+bash scripts/gate.sh
+uv build --no-sources
+```
+
+The gate checks the actual interpreter/import path, lockfile, Ruff, strict mypy
+and the entire test suite. Browser tests refuse absent acceptance prerequisites
+rather than pretending they ran. Evidence records distinguish protocol fixtures,
+installed-artifact checks, public-corpus acceptance and production activation.
+
+## Migration from go-spider
+
+Install `ghimera==0.3.0` explicitly; this is a new distribution, not an in-place
+rename of old PyPI releases. Use `from ghimera import Collector, GhimeraConfig`,
+`ghimera.*` for submodules, and `ghimera` or `python -m ghimera` for the command.
+The legacy root `from chimera import Collector, ChimeraConfig` and
+`python -m chimera` forward to the same implementation. Old nested
+`chimera.*` imports must migrate; there is no second implementation or import hook.
+Existing versioned `chimera.*` configuration, graph, harvest, journal and result
+schemas are retained so existing saved evidence does not change identity.
+
+The v0.1.0 `spider_core` API and old `spider` command are not supplied; keep
+`go-spider==0.1.0` while migrating those applications. The old cloud-client,
+VPN-manager and implicit fallback design is not retained. Prototype source
+remains in Git history.
+
+See [CHANGELOG.md](https://github.com/ginkorea/ghimera/blob/v0.3.0/CHANGELOG.md) for release changes. Josh Gompert maintains
+the project at [ginkorea/ghimera](https://github.com/ginkorea/ghimera).
+Licensed under [MIT](https://github.com/ginkorea/ghimera/blob/v0.3.0/LICENSE), matching the existing PyPI licence declaration.