likhit 0.1.0__tar.gz

likhit-0.1.0/PKG-INFO

@@ -0,0 +1,104 @@
+Metadata-Version: 2.4
+Name: likhit
+Version: 0.1.0
+Summary: Nepali PDF-to-Markdown converter powered by MarkItDown
+Author: notashwinii
+Author-email: ashwinii.subedi@gmail.com
+Requires-Python: >=3.12,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Dist: fonttools (>=4.47.0,<5.0.0)
+Requires-Dist: markitdown[pdf] (==0.1.5)
+Requires-Dist: npttf2utf (>=0.3.7,<1.0.0)
+Requires-Dist: pymupdf (==1.27.2)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Description-Content-Type: text/markdown
+
+# likhit
+
+`likhit` is a public PDF-to-Markdown tool for Nepali government documents.
+
+The default path is powered by [MarkItDown](https://github.com/microsoft/markitdown), with `likhit` intercepting born-digital Nepali PDFs that need Nepal-specific repair before Markdown is emitted. That repair layer handles Kalimati broken-CMap fixes, Devanagari reordering and spacing normalization, and legacy Nepali font remapping where applicable.
+
+## Installation
+
+### With Poetry
+
+```bash
+poetry install
+```
+
+Run project commands with `poetry run`:
+
+```bash
+poetry run pytest
+poetry run ruff check .
+poetry run black --check .
+```
+
+## Recommended Usage
+
+Convert a single PDF to editable Markdown:
+
+```bash
+poetry run likhit convert path/to/document.pdf --out path/to/document.md
+```
+
+Convert multiple PDFs at once:
+
+```bash
+poetry run likhit convert path/to/a.pdf path/to/b.pdf --out-dir path/to/output-dir
+```
+
+If `--out` or `--out-dir` is omitted, `likhit` writes Markdown files in the current directory using the input filename stem.
+
+## Usage
+
+`convert` is the public path.
+
+- Input scope: born-digital PDFs only
+- Output: generic editable Markdown
+- Engine: MarkItDown by default
+- `likhit` value-add: Nepali PDF repair before Markdown output when needed
+- Recognized document layouts such as Kanun Patrika and CIAA-style PDFs are auto-detected internally so `likhit` can preserve better text order and structure without a `--type` flag
+- No OCR support is included in this branch
+
+## Architecture
+
+The new default pipeline is:
+
+1. `likhit convert` opens the PDF and checks whether it matches a known structure-aware document type.
+2. If the PDF matches a known layout such as Kanun Patrika or a CIAA-style document, `likhit` reuses its existing structure-aware extraction logic internally.
+3. Otherwise, MarkItDown handles the default conversion path.
+4. When the PDF needs Nepali repair, `likhit` repairs the text first:
+   - Kalimati broken-CMap repair
+   - Devanagari reordering
+   - Devanagari spacing normalization
+   - Legacy-font remapping through `npttf2utf`
+5. `likhit` assembles repaired text blocks into Markdown.
+
+This keeps the public product story simple: `likhit` is the tool users call, while MarkItDown is embedded infrastructure.
+
+## Current Scope
+
+- Supported default input: PDF only
+- Supported default output: Markdown only
+- Supported default document class: born-digital PDFs
+- Unsupported in this branch: OCR, scanned/image-only PDFs, `.doc`, `.docx`, and image inputs
+
+## Project Layout
+
+- `src/likhit/core.py`: public `convert` and `convert_many` entry points
+- `src/likhit/markitdown_integration.py`: MarkItDown instance setup and custom PDF converter
+- `src/likhit/nepali_pdf_repair.py`: reusable Nepal-specific PDF repair layer
+- `src/likhit/markdown_assembly.py`: generic Markdown assembly for the default conversion path
+- `src/likhit/extractors/`, `src/likhit/handlers/`, `src/likhit/renderers/`: internal Nepali PDF repair and legacy extraction internals
+- `tests/`: conversion, extraction, and CLI coverage
+
+## References
+
+- MarkItDown: https://github.com/microsoft/markitdown
+- MarkItDown sample plugin: https://github.com/microsoft/markitdown/tree/main/packages/markitdown-sample-plugin
+

likhit-0.1.0/README.md

@@ -0,0 +1,85 @@
+# likhit
+
+`likhit` is a public PDF-to-Markdown tool for Nepali government documents.
+
+The default path is powered by [MarkItDown](https://github.com/microsoft/markitdown), with `likhit` intercepting born-digital Nepali PDFs that need Nepal-specific repair before Markdown is emitted. That repair layer handles Kalimati broken-CMap fixes, Devanagari reordering and spacing normalization, and legacy Nepali font remapping where applicable.
+
+## Installation
+
+### With Poetry
+
+```bash
+poetry install
+```
+
+Run project commands with `poetry run`:
+
+```bash
+poetry run pytest
+poetry run ruff check .
+poetry run black --check .
+```
+
+## Recommended Usage
+
+Convert a single PDF to editable Markdown:
+
+```bash
+poetry run likhit convert path/to/document.pdf --out path/to/document.md
+```
+
+Convert multiple PDFs at once:
+
+```bash
+poetry run likhit convert path/to/a.pdf path/to/b.pdf --out-dir path/to/output-dir
+```
+
+If `--out` or `--out-dir` is omitted, `likhit` writes Markdown files in the current directory using the input filename stem.
+
+## Usage
+
+`convert` is the public path.
+
+- Input scope: born-digital PDFs only
+- Output: generic editable Markdown
+- Engine: MarkItDown by default
+- `likhit` value-add: Nepali PDF repair before Markdown output when needed
+- Recognized document layouts such as Kanun Patrika and CIAA-style PDFs are auto-detected internally so `likhit` can preserve better text order and structure without a `--type` flag
+- No OCR support is included in this branch
+
+## Architecture
+
+The new default pipeline is:
+
+1. `likhit convert` opens the PDF and checks whether it matches a known structure-aware document type.
+2. If the PDF matches a known layout such as Kanun Patrika or a CIAA-style document, `likhit` reuses its existing structure-aware extraction logic internally.
+3. Otherwise, MarkItDown handles the default conversion path.
+4. When the PDF needs Nepali repair, `likhit` repairs the text first:
+   - Kalimati broken-CMap repair
+   - Devanagari reordering
+   - Devanagari spacing normalization
+   - Legacy-font remapping through `npttf2utf`
+5. `likhit` assembles repaired text blocks into Markdown.
+
+This keeps the public product story simple: `likhit` is the tool users call, while MarkItDown is embedded infrastructure.
+
+## Current Scope
+
+- Supported default input: PDF only
+- Supported default output: Markdown only
+- Supported default document class: born-digital PDFs
+- Unsupported in this branch: OCR, scanned/image-only PDFs, `.doc`, `.docx`, and image inputs
+
+## Project Layout
+
+- `src/likhit/core.py`: public `convert` and `convert_many` entry points
+- `src/likhit/markitdown_integration.py`: MarkItDown instance setup and custom PDF converter
+- `src/likhit/nepali_pdf_repair.py`: reusable Nepal-specific PDF repair layer
+- `src/likhit/markdown_assembly.py`: generic Markdown assembly for the default conversion path
+- `src/likhit/extractors/`, `src/likhit/handlers/`, `src/likhit/renderers/`: internal Nepali PDF repair and legacy extraction internals
+- `tests/`: conversion, extraction, and CLI coverage
+
+## References
+
+- MarkItDown: https://github.com/microsoft/markitdown
+- MarkItDown sample plugin: https://github.com/microsoft/markitdown/tree/main/packages/markitdown-sample-plugin

likhit-0.1.0/pyproject.toml

@@ -0,0 +1,31 @@
+[project]
+name = "likhit"
+version = "0.1.0"
+description = "Nepali PDF-to-Markdown converter powered by MarkItDown"
+authors = [
+    { name = "notashwinii", email = "ashwinii.subedi@gmail.com" }
+]
+readme = "README.md"
+requires-python = ">=3.12,<4.0"
+dependencies = [
+    "fonttools>=4.47.0,<5.0.0",
+    "npttf2utf>=0.3.7,<1.0.0",
+    "pymupdf==1.27.2",
+    "pyyaml>=6.0,<7.0",
+    "markitdown[pdf]==0.1.5"
+]
+
+[project.scripts]
+likhit = "likhit.cli:main"
+
+[tool.poetry]
+packages = [{ include = "likhit", from = "src" }]
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^8.0.0"
+ruff = "^0.11.0"
+black = "^25.0.0"
+
+[build-system]
+requires = ["poetry-core>=2.0.0,<3.0.0"]
+build-backend = "poetry.core.masonry.api"

likhit-0.1.0/src/likhit/__init__.py

@@ -0,0 +1,6 @@
+"""Likhit package."""
+
+from likhit.core import convert, convert_many
+from likhit.version import __version__
+
+__all__ = ["__version__", "convert", "convert_many"]

likhit-0.1.0/src/likhit/cli.py

@@ -0,0 +1,84 @@
+"""CLI entry point for conversion."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from likhit.core import convert_many, derive_convert_output_name
+from likhit.errors import LikhitError, ValidationError
+
+
+def build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="likhit",
+        description="Convert born-digital PDFs to editable Markdown.",
+    )
+    subparsers = parser.add_subparsers(dest="command")
+
+    convert_parser = subparsers.add_parser(
+        "convert",
+        help="Convert PDFs to editable Markdown",
+    )
+    convert_parser.add_argument("inputs", nargs="+", help="One or more input PDF files")
+    convert_parser.add_argument("--out", help="Output path for a single input file")
+    convert_parser.add_argument(
+        "--out-dir",
+        help="Directory for generated Markdown files; defaults to current directory",
+    )
+    return parser
+
+
+def _write_convert_outputs(args: argparse.Namespace) -> int:
+    if len(args.inputs) > 1 and args.out:
+        raise ValidationError("--out can only be used with a single input file")
+
+    for input_path in args.inputs:
+        path = Path(input_path)
+        if path.suffix.lower() != ".pdf":
+            raise ValidationError(
+                "Unsupported input format for convert. Only born-digital PDF files are supported."
+            )
+
+    output_dir = Path(args.out_dir or ".")
+    output_dir.mkdir(parents=True, exist_ok=True)
+    results = convert_many(args.inputs)
+
+    existing_names: set[str] = {
+        path.name for path in output_dir.iterdir() if path.is_file()
+    }
+    for source_path, markdown in results:
+        if args.out:
+            destination = Path(args.out)
+            destination.parent.mkdir(parents=True, exist_ok=True)
+            if destination.exists():
+                raise ValidationError(f"Output file already exists: {destination}")
+        else:
+            destination = output_dir / derive_convert_output_name(
+                source_path,
+                existing_names,
+            )
+        destination.write_text(markdown, encoding="utf-8")
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = build_parser()
+    args = parser.parse_args(argv)
+    if args.command is None:
+        parser.print_help()
+        return 1
+
+    try:
+        if args.command == "convert":
+            return _write_convert_outputs(args)
+        parser.print_help()
+        return 1
+    except LikhitError as exc:
+        print(str(exc), file=sys.stderr)
+        return 1
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

likhit-0.1.0/src/likhit/core.py

@@ -0,0 +1,126 @@
+"""Public conversion entry points."""
+
+from __future__ import annotations
+
+from pathlib import Path
+import re
+
+from likhit.errors import ExtractionError, ValidationError
+from likhit.handlers import CIAAPressReleaseHandler, KanunPatrikaHandler
+from likhit.markitdown_integration import convert_pdf_to_markdown
+from likhit.models import DocumentType, ExtractionResult
+from likhit.renderers import MarkdownRenderer
+from likhit.extractors.font_based import FontBasedStrategy
+
+_KANUN_MARKERS = (
+    "नेपालकानूनपत्रिका",
+    "नेपालकानुनपत्रिका",
+    "निर्णय नं",
+    "ने.का.प.",
+)
+_CIAA_MARKERS = (
+    "अख्तियार दुरुपयोग अनुसन्धान आयोग",
+    "प्रेस विज्ञ",
+    "प्रेस मिज्ञ",
+    "विषय:",
+    "मिषय:",
+    "आयोगकोनिर्णय",
+    "अनुसन्धानबाट पुष्टि भएको",
+)
+
+
+def _metadata_from_options(
+    title: str | None,
+    publication_date: str | None,
+    source_url: str | None,
+) -> dict[str, str | None]:
+    return {
+        "title": title,
+        "publication_date": publication_date,
+        "source_url": source_url,
+    }
+
+
+def _resolve_handler(
+    doc_type: DocumentType,
+) -> CIAAPressReleaseHandler | KanunPatrikaHandler:
+    if doc_type is DocumentType.CIAA_PRESS_RELEASE:
+        return CIAAPressReleaseHandler()
+    if doc_type is DocumentType.KANUN_PATRIKA:
+        return KanunPatrikaHandler()
+    raise ValidationError(f"Unsupported document type: {doc_type.value}")
+
+
+def _normalize_detection_text(text: str) -> str:
+    return re.sub(r"\s+", "", text).casefold()
+
+
+def _detect_document_type(raw_text: str) -> DocumentType | None:
+    normalized = _normalize_detection_text(raw_text)
+
+    if any(
+        _normalize_detection_text(marker) in normalized for marker in _KANUN_MARKERS
+    ):
+        return DocumentType.KANUN_PATRIKA
+    if any(_normalize_detection_text(marker) in normalized for marker in _CIAA_MARKERS):
+        return DocumentType.CIAA_PRESS_RELEASE
+    return None
+
+
+def _render_markdown_without_frontmatter(result: ExtractionResult) -> str:
+    markdown = MarkdownRenderer().render(result).lstrip()
+    if not markdown.startswith("---\n"):
+        return markdown.strip()
+
+    parts = markdown.split("\n---\n", 1)
+    if len(parts) != 2:
+        return markdown.strip()
+    return parts[1].strip()
+
+
+def _convert_with_detected_structure(file_path: str) -> str | None:
+    strategy = FontBasedStrategy()
+    try:
+        raw_document = strategy.extract_text(file_path)
+    except ExtractionError as exc:
+        if str(exc) == "No text content found in document":
+            return None
+        raise
+    doc_type = _detect_document_type(raw_document.raw_text)
+    if doc_type is None:
+        return None
+
+    handler = _resolve_handler(doc_type)
+    result = handler.build_result(
+        raw_document,
+        _metadata_from_options(None, None, None),
+    )
+    return _render_markdown_without_frontmatter(result)
+
+
+def convert(file_path: str) -> str:
+    path = Path(file_path)
+    if path.suffix.lower() != ".pdf":
+        raise ValidationError(
+            "Unsupported input format for convert. Only born-digital PDF files are supported."
+        )
+
+    structured_markdown = _convert_with_detected_structure(file_path)
+    if structured_markdown is not None:
+        return structured_markdown
+    return convert_pdf_to_markdown(file_path)
+
+
+def derive_convert_output_name(source_path: str, existing: set[str]) -> str:
+    base_name = Path(source_path).stem or "document"
+    candidate = f"{base_name}.md"
+    counter = 2
+    while candidate in existing:
+        candidate = f"{base_name}-{counter}.md"
+        counter += 1
+    existing.add(candidate)
+    return candidate
+
+
+def convert_many(file_paths: list[str]) -> list[tuple[str, str]]:
+    return [(file_path, convert(file_path)) for file_path in file_paths]

likhit-0.1.0/src/likhit/errors.py

@@ -0,0 +1,13 @@
+"""Project-specific exceptions."""
+
+
+class LikhitError(Exception):
+    """Base exception for extraction failures."""
+
+
+class ValidationError(LikhitError):
+    """Raised when user input or extracted content is invalid."""
+
+
+class ExtractionError(LikhitError):
+    """Raised when document extraction fails."""

likhit-0.1.0/src/likhit/extractors/__init__.py

@@ -0,0 +1,5 @@
+"""Extraction strategy exports."""
+
+from likhit.extractors.font_based import FontBasedStrategy
+
+__all__ = ["FontBasedStrategy"]

likhit-0.1.0/src/likhit/extractors/base.py

@@ -0,0 +1,45 @@
+"""Shared extraction abstractions."""
+
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from dataclasses import dataclass, field
+
+from likhit.models import Table
+
+
+@dataclass(slots=True)
+class TextFragment:
+    """A normalized text block plus its page/layout metadata."""
+
+    text: str
+    page_number: int
+    x0: float
+    y0: float
+    x1: float
+    y1: float
+    block_number: int = 0
+    line_number: int = 0
+    gap_before: float | None = None
+
+
+@dataclass(slots=True)
+class RawDocument:
+    """Unstructured text extracted from a document."""
+
+    paragraphs: list[str]
+    raw_text: str
+    fragments: list[TextFragment]
+    tables: list[Table] = field(default_factory=list)
+
+
+class ExtractionStrategy(ABC):
+    """Base interface for extraction strategies."""
+
+    @abstractmethod
+    def extract_text(self, file_path: str, pages: str | None = None) -> RawDocument:
+        """Extract unstructured text from a document."""
+
+    @abstractmethod
+    def extract_tables(self, file_path: str) -> list[Table]:
+        """Extract structured tables from a document."""