fmjl 0.5.0__tar.gz

fmjl-0.5.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Rupak Kumar
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

fmjl-0.5.0/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.4
+Name: fmjl
+Version: 0.5.0
+Summary: FMJL (.fmjl): one document, two forms. JSON Lines container, Markdown text, LaTeX formulas, HTML merged-cell tables, built for RAG. Converter, checker, PDF and Word importers, retriever chunks.
+Author: Rupak Kumar
+License: MIT
+Project-URL: Homepage, https://github.com/rupakkumar-76319/fmjl
+Project-URL: Rulebook, https://github.com/rupakkumar-76319/fmjl/blob/main/rulebook/fmjl_rulebook_v0.5.md
+Project-URL: Issues, https://github.com/rupakkumar-76319/fmjl/issues
+Keywords: fmjl,jsonl,markdown,rag,document,pdf,docx,retrieval
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: jsonschema>=4
+Provides-Extra: pdf
+Requires-Dist: pymupdf>=1.24; extra == "pdf"
+Dynamic: license-file
+
+# FMJL
+
+FMJL, short for Format, Markdown, JSON Lines, stores a document as one JSON object per line: one element (heading, paragraph, table, formula, image, caption...) with a stable id, a content hash, its page and position, and who may read it. People write and edit the same document as normal Markdown; the tool converts both ways without renumbering anything.
+
+    pip install fmjl            # converter, checker, Word importer, chunks
+    pip install fmjl[pdf]       # adds the PDF importer (PyMuPDF)
+
+    fmjl notes.md               # Markdown  -> notes.fmjl, then checks it
+    fmjl notes.fmjl             # .fmjl     -> notes.md, ids kept
+    fmjl report.pdf             # PDF       -> report.fmjl, report.md, images/
+    fmjl report.docx            # Word      -> the same
+    fmjl chunks report.fmjl     # retriever-ready chunks as JSON Lines
+
+From Python: `import fmjl`; `fmjl.load`, `fmjl.chunks`, `fmjl.changed_chunks`, `fmjl.check`.
+
+Rulebook, examples, benchmark and the VS Code extension: https://github.com/rupakkumar-76319/fmjl

fmjl-0.5.0/README.md

@@ -0,0 +1,211 @@
+---
+fmjl: "0.5"
+doc: readme
+lang: en
+access: ["all"]
+source: README.md
+sha256: 0afdb86ddae8de9cd5d5b8c778609ee6d35a3553b9fc143c1890856764b01045
+protection: none
+signed: false
+converter: fmjl 0.5
+created: 2026-09-28T20:11:27Z
+last_id: 25
+---
+
+<!-- e1 -->
+# FMJL
+
+<!-- e2 -->
+FMJL, short for Format, Markdown, JSON Lines, is a document format that combines four
+languages, each doing the one job it is best at:
+
+<!-- e3 -->
+| Language | Job in FMJL | Why that one |
+| --- | --- | --- |
+| JSON Lines | The container: one element per line, with its id, hash, page, position and permissions | Every programming language reads JSON; one line per element means a file can be streamed, searched and cut without a parser |
+| Markdown | All readable text: headings, paragraphs, lists, simple tables, image descriptions | People can read and write it directly; it is the language of GitHub, VS Code and every editor |
+| LaTeX | Math formulas | The only language that writes every formula exactly |
+| HTML | Tables with merged cells or multi-level headers | Markdown tables cannot merge cells; HTML can |
+
+<!-- e4 -->
+No single format does this on its own. Markdown cannot carry an image's page and position,
+cannot tie a caption to its image, and cannot say who may read a paragraph. JSON can hold
+those facts but nobody wants to write or read a document as JSON. LaTeX writes formulas
+perfectly but is slow to parse and hard for most people to write. HTML has the tables but
+is heavy everywhere else. FMJL takes the strength of each and leaves the rest out.
+
+<!-- e5 -->
+A document becomes a list of small parts called elements: headings, paragraphs, lists,
+tables, formulas, code, images, captions, footnotes, form fields, stamps, watermarks and
+more. Every element has a stable id, a content hash, a page and position, and a permission
+list. That is what RAG systems and search engines need, and it is why FMJL reads a document
+in about a tenth of a millisecond (see `benchmark/`).
+
+<!-- e6 -->
+One document has two forms that hold the same information:
+
+<!-- e7 -->
+| Form | File | Made for |
+| --- | --- | --- |
+| Storage | `name.fmjl` | Machines: search, RAG, databases |
+| Authoring | `name.md` | People: reading, writing, reviewing |
+
+<!-- e8 -->
+## Folder layout
+
+<!-- e9 -->
+```text
+fmjl.py          the reference tool (needs Python 3.9+ and: pip install jsonschema)
+fmjl_pdf.py      the PDF importer (needs: pip install pymupdf)
+fmjl_docx.py     the Word importer (needs nothing else)
+rulebook/        the specification, version 0.5, in both forms
+examples/        sample documents (Markdown, PDF, Word) with their images/, and rag_demo.py
+fmjl-vscode/     the VS Code extension: convert, syntax coloring and live checking
+benchmark/       the same document in FMJL, Markdown, JSON and LaTeX, and the scores
+archive/         older versions (0.1, 0.2), the evaluation, and the first converter
+```
+
+<!-- e10 -->
+## Everyday use
+
+<!-- e11 -->
+Write a normal Markdown file, then turn it into the storage form. The command is
+`fmjl <input> <output>`; the file extensions tell the tool which way to convert:
+
+<!-- e12 -->
+```powershell
+fmjl notes.md  store\notes.fmjl    # Markdown to storage form, then checks it
+fmjl store\notes.fmjl  notes.md    # storage form back to Markdown; ids are kept
+fmjl notes.md                      # output left out: notes.fmjl next to the input
+fmjl notes.fmjl                    # same shortcut the other way
+```
+
+<!-- e13 -->
+`fmjl` is the small `fmjl.cmd` file in this folder. Add the folder to your PATH once and
+the word works from anywhere. Without it, write `python fmjl.py new notes.md`.
+
+## From a PDF or a Word file
+
+```powershell
+fmjl report.pdf          # writes report.fmjl, report.md and images/, then checks
+fmjl report.docx         # the same from Word
+```
+
+The importer reads the text layer of the PDF and turns it into elements: headings by
+font size, paragraphs, lists, tables (with or without ruling lines), images with their
+captions linked by `reference`, and repeated headers, footers and page numbers marked as
+`noise`. Every element carries `page` (counted from 0) and `bbox` (0 to 1000), so a RAG
+system can cite the exact place on the page. A paragraph that runs over a page break is
+linked to its first half with `continues`. Open the `.md` afterwards, fix what the
+importer got wrong, and run `fmjl report.md`; the ids stay.
+
+Pages without a text layer (scans) need OCR. Install Tesseract and the importer uses it;
+without it, the page is reported and skipped. `examples/solar_report.pdf` is a sample
+with its imported `.fmjl` and `.md`.
+
+A Word file already knows its structure, so the Word importer reads it directly: headings
+from the Heading styles, bullet and numbered lists, tables with merged cells, images with
+captions, footnotes, Word formulas as LaTeX, the header and footer as `noise`, and page
+numbers from the page breaks Word recorded. `examples/maintenance_guide.docx` is a sample.
+
+Merged cells never need HTML by hand. In a Markdown table a cell holding only `^` joins
+the cell above it and a cell holding only `<` joins the cell to its left; the tool writes
+the HTML with `rowspan` and `colspan` for you, and shows the same shortcut when you
+convert back:
+
+```markdown
+| Name | Score | < |
+| --- | --- | --- |
+| ^ | Math | AI |
+| Rupak Kumar | 9 | 10 |
+```
+
+<!-- e14 -->
+All commands:
+
+<!-- e15 -->
+| Command | What it does |
+| --- | --- |
+| `python fmjl.py new notes.md` | Authoring form to storage form |
+| `python fmjl.py md notes.fmjl` | Storage form to authoring form |
+| `python fmjl.py fill notes.fmjl` | Fills in `id`, `hash`, `characters` and `parent`; makes `md` canonical |
+| `python fmjl.py check notes.fmjl` | Checks every rule of the rulebook |
+| `python fmjl.py view notes.fmjl` | Prints the document as clean Markdown |
+| `python fmjl.py info notes.fmjl` | Prints the title, element counts and an outline |
+| `python fmjl.py upgrade old.fmjl` | Turns a version 0.1 to 0.4 file into 0.5 |
+| `python fmjl.py pdf report.pdf` | PDF to storage form, authoring form and `images/` |
+| `python fmjl.py docx report.docx` | Word to storage form, authoring form and `images/` |
+| `python fmjl.py chunks report.fmjl` | Retriever-ready chunks as JSON Lines |
+
+<!-- e16 -->
+## In a RAG pipeline
+
+```powershell
+fmjl chunks report.fmjl                     # one chunk per element, as JSON Lines
+fmjl chunks report.fmjl --by section        # one chunk per heading and what is under it
+fmjl chunks new.fmjl --since old.fmjl       # only the chunks that changed: re-embed just those
+```
+
+Every chunk carries `id`, `text` (the headings above it, then the content), `page`, `bbox`,
+`hash`, `access` and `path`. Noise, tables of contents and redactions are never included.
+The same is available from Python:
+
+```python
+import fmjl
+rows = fmjl.load("report.fmjl")
+for c in fmjl.chunks(rows):
+    embed(c["text"], metadata={"id": c["id"], "page": c["page"], "access": c["access"]})
+```
+
+`examples/rag_demo.py` is the whole pipeline in one file with nothing to install: it loads
+every `.fmjl` in a folder, scores the chunks against a question, prints the best ones with
+their citation (document, page, element id), and, when the `anthropic` package and a key
+are present, asks Claude to answer from those chunks only:
+
+```powershell
+python examples\rag_demo.py "How much did electricity bills fall?"
+```
+
+## What the benchmark shows
+
+<!-- e17 -->
+The same 75-sentence report with two formulas, a table, an image and a caption was written
+in FMJL, Markdown, JSON and LaTeX, then read back and scored (`benchmark/results.json`):
+
+<!-- e18 -->
+|  | FMJL | Markdown | JSON | LaTeX |
+| --- | --- | --- | --- | --- |
+| Page number of every element | 23/23 | 0/23 | 23/23 | 0/23 |
+| Caption linked to its image | yes | no | yes | yes |
+| Elements keeping their id after an edit | 23/23 | 0/23 | 21/23 | 5/23 |
+| Detects a silently changed letter | yes | no | no | no |
+| Read one document | 0.11 ms | 3.85 ms | 0.07 ms | 29.32 ms |
+
+<!-- e19 -->
+## VS Code
+
+<!-- e20 -->
+Search for **FMJL** in the Extensions view and install it (publisher rupakkumar). It needs
+no Python. Right-click a `.md` file for **FMJL: Convert Markdown to .fmjl**, or a `.fmjl`
+file for **FMJL: Convert .fmjl to Markdown**. Any `.fmjl` file gets coloring, red
+underlines for rule violations, and the command **FMJL: Check current file**.
+
+<!-- e21 -->
+The extension's converter is a JavaScript port of `fmjl.py`. Running
+`node fmjl-vscode/test/roundtrip.js` proves the two give byte-identical output on every
+document in this repository.
+
+<!-- e22 -->
+## Status
+
+<!-- e23 -->
+Draft, version 0.5. The rulebook is the authority; if `fmjl.py` and the rulebook
+disagree, the tool has a bug.
+
+<!-- e24 -->
+## Author and license
+
+<!-- e25 -->
+Created by Rupak Kumar. Released under the MIT License (see `LICENSE`): use it freely,
+keep the copyright line. Suggestions and bug reports go to
+https://github.com/rupakkumar-76319/fmjl/issues.

fmjl-0.5.0/fmjl.egg-info/PKG-INFO

@@ -0,0 +1,40 @@
+Metadata-Version: 2.4
+Name: fmjl
+Version: 0.5.0
+Summary: FMJL (.fmjl): one document, two forms. JSON Lines container, Markdown text, LaTeX formulas, HTML merged-cell tables, built for RAG. Converter, checker, PDF and Word importers, retriever chunks.
+Author: Rupak Kumar
+License: MIT
+Project-URL: Homepage, https://github.com/rupakkumar-76319/fmjl
+Project-URL: Rulebook, https://github.com/rupakkumar-76319/fmjl/blob/main/rulebook/fmjl_rulebook_v0.5.md
+Project-URL: Issues, https://github.com/rupakkumar-76319/fmjl/issues
+Keywords: fmjl,jsonl,markdown,rag,document,pdf,docx,retrieval
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Markup :: Markdown
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: jsonschema>=4
+Provides-Extra: pdf
+Requires-Dist: pymupdf>=1.24; extra == "pdf"
+Dynamic: license-file
+
+# FMJL
+
+FMJL, short for Format, Markdown, JSON Lines, stores a document as one JSON object per line: one element (heading, paragraph, table, formula, image, caption...) with a stable id, a content hash, its page and position, and who may read it. People write and edit the same document as normal Markdown; the tool converts both ways without renumbering anything.
+
+    pip install fmjl            # converter, checker, Word importer, chunks
+    pip install fmjl[pdf]       # adds the PDF importer (PyMuPDF)
+
+    fmjl notes.md               # Markdown  -> notes.fmjl, then checks it
+    fmjl notes.fmjl             # .fmjl     -> notes.md, ids kept
+    fmjl report.pdf             # PDF       -> report.fmjl, report.md, images/
+    fmjl report.docx            # Word      -> the same
+    fmjl chunks report.fmjl     # retriever-ready chunks as JSON Lines
+
+From Python: `import fmjl`; `fmjl.load`, `fmjl.chunks`, `fmjl.changed_chunks`, `fmjl.check`.
+
+Rulebook, examples, benchmark and the VS Code extension: https://github.com/rupakkumar-76319/fmjl

fmjl-0.5.0/fmjl.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+fmjl.py
+fmjl_docx.py
+fmjl_pdf.py
+pyproject.toml
+fmjl.egg-info/PKG-INFO
+fmjl.egg-info/SOURCES.txt
+fmjl.egg-info/dependency_links.txt
+fmjl.egg-info/entry_points.txt
+fmjl.egg-info/requires.txt
+fmjl.egg-info/top_level.txt

fmjl-0.5.0/fmjl.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

fmjl-0.5.0/fmjl.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+fmjl = fmjl:main

fmjl-0.5.0/fmjl.egg-info/requires.txt

@@ -0,0 +1,4 @@
+jsonschema>=4
+
+[pdf]
+pymupdf>=1.24

fmjl-0.5.0/fmjl.egg-info/top_level.txt

@@ -0,0 +1,3 @@
+fmjl
+fmjl_docx
+fmjl_pdf