structured-data-extraction 0.1.0__tar.gz → 0.2.0__tar.gz

{structured_data_extraction-0.1.0/src/structured_data_extraction.egg-info → structured_data_extraction-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: structured-data-extraction
-Version: 0.1.0
+Version: 0.2.0
 Summary: Pipeline for extracting structured data (tables, sections, key/value fields) from PDF documents using docling, embeddings, and LLMs.
 Author-email: Dimitrios Avellas <dimitrisavellas8@gmail.com>
 License-Expression: MIT
@@ -140,6 +140,7 @@ Notes:
 ## Usage
 
 ```bash
+CLI:
 # one PDF
 structured-extract pdf/1167fc.pdf --keys gain CMRR "supply voltage" #when key is more than one word then it should be inside ""
 
@@ -148,7 +149,65 @@ structured-extract pdf/ --keys revenue "net income"
 
 ```
 
-Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list. A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
+```python file
+Script:
+from pathlib import Path
+from structured_data_extraction.main import process_single_pdf, process_batch, finalize_outputs
+
+path = Path("pdf/example.pdf")
+# OR dir = Path("pdf_dir")
+keys = ["key_1", "key_2", "key_3"]
+
+results = process_single_pdf(path, keys)
+# OR process_batch(pdf_dir, user_keys)
+
+
+
+finalize_outputs(levels=["all"])
+
+print("Done. See output/fused.json and output/records.json")
+```
+
+Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list.
+
+### Custom schema (skip schema discovery)
+
+Pass `--schema` with a JSON file to bypass the schema-discovery LLM call entirely. The file must contain `document_type` and `schema` keys, using the same format as `intermediate/*_schema.json`. Keys are derived from the schema, so `--keys` is optional.
+
+Use `"fixed"` on any sub-field to constrain the LLM to a single value — the field becomes a `Literal` type in the Pydantic model, so the model can only emit that exact value. This lets you slice a table down to specific variants, grades, or qualifiers:
+
+```bash
+structured-extract pdf/AD840.pdf --schema my_schema.json
+```
+
+```json
+{
+  "document_type": "operational amplifier datasheet",
+  "schema": {
+    "quiescent_current_s_typ": {
+      "type": "list",
+      "description": "Quiescent current typical value for the AD840S variant",
+      "fields": {
+        "differentiator": {
+          "type": "string",
+          "description": "Product variant",
+          "fixed": "AD840S"
+        },
+        "qualifier": {
+          "type": "string",
+          "description": "Column label: min, typ, or max",
+          "fixed": "Typ"
+        },
+        "condition": { "type": "string", "description": "Operating condition" },
+        "value": { "type": "number", "description": "Quiescent current value", "unit": "mA" },
+        "metadata": { "type": "object", "description": "Source and provenance" }
+      }
+    }
+  }
+}
+```
+
+A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
 
 A multi-model run then ends by printing the confidence-tier histogram and offering to keep a subset of tiers — pass `--levels` (e.g. `--levels 1.0 0.75`, or `all`) to choose up front, or omit it for an interactive prompt. Single-model runs skip this step, since every record is confidence 1.0.
 

{structured_data_extraction-0.1.0 → structured_data_extraction-0.2.0}/README.md

@@ -108,6 +108,7 @@ Notes:
 ## Usage
 
 ```bash
+CLI:
 # one PDF
 structured-extract pdf/1167fc.pdf --keys gain CMRR "supply voltage" #when key is more than one word then it should be inside ""
 
@@ -116,7 +117,65 @@ structured-extract pdf/ --keys revenue "net income"
 
 ```
 
-Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list. A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
+```python file
+Script:
+from pathlib import Path
+from structured_data_extraction.main import process_single_pdf, process_batch, finalize_outputs
+
+path = Path("pdf/example.pdf")
+# OR dir = Path("pdf_dir")
+keys = ["key_1", "key_2", "key_3"]
+
+results = process_single_pdf(path, keys)
+# OR process_batch(pdf_dir, user_keys)
+
+
+
+finalize_outputs(levels=["all"])
+
+print("Done. See output/fused.json and output/records.json")
+```
+
+Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list.
+
+### Custom schema (skip schema discovery)
+
+Pass `--schema` with a JSON file to bypass the schema-discovery LLM call entirely. The file must contain `document_type` and `schema` keys, using the same format as `intermediate/*_schema.json`. Keys are derived from the schema, so `--keys` is optional.
+
+Use `"fixed"` on any sub-field to constrain the LLM to a single value — the field becomes a `Literal` type in the Pydantic model, so the model can only emit that exact value. This lets you slice a table down to specific variants, grades, or qualifiers:
+
+```bash
+structured-extract pdf/AD840.pdf --schema my_schema.json
+```
+
+```json
+{
+  "document_type": "operational amplifier datasheet",
+  "schema": {
+    "quiescent_current_s_typ": {
+      "type": "list",
+      "description": "Quiescent current typical value for the AD840S variant",
+      "fields": {
+        "differentiator": {
+          "type": "string",
+          "description": "Product variant",
+          "fixed": "AD840S"
+        },
+        "qualifier": {
+          "type": "string",
+          "description": "Column label: min, typ, or max",
+          "fixed": "Typ"
+        },
+        "condition": { "type": "string", "description": "Operating condition" },
+        "value": { "type": "number", "description": "Quiescent current value", "unit": "mA" },
+        "metadata": { "type": "object", "description": "Source and provenance" }
+      }
+    }
+  }
+}
+```
+
+A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
 
 A multi-model run then ends by printing the confidence-tier histogram and offering to keep a subset of tiers — pass `--levels` (e.g. `--levels 1.0 0.75`, or `all`) to choose up front, or omit it for an interactive prompt. Single-model runs skip this step, since every record is confidence 1.0.
 

{structured_data_extraction-0.1.0 → structured_data_extraction-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "structured-data-extraction"
-version = "0.1.0"
+version = "0.2.0"
 description = "Pipeline for extracting structured data (tables, sections, key/value fields) from PDF documents using docling, embeddings, and LLMs."
 readme = "README.md"
 requires-python = ">=3.11"

{structured_data_extraction-0.1.0 → structured_data_extraction-0.2.0}/src/structured_data_extraction/extract.py

@@ -23,13 +23,13 @@ import os
 from concurrent.futures import ThreadPoolExecutor, as_completed
 from datetime import datetime, timezone
 from pathlib import Path
-from typing import Any, List, Optional
+from typing import Any, List, Literal, Optional
 import instructor
 from jinja2 import Environment, ChoiceLoader, FileSystemLoader, PackageLoader
 from openai import OpenAI
 from pydantic import BaseModel, Field, create_model
 from . import config, matching
-from .paths import inter, read_json, cached
+from .paths import inter, read_json, write_json, cached
 from .tables import strip_code_fence
 
 logger = logging.getLogger(__name__)
@@ -110,10 +110,15 @@ def _build_extraction_model(schema, user_keys):
         for sub_name, sub_def in fields_spec.items():
             if sub_name == "metadata":
                 continue
-            field_type = type_map.get(sub_def.get("type", "string"), str)
             sub_desc = sub_def.get("description", "")
-            default = "" if field_type is str else None
-            sub_fields[sub_name] = (field_type, Field(default=default, description=sub_desc))
+            fixed = sub_def.get("fixed")
+            if fixed is not None:
+                field_type = Literal[fixed]
+                sub_fields[sub_name] = (field_type, Field(default=fixed, description=sub_desc))
+            else:
+                field_type = type_map.get(sub_def.get("type", "string"), str)
+                default = "" if field_type is str else None
+                sub_fields[sub_name] = (field_type, Field(default=default, description=sub_desc))
             if sub_def.get("unit") and not has_unit_field:
                 sub_fields[f"{sub_name}_unit"] = (str, Field(default="", description=f"Unit for {sub_name}"))
 
@@ -368,7 +373,7 @@ def _run_one_model(model_name, stem, out_dir, unique, vocab, user_keys, schema,
     return extraction
 
 
-def extract_parallel(stem, user_keys):
+def extract_parallel(stem, user_keys, user_schema=None):
     """Run every configured extraction model over one document in parallel.
 
     Stamps run provenance (run.json), discovers the schema once for all models, then fans
@@ -387,11 +392,15 @@ def extract_parallel(stem, user_keys):
 
     md_text, vocab, unique = _prepare_sections(stem)
 
-    # schema discovery once for all models — cached to intermediate/<stem>_schema.json
-    disc = cached(inter(stem, "schema.json"),
-                  lambda: dict(zip(("document_type", "schema"),
-                                   discover_schema(md_text, unique, vocab, user_keys))))
-    doc_type, schema = disc["document_type"], disc["schema"]
+    if user_schema is not None:
+        doc_type, schema = user_schema["document_type"], user_schema["schema"]
+        write_json(inter(stem, "schema.json"), user_schema)
+        logger.info("Using user-supplied schema (skipped schema discovery)")
+    else:
+        disc = cached(inter(stem, "schema.json"),
+                      lambda: dict(zip(("document_type", "schema"),
+                                       discover_schema(md_text, unique, vocab, user_keys))))
+        doc_type, schema = disc["document_type"], disc["schema"]
     logger.info(f"Schema: document_type='{doc_type}'")
 
     results = []

{structured_data_extraction-0.1.0 → structured_data_extraction-0.2.0}/src/structured_data_extraction/main.py

@@ -49,7 +49,7 @@ def _invalidate_stale_key_caches(stem, user_keys):
         p.unlink()
 
 
-def process_single_pdf(pdf_path, user_keys):
+def process_single_pdf(pdf_path, user_keys, user_schema=None):
     """Run the full extraction pipeline on one PDF.
 
     Executes every stage in order: convert → vocab → classify → tables →
@@ -150,7 +150,7 @@ def process_single_pdf(pdf_path, user_keys):
             refine_matched_tables(str(tables_file), matched_idxs, pdf_path=str(pdf_path))
 
     with timed("extract"):
-        results = extract_parallel(stem, user_keys)
+        results = extract_parallel(stem, user_keys, user_schema=user_schema)
 
     _log_run_summary(stem, timings, results, matches)
     logger.info(f"=== Done with {pdf_path.name} ===")
@@ -233,7 +233,7 @@ def _offer_filter(records_path, levels):
     filter_records.run(records_path, levels=levels)
 
 
-def process_batch(pdf_dir, user_keys):
+def process_batch(pdf_dir, user_keys, user_schema=None):
     pdf_dir = Path(pdf_dir)
     if not pdf_dir.exists():
         logger.error(f"Directory not found: {pdf_dir}")
@@ -248,7 +248,7 @@ def process_batch(pdf_dir, user_keys):
     results = []
     for pdf in pdfs:
         try:
-            result = process_single_pdf(str(pdf), user_keys)
+            result = process_single_pdf(str(pdf), user_keys, user_schema=user_schema)
             if result:
                 results.append(result)
         except Exception as e:
@@ -280,6 +280,9 @@ def main():
     parser.add_argument("--levels", nargs="+", metavar="LVL",
                         help="confidence tiers to keep (e.g. 1.0 0.75, or 'all'); "
                              "omit to be prompted after extraction")
+    parser.add_argument("--schema", metavar="FILE",
+                        help="Path to a JSON schema file (same format as intermediate/*_schema.json); "
+                             "skips schema discovery. Keys are derived from the schema, so --keys is optional")
     parser.add_argument("--dump-templates", metavar="DIR",
                         help="Write the built-in prompt templates to DIR for editing, then exit; "
                              "point TEMPLATES_DIR at that folder to use your edits")
@@ -293,8 +296,22 @@ def main():
         parser.error("a PDF file or directory is required (unless using --dump-templates)")
 
     config.FORCE = args.force
+
+    user_schema = None
+    if args.schema:
+        schema_path = Path(args.schema)
+        if not schema_path.exists():
+            print(f"Schema file not found: {schema_path}")
+            sys.exit(1)
+        user_schema = json.loads(schema_path.read_text(encoding="utf-8"))
+        if not isinstance(user_schema, dict) or "schema" not in user_schema or "document_type" not in user_schema:
+            print("Schema file must contain 'document_type' and 'schema' keys")
+            sys.exit(1)
+
     user_keys = args.keys
-    if not user_keys:
+    if not user_keys and user_schema:
+        user_keys = list(user_schema["schema"].keys())
+    elif not user_keys:
         print("Enter keys to extract (comma-separated):")
         raw = input("> ").strip()
         user_keys = [k.strip() for k in raw.split(",") if k.strip()]
@@ -308,7 +325,7 @@ def main():
     config.OUTPUT_DIR = args.output
 
     if Path(args.path).is_dir():
-        results = process_batch(args.path, user_keys)
+        results = process_batch(args.path, user_keys, user_schema=user_schema)
         if results:
             finalize_outputs(args.levels)
         print(f"Extracted from {len(results)} files → {config.OUTPUT_DIR}")
@@ -316,7 +333,7 @@ def main():
             print("No documents produced any extraction.")
             sys.exit(1)
     else:
-        result = process_single_pdf(args.path, user_keys)
+        result = process_single_pdf(args.path, user_keys, user_schema=user_schema)
         if result:
             finalize_outputs(args.levels)
             print(json.dumps(result, indent=2, ensure_ascii=False))  #document them well

{structured_data_extraction-0.1.0 → structured_data_extraction-0.2.0/src/structured_data_extraction.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: structured-data-extraction
-Version: 0.1.0
+Version: 0.2.0
 Summary: Pipeline for extracting structured data (tables, sections, key/value fields) from PDF documents using docling, embeddings, and LLMs.
 Author-email: Dimitrios Avellas <dimitrisavellas8@gmail.com>
 License-Expression: MIT
@@ -140,6 +140,7 @@ Notes:
 ## Usage
 
 ```bash
+CLI:
 # one PDF
 structured-extract pdf/1167fc.pdf --keys gain CMRR "supply voltage" #when key is more than one word then it should be inside ""
 
@@ -148,7 +149,65 @@ structured-extract pdf/ --keys revenue "net income"
 
 ```
 
-Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list. A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
+```python file
+Script:
+from pathlib import Path
+from structured_data_extraction.main import process_single_pdf, process_batch, finalize_outputs
+
+path = Path("pdf/example.pdf")
+# OR dir = Path("pdf_dir")
+keys = ["key_1", "key_2", "key_3"]
+
+results = process_single_pdf(path, keys)
+# OR process_batch(pdf_dir, user_keys)
+
+
+
+finalize_outputs(levels=["all"])
+
+print("Done. See output/fused.json and output/records.json")
+```
+
+Wrap multi-word keys in quotes. If you omit `--keys`, it prompts for a comma-separated list.
+
+### Custom schema (skip schema discovery)
+
+Pass `--schema` with a JSON file to bypass the schema-discovery LLM call entirely. The file must contain `document_type` and `schema` keys, using the same format as `intermediate/*_schema.json`. Keys are derived from the schema, so `--keys` is optional.
+
+Use `"fixed"` on any sub-field to constrain the LLM to a single value — the field becomes a `Literal` type in the Pydantic model, so the model can only emit that exact value. This lets you slice a table down to specific variants, grades, or qualifiers:
+
+```bash
+structured-extract pdf/AD840.pdf --schema my_schema.json
+```
+
+```json
+{
+  "document_type": "operational amplifier datasheet",
+  "schema": {
+    "quiescent_current_s_typ": {
+      "type": "list",
+      "description": "Quiescent current typical value for the AD840S variant",
+      "fields": {
+        "differentiator": {
+          "type": "string",
+          "description": "Product variant",
+          "fixed": "AD840S"
+        },
+        "qualifier": {
+          "type": "string",
+          "description": "Column label: min, typ, or max",
+          "fixed": "Typ"
+        },
+        "condition": { "type": "string", "description": "Operating condition" },
+        "value": { "type": "number", "description": "Quiescent current value", "unit": "mA" },
+        "metadata": { "type": "object", "description": "Source and provenance" }
+      }
+    }
+  }
+}
+```
+
+A full run writes per-model extractions to `output/<stem>_extraction_<model>.json`, then fuses them into `fused.json` (every fact, sorted by confidence) plus `rating_report.json`, and wraps `fused.json` into the provenance-stamped `records.json`.
 
 A multi-model run then ends by printing the confidence-tier histogram and offering to keep a subset of tiers — pass `--levels` (e.g. `--levels 1.0 0.75`, or `all`) to choose up front, or omit it for an interactive prompt. Single-model runs skip this step, since every record is confidence 1.0.