falcon-perception 1.0.0__tar.gz

falcon_perception-1.0.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 Technology Innovation Institute (TII), UAE
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

falcon_perception-1.0.0/NOTICE

@@ -0,0 +1,21 @@
+Falcon Perception
+Copyright 2026 Technology Innovation Institute (TII), UAE
+https://github.com/tiiuae/falcon-perception
+
+This product includes software developed at the Technology Innovation
+Institute (TII), Abu Dhabi, UAE.
+
+If you use Falcon Perception — including its model weights, inference engine,
+or derivative works (e.g. fine-tuned models, adapted architectures) — in any
+public-facing work, publication, or product, please acknowledge the project by:
+
+  1. Retaining this NOTICE file in any redistribution.
+  2. Citing the project in academic or technical publications:
+
+     @preprint{falcon-perception2026,
+       title     = {Falcon Perception: Natively Multimodal Dense Autoregressive
+                    Vision-Language Transformer},
+       author    = {TII Falcon Perception Team},
+       year      = {2025},
+       url       = {https://github.com/tiiuae/falcon-perception}
+     }

falcon_perception-1.0.0/PKG-INFO

@@ -0,0 +1,411 @@
+Metadata-Version: 2.4
+Name: falcon-perception
+Version: 1.0.0
+Summary: Falcon Perception is a dense vision-language model that performs object detection, instance segmentation, and OCR from natural language queries.
+Author: Technology Innovation Institute
+Maintainer-email: "Phuc H. Le Khac" <phuc.lekhac@tii.ae>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/tiiuae/Falcon-Perception
+Project-URL: Repository, https://github.com/tiiuae/Falcon-Perception
+Project-URL: Issues, https://github.com/tiiuae/Falcon-Perception/issues
+Project-URL: Hugging Face, https://huggingface.co/tiiuae/Falcon-Perception
+Keywords: vision-language-model,object-detection,instance-segmentation,ocr,falcon,deep-learning,computer-vision
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: einops>=0.8.1
+Requires-Dist: numpy>=1.24
+Requires-Dist: pillow>=12.0.0
+Requires-Dist: tokenizers>=0.21.0
+Requires-Dist: datasets>=4.5.0
+Requires-Dist: pycocotools>=2.0.11
+Requires-Dist: opencv-python>=4.12.0.88
+Requires-Dist: scipy>=1.15.3
+Requires-Dist: tyro>=1.0.3
+Requires-Dist: hf-transfer>=0.1.9
+Requires-Dist: requests>=2.31.0
+Requires-Dist: hf-xet>=1.3.1
+Requires-Dist: mlx>=0.30.0; sys_platform == "darwin" and platform_machine == "arm64"
+Requires-Dist: torch>=2.11.0; sys_platform != "darwin"
+Requires-Dist: torchvision>=0.26.0; sys_platform != "darwin"
+Requires-Dist: safetensors>=0.7.0
+Provides-Extra: torch
+Requires-Dist: torch>=2.11.0; extra == "torch"
+Requires-Dist: torchvision>=0.26.0; extra == "torch"
+Provides-Extra: mlx
+Requires-Dist: mlx>=0.30.0; (sys_platform == "darwin" and platform_machine == "arm64") and extra == "mlx"
+Provides-Extra: ocr
+Requires-Dist: falcon-perception[torch]; extra == "ocr"
+Requires-Dist: transformers>=4.57.3; extra == "ocr"
+Provides-Extra: dev
+Requires-Dist: tensorboard>=2.20.0; extra == "dev"
+Requires-Dist: ipykernel>=7.2.0; extra == "dev"
+Requires-Dist: matplotlib>=3.10.8; extra == "dev"
+Provides-Extra: server
+Requires-Dist: fastapi>=0.129.0; extra == "server"
+Requires-Dist: uvicorn[standard]>=0.40.0; extra == "server"
+Requires-Dist: python-multipart>=0.0.22; extra == "server"
+Provides-Extra: demo
+Requires-Dist: streamlit>=1.52.2; extra == "demo"
+Requires-Dist: requests>=2.31.0; extra == "demo"
+Dynamic: license-file
+
+
+<p align="center">
+  <img src="assets/logo.png" width="70%" alt="Falcon Perception and Falcon OCR" />
+</p>
+
+<p align="center">
+  <a href="http://arxiv.org/abs/2603.27365"><img src="https://img.shields.io/badge/arXiv-Tech%20Report-b31b1b?logo=arxiv&logoColor=white" alt="Tech Report" /></a>&nbsp;&nbsp;
+  <a href="https://huggingface.co/blog/tiiuae/falcon-perception"><img src="https://img.shields.io/badge/Blogpost-2196F3?logo=googledocs&logoColor=white" alt="Blogpost" /></a>&nbsp;&nbsp;
+  <a href="https://vision.falcon.aidrc.tii.ae"><img src="https://img.shields.io/badge/%F0%9F%9A%80-Playground-blue" alt="Playground" /></a>
+</p>
+<p align="center">
+  <a href="https://huggingface.co/datasets/tiiuae/PBench"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-PBench-orange" alt="PBench" /></a>&nbsp;&nbsp;
+  <a href="https://huggingface.co/tiiuae/Falcon-OCR"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-OCR%20Model-orange" alt="OCR Model" /></a>&nbsp;&nbsp;
+  <a href="https://huggingface.co/tiiuae/Falcon-Perception"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Perception%20Model-orange" alt="Perception Model" /></a>
+</p>
+
+# Falcon Perception
+A minimal, readable yet performant PyTorch inference engine implementation of **Falcon Perception** — a natively multimodal, dense, autoregressive Transformer model that performs **object detection**, **instance segmentation**, or **OCR** from natural language queries.
+
+![falcon_inference](https://github.com/user-attachments/assets/1e45051b-ae05-4170-b326-d61d79a47ba2)
+
+
+> *"Segment these expressions in the image: the cat on the left"* → bounding boxes + pixel-level masks
+
+> *"Extract the text content from this image."* → text / latex formulas / html table ... 
+
+
+## Contents
+- [Quick Start](#quick-start)
+  - [Installation](#installation)
+  - [Run Perception (detection / segmentation)](#run-perception-detection--segmentation)
+  - [Run OCR (text extraction)](#run-ocr-text-extraction)
+  - [Run Multiple Samples with Paged Inference Engine](#run-multiple-samples-with-paged-inference-engine)
+  - [Interactive Notebooks](#interactive-notebooks)
+- [Inference Engines](#inference-engines)
+  - [PyTorch Inference Engines](#pytorch-inference-engines)
+    - [Paged Inference Engine](#paged-inference-engine)
+    - [Paged OCR Inference Engine](#paged-ocr-inference-engine)
+    - [Batch Inference Engine](#batch-inference-engine)
+  - [MLX Batch Inference Engine (Apple Silicon)](#mlx-batch-inference-engine-apple-silicon)
+- [Inference Server](#inference-server)
+  - [Launch Server](#launch-server)
+  - [Launch Streamlit Demo App](#launch-streamlit-demo-app)
+- [vLLM Docker Server (FalconOCR Only)](#vllm-docker-server-falconocr-only)
+  - [Serving Throughput](#serving-throughput)
+  - [Quick Start](#quick-start-1)
+- [Citation](#citation)
+- [Acknowledgments](#acknowledgments)
+
+
+## Quick Start
+
+### Installation
+
+The package supports two backends: **PyTorch** (CUDA GPUs) and **MLX** (Apple Silicon Macs).
+A bare `pip install` auto-detects your platform, or you can pick an explicit extra.
+
+| Install command | Backend | When to use |
+|---|---|---|
+| `pip install -e .` | Auto-detect | Mac -> MLX, Linux -> Torch |
+| `pip install -e ".[torch]"` | PyTorch + CUDA | GPU server or explicit Torch on Mac |
+| `pip install -e ".[mlx]"` | MLX | Apple Silicon Mac |
+| `pip install -e ".[ocr]"` | Torch + transformers | Layout-aware OCR (needs a layout detection model) |
+| `pip install -e ".[dev]"` | -- | Adds tensorboard, matplotlib, ipykernel |
+| `pip install -e ".[server]"` | -- | Adds FastAPI / Uvicorn for the paged inference server |
+
+```bash
+# Example: from source with uv (recommended)
+uv sync                       # installs torch with CUDA 12.8 wheels by default
+source .venv/bin/activate
+```
+
+#### CUDA version
+
+`uv sync` defaults to **CUDA 12.8** wheels for PyTorch (works with NVIDIA driver >= 570.x).
+If you need a different CUDA version, edit the `[[tool.uv.index]]` URL in `pyproject.toml`:
+
+```bash
+# CUDA 12.6 (driver >= 560.x)
+url = "https://download.pytorch.org/whl/cu126"
+# CUDA 13.0 (driver >= 575.x) — or remove [tool.uv.sources] entirely
+url = "https://download.pytorch.org/whl/cu130"
+```
+
+Then re-run `uv lock && uv sync`.
+
+If installing with **pip** instead of uv, install PyTorch first with the correct CUDA version:
+
+```bash
+pip install torch torchvision --index-url https://download.pytorch.org/whl/cu128
+pip install -e .
+```
+
+> **Note:** The MLX backend does not require PyTorch or transformers at all.
+> The core data pipeline and tokenizer run on numpy/PIL and the lightweight
+> [tokenizers](https://github.com/huggingface/tokenizers) library.
+
+### Run Perception (detection / segmentation)
+
+**PyTorch (GPU)**
+```bash
+# Auto-downloads model + streaming sample image from Huggingface
+python demo/perception_single.py
+
+# Custom image and query, can be path or url
+python demo/perception_single.py --image photo.jpg --query "cat"
+
+# Detection only (no masks)
+python demo/perception_single.py --image photo.jpg --query "cat" --task detection
+```
+
+**MLX (Apple Silicon)**
+```bash
+python demo/perception_single_mlx.py --image photo.jpg --query "cat"
+
+# Detection only
+python demo/perception_single_mlx.py --image photo.jpg --query "cat" --task detection
+```
+
+### Run OCR (text extraction)
+
+**PyTorch (GPU)**
+```bash
+# Auto-downloads model + stream sample image from HuggingFace
+python demo/ocr_single.py
+
+# Custom document image
+python demo/ocr_single.py --image document.png
+
+# Layout-aware OCR (detects regions first, then extracts text per region)
+# Requires the [ocr] extra: pip install -e ".[ocr]"
+# This will lazily download and run a 3rd party layout detection model
+# PaddlePaddle/PP-DocLayoutV3_safetensors from Huggingface
+python demo/ocr_single.py --image document.png --task ocr_layout
+```
+
+**MLX (Apple Silicon)**
+```bash
+python demo/ocr_single_mlx.py --image document.png
+python demo/ocr_single_mlx.py  # loads a demo sample from OCRBench-v2
+```
+
+**OCR modes**
+
+| Mode | Best for | How |
+|------|----------|-----|
+| **Plain OCR** | Simple documents, real-world photos, slides, receipts, invoices | `--task ocr_plain` |
+| **Layout + OCR** | Complex multi-column documents, academic papers, reports, dense pages | `--task ocr_layout` |
+
+### Run Multiple Samples with Paged Inference Engine
+
+```bash
+# Perception — PBench dataset
+python demo/perception_benchmark.py  # stream 50 samples from HF
+python demo/perception_benchmark.py --limit -1  # stream entire benchmark
+
+# OCR — OCRBench-v2 dataset
+python demo/ocr_benchmark.py
+python demo/ocr_benchmark.py --limit 200
+```
+
+### Interactive Notebooks
+
+Step-by-step walkthroughs with inline visualizations:
+
+| Notebook | Description | Colab |
+|---|---|---|
+| [`demo/perception.ipynb`](demo/perception.ipynb) | **Falcon Perception** — engine setup, detection vs segmentation, HR cache, dense tuning, PBench level showcase, and benchmark with IoU metrics | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1Jy6lRYuGMKJEt9--KBLm6kjbC-vQ0Uwe) |
+| [`demo/ocr.ipynb`](demo/ocr.ipynb) | **Falcon OCR** — full-page and layout-based OCR across handwriting, formulas, tables, scanned documents, and scientific papers | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1HYOcYWqmeUT9j2h6_dNM2lX1mlI34Bfa) |
+| [`demo/perception_agent.ipynb`](demo/perception_agent.ipynb) | **Perception Agent** — grounded visual reasoning agent using Falcon Perception as a tool with an orchestrator VLM for multi-step scene understanding | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1wg4EbNDKllAxoK1v5gbdRzkrKjdjIwbj) |
+| [`demo/perception_ov_mot.ipynb`](demo/perception_ov_mot.ipynb) | **Open-Vocab Multi-Object Tracking** — video object tracking pipeline using Falcon Perception in detection and segmentation modes | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/13Hiei8As2JbiZuYllXQTNX_ZelysZSpH) |
+
+
+## Inference Engines
+
+### PyTorch Inference Engines
+
+**We use FlexAttention for both inference engines and training.**
+The hybrid attention mask (bidirectional image + causal text) is expressed as composable mask functions. 
+The FlexAttention's `maskmod` also make it easy to implement continuous batching with paged attention via simple Python.
+PyTorch's `flex_attention` compiles them into fused Triton kernel — no custom attention code needed.
+
+#### Paged Inference Engine
+
+Performant engine with CUDAGraph and continuous batching via a paged KV cache:
+
+- **Paged KV cache** with virtual page tables (no wasted memory from padding)
+- **Continuous batching**: new sequences enter mid-generation, finished ones release pages immediately
+- **Torch compile**: piece-wise region outside the flex attention kernel.
+- **CUDA graph capture** for the decode loop (eliminates kernel launch overhead, important for small models)
+- **Background tokenization**: CPU thread pool overlapped with GPU compute
+- **Preemption**: if memory is tight, running sequences can be paused and re-prefilled later
+- **High-Resolution image feature cache** (for segmentation): LRU cache with pre-allocated pinned memory buffers for async GPU↔CPU transfer of high-resolution image features. Help reduce prefill time for subsequent query of the same image.
+
+Please check `demo/perception_single.py` and `demo/perception_benchmark.py` on how to directly instantiate and use the engine.
+
+> **NOTE**: First run will takes longer ~10-30s for torch compile and CUDAGraph capture. Subsequent run will be faster, around ~100ms for prefill, ~200ms for upsampling (0ms if cached), and ~50ms for decode a couple of instances (~10 tokens). (measured on H100)
+
+#### Paged OCR Inference Engine
+
+Extends the Perception's paged engine for document understanding:
+
+- **Layout detection**: runs a lightweight detector to find text regions, tables, figures, headers
+- **Per-region OCR**: crops each region and runs OCR inference with a category-specific prompt
+- **Continuous batching and gather**: all crops of the image are sent to the engine for continuous batched extraction. Once all crops are completed, the output are gathered and assembed into a structured output.
+
+Please check `demo/ocr_single.py` and `demo/ocr_benchmark.py` on how to directly instantiate and use the engine.
+
+> **NOTE**: First `layout_ocr` run will lazily download and run the document layout detection model.
+
+#### Batch Inference Engine
+
+The simplest and closest to training code path, make it easier to understand the model's forward pass without all the optimization.
+All sequences are left-padded to the same length (with correct rope indices and attention mask), runs a single prefill, then decodes token-by-token with a dense KV cache until all sequences are completed.
+
+Please check the `demo/perception_single.py --engine-type batch` path for usage.
+
+
+### MLX Batch Inference Engine (Apple Silicon)
+
+The MLX backend provides batch inference on Apple Silicon Macs using the
+[MLX](https://github.com/ml-explore/mlx) framework. It shares the same
+model architecture and weights (auto-converted from safetensors on first load)
+and produces equivalent results.
+
+- Dense KV cache, left-padded batch inference
+- `mx.fast.scaled_dot_product_attention` with native sink support
+- Tiled windowed cross-attention in the AnyUp upsampler for memory efficiency
+- No PyTorch or transformers dependency
+
+See `demo/perception_single_mlx.py` for usage.
+
+## Inference Server
+
+The server provides a REST API to the continuous batching Paged Inference Engine across multiple GPUs.
+
+### Launch server
+
+```bash
+# Install server and streamlit demo dependencies
+uv sync --extra server --extra demo
+
+# Auto-detects all available GPUs, compiles model, captures CUDA graphs
+python -m falcon_perception.server
+
+# Explicit config
+python -m falcon_perception.server --config.num-gpus 2 --config.port 7680
+
+# Or with the OCR model
+python -m falcon_perception.server --config.hf-model-id tiiuae/Falcon-OCR --config.port 7681
+```
+
+The server starts one engine worker per GPU in a separate process (i.e. Data Parallel). Each worker builds its own model, runs `torch.compile`, and captures CUDA graphs for the decode loop. Workers communicate with the main FastAPI process via `multiprocessing.Queue` and the server will assign new request to the worker with lowest number of queuing requests.
+
+Please check the [`server/README.md`](falcon_perception/server/README.md) for detailed usage.
+
+
+### Launch Streamlit Demo App
+
+A browser-based demo UI that connects to the inference server.
+
+```bash
+# With a server already launched in a separate terminal, launch the Streamlit app
+streamlit run demo/streamlit_app.py
+```
+
+The app provides:
+- Image upload and URL input
+- Automatic model detection — the task selector shows only tasks supported by the loaded model (segmentation/detection for Falcon-Perception, OCR plain/layout for Falcon-OCR)
+- Interactive mask and bounding box overlay visualization
+- Adjustable image resolution settings
+- Per-request timing breakdown
+
+
+## vLLM Docker Server (FalconOCR Only)
+For FalconOCR without the extra perception heads, we also provide a vLLM docker server for fast deployment.
+
+### Serving Throughput
+
+Measured on a single A100-80GB GPU with vLLM, processing document images from olmOCR-Bench at very high concurrency for optimal utilisation of vllm.
+
+- **Layout + OCR** — The full end-to-end pipeline: layout detection finds regions on each page, crops them, and the VLLM runs on every crop. This is the real-world serving number that includes both layout and OCR time.
+
+| Mode | tok/s | img/s | Description |
+|------|------:|------:|-------------|
+| **Layout + OCR** | 5,825 | 2.9 | Full pipeline: layout detection → crop → per-region OCR |
+
+At 0.3B parameters, Falcon OCR is roughly 3x smaller than 0.9B-class OCR VLMs (e.g. PaddleOCR VL), which translates directly into higher serving throughput at competitive accuracy.
+
+### Quick Start
+For production OCR serving, we also provide a Docker image (~6,000 tok/s on a single A100) that bundles vLLM with an end-to-end parsing pipeline. It exposes two services: **vLLM** (port 8000, OpenAI-compatible API) and a **Pipeline** (port 5002, layout → crop → OCR → markdown).
+
+```bash
+# Two GPUs (best throughput): vLLM on one, layout model on the other
+docker run -d --name falcon-ocr \
+  --gpus '"device=0,1"' \
+  -e EXPOSED_GPU_IDS=0,1 -e VLLM_GPU=0 -e PIPELINE_GPU=1 \
+  -e VLLM_GPU_MEM_UTIL=0.90 \
+  -p 8000:8000 -p 5002:5002 \
+  ghcr.io/tiiuae/falcon-ocr:latest
+
+# Single GPU (share memory — lower VLLM_GPU_MEM_UTIL to leave room for layout model)
+docker run -d --name falcon-ocr \
+  --gpus '"device=0"' \
+  -e EXPOSED_GPU_IDS=0 -e VLLM_GPU=0 -e PIPELINE_GPU=0 \
+  -e VLLM_GPU_MEM_UTIL=0.55 -e MAX_NUM_SEQS=512 \
+  -p 8000:8000 -p 5002:5002 \
+  ghcr.io/tiiuae/falcon-ocr:latest
+```
+
+```bash
+# Upload (images or PDFs)
+curl -X POST http://localhost:5002/falconocr/upload \
+  -F "files=@document.pdf;type=application/pdf"
+
+# Parse with layout detection
+curl -X POST http://localhost:5002/falconocr/parse \
+  -H "Content-Type: application/json" \
+  -d '{"images": ["data:image/jpeg;base64,<...>"], "skip_layout": false}'
+
+# Parse without layout (full image → VLM directly)
+curl -X POST http://localhost:5002/falconocr/parse \
+  -H "Content-Type: application/json" \
+  -d '{"images": ["data:image/jpeg;base64,<...>"], "skip_layout": true}'
+```
+
+See the [Falcon-OCR model card](https://huggingface.co/tiiuae/Falcon-OCR) for the full API reference, configuration variables, and deployment options.
+
+## Citation
+If you use Falcon-Perception, please cite
+```
+@article{bevli2026falcon,
+  title   = {Falcon Perception},
+  author  = {Bevli, Aviraj and Chaybouti, Sofian and Dahou, Yasser and Hacid, Hakim and Huynh, Ngoc Dung and Le Khac, Phuc H. and Narayan, Sanath and Para, Wamiq Reyaz and Singh, Ankit},
+  journal = {arXiv preprint arXiv:2603.27365},
+  year    = {2026},
+  url     = {https://arxiv.org/abs/2603.27365}
+}
+```
+## Acknowledgments
+We based on, built upon and are inspired by the following excellent work:
+- https://github.com/pytorch/torchtitan
+- https://github.com/karpathy/nanochat
+- https://github.com/meta-pytorch/attention-gym
+- https://github.com/changjonathanc/flex-nano-vllm
+- https://github.com/vikhyat/moondream
+- https://github.com/wimmerth/anyup
+- https://github.com/roboflow/trackers