deduplicate-it 1.2.0__tar.gz

deduplicate_it-1.2.0/LICENSE

@@ -0,0 +1,152 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship made available under
+      the License, as indicated by a copyright notice that is included
+      in or attached to the work.
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other
+      transformations represent, as a whole, an original work of authorship.
+
+      "Contribution" shall mean, as submitted to the Licensor for inclusion
+      in the Work by the copyright owner or by an individual or Legal Entity
+      authorized to submit on behalf of the copyright owner.
+
+      "Contributor" shall mean Licensor and any Legal Entity on behalf of
+      whom a Contribution has been received by the Licensor and included
+      within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent contributions
+      Licensable by such Contributor that are necessarily infringed by
+      their Contribution(s) alone or by the combination of their
+      Contribution(s) with the Work to which such Contribution(s) was
+      submitted. If You institute patent litigation against any entity
+      (including a cross-claim or counterclaim in a lawsuit) alleging that
+      the Work or any such patent application constitutes direct or
+      contributory patent infringement, then any patent rights granted to
+      You under this License for that Work shall terminate as of the date
+      such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the Work
+      or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You meet
+      the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative Works
+          a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works that
+          You distribute, all copyright, patent, trademark, and attribution
+          notices from the Source form of the Work, excluding those notices
+          that do not pertain to any part of the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file, You must include a
+          readable copy of the attribution notices contained within such
+          NOTICE file, in at least one of the following places: within a
+          NOTICE text provided with the Derivative Works; within the Source
+          form or documentation, if provided along with the Derivative Works;
+          or, within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or agreed
+      to in writing, Licensor provides the Work (and each Contributor
+      provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES
+      OR CONDITIONS OF ANY KIND, either express or implied, including,
+      without limitation, any warranties or conditions of TITLE,
+      NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE.
+      You are solely responsible for determining the appropriateness of using
+      or redistributing the Work and assume any risks associated with Your
+      exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise, unless
+      required by applicable law (such as deliberate and grossly negligent
+      acts) or agreed to in writing, shall any Contributor be liable to You
+      for damages, including any direct, indirect, special, incidental, or
+      consequential damages of any character arising as a result of this
+      License or out of the use or inability to use the Work.
+
+   9. Accepting Warranty or Additional Liability. While redistributing the
+      Work or Derivative Works thereof, You may choose to offer, and charge
+      a fee for, acceptance of support, warranty, indemnity, or other
+      liability obligations and/or rights consistent with this License.
+      However, in accepting such obligations, You may act only on Your own
+      behalf and on Your sole responsibility, not on behalf of any other
+      Contributor, and only if You agree to indemnify, defend, and hold each
+      Contributor harmless for any liability incurred by, or claims asserted
+      against, such Contributor by reason of your accepting any such warranty
+      or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2025 David Purkarthofer and Sebastian Labenbacher
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

deduplicate_it-1.2.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: deduplicate-it
+Version: 1.2.0
+Summary: Free, auditable removal of duplicate records from literature search exports for systematic reviews, scoping reviews and meta-analyses.
+Author: David Purkarthofer, Sebastian Labenbacher
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://deduplicate.it
+Project-URL: Repository, https://github.com/dpurkarthofer/deduplicate.it
+Project-URL: Paper, https://doi.org/10.1177/18911803261484937
+Project-URL: Archive, https://doi.org/10.5281/zenodo.18835297
+Project-URL: Changelog, https://github.com/dpurkarthofer/deduplicate.it/blob/main/CHANGELOG.md
+Keywords: systematic-review,scoping-review,meta-analysis,deduplication,evidence-synthesis,literature-search,PRISMA,RIS,BibTeX,MEDLINE
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: Intended Audience :: Healthcare Industry
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Topic :: Text Processing :: Filters
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# deduplicate.it
+
+Free, open-source tool for automated deduplication of literature search exports for systematic reviews, scoping reviews, and meta-analyses.
+
+**Web interface:** [deduplicate.it](https://deduplicate.it)
+
+[![Paper](https://img.shields.io/badge/Campbell%20Syst%20Rev-10.1177%2F18911803261484937-1a7f37)](https://doi.org/10.1177/18911803261484937)
+[![Software archive](https://img.shields.io/badge/Zenodo-10.5281%2Fzenodo.18835297-1682d4)](https://doi.org/10.5281/zenodo.18835297)
+[![License](https://img.shields.io/badge/license-Apache--2.0-lightgrey)](LICENSE)
+
+**Peer-reviewed and published.** The matching rule, the evaluation and the limitations are described in full in the methods paper:
+
+> Purkarthofer D, Labenbacher S, Bornemann-Cimenti H, Landoni G. deduplicate.it: A simple open-source tool to remove duplicates from literature searches. *Campbell Systematic Reviews*. 2026;22(3). doi:[10.1177/18911803261484937](https://doi.org/10.1177/18911803261484937)
+
+Across five real-world searches (958–2,408 records) it removed **76.6–89.6%** of the duplicates found by Rayyan-assisted manual deduplication and cut deduplication time by **68.4–86.7%**. Records without a valid DOI, and records whose DOIs collide, are never silently merged: they are retained, exported separately and sorted to the top of the output for manual review.
+
+See [Citation](#citation) for when to cite the software archive instead.
+
+## Contents
+
+### `web/` — PHP web application
+Deploy to any PHP 8.x server. Place `index.php`, `legal.php`, and both `.drawio.html` flowchart templates in the same directory.
+
+`.user.ini` raises the upload and memory limits the tool needs — PHP's default
+`upload_max_filesize` of 2 MB is smaller than a typical database export. On shared hosting
+running PHP-FPM, keep it beside `index.php`; on other setups apply the equivalent
+directives in `php.ini`.
+
+### `deduplicate_it/` — Python command-line tool
+No external dependencies (Python standard library only). Requires Python 3.11 or newer.
+
+**Install:**
+
+```bash
+pip install deduplicate-it
+```
+
+**Usage:**
+
+```bash
+deduplicate-it --source path/to/exports --outdir path/to/results
+```
+
+With no arguments it reads `./source` and writes to the current directory. Other options:
+`--format` (repeatable: `ris`, `csv`, `medline`, `xml`; default `ris`), `--version`, `--help`.
+
+Files are auto-detected by format; processed in alphabetical order (sets tie-break priority).
+
+**Running from a clone instead of installing:** `python3 cli/literature_deduplication.py` still
+works from a checkout — it is a thin wrapper around the same code. It needs the `deduplicate_it/`
+package beside it, so download the repository (or a release archive), not that one file on its own.
+Before 1.2.0 the script was self-contained; see the note in [CHANGELOG.md](CHANGELOG.md).
+
+## Supported input formats
+
+| Format | Extensions |
+|--------|-----------|
+| MEDLINE / PubMed NBIB | `.txt`, `.nbib` |
+| RIS (Embase, Cochrane databases, CINAHL, Scopus, Web of Science) | `.ris`, `.enw` |
+| Web of Science ISI-tagged | `.ciw` |
+| BibTeX | `.bib` |
+| CSV / TSV | `.csv`, `.tsv` |
+
+Format is inferred from file content, not file extension. ERIC's MEDLINE-style `.nbib`
+export is recognised from its own tags, so it is handled correctly whatever the file is
+named.
+
+## Output files
+
+| File | Description |
+|------|-------------|
+| `deduplicated.ris` | Deduplicated references — import into Rayyan, Covidence, or EndNote |
+| `deduplicated.csv` | Deduplicated references — tabular |
+| `deduplicated.txt` | Deduplicated references — MEDLINE tagged text |
+| `deduplicated.xml` | Deduplicated references — XML |
+| `excluded_duplicates.csv` | Full audit trail: one row per excluded duplicate paired with the retained record |
+| `doi_collisions.csv` | Records with the same DOI but different titles — retained in output, flagged for manual review |
+| `deduplicated_prisma_flowchart.html` | Editable PRISMA-style flowchart, pre-filled with record counts |
+| `deduplicated_prisma_flowchart_extended.html` | As above, with the per-database breakdown |
+
+The web application offers all four deduplicated-reference formats for download. The
+command-line tool writes RIS only by default. `--format` **replaces** that default rather than
+adding to it, so name every format you want, RIS included: `--format ris --format csv` writes
+both, while `--format csv` on its own writes no RIS file.
+
+## Algorithm
+
+Deduplication uses a compound key of **normalised DOI** and **normalised title**. Records are excluded only when both fields agree after normalisation. Records without a valid DOI, or without a normalised title, pass through unchanged. DOI collisions (same DOI, different titles — common in conference supplement publications) are kept in the output and exported to a separate log for manual review.
+
+Where a database publishes no DOI field but stores the DOI inside a link field — ERIC is the
+common case — the DOI is recovered from that link. Candidates must still pass full
+structural validation, so record pages, PDFs and other non-DOI URLs are ignored.
+
+See the accompanying manuscript and Supplement A for a full step-by-step technical description.
+
+## Version history
+
+See [CHANGELOG.md](CHANGELOG.md). The **release version** (currently 1.2.0) and the
+**algorithm generation** (`v6`) are tracked separately: the algorithm generation changes
+only when deduplication decisions change. The web application and the Python package share
+the release version, so 1.2.0 means the same algorithm in both.
+
+## License
+
+Copyright 2025 David Purkarthofer and Sebastian Labenbacher.
+Apache License 2.0 — see [LICENSE](LICENSE).
+
+## Citation
+
+**Please cite the methods paper.** This is the citation to use when deduplicate.it
+contributed to a review:
+
+> Purkarthofer, D., Labenbacher, S., Bornemann-Cimenti, H., & Landoni, G. (2026).
+> *deduplicate.it: A simple open-source tool to remove duplicates from literature searches.*
+> Campbell Systematic Reviews, 22(3). https://doi.org/10.1177/18911803261484937
+
+A `CITATION.cff` file is included, so GitHub's "Cite this repository" button returns the
+reference above.
+
+**Citing the software itself** is only needed when you have to point at a specific version
+of the code — for a reproducibility statement, say:
+
+> Purkarthofer D, Labenbacher S. *deduplicate.it: Automated deduplication of literature searches* [Software]. Zenodo. https://doi.org/10.5281/zenodo.18835297
+
+That DOI covers all versions and always resolves to the most recent one. To pin the exact
+version you used, take its version-specific DOI from the Zenodo record.

deduplicate_it-1.2.0/README.md

@@ -0,0 +1,126 @@
+# deduplicate.it
+
+Free, open-source tool for automated deduplication of literature search exports for systematic reviews, scoping reviews, and meta-analyses.
+
+**Web interface:** [deduplicate.it](https://deduplicate.it)
+
+[![Paper](https://img.shields.io/badge/Campbell%20Syst%20Rev-10.1177%2F18911803261484937-1a7f37)](https://doi.org/10.1177/18911803261484937)
+[![Software archive](https://img.shields.io/badge/Zenodo-10.5281%2Fzenodo.18835297-1682d4)](https://doi.org/10.5281/zenodo.18835297)
+[![License](https://img.shields.io/badge/license-Apache--2.0-lightgrey)](LICENSE)
+
+**Peer-reviewed and published.** The matching rule, the evaluation and the limitations are described in full in the methods paper:
+
+> Purkarthofer D, Labenbacher S, Bornemann-Cimenti H, Landoni G. deduplicate.it: A simple open-source tool to remove duplicates from literature searches. *Campbell Systematic Reviews*. 2026;22(3). doi:[10.1177/18911803261484937](https://doi.org/10.1177/18911803261484937)
+
+Across five real-world searches (958–2,408 records) it removed **76.6–89.6%** of the duplicates found by Rayyan-assisted manual deduplication and cut deduplication time by **68.4–86.7%**. Records without a valid DOI, and records whose DOIs collide, are never silently merged: they are retained, exported separately and sorted to the top of the output for manual review.
+
+See [Citation](#citation) for when to cite the software archive instead.
+
+## Contents
+
+### `web/` — PHP web application
+Deploy to any PHP 8.x server. Place `index.php`, `legal.php`, and both `.drawio.html` flowchart templates in the same directory.
+
+`.user.ini` raises the upload and memory limits the tool needs — PHP's default
+`upload_max_filesize` of 2 MB is smaller than a typical database export. On shared hosting
+running PHP-FPM, keep it beside `index.php`; on other setups apply the equivalent
+directives in `php.ini`.
+
+### `deduplicate_it/` — Python command-line tool
+No external dependencies (Python standard library only). Requires Python 3.11 or newer.
+
+**Install:**
+
+```bash
+pip install deduplicate-it
+```
+
+**Usage:**
+
+```bash
+deduplicate-it --source path/to/exports --outdir path/to/results
+```
+
+With no arguments it reads `./source` and writes to the current directory. Other options:
+`--format` (repeatable: `ris`, `csv`, `medline`, `xml`; default `ris`), `--version`, `--help`.
+
+Files are auto-detected by format; processed in alphabetical order (sets tie-break priority).
+
+**Running from a clone instead of installing:** `python3 cli/literature_deduplication.py` still
+works from a checkout — it is a thin wrapper around the same code. It needs the `deduplicate_it/`
+package beside it, so download the repository (or a release archive), not that one file on its own.
+Before 1.2.0 the script was self-contained; see the note in [CHANGELOG.md](CHANGELOG.md).
+
+## Supported input formats
+
+| Format | Extensions |
+|--------|-----------|
+| MEDLINE / PubMed NBIB | `.txt`, `.nbib` |
+| RIS (Embase, Cochrane databases, CINAHL, Scopus, Web of Science) | `.ris`, `.enw` |
+| Web of Science ISI-tagged | `.ciw` |
+| BibTeX | `.bib` |
+| CSV / TSV | `.csv`, `.tsv` |
+
+Format is inferred from file content, not file extension. ERIC's MEDLINE-style `.nbib`
+export is recognised from its own tags, so it is handled correctly whatever the file is
+named.
+
+## Output files
+
+| File | Description |
+|------|-------------|
+| `deduplicated.ris` | Deduplicated references — import into Rayyan, Covidence, or EndNote |
+| `deduplicated.csv` | Deduplicated references — tabular |
+| `deduplicated.txt` | Deduplicated references — MEDLINE tagged text |
+| `deduplicated.xml` | Deduplicated references — XML |
+| `excluded_duplicates.csv` | Full audit trail: one row per excluded duplicate paired with the retained record |
+| `doi_collisions.csv` | Records with the same DOI but different titles — retained in output, flagged for manual review |
+| `deduplicated_prisma_flowchart.html` | Editable PRISMA-style flowchart, pre-filled with record counts |
+| `deduplicated_prisma_flowchart_extended.html` | As above, with the per-database breakdown |
+
+The web application offers all four deduplicated-reference formats for download. The
+command-line tool writes RIS only by default. `--format` **replaces** that default rather than
+adding to it, so name every format you want, RIS included: `--format ris --format csv` writes
+both, while `--format csv` on its own writes no RIS file.
+
+## Algorithm
+
+Deduplication uses a compound key of **normalised DOI** and **normalised title**. Records are excluded only when both fields agree after normalisation. Records without a valid DOI, or without a normalised title, pass through unchanged. DOI collisions (same DOI, different titles — common in conference supplement publications) are kept in the output and exported to a separate log for manual review.
+
+Where a database publishes no DOI field but stores the DOI inside a link field — ERIC is the
+common case — the DOI is recovered from that link. Candidates must still pass full
+structural validation, so record pages, PDFs and other non-DOI URLs are ignored.
+
+See the accompanying manuscript and Supplement A for a full step-by-step technical description.
+
+## Version history
+
+See [CHANGELOG.md](CHANGELOG.md). The **release version** (currently 1.2.0) and the
+**algorithm generation** (`v6`) are tracked separately: the algorithm generation changes
+only when deduplication decisions change. The web application and the Python package share
+the release version, so 1.2.0 means the same algorithm in both.
+
+## License
+
+Copyright 2025 David Purkarthofer and Sebastian Labenbacher.
+Apache License 2.0 — see [LICENSE](LICENSE).
+
+## Citation
+
+**Please cite the methods paper.** This is the citation to use when deduplicate.it
+contributed to a review:
+
+> Purkarthofer, D., Labenbacher, S., Bornemann-Cimenti, H., & Landoni, G. (2026).
+> *deduplicate.it: A simple open-source tool to remove duplicates from literature searches.*
+> Campbell Systematic Reviews, 22(3). https://doi.org/10.1177/18911803261484937
+
+A `CITATION.cff` file is included, so GitHub's "Cite this repository" button returns the
+reference above.
+
+**Citing the software itself** is only needed when you have to point at a specific version
+of the code — for a reproducibility statement, say:
+
+> Purkarthofer D, Labenbacher S. *deduplicate.it: Automated deduplication of literature searches* [Software]. Zenodo. https://doi.org/10.5281/zenodo.18835297
+
+That DOI covers all versions and always resolves to the most recent one. To pin the exact
+version you used, take its version-specific DOI from the Zenodo record.

deduplicate_it-1.2.0/deduplicate_it/__init__.py

@@ -0,0 +1,11 @@
+"""deduplicate.it — remove duplicate records from literature search exports.
+
+Release 1.2.0 · algorithm generation v6.
+See CHANGELOG.md for the release history.
+"""
+
+__version__ = "1.2.0"
+
+from .core import main  # noqa: E402,F401
+
+__all__ = ["main", "__version__"]