geosint 1.0.0__tar.gz

geosint-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jenil Mistry
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

geosint-1.0.0/PKG-INFO

@@ -0,0 +1,436 @@
+Metadata-Version: 2.4
+Name: geosint
+Version: 1.0.0
+Summary: Geospatial OSINT collection and monitoring for points of interest and organisations - OpenStreetMap by default, optionally Google Places.
+Author: Jenil Mistry
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/jenilmistryhq/geosint
+Project-URL: Repository, https://github.com/jenilmistryhq/geosint
+Project-URL: Issues, https://github.com/jenilmistryhq/geosint/issues
+Keywords: osint,geospatial,openstreetmap,overpass,geojson,reconnaissance,monitoring,investigation
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Information Technology
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: GIS
+Classifier: Topic :: Security
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Environment :: Console
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.31
+Provides-Extra: xlsx
+Requires-Dist: openpyxl>=3.1; extra == "xlsx"
+Provides-Extra: dotenv
+Requires-Dist: python-dotenv>=1.0; extra == "dotenv"
+Provides-Extra: bulk
+Requires-Dist: osmium>=3.7; extra == "bulk"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: osmium>=3.7; extra == "dev"
+Dynamic: license-file
+
+# geosint
+
+Geospatial OSINT collection and monitoring for points of interest and
+organisations. Collects from public map data, records **where every fact came
+from**, reports **what changed between collections**, and correlates entities
+by shared infrastructure.
+
+[![Python](https://img.shields.io/pypi/pyversions/geosint.svg)](https://pypi.org/project/geosint/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+```
+$ geosint collect --what "bureau de change" --area "Austin, TX" --case meridian
+
+3 entities collected (0 new to this case)
+
+ENTITY             CATEGORY          ADDRESS    PHONE            DOMAIN             CONF  TRIAGE
+-----------------  ----------------  ---------  ---------------  -----------------  ----  ----------
+Alpha Exchange     bureau_de_change  1 Main St  +1 555 077 2200  newholdco.example  83    Unreviewed
+Beta Money Ser...  bureau_de_change  1 Main St  +1 555 010 0100  holdco.example     83    Unreviewed
+Cedar Currency     bureau_de_change  Cedar Ave  +1 555 044 7788                     67    Unreviewed
+
+  3/3 have usable coordinates.
+
+3 field change(s) since the last collection:
+
+WHEN              ENTITY          FIELD    WAS                     NOW                    SOURCE
+----------------  --------------  -------  ----------------------  ---------------------  ------
+2026-09-09 19:31  Alpha Exchange  phone    +1 555 010 0100         +1 555 077 2200        osm
+2026-09-09 19:31  Alpha Exchange  website  https://holdco.example  https://newholdco.e... osm
+2026-09-09 19:31  Alpha Exchange  domain   holdco.example          newholdco.example      osm
+
+1 previously-seen entity did not reappear:
+    - Northside Exchange
+```
+
+## Scope
+
+This collects **organisations, businesses, and infrastructure** from public map
+data — entities that publish an address and a phone number precisely so they
+can be found. It queries each source's own official public API, at documented
+rate limits, identifying itself with a contact address.
+
+**It is not a people-search tool.** There is no person lookup, no
+cross-referencing of individuals, and none should be added. Everything it
+returns describes a *place* or an *organisation*.
+
+## Start here
+
+```bash
+pip install geosint
+export GEOSINT_CONTACT="you@example.com"   # required, see below
+geosint start
+```
+
+`start` asks five plain questions, remembers the answers, and runs your first
+collection. Everything below is the same thing with flags.
+
+---
+
+## Why this rather than a scraper
+
+Three things, and they are the whole design.
+
+**1. Provenance on every field.** Each record carries its source, source
+reference, collection time, case, and the query that found it. `geosint show`
+prints all of it. An answer you cannot attribute is not intelligence, it is
+trivia.
+
+**2. Change is the product.** Re-run a collection and the tool diffs it against
+what is stored, writing an append-only observation for every field that moved.
+A phone number changing, a website going dark, an entity vanishing from the
+map — these are findings, and a tool that just overwrites its database destroys
+them.
+
+```bash
+geosint collect --what pharmacy --area "Austin, TX" --case alpha   # week 1
+geosint collect --what pharmacy --area "Austin, TX" --case alpha   # week 2
+geosint changes --since 7d --case alpha
+```
+
+**3. Correlation between entities.** Everything else describes entities one at
+a time. `geosint links` looks for what connects them:
+
+| Link | Meaning |
+|---|---|
+| shared phone | Two "different" organisations answering one number |
+| shared domain | Two entities publishing the same web host |
+| operator / brand | Affiliation the source states outright — the strongest signal available |
+| co-location | Entities within metres of each other |
+
+```
+$ geosint links --case meridian
+
+Stated operator / brand affiliations
+
+  Affiliation: Meridian Holdings   (2 entities)
+      - Alpha Exchange
+      - Beta Money Services
+
+Co-located entities (within 25m)
+
+  Cluster: 2 entities within 25m   (2 entities)
+      - Alpha Exchange
+      - Beta Money Services
+```
+
+A shared value is **a lead, not a conclusion**. Shared numbers and domains have
+mundane explanations — a shared reception, a franchise call centre, a serviced
+office — at least as often as interesting ones. The tool reports the evidence
+and leaves the judgement to you.
+
+---
+
+## Setup
+
+```bash
+pip install geosint                     # core
+pip install "geosint[xlsx,dotenv]"      # + Excel export, .env support
+pip install "geosint[bulk]"             # + offline .osm.pbf extraction
+```
+
+### The contact address is required
+
+OpenStreetMap's Nominatim and Overpass are run by volunteers, and their usage
+policy requires requests to carry a real contact so an operator can email you
+rather than block you.
+
+```bash
+export GEOSINT_CONTACT="you@example.com"          # macOS / Linux
+$env:GEOSINT_CONTACT="you@example.com"            # Windows PowerShell
+```
+
+Or pass `--contact` per run. It is sent to OSM and to any site you explicitly
+choose to check with `--check-websites`, and nowhere else.
+
+### Optional: the Google collector
+
+`--provider google` has better contact coverage but no `operator`, `brand`,
+`ref`, or `wikidata` fields — the metadata that makes OSM more useful for
+investigation. It also costs money past a free monthly allowance, and requires
+billing enabled with a card on file even to stay inside it. Google prices a
+whole response at the tier of its most expensive requested field, so
+`--include-ratings` drops the free allowance from 5,000 to 1,000 calls/month.
+
+1. https://console.cloud.google.com → new project
+2. Enable **Places API (New)**, enable billing
+3. Create an API key, restrict it to "Places API (New)"
+4. `export GOOGLE_PLACES_API_KEY="your-key"`
+
+`--near` / `--radius` become a Places `locationRestriction` rectangle around the
+circle (the only shape Text Search accepts there), and results are then narrowed
+to the true radius. The radius is capped at 50 km, since Text Search returns at
+most 60 results - a wider radius is refused rather than quietly clamped. Use
+`--provider osm` for a bigger sweep. `--tile` is osm-only.
+
+---
+
+## Commands
+
+### Collect
+
+```bash
+# A named area (geocoded to its bounding box)
+geosint collect --what pharmacy --area "Austin, TX" --case alpha
+
+# A point and radius - more precise, and usually what you want.
+# A box around a city centre is mostly not the city centre.
+geosint collect --what pharmacy --near 30.2672,-97.7431 --radius 2
+
+# Any OSM tag, for anything not in the built-in category list
+geosint collect --osm-tag "office=diplomatic" --area "Vienna, Austria"
+geosint collect --osm-tag "man_made=mast" --near 51.5,-0.12 --radius 10
+
+# Dense area under-returning? Split it into a 3x3 grid (9 queries, deduped)
+geosint collect --what pharmacy --area "Chicago, IL" --tile 3
+
+# Many collections from a CSV (header: what,area[,osm_tag]).
+# Each row is its own tracked collection, so re-running the batch diffs
+# every row and reports what stopped appearing, per row.
+geosint batch --input targets.csv --case alpha
+
+# A whole country or region, offline, no rate limit
+geosint pbf --pbf texas-latest.osm.pbf --what pharmacy --case alpha
+```
+
+For `pbf`, download the smallest extract covering your region from
+<https://download.geofabrik.de/>.
+
+### Analyse
+
+```bash
+geosint entities --case alpha                    # browse
+geosint entities --indicator shared_phone        # filter by indicator
+geosint entities --missing                       # what stopped appearing
+geosint show "Alpha Exchange"                    # one entity + full provenance
+geosint links --case alpha --tag                 # correlate, and tag the results
+geosint near 30.2672,-97.7431 2                  # what's within 2km
+```
+
+### Record judgement
+
+```bash
+geosint triage "Alpha Exchange" Flagged --note "same address as Beta"
+geosint note "Alpha Exchange" "registration MSB-11021 predates the website"
+geosint recheck "Alpha Exchange" +7d             # also +2w, +3m, +1y, 2026-10-01
+geosint recheck                                  # what's due
+```
+
+Triage states: `Unreviewed`, `Reviewing`, `Corroborated`, `Dismissed`,
+`Flagged`. Anything you write — triage, notes, recheck dates, analyst — is
+**never overwritten by a re-collection**.
+
+### Monitor and report
+
+```bash
+geosint changes --since 7d --case alpha          # what moved
+geosint runs                                     # collection history
+geosint report --case alpha                      # indicator + link summary
+geosint cases                                    # all cases
+geosint export --case alpha                      # GeoJSON by default
+```
+
+`export` writes **GeoJSON** by default because the data is inherently
+geospatial — drop the file into [geojson.io](https://geojson.io), QGIS,
+kepler.gl, or Google Earth with no conversion. `--format csv|xlsx|json` also
+available.
+
+---
+
+## Indicators
+
+Findings are named indicators, not a single score. A score collapses independent
+facts into one number and hides which fact drove it.
+
+| Indicator | Meaning |
+|---|---|
+| `no_web_presence` | No website published |
+| `website_dead` | Website published but unreachable (needs `--check-websites`) |
+| `no_contact_details` | No phone, website, or email at all |
+| `no_address` / `no_coordinates` | Incomplete location data |
+| `operator_named` | The source names who runs it |
+| `chain_affiliated` | Carries a brand tag |
+| `registration_ref` | Carries a licence/registration reference |
+| `wikidata_linked` | Hard link to a structured record |
+| `stale_survey` | Map data not verified recently |
+| `not_operational` | Source reports it closed |
+| `shared_phone` / `shared_domain` | Written by `geosint links --tag` |
+
+`confidence` is a separate number: the percentage of core fields actually
+populated. It measures **record completeness**, not how interesting the entity
+is — those are different questions and should not share a scale.
+
+---
+
+## What the sources can and cannot tell you
+
+- **OSM has no rating, review, or operational-status data.** Those come back as
+  unknown, and every check skips rather than inventing a value. A blank never
+  reads as a zero.
+- **Ways carry tags but not coordinates** in offline `pbf` mode (full geometry
+  assembly needs a location index and far more memory). They are flagged
+  `no_coordinates` so the gap is visible in the data rather than silently absent.
+  Nodes — the common case for POIs — carry their coordinates.
+- **`--check-websites` is not an audit.** One GET to each entity's own published
+  homepage, once, recording only whether it answered. Nothing is read out of the
+  page, and it is never a crawl. Sites whose host resolves into private or
+  loopback address space are declined rather than fetched — the URL comes from a
+  world-writable OSM tag.
+- **Coverage varies enormously by region.** OSM is crowdsourced; absence of an
+  entity is weak evidence of absence in reality.
+- **This tool does not scrape.** `pbf` reaches country scale through OSM's own
+  bulk distribution channel — a bigger free, ToS-clean source, not evasion of
+  anyone's rate limits. It does not harvest contact details off web pages
+  either: `email` comes only from the source's own `email`/`contact:email` tag,
+  published as a contact detail by a mapper.
+- **Spreadsheet exports are formula-escaped.** An OSM `name` tag is
+  world-writable, and Excel evaluates a cell beginning `=`, `+`, `-` or `@`.
+  Values that would execute are prefixed so they stay text. GeoJSON and JSON are
+  untouched — nothing that reads them executes them.
+
+---
+
+## Options (`collect` / `batch` / `pbf`)
+
+| Flag | Default | What it does |
+|---|---|---|
+| `--what` | — | Category, e.g. `"pharmacy"` |
+| `--osm-tag` | — | Any OSM tag, e.g. `"office=ngo"` (overrides `--what`) |
+| `--area` | — | Place name, geocoded to a bounding box |
+| `--near` / `--radius` | — | Centre point (`lat,lon`) and radius in km. google caps the radius at 50km |
+| `--case` | `default` | Scopes entities, changes, and correlation |
+| `--contact` | `$GEOSINT_CONTACT` | Your email/URL, sent as User-Agent |
+| `--provider` | `osm` | `osm` (free, richer metadata) or `google` |
+| `--tile` | 1 | osm only — N x N grid split for fuller coverage. Raise `--max-results` with it |
+| `--check-websites` | off | Record whether each published site answers. 8 at a time |
+| `--no-store` | off | One-off collection; disables change detection |
+| `--db-path` | `.geosint.db` | Database location |
+| `--max-results` | 60 | osm ceiling 3000; google hard cap 60 |
+| `--min-confidence` | 0 | Drop records less than N% complete |
+| `--format` | geojson | `geojson`, `csv`, `xlsx`, `json` |
+| `--output` | — | Write results to this file |
+| `--cache-ttl` | 24 | Hours before a cached query refetches (0 = never reuse) |
+| `--no-cache` | off | Skip the cache entirely |
+| `--dry-run` | off | Collect and analyse, write nothing |
+| `--verbose` / `--quiet` | normal | Debug logging / warnings only |
+
+A stored collection always bypasses the cache on read: it exists to answer
+"what changed?", and only the network can answer that. The cache key covers
+every flag that changes the result set, `--max-results` included, so raising a
+cap refetches rather than replaying the smaller answer.
+
+`--max-results` is applied **after** tiling merges and dedupes, so `--tile 3`
+with the default 60 still keeps 60 — of the whole area, ranked by record
+completeness, not the first tiles queried. When results are dropped to fit the
+cap the run says so and names the flag to raise. A radius collection keeps the
+nearest N instead, because there distance is a meaningful order.
+
+---
+
+## Development
+
+```bash
+git clone https://github.com/jenilmistryhq/geosint.git
+cd geosint
+pip install -e ".[dev,xlsx,dotenv]"
+pytest -q
+```
+
+388 tests, all against mocked HTTP and temporary SQLite, or a real synthetic
+`.osm.pbf` fixture — no API key, no network, no cost. (One skips unless the
+`bulk` extra is installed.) Each test runs in its own working directory with a
+neutralised `$HOME`, so nothing leaks between tests or into the repo.
+
+Two rules this suite learned the hard way:
+
+- **Build fixtures from a real normalizer, not by hand.** `dedupe` once keyed on
+  field names no normalizer produced. Every test passed; every branch of every
+  chain silently merged into one record. `tests/test_dedupe.py` now builds its
+  input by calling the collectors' own `_normalize`.
+- **Mock the transport, assert on the contract.** Tests that mock a client
+  wholesale prove only that the mock works — `--provider google` had no
+  end-to-end test and silently discarded `--near` for as long as the flag
+  existed.
+
+`tests/test_regressions.py` holds one named test per bug that produced
+confident, wrong output.
+
+### Layout
+
+```
+geosint/
+├── cli.py             argument parsing + command orchestration
+├── wizard.py          `start` - guided setup
+├── display.py         tables, bar charts, entity cards
+├── config.py          env/.env/CLI resolution + validation (one path, all commands)
+├── useragent.py       required contact -> User-Agent
+├── geo.py             coordinates, radius maths, GeoJSON
+├── entities.py        Entity record, indicators, confidence
+├── analysis.py        correlation: shared phone/domain/operator/location
+├── db.py              SQLite: entities, observations, collections, aliases, actions
+├── osm_client.py      Overpass + Nominatim collector
+├── api_client.py      Google Places API (New) collector
+├── bulk_extract.py    offline .osm.pbf extraction
+├── dedupe.py          identity keys, in-memory and at the DB boundary
+├── website_health.py  does the published site resolve
+├── batch.py           CSV multi-collection runner
+├── cache.py           per-query cache with a TTL
+└── exporters.py       GeoJSON / CSV / XLSX / JSON
+```
+
+### Data model
+
+```
+collections   one row per run: query, source, area, counts, timestamps
+     |
+entities      current best-known state, one row per entity
+     |
+observations  APPEND-ONLY. Every field change, with source and timestamp.
+aliases       maps a second source's id onto the canonical entity
+actions       analyst triage changes and notes
+```
+
+Adding a category? Add it to `CATEGORY_TAG_MAP` in `osm_client.py` plus a line
+in `test_osm_client.py`.
+
+## Legal and ethical note
+
+This queries each data source's own official public API (OpenStreetMap's
+Nominatim and Overpass, Google's Places API) and, with `--check-websites`, makes
+a plain GET to an entity's own published homepage. It does not scrape search
+results, bypass authentication or CAPTCHAs, evade rate limits, or attempt to
+disguise itself — the contact address is mandatory precisely so it does not.
+
+You are responsible for the lawfulness and proportionality of your collection.
+Public availability is not the same as a right to compile, and the answer
+differs by jurisdiction and by purpose. Where collected records touch personal
+data, data-protection law generally applies to you as the one compiling them.
+
+## License
+
+[MIT](LICENSE)