insta-collect-data 0.2.0__tar.gz

insta_collect_data-0.2.0/PKG-INFO

@@ -0,0 +1,313 @@
+Metadata-Version: 2.4
+Name: insta-collect-data
+Version: 0.2.0
+Summary:  Modular Instagram Data Collector
+Author-email: Alam Mahadika <alam.mahadika.psm@umy.ac.id>
+License: MIT
+Project-URL: Homepage, https://github.com/Alammahadika/insta-collect
+Keywords: instagram,scraper,hashtag,profile,engagement,analytics,comments
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: beautifulsoup4>=4.12.2
+Requires-Dist: lxml>=4.9.3
+Requires-Dist: pandas>=2.1.0
+Requires-Dist: openpyxl>=3.1.2
+Requires-Dist: requests>=2.31.0
+Requires-Dist: tqdm>=4.65.0
+Requires-Dist: playwright>=1.40.0
+
+# insta-collect-data: Modular Instagram Data Collector
+
+## Project Status: In Development
+
+insta-collect-data is a Playwright-based collector for Instagram **hashtags** and **profiles**. It supports photo and carousel posts as well as Reels, extracts post-level engagement metrics, and can compute per-profile engagement rates for influencer research.
+
+The live scraper is functional but depends on Instagram's public markup. Expect occasional breakage when Instagram changes its HTML or tightens its login wall. The HTML comment parser is still experimental.
+
+***
+
+## Project Overview
+
+insta-collect-data is a small, modular Python project that automates Instagram in a real browser via **Playwright**. It is intended for structured data collection for analysis and research: hashtag sampling, creator profiling, engagement comparisons, and small-batch archival work.
+
+***
+
+## Current Key Features
+
+* **Hashtag Scraping:** Collect posts from an explore/tag page.
+* **Profile Scraping:** Collect recent posts for one handle or a list of handles in a single browser session.
+* **Engagement Metrics:** Extract `likes`, `comments`, and `views` (when Instagram exposes them), plus profile `followers`, `following`, and total `posts`.
+* **Engagement Rate:** Compute average engagement per post and ER% for each scraped profile.
+* **Reels Included:** Both `/p/` posts and `/reel/` links are collected. Older versions filtered videos out; that is no longer the case.
+* **Rich Post Data:** `url`, `username`, `caption`, `caption_status`, `timestamp`, `is_video`, `hashtags`, `mentions`, `likes`, `comments`, `views`.
+* **Session Reuse:** Log in once with a visible browser and reuse the saved Playwright profile on later headless runs.
+* **Login-Wall Handling:** Detects Instagram login walls, dismisses cookie/sign-up modals when possible, and gives an actionable error in headless mode.
+* **Dual Output:** Every run writes both **JSON** and **CSV**.
+* **HTML Comment Extraction:** Parse comments from a previously saved Instagram HTML page into JSON and XLSX.
+
+***
+
+## Installation
+
+```bash
+pip install insta-collect-data
+```
+
+### or install directly from GitHub
+
+```bash
+pip install git+https://github.com/Alammahadika/insta-collect.git
+```
+
+This distribution installs the `insta_collect` Python package and the `insta-collect` CLI command.
+
+> `insta-collect-data` is the continuation of the earlier `insta-collect` package (last release 0.1.2) under a new PyPI name.
+
+Requirements: **Python 3.10+** and a Chromium browser for Playwright.
+
+### Playwright Setup (Required)
+
+Install the browser binaries once after installing the package:
+
+```bash
+playwright install chromium
+```
+
+### First-Time Login (Recommended)
+
+Instagram often blocks logged-out scraping with a login wall. Log in once using a visible browser; the session is stored in the Playwright profile directory (`ig_profile` by default) and reused afterwards.
+
+```bash
+insta-collect --user natgeo --limit 6 --headed
+```
+
+Log in manually in the window that opens, then let the run finish. Later runs work headless by default.
+
+> Do not commit the `ig_profile/` folder — it contains your authenticated session.
+
+***
+
+## Usage
+
+The installed CLI is `insta-collect`. It has two modes, and they are mutually exclusive.
+
+### Arguments
+
+| Argument | Description | Default |
+|---|---|---|
+| `--tag` | Hashtag to scrape (without `#`) | — |
+| `--user` | One handle, or a comma-separated list (`natgeo,nasa`) | — |
+| `--users-file` | Text file with one handle per line; `#` starts a comment | — |
+| `--limit` | Max posts per target (6–12 is a good range for engagement analysis) | `10` |
+| `--profile` | Playwright persistent profile directory (session storage) | `ig_profile` |
+| `--headed` | Show the browser window (first-time login / debugging) | headless |
+
+Provide either `--tag` **or** `--user` / `--users-file`.
+
+### Hashtag mode
+
+```bash
+insta-collect --tag donaldtrump --limit 10
+```
+
+### Single profile
+
+```bash
+insta-collect --user natgeo --limit 10
+```
+
+### Multiple profiles
+
+```bash
+insta-collect --user natgeo,nasa --limit 8
+```
+
+Or from a file (`handles.txt`):
+
+```text
+# one handle per line
+natgeo
+nasa
+@somecreator
+```
+
+```bash
+insta-collect --users-file handles.txt --limit 8
+```
+
+### Visible browser
+
+```bash
+insta-collect --tag donaldtrump --limit 10 --headed
+```
+
+***
+
+## Output
+
+Each run writes two files in the current directory.
+
+| Mode | Files |
+|---|---|
+| Hashtag | `result_<tag>.json`, `result_<tag>.csv` |
+| Single profile | `result_<handle>.json`, `result_<handle>.csv` |
+| Multiple profiles | `result_profiles.json`, `result_profiles.csv` |
+
+### Post fields (hashtag mode)
+
+```json
+[
+  {
+    "url": "https://www.instagram.com/p/DS3UWMfCRA5/",
+    "username": "cdn37",
+    "caption": "El presidente de Estados Unidos...",
+    "caption_status": "long_text",
+    "timestamp": "2025-12-29T22:35:50.000Z",
+    "is_video": false,
+    "hashtags": ["CDN", "Noticias", "DonaldTrump"],
+    "mentions": [],
+    "likes": 1200,
+    "comments": 34,
+    "views": null,
+    "source_tag": "donaldtrump"
+  }
+]
+```
+
+`caption_status` is one of `long_text`, `short_text`, `no_text`, `login_required`, or `error`.
+
+### Profile fields (profile mode)
+
+```json
+[
+  {
+    "handle": "natgeo",
+    "url": "https://www.instagram.com/natgeo/",
+    "followers": 283000000,
+    "following": 120,
+    "posts_total": 31000,
+    "posts_scraped": 8,
+    "avg_engagement": 45210.5,
+    "engagement_rate_pct": 0.016,
+    "posts": [
+      {
+        "url": "https://www.instagram.com/p/...",
+        "username": "natgeo",
+        "caption": "...",
+        "likes": 40000,
+        "comments": 5210,
+        "views": null,
+        "source_handle": "natgeo"
+      }
+    ]
+  }
+]
+```
+
+### Engagement rate definition
+
+For each scraped post, engagement is `likes + comments`, plus `views` when that post reported a view count. Then:
+
+```text
+avg_engagement     = mean(per-post engagement)
+engagement_rate_%  = avg_engagement / followers * 100
+```
+
+Instagram displays rounded counts (`12K`, `1.5M`), so parsed metrics are approximations, not exact values.
+
+### CSV note
+
+CSV output is flat. In profile mode the nested `posts` list is written as a single serialized cell per profile. Use the JSON file when you need post-level analysis.
+
+***
+
+## How It Works
+
+1. **Session resumption:** Opens Chromium with the persistent `--profile` directory, reusing any saved login.
+2. **Link collection:** Navigates to the hashtag or profile page, dismisses the cookie banner, and scrolls until enough `/p/` and `/reel/` links are found (query strings stripped, duplicates removed).
+3. **Post extraction:** Visits each post, closes the sign-up modal when present, and reads `og:description` plus page selectors for caption, username, timestamp, and engagement metrics.
+4. **Filtering:** Skips posts that hit a login wall or throw an error. Reels and videos are kept.
+5. **Aggregation:** In profile mode, computes average engagement and ER% from the scraped posts.
+6. **Export:** Writes JSON and CSV.
+
+***
+
+## Limitations
+
+* Instagram's login wall is the most common failure. If a headless run reports it, rerun once with `--headed` and log in.
+* Logged-out hashtag grids often expose only Reels; results may be Reel-heavy.
+* Metrics are rounded by Instagram (`K`/`M`/`B`) and may be missing entirely.
+* Keep `--limit` modest and expect random delays between requests. This is not a high-volume crawler.
+* The persistent profile directory contains credentials; exclude it from version control.
+
+***
+
+## Naming Conventions: `_` vs `-` in Python Projects
+
+| Context | Use | Example |
+|---|---|---|
+| Package name (PyPI / GitHub) | `-` (hyphen) | insta-collect-data |
+| Python package directory | `_` (underscore) | insta_collect |
+| Python import statement | `_` (underscore) | import insta_collect |
+| CLI execution via module | `_` (underscore) | python -m insta_collect.cli |
+| Script / file name | flexible | insta-collect.py |
+
+***
+
+## Collecting Comments from an HTML File
+
+In addition to live scraping, insta-collect-data can parse comments from a saved Instagram HTML page made in a logged-in browser.
+
+This is useful when:
+
+* You already saved the page and want structured comments.
+* You want to avoid live requests.
+* You need an XLSX deliverable.
+
+### Example
+
+```bash
+cd path/to/insta_collect   # the package directory in a repo checkout
+python3 insta-collect.py ~/Downloads/obama.html --preview 20
+```
+
+The script writes `instagram_comments.json` and `instagram_comments.xlsx` to the current directory.
+
+### Output
+
+```text
+[+] Total entries saved: 239
+[+] JSON output: instagram_comments.json
+[+] XLSX output: instagram_comments.xlsx
+```
+
+### Comment fields
+
+* `username`
+* `text` (`comment_text` in the README of older releases)
+* `timestamp` when available
+* `source_file`
+
+### Notes
+
+* Output filenames are fixed; existing files are overwritten.
+* `--preview` only affects terminal output, not the saved data.
+* This feature is experimental and may change.
+
+***
+
+## Roadmap
+
+* Live comment scraping.
+* XLSX export for live post/profile runs.
+* Retry/backoff and proxy support for larger batches.
+* Optional pagination beyond the recent grid.
+
+Completed in this release: profile scraping, engagement metrics/ER%, CSV output, Reels support, headless-by-default with persistent login.
+
+***
+
+## Ethical Use Notice
+
+This tool is intended strictly for academic research, journalism, and public-interest analysis.
+Users are responsible for complying with Instagram's Terms of Service and applicable laws.

insta_collect_data-0.2.0/README.md

@@ -0,0 +1,295 @@
+# insta-collect-data: Modular Instagram Data Collector
+
+## Project Status: In Development
+
+insta-collect-data is a Playwright-based collector for Instagram **hashtags** and **profiles**. It supports photo and carousel posts as well as Reels, extracts post-level engagement metrics, and can compute per-profile engagement rates for influencer research.
+
+The live scraper is functional but depends on Instagram's public markup. Expect occasional breakage when Instagram changes its HTML or tightens its login wall. The HTML comment parser is still experimental.
+
+***
+
+## Project Overview
+
+insta-collect-data is a small, modular Python project that automates Instagram in a real browser via **Playwright**. It is intended for structured data collection for analysis and research: hashtag sampling, creator profiling, engagement comparisons, and small-batch archival work.
+
+***
+
+## Current Key Features
+
+* **Hashtag Scraping:** Collect posts from an explore/tag page.
+* **Profile Scraping:** Collect recent posts for one handle or a list of handles in a single browser session.
+* **Engagement Metrics:** Extract `likes`, `comments`, and `views` (when Instagram exposes them), plus profile `followers`, `following`, and total `posts`.
+* **Engagement Rate:** Compute average engagement per post and ER% for each scraped profile.
+* **Reels Included:** Both `/p/` posts and `/reel/` links are collected. Older versions filtered videos out; that is no longer the case.
+* **Rich Post Data:** `url`, `username`, `caption`, `caption_status`, `timestamp`, `is_video`, `hashtags`, `mentions`, `likes`, `comments`, `views`.
+* **Session Reuse:** Log in once with a visible browser and reuse the saved Playwright profile on later headless runs.
+* **Login-Wall Handling:** Detects Instagram login walls, dismisses cookie/sign-up modals when possible, and gives an actionable error in headless mode.
+* **Dual Output:** Every run writes both **JSON** and **CSV**.
+* **HTML Comment Extraction:** Parse comments from a previously saved Instagram HTML page into JSON and XLSX.
+
+***
+
+## Installation
+
+```bash
+pip install insta-collect-data
+```
+
+### or install directly from GitHub
+
+```bash
+pip install git+https://github.com/Alammahadika/insta-collect.git
+```
+
+This distribution installs the `insta_collect` Python package and the `insta-collect` CLI command.
+
+> `insta-collect-data` is the continuation of the earlier `insta-collect` package (last release 0.1.2) under a new PyPI name.
+
+Requirements: **Python 3.10+** and a Chromium browser for Playwright.
+
+### Playwright Setup (Required)
+
+Install the browser binaries once after installing the package:
+
+```bash
+playwright install chromium
+```
+
+### First-Time Login (Recommended)
+
+Instagram often blocks logged-out scraping with a login wall. Log in once using a visible browser; the session is stored in the Playwright profile directory (`ig_profile` by default) and reused afterwards.
+
+```bash
+insta-collect --user natgeo --limit 6 --headed
+```
+
+Log in manually in the window that opens, then let the run finish. Later runs work headless by default.
+
+> Do not commit the `ig_profile/` folder — it contains your authenticated session.
+
+***
+
+## Usage
+
+The installed CLI is `insta-collect`. It has two modes, and they are mutually exclusive.
+
+### Arguments
+
+| Argument | Description | Default |
+|---|---|---|
+| `--tag` | Hashtag to scrape (without `#`) | — |
+| `--user` | One handle, or a comma-separated list (`natgeo,nasa`) | — |
+| `--users-file` | Text file with one handle per line; `#` starts a comment | — |
+| `--limit` | Max posts per target (6–12 is a good range for engagement analysis) | `10` |
+| `--profile` | Playwright persistent profile directory (session storage) | `ig_profile` |
+| `--headed` | Show the browser window (first-time login / debugging) | headless |
+
+Provide either `--tag` **or** `--user` / `--users-file`.
+
+### Hashtag mode
+
+```bash
+insta-collect --tag donaldtrump --limit 10
+```
+
+### Single profile
+
+```bash
+insta-collect --user natgeo --limit 10
+```
+
+### Multiple profiles
+
+```bash
+insta-collect --user natgeo,nasa --limit 8
+```
+
+Or from a file (`handles.txt`):
+
+```text
+# one handle per line
+natgeo
+nasa
+@somecreator
+```
+
+```bash
+insta-collect --users-file handles.txt --limit 8
+```
+
+### Visible browser
+
+```bash
+insta-collect --tag donaldtrump --limit 10 --headed
+```
+
+***
+
+## Output
+
+Each run writes two files in the current directory.
+
+| Mode | Files |
+|---|---|
+| Hashtag | `result_<tag>.json`, `result_<tag>.csv` |
+| Single profile | `result_<handle>.json`, `result_<handle>.csv` |
+| Multiple profiles | `result_profiles.json`, `result_profiles.csv` |
+
+### Post fields (hashtag mode)
+
+```json
+[
+  {
+    "url": "https://www.instagram.com/p/DS3UWMfCRA5/",
+    "username": "cdn37",
+    "caption": "El presidente de Estados Unidos...",
+    "caption_status": "long_text",
+    "timestamp": "2025-12-29T22:35:50.000Z",
+    "is_video": false,
+    "hashtags": ["CDN", "Noticias", "DonaldTrump"],
+    "mentions": [],
+    "likes": 1200,
+    "comments": 34,
+    "views": null,
+    "source_tag": "donaldtrump"
+  }
+]
+```
+
+`caption_status` is one of `long_text`, `short_text`, `no_text`, `login_required`, or `error`.
+
+### Profile fields (profile mode)
+
+```json
+[
+  {
+    "handle": "natgeo",
+    "url": "https://www.instagram.com/natgeo/",
+    "followers": 283000000,
+    "following": 120,
+    "posts_total": 31000,
+    "posts_scraped": 8,
+    "avg_engagement": 45210.5,
+    "engagement_rate_pct": 0.016,
+    "posts": [
+      {
+        "url": "https://www.instagram.com/p/...",
+        "username": "natgeo",
+        "caption": "...",
+        "likes": 40000,
+        "comments": 5210,
+        "views": null,
+        "source_handle": "natgeo"
+      }
+    ]
+  }
+]
+```
+
+### Engagement rate definition
+
+For each scraped post, engagement is `likes + comments`, plus `views` when that post reported a view count. Then:
+
+```text
+avg_engagement     = mean(per-post engagement)
+engagement_rate_%  = avg_engagement / followers * 100
+```
+
+Instagram displays rounded counts (`12K`, `1.5M`), so parsed metrics are approximations, not exact values.
+
+### CSV note
+
+CSV output is flat. In profile mode the nested `posts` list is written as a single serialized cell per profile. Use the JSON file when you need post-level analysis.
+
+***
+
+## How It Works
+
+1. **Session resumption:** Opens Chromium with the persistent `--profile` directory, reusing any saved login.
+2. **Link collection:** Navigates to the hashtag or profile page, dismisses the cookie banner, and scrolls until enough `/p/` and `/reel/` links are found (query strings stripped, duplicates removed).
+3. **Post extraction:** Visits each post, closes the sign-up modal when present, and reads `og:description` plus page selectors for caption, username, timestamp, and engagement metrics.
+4. **Filtering:** Skips posts that hit a login wall or throw an error. Reels and videos are kept.
+5. **Aggregation:** In profile mode, computes average engagement and ER% from the scraped posts.
+6. **Export:** Writes JSON and CSV.
+
+***
+
+## Limitations
+
+* Instagram's login wall is the most common failure. If a headless run reports it, rerun once with `--headed` and log in.
+* Logged-out hashtag grids often expose only Reels; results may be Reel-heavy.
+* Metrics are rounded by Instagram (`K`/`M`/`B`) and may be missing entirely.
+* Keep `--limit` modest and expect random delays between requests. This is not a high-volume crawler.
+* The persistent profile directory contains credentials; exclude it from version control.
+
+***
+
+## Naming Conventions: `_` vs `-` in Python Projects
+
+| Context | Use | Example |
+|---|---|---|
+| Package name (PyPI / GitHub) | `-` (hyphen) | insta-collect-data |
+| Python package directory | `_` (underscore) | insta_collect |
+| Python import statement | `_` (underscore) | import insta_collect |
+| CLI execution via module | `_` (underscore) | python -m insta_collect.cli |
+| Script / file name | flexible | insta-collect.py |
+
+***
+
+## Collecting Comments from an HTML File
+
+In addition to live scraping, insta-collect-data can parse comments from a saved Instagram HTML page made in a logged-in browser.
+
+This is useful when:
+
+* You already saved the page and want structured comments.
+* You want to avoid live requests.
+* You need an XLSX deliverable.
+
+### Example
+
+```bash
+cd path/to/insta_collect   # the package directory in a repo checkout
+python3 insta-collect.py ~/Downloads/obama.html --preview 20
+```
+
+The script writes `instagram_comments.json` and `instagram_comments.xlsx` to the current directory.
+
+### Output
+
+```text
+[+] Total entries saved: 239
+[+] JSON output: instagram_comments.json
+[+] XLSX output: instagram_comments.xlsx
+```
+
+### Comment fields
+
+* `username`
+* `text` (`comment_text` in the README of older releases)
+* `timestamp` when available
+* `source_file`
+
+### Notes
+
+* Output filenames are fixed; existing files are overwritten.
+* `--preview` only affects terminal output, not the saved data.
+* This feature is experimental and may change.
+
+***
+
+## Roadmap
+
+* Live comment scraping.
+* XLSX export for live post/profile runs.
+* Retry/backoff and proxy support for larger batches.
+* Optional pagination beyond the recent grid.
+
+Completed in this release: profile scraping, engagement metrics/ER%, CSV output, Reels support, headless-by-default with persistent login.
+
+***
+
+## Ethical Use Notice
+
+This tool is intended strictly for academic research, journalism, and public-interest analysis.
+Users are responsible for complying with Instagram's Terms of Service and applicable laws.