catui-agent 1.0.0-beta.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/quora/scraping.md

@@ -0,0 +1,364 @@
+# Quora — Data Extraction
+
+`https://www.quora.com` — Q&A platform. One reliable access path: `http_get` with a Chrome UA against question, answer, topic, and profile pages. Quora SSR-renders all public data into `window.ansFrontendGlobals.data.inlineQueryResults` via `.push()` calls. No browser needed for read-only tasks.
+
+## Do this first: pick your access path
+
+| Goal | Best approach | Latency |
+|------|--------------|---------|
+| Question metadata + first ~3 ranked answers | `http_get` question page + parse push payloads | ~600ms |
+| Single answer (full text + upvotes + views) | `http_get` answer permalink | ~400ms |
+| Answer count for a question | question page, payload with `answerCount` | same request as above |
+| Topic metadata (id, name, follower count) | `http_get` topic page + parse push payloads | ~400ms |
+| User profile (name, follower/following, credential) | `http_get` profile page + parse push payloads | ~500ms |
+| Keyword search results | NOT available via http_get — server returns no result data | N/A |
+
+**Never use a browser for read-only Quora tasks.** All question, answer, topic, and profile data is server-rendered. Browser is only needed for authenticated actions (posting, upvoting, following) or for getting more than the first ~3 answers on a question page (the rest load via XHR pagination).
+
+---
+
+## UA requirement: Chrome or Firefox — NOT bare Mozilla/5.0
+
+```
+bare "Mozilla/5.0"  -> HTTP 403
+Googlebot UA        -> HTTP 403
+Chrome UA           -> HTTP 200  (confirmed working)
+Firefox UA          -> HTTP 200  (confirmed working)
+```
+
+Use this header bundle for all requests:
+
+```python
+import urllib.request, gzip, json, re
+
+CHROME_UA = (
+    "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) "
+    "AppleWebKit/537.36 (KHTML, like Gecko) "
+    "Chrome/123.0.0.0 Safari/537.36"
+)
+
+def quora_get(url):
+    """Fetch any public Quora page. Returns HTML string.
+    Requires Chrome/Firefox UA — bare Mozilla/5.0 returns 403.
+    """
+    req = urllib.request.Request(url, headers={
+        "User-Agent": CHROME_UA,
+        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+        "Accept-Encoding": "gzip",
+        "Accept-Language": "en-US,en;q=0.9",
+    })
+    with urllib.request.urlopen(req, timeout=20) as r:
+        data = r.read()
+        if r.headers.get("Content-Encoding") == "gzip":
+            data = gzip.decompress(data)
+        return data.decode()
+```
+
+---
+
+## The data format: `ansFrontendGlobals.data.inlineQueryResults`
+
+Quora SSR embeds all page data as a series of `.push("...")` calls inside `<script>` blocks. Each call pushes a JSON-encoded string (with escaped quotes) into `inlineQueryResults`. There are no JSON-LD blocks, no `__NEXT_DATA__`, no React hydration state — only these push calls.
+
+```python
+def extract_quora_payloads(html):
+    """Extract and parse all push() payloads from a Quora page.
+    Returns list of dicts (already decoded from double JSON encoding).
+    """
+    raw_payloads = re.findall(r'\.push\("((?:[^"\\]|\\.)*)"\)', html)
+    results = []
+    for raw in raw_payloads:
+        try:
+            # Two levels of encoding: outer JS string escape, inner JSON
+            inner = json.loads('"' + raw + '"')   # decode JS string escaping
+            results.append(json.loads(inner))      # decode actual JSON
+        except Exception:
+            pass
+    return results
+```
+
+A question page returns **16 payloads**. A profile or topic page returns **3 payloads**. The payloads that matter are identified by their `data` keys, not by position (positions are stable across requests for the same page type, but best to key on content).
+
+---
+
+## Path 1: Question page — metadata + first answers (fastest)
+
+```python
+def quora_question(url):
+    """
+    Scrape a Quora question page.
+    Returns:
+      question: {qid, id, title, url, slug, topics}
+      answers:  list of answer dicts (first ~3 ranked answers only)
+      answer_count: total answer count (all answers, not just loaded)
+      related_questions: list of question title strings
+    Only the first ~3 highest-ranked answers are SSR'd.
+    The rest require XHR pagination (browser or session cookies needed).
+    """
+    html = quora_get(url)
+    payloads = extract_quora_payloads(html)
+
+    def spans_to_text(json_str):
+        """Quora stores all text as serialized span objects."""
+        try:
+            doc = json.loads(json_str)
+            parts = []
+            for sec in doc.get('sections', []):
+                for span in sec.get('spans', []):
+                    if span.get('text'):
+                        parts.append(span['text'])
+                parts.append('\n')
+            return ''.join(parts).strip()
+        except Exception:
+            return json_str
+
+    def author_display_name(author_dict):
+        names = author_dict.get('names', [])
+        if names:
+            n = names[0]
+            return f"{n.get('givenName', '')} {n.get('familyName', '')}".strip()
+        return None
+
+    result = {'question': {}, 'answers': [], 'answer_count': None, 'related_questions': []}
+
+    for payload in payloads:
+        data = payload.get('data', payload)
+
+        # Question metadata — keyed by presence of 'qid' inside 'question'
+        if 'question' in data and isinstance(data['question'], dict):
+            q = data['question']
+            if q.get('qid') and not result['question']:
+                result['question'] = {
+                    'qid':   q.get('qid'),
+                    'id':    q.get('id'),
+                    'title': spans_to_text(q.get('title', '')),
+                    'url':   q.get('url'),
+                    'slug':  q.get('slug'),
+                    'topics': [t['name'] for t in q.get('navigationTopics', [])],
+                }
+
+        # Total answer count — keyed by 'answerCount'
+        if 'answerCount' in data:
+            result['answer_count'] = data['answerCount']
+            rq = (data.get('bottomRelatedQuestionsInfo') or {}).get('relatedQuestions', [])
+            result['related_questions'] = [spans_to_text(r['title']) for r in rq]
+
+        # Answer nodes — keyed by node.__typename == 'QuestionAnswerItem2'
+        node = data.get('node', {})
+        if isinstance(node, dict) and node.get('__typename') == 'QuestionAnswerItem2':
+            answer = node.get('answer', {})
+            if answer.get('aid'):
+                a_author = answer.get('author') or {}
+                cred = answer.get('authorCredential') or {}
+                result['answers'].append({
+                    'aid':              answer.get('aid'),
+                    'index':            node.get('index'),
+                    'author_name':      author_display_name(a_author),
+                    'author_profile':   a_author.get('profileUrl'),
+                    'author_uid':       a_author.get('uid'),
+                    'author_credential': cred.get('translatedString'),
+                    'num_upvotes':      answer.get('numUpvotes'),
+                    'num_views':        answer.get('numViews'),
+                    'num_shares':       answer.get('numShares'),
+                    'num_comments':     answer.get('numDisplayComments'),
+                    'creation_time_us': answer.get('creationTime'),  # microseconds since epoch
+                    'viewer_has_access': answer.get('viewerHasAccess'),
+                    'perma_url':        answer.get('permaUrl'),
+                    'text':             spans_to_text(answer.get('content', '{}')),
+                })
+
+    return result
+```
+
+### Example output
+
+```python
+result = quora_question("https://www.quora.com/What-is-the-meaning-of-life")
+
+# result['question']:
+# {
+#   'qid': 2861,
+#   'id': 'UXVlc3Rpb25AMDoyODYx',
+#   'title': 'What is the meaning of life?',
+#   'url': '/What-is-the-meaning-of-life',
+#   'slug': 'What-is-the-meaning-of-life',
+#   'topics': ['Philosophy', 'The Big Unanswered Questions', 'Meaning of Life', ...]
+# }
+
+# result['answer_count']:  413
+
+# result['answers'][0]:
+# {
+#   'aid': 2779675,
+#   'index': 1,
+#   'author_name': 'Shubhankar Srivastava',
+#   'author_profile': '/profile/Shubhankar-Srivastava',
+#   'author_uid': 5381038,
+#   'author_credential': 'works at D. E. Shaw',
+#   'num_upvotes': 589,
+#   'num_views': 24085,
+#   'num_shares': 0,
+#   'num_comments': 8,
+#   'creation_time_us': 1373364681312036,   # divide by 1e6 for seconds
+#   'viewer_has_access': True,
+#   'perma_url': '/What-is-the-meaning-of-life/answer/Shubhankar-Srivastava',
+#   'text': 'Every morning in Africa, a deer wakes up...'
+# }
+```
+
+### Convert creation_time_us to datetime
+
+```python
+from datetime import datetime, timezone
+ts_sec = result['answers'][0]['creation_time_us'] / 1_000_000
+dt = datetime.fromtimestamp(ts_sec, tz=timezone.utc)
+# datetime(2013, 7, 9, 9, 31, 21, tzinfo=timezone.utc)
+```
+
+---
+
+## Path 2: Single answer permalink
+
+Fetching `quora.com/{question-slug}/answer/{author-slug}` directly returns only that one answer's full data in 3 payloads instead of 16. Use this when you already know the answer URL.
+
+```python
+def quora_answer(answer_url):
+    """
+    Fetch a single answer by its permalink.
+    URL format: https://www.quora.com/{question-slug}/answer/{author-profile-slug}
+    Returns answer dict with: aid, num_upvotes, num_views, text, author info.
+    """
+    html = quora_get(answer_url)
+    payloads = extract_quora_payloads(html)
+
+    for payload in payloads:
+        data = payload.get('data', {})
+        if 'answer' in data and isinstance(data['answer'], dict):
+            a = data['answer']
+            author = a.get('author') or {}
+            names = author.get('names', [{}])
+            n = names[0] if names else {}
+            return {
+                'aid':          a.get('aid'),
+                'num_upvotes':  a.get('numUpvotes'),
+                'num_views':    a.get('numViews'),
+                'author_name':  f"{n.get('givenName','')} {n.get('familyName','')}".strip(),
+                'author_uid':   author.get('uid'),
+                'text':         _spans_to_text(a.get('content', '{}')),
+            }
+    return {}
+
+# Example:
+# quora_answer("https://www.quora.com/What-is-the-meaning-of-life/answer/Pararth-Shah")
+# -> {'aid': 4734237, 'num_upvotes': 234, 'num_views': 100643, 'author_name': 'Pararth Shah', ...}
+```
+
+---
+
+## Path 3: Topic page
+
+```python
+def quora_topic(topic_url):
+    """
+    Fetch topic metadata from a Quora topic page.
+    URL format: https://www.quora.com/topic/{topic-slug}
+    Returns: tid, name, num_followers, url, is_following, has_leaderboard.
+    NOTE: The topic page itself only renders topic metadata, NOT the question feed.
+    Question feed requires browser (XHR-loaded via React).
+    """
+    html = quora_get(topic_url)
+    payloads = extract_quora_payloads(html)
+
+    for payload in payloads:
+        data = payload.get('data', {})
+        if 'topic' in data and isinstance(data['topic'], dict):
+            t = data['topic']
+            return {
+                'tid':           t.get('tid'),
+                'id':            t.get('id'),
+                'name':          t.get('name'),
+                'url':           t.get('url'),
+                'num_followers': t.get('numFollowers'),
+                'is_following':  t.get('isFollowing'),
+                'has_leaderboard': t.get('hasLeaderboard'),
+                'photo_url':     t.get('photoUrl'),
+                'is_locked':     t.get('isLocked'),
+            }
+    return {}
+
+# Example:
+# quora_topic("https://www.quora.com/topic/Python-programming-language")
+# -> {'tid': 13292, 'name': 'Python Programming Language', 'num_followers': 10, ...}
+```
+
+---
+
+## Path 4: User profile page
+
+```python
+def quora_profile(profile_url):
+    """
+    Fetch user profile data from https://www.quora.com/profile/{username}
+    Returns: uid, name, credential, follower_count, following_count, profile_image_url.
+    """
+    html = quora_get(profile_url)
+    payloads = extract_quora_payloads(html)
+
+    for payload in payloads:
+        data = payload.get('data', {})
+        if 'user' in data and isinstance(data['user'], dict):
+            u = data['user']
+            names = u.get('names', [{}])
+            n = names[0] if names else {}
+            cred = u.get('profileCredential') or {}
+            return {
+                'uid':             u.get('uid'),
+                'id':              u.get('id'),
+                'name':            f"{n.get('givenName','')} {n.get('familyName','')}".strip(),
+                'profile_url':     u.get('profileUrl'),
+                'follower_count':  u.get('followerCount'),
+                'following_count': u.get('followingCount'),
+                'profile_image':   u.get('profileImageUrl'),
+                'credential':      cred.get('experience'),
+                'is_verified':     u.get('isVerified'),
+                'is_anon':         u.get('isAnon'),
+                'is_ai_account':   u.get('isAiAccount'),
+                'deactivated':     u.get('deactivated'),
+            }
+    return {}
+
+# Example:
+# quora_profile("https://www.quora.com/profile/Pararth-Shah")
+# -> {'uid': 4683832, 'name': 'Pararth Shah', 'follower_count': 5154,
+#     'following_count': 83, 'credential': 'Unfinished symphony.', ...}
+```
+
+---
+
+## Gotchas
+
+- **Bare Mozilla/5.0 UA returns HTTP 403** — Always use a full Chrome or Firefox UA string. The default `http_get` helper's `"User-Agent": "Mozilla/5.0"` will be blocked. Do not use `http_get` directly; use the `quora_get` wrapper above.
+
+- **Googlebot UA returns HTTP 403** — Quora blocks crawler UAs. Only real browser UAs work.
+
+- **Double JSON encoding** — Each `.push()` argument is a JavaScript string literal containing JSON. To parse: first `json.loads('"' + raw + '"')` to decode the JS string escaping (converts `\\"` to `"`), then `json.loads(inner)` to parse the actual JSON object. Skipping either step produces parse errors.
+
+- **All text fields are serialized span objects** — `question.title`, `answer.content`, `user.descriptionQtextDocument.legacyJson`, etc. are all JSON strings containing a `{"sections": [{"spans": [...]}]}` document, not plain text. Always parse through `spans_to_text()`.
+
+- **Question page only SSR's the first ~3 answers** — The `answers` list in the result will contain at most 3 entries (the top-ranked answers). The `answer_count` field shows the true total (e.g. 413). To get more answers you need browser-based XHR pagination (Quora sends additional answers via GraphQL calls that require session auth in practice).
+
+- **`viewer_has_access: false` still includes full content** — Even when `viewerHasAccess` is `False` (answers from Quora+ Spaces / tribe-only content), the `content` field is still present in the SSR payload and the full text is readable. The flag only controls client-side gating in the browser.
+
+- **`creation_time_us` is microseconds, not milliseconds** — Divide by `1_000_000` (not `1_000`) to get a Unix timestamp in seconds. Confirmed: `1373364681312036 / 1e6 = 1373364681.3` (July 2013).
+
+- **`numFollowers` on topic pages may be 0 even for major topics** — The field reflects the logged-in user's follow state for some topics and appears to undercount. Treat as approximate.
+
+- **Search pages do not yield result data** — `https://www.quora.com/search?q=...` returns 3 payloads with viewer/network info only — no search results in the SSR payload. Search results are loaded client-side and are not accessible via http_get.
+
+- **Profile pages do not include the user's answer list** — The profile page SSR payload returns user metadata only. The list of a user's answers is loaded via XHR pagination. To get answers for a specific question, use the question URL directly.
+
+- **IDs are base64-encoded Relay global IDs** — `id: "UXVlc3Rpb25AMDoyODYx"` decodes to `"Question@0:2861"`. The numeric `qid`/`uid`/`aid`/`tid` fields are more useful for constructing URLs and deduplication. Use `qid` and `aid` as stable identifiers.
+
+- **`permaUrl` may be an absolute URL for Spaces answers** — Most answers have `permaUrl: "/Question-slug/answer/Author-Name"` (relative). Answers posted in a Quora Space have a full absolute URL like `"https://spacename.quora.com/Question-slug"`. Handle both forms.
+
+- **No public REST or GraphQL API** — Quora's internal `graphql/gql_para_POST` endpoint requires a valid `quora-formkey` header derived from the session, making it inaccessible without a real authenticated session. The SSR push-payload approach is the only reliable unauthenticated path.