webcheck-strings 1.59__tar.gz → 1.601__tar.gz

{webcheck_strings-1.59 → webcheck_strings-1.601}/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.1
 Name: webcheck_strings
-Version: 1.59
+Version: 1.601
 Summary: Monitor text strings on websites
 Home-page: http://ssb22.user.srcf.net/setup/webcheck.html
 Author: Silas S. Brown
 Author-email: ssb22@cam.ac.uk
 License: Apache 2
-Platform: all
+Platform: any
 Classifier: Programming Language :: Python :: 2
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
@@ -20,8 +20,7 @@ WebCheck
 
 From http://ssb22.user.srcf.net/setup/webcheck.html 
 
-webcheck.py requires Python, and is compatible with both Python 2 and Python 3.
-It is a program to check Web pages for changes to specific phrases of text. I currently use it to:
+WebCheck is a Python program (compatible with both Python 2 and Python 3) to check Web pages for changes to specific phrases (strings) of text. I currently use it to:
 
 * check that external links from my website still lead to the information that was there when I added them (I might need to update my links otherwise), and check whether or not I need to warn Lynx users about misguided blocking,
 * check the websites of companies, organisations or governments for changes to a specific rule to which I'd be interested in changes (for example because it applies to myself or someone I know, or because I've commented on it somewhere and my comment might need updating),
@@ -89,7 +88,7 @@ If the site lists new items but does not support RSS, you can also *extract* ite
 
 Besides checking `http://`, `https://` and `gemini://` URLs, you can check for:
 
-  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.216.34` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.216.34)`
+  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.215.14` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.215.14)`
   * Server reachability: if a server has been unreachable for a long time and you want to be alerted if it ever becomes reachable again, you can place `up://` before the URL (e.g. `up://http://www.example.com`) which will return `yes` or `no` and not report an error if the server is not reachable.
   * Misguided blocking of the Lynx browser. Prepend `blocks-lynx://` to a URL to have Webcheck try to fetch it with a Lynx user agent and return `yes` if it gets an error or timeout only with that agent, or `no` otherwise, so you can, at least in a `noscript` tag, warn your visitors that the server to which you link has accidentally started discriminating against blind and other users of the text-only Lynx browser due to misguided security settings—some “example” security configurations of web servers incorrectly assume Lynx implies automation and block it, and staff have been known to copy such examples without review or without realising what blocking Lynx implies.
   * Items in the HTTP HEAD response: prepend `head://` to a URL. This might be useful for checking `Last-Modified` to see if a large download has changed, like `wget -N` but without needing to keep a copy of the file on the machine where your WebCheck runs. 

{webcheck_strings-1.59 → webcheck_strings-1.601}/setup.py

@@ -1,10 +1,9 @@
-from setuptools import setup, find_packages;setup(name='webcheck_strings',version='1.59',entry_points={'console_scripts':['webcheck=webcheck.__init__:main']},license='Apache 2',platforms='all',url='http://ssb22.user.srcf.net/setup/webcheck.html',author='Silas S. Brown',author_email='ssb22@cam.ac.uk',description='Monitor text strings on websites',long_description=r'''WebCheck
+from setuptools import setup, find_packages;setup(name='webcheck_strings',version='1.601',entry_points={'console_scripts':['webcheck=webcheck.__init__:main']},license='Apache 2',platforms='any',url='http://ssb22.user.srcf.net/setup/webcheck.html',author='Silas S. Brown',author_email='ssb22@cam.ac.uk',description='Monitor text strings on websites',long_description=r'''WebCheck
 ========
 
 From http://ssb22.user.srcf.net/setup/webcheck.html 
 
-webcheck.py requires Python, and is compatible with both Python 2 and Python 3.
-It is a program to check Web pages for changes to specific phrases of text. I currently use it to:
+WebCheck is a Python program (compatible with both Python 2 and Python 3) to check Web pages for changes to specific phrases (strings) of text. I currently use it to:
 
 * check that external links from my website still lead to the information that was there when I added them (I might need to update my links otherwise), and check whether or not I need to warn Lynx users about misguided blocking,
 * check the websites of companies, organisations or governments for changes to a specific rule to which I'd be interested in changes (for example because it applies to myself or someone I know, or because I've commented on it somewhere and my comment might need updating),
@@ -72,7 +71,7 @@ If the site lists new items but does not support RSS, you can also *extract* ite
 
 Besides checking `http://`, `https://` and `gemini://` URLs, you can check for:
 
-  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.216.34` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.216.34)`
+  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.215.14` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.215.14)`
   * Server reachability: if a server has been unreachable for a long time and you want to be alerted if it ever becomes reachable again, you can place `up://` before the URL (e.g. `up://http://www.example.com`) which will return `yes` or `no` and not report an error if the server is not reachable.
   * Misguided blocking of the Lynx browser. Prepend `blocks-lynx://` to a URL to have Webcheck try to fetch it with a Lynx user agent and return `yes` if it gets an error or timeout only with that agent, or `no` otherwise, so you can, at least in a `noscript` tag, warn your visitors that the server to which you link has accidentally started discriminating against blind and other users of the text-only Lynx browser due to misguided security settings—some “example” security configurations of web servers incorrectly assume Lynx implies automation and block it, and staff have been known to copy such examples without review or without realising what blocking Lynx implies.
   * Items in the HTTP HEAD response: prepend `head://` to a URL. This might be useful for checking `Last-Modified` to see if a large download has changed, like `wget -N` but without needing to keep a copy of the file on the machine where your WebCheck runs. 

{webcheck_strings-1.59 → webcheck_strings-1.601}/webcheck/__init__.py

@@ -1,7 +1,7 @@
 #!/usr/bin/env python
 # (compatible with both Python 2 and Python 3)
 
-# webcheck.py v1.59 (c) 2014-24 Silas S. Brown.
+# webcheck.py v1.601 (c) 2014-24 Silas S. Brown.
 # See webcheck.html for description and usage instructions
 
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -29,7 +29,6 @@
 # and in China: https://gitee.com/ssb22/web-imap-etc
 
 max_threads = 10
-delay = 80 # seconds between fetches on same domain-set
 keep_etags = False # if True, will also keep any ETag headers as well as Last-Modified
 verify_SSL_certificates = False # webcheck's non-Webdriver URLs are for monitoring public services and there's not a lot of point in SSL authentication; failures due to server/client certificate misconfigurations are more trouble than they're worth
 
@@ -104,7 +103,7 @@ def read_input():
   days = 0 ; extraHeaders = []
   url = mainDomain = None
   lines = read_input_file()
-  lastList = None
+  lastList = None # refs a list within ret, of [(days,text,elseLogic)]
   while lines:
     line = line_withComment = " ".join(lines.pop().split())
     if " #" in line: line = line[:line.index(" #")].strip()
@@ -168,7 +167,7 @@ def read_input():
     if isElse:
       assert lastList, "else without suitable rule before it"
       lastList[-1] = lastList[-1][:2] + ((url,[(0,text,None)]),) # must be days=0 because don't want to re-check the days count when just retrieved and failed something possibly on same URL ('else:' can be used for simple retrying)
-      lastList = lastList[-1][2][1] # so 'else' can be used as 'else if'
+      lastList = lastList[-1][2][1] # ref the above list (the one after url), so 'else' can be used as 'else if' and next iteration extends it
     else:
       lastList = ret.setdefault({
         # domains to treat as equivalent for rate reduce
@@ -237,7 +236,8 @@ def main():
 
     # 1 job per domain:
     global jobs ; jobs = Queue.Queue()
-    for v in read_input().values(): jobs.put(v)
+    for mainDomain,jobItems in read_input().items():
+      jobs.put((mainDomain,jobItems))
     
     global previous_timestamps
     try: previous_timestamps = pickle.Unpickler(open(".webcheck-last","rb")).load()
@@ -268,19 +268,24 @@ default_ua = 'Mozilla/5.0 or whatever you like (actually Webcheck)'
 # people to hide their tools from webmasters unnecessarily.
 
 class Delayer:
-  def __init__(self): self.last_fetch_finished = 0
+  def __init__(self,mainDomain):
+    self.last_fetch_finished = 0
+    if mainDomain=="stackoverflow.com":
+      # (or other Stack Exchange sites)
+      self.delay = 80 # seconds between fetches to these sites
+    else: self.delay = 3 # (want small if checking many pages and the server doesn't mind, but still non-0)
   def wait(self):
-    time.sleep(max(0,self.last_fetch_finished+delay-time.time()))
+    time.sleep(max(0,self.last_fetch_finished+self.delay-time.time()))
     if sys.stderr.isatty(): sys.stderr.write('.'),sys.stderr.flush()
   def done(self): self.last_fetch_finished = time.time()
 
 def worker_thread(*args):
     opener = [None]
     while True:
-      try: job = jobs.get(False)
+      try: mainDomain,job = jobs.get(False)
       except: return # no more jobs left
       try:
-        delayer = Delayer()
+        delayer = Delayer(mainDomain)
         items = sorted(job.items()) # sorted will group http and https together
         items.reverse()
         while items:
@@ -306,6 +311,7 @@ def worker_thread(*args):
       jobs.task_done()
 class CDNBackoff(Exception): pass
 
+import threading ; lock = threading.Lock() # webcheck-debug
 def doJob(opener,delayer,url,checklist,extraHeaders):
   failRet = [c[2] for c in checklist if c[2]]
   delayer.wait()
@@ -413,13 +419,13 @@ def doJob(opener,delayer,url,checklist,extraHeaders):
           out=check(t[1:],content,"Source of "+url,"")
       elif not t or t.startswith('#'):
           parseRSS(url,content,t.replace('#','',1).strip())
-          out = None
+          out = None # (no else: after real RSS, but you can do else: for no extracted items: that goes through check() above)
       else:
         if textContent==None:
           textContent,errmsg=htmlStrings(content)
         out=check(t,textContent,url,errmsg)
-      if out:
-        if item[2]: toRet.append(item[2])
+      if out: # something to alert, unless else:
+        if item[2]: toRet.append(item[2]) # else:
         else: sys.stdout.write(out) # don't use 'print' or may have problems with threads
   return toRet
 
@@ -502,7 +508,7 @@ def run_webdriver_inner(actionList,browser):
             # wait for "string" to appear in the source
             tries = 30
             while tries and not myFind(a[1:-1],getSrc()):
-              time.sleep(delay) ; tries -= 1
+              time.sleep(2) ; tries -= 1
             if not tries:
               try: current_url = browser.current_url
               except: current_url = "(unable to obtain)"
@@ -586,7 +592,7 @@ def run_webdriver_inner(actionList,browser):
         elif re.match("[0-9]+$",a): time.sleep(int(a))
         else: sys.stdout.write("Ignoring webdriver unknown action "+repr(a)+'\n')
         if sys.stderr.isatty(): sys.stderr.write(':'),sys.stderr.flush() # webdriver's '.'
-        time.sleep(delay)
+        time.sleep(2)
     snippets.append(getSrc())
     return B('\n').join(snippets)
 
@@ -708,7 +714,7 @@ def check(text,content,url,errmsg):
     if comment: comment="\n  "+paren(comment)
     text = text.strip()
     assert text # or should have gone to parseRSS instead
-    if text.startswith('{') and text.endswith('}') and '...' in text: extract(url,content,text[1:-1].split('...'),orig_comment)
+    if text.startswith('{') and text.endswith('}') and '...' in text: return extract(url,content,text[1:-1].split('...'),orig_comment)
     elif text.startswith("!"): # 'not', so alert if DOES contain
         if len(text)==1: return # TODO: print error?
         if myFind(text[1:],content):
@@ -814,8 +820,8 @@ def extract(url,content,startEndMarkers,comment):
     c = content[i+len(start):j].decode('utf-8').strip()
     if c: items.append(('Auto-extracted text:','',c,"")) # NB the 'title' field must not be empty (unless we relocate that logic to parseRSS instead of handleRSS)
     i = j+len(end)
-  if not items: print ("No items were extracted from "+url+" via "+S(start)+"..."+S(end)+" (check that site changes haven't invalidated this extraction rule)")
-  handleRSS(url,items,comment,"extracted")
+  if not items: return ("No items were extracted from "+url+" via "+S(start)+"..."+S(end)+" (check that site changes haven't invalidated this extraction rule)")
+  else: handleRSS(url,items,comment,"extracted")
 
 def myFind(text,content):
   text,content = B(text),B(content)

{webcheck_strings-1.59 → webcheck_strings-1.601}/webcheck_strings.egg-info/PKG-INFO

@@ -1,12 +1,12 @@
 Metadata-Version: 2.1
 Name: webcheck-strings
-Version: 1.59
+Version: 1.601
 Summary: Monitor text strings on websites
 Home-page: http://ssb22.user.srcf.net/setup/webcheck.html
 Author: Silas S. Brown
 Author-email: ssb22@cam.ac.uk
 License: Apache 2
-Platform: all
+Platform: any
 Classifier: Programming Language :: Python :: 2
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: Apache Software License
@@ -20,8 +20,7 @@ WebCheck
 
 From http://ssb22.user.srcf.net/setup/webcheck.html 
 
-webcheck.py requires Python, and is compatible with both Python 2 and Python 3.
-It is a program to check Web pages for changes to specific phrases of text. I currently use it to:
+WebCheck is a Python program (compatible with both Python 2 and Python 3) to check Web pages for changes to specific phrases (strings) of text. I currently use it to:
 
 * check that external links from my website still lead to the information that was there when I added them (I might need to update my links otherwise), and check whether or not I need to warn Lynx users about misguided blocking,
 * check the websites of companies, organisations or governments for changes to a specific rule to which I'd be interested in changes (for example because it applies to myself or someone I know, or because I've commented on it somewhere and my comment might need updating),
@@ -89,7 +88,7 @@ If the site lists new items but does not support RSS, you can also *extract* ite
 
 Besides checking `http://`, `https://` and `gemini://` URLs, you can check for:
 
-  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.216.34` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.216.34)`
+  * DNS changes (useful if you’re maintaining a hosts file somewhere due to unreliable DNS or an awkward proxy situation): URLs starting `dns://` will return a list of all current IPs, each enclosed in parentheses. So for example to be alerted if `93.184.215.14` ceases to be one of the IP addresses of `example.com`, use `dns://example.com (93.184.215.14)`
   * Server reachability: if a server has been unreachable for a long time and you want to be alerted if it ever becomes reachable again, you can place `up://` before the URL (e.g. `up://http://www.example.com`) which will return `yes` or `no` and not report an error if the server is not reachable.
   * Misguided blocking of the Lynx browser. Prepend `blocks-lynx://` to a URL to have Webcheck try to fetch it with a Lynx user agent and return `yes` if it gets an error or timeout only with that agent, or `no` otherwise, so you can, at least in a `noscript` tag, warn your visitors that the server to which you link has accidentally started discriminating against blind and other users of the text-only Lynx browser due to misguided security settings—some “example” security configurations of web servers incorrectly assume Lynx implies automation and block it, and staff have been known to copy such examples without review or without realising what blocking Lynx implies.
   * Items in the HTTP HEAD response: prepend `head://` to a URL. This might be useful for checking `Last-Modified` to see if a large download has changed, like `wget -N` but without needing to keep a copy of the file on the machine where your WebCheck runs.