pygments.rb 1.2.1 → 2.0.0.rc2

data/CHANGELOG.md

@@ -1,111 +0,0 @@
-CHANGELOG
-===========
-
-Version 1.2.1 (2017/12/07)
------------------------------
-
-* Automatically update `lexers` cache on build [186](https://github.com/tmm1/pygments.rb/pull/186)
-  * See [#185](https://github.com/tmm1/pygments.rb/pull/185) for the reason
-
-Version 1.2.0 (2017/09/13)
------------------------------
-
-* Exclude symlinks from the gem package to solve Windows issues [#181](https://github.com/tmm1/pygments.rb/pull/181)
-* Upgrade pygments to 2.0.0 [#180](https://github.com/tmm1/pygments.rb/pull/180)
-
-Version 1.1.2 (2017/04/03)
------------------------------
-
-* Resolves #176 exclude find_error.py symlink from gem [#177](https://github.com/tmm1/pygments.rb/pull/177)
-
-Version 1.1.1 (2016/12/28)
------------------------------
-
-* Suppress Ruby 2.4.0's warnings [#172](https://github.com/tmm1/pygments.rb/pull/172)
-* Enable `frozen_string_literal` [#173](https://github.com/tmm1/pygments.rb/pull/173)
-
-Version 1.1.0 (2016/12/24)
------------------------------
-
-* Support JRuby [#170](https://github.com/tmm1/pygments.rb/pull/170)
-* Make pygments.rb thread safe [#171](https://github.com/tmm1/pygments.rb/pull/171)
-
-Version 1.0.0 (2016/12/11)
------------------------------
-
-* Upgrade bundled pygments to 2.2.0-HEAD [#167](https://github.com/tmm1/pygments.rb/pull/167)
-  * This includes **incompatible changes* because of upgrade of pygments.
-    See http://pygments.org/ for details.
-* Relax yajl-ruby dependency to "~> 1.2" [#164](https://github.com/tmm1/pygments.rb/pull/164)
-* Python binary can be configured by `PYTMENTS_RB_PYTHON` env [#168](https://github.com/tmm1/pygments.rb/pull/168)
-* Improved error messages when python binary is missing [#158](https://github.com/tmm1/pygments.rb/pull/158)
-
-
-Version 0.5.4 (Nov 3, 2013)
------------------------------
-
-* Update lexers file
-
-Version 0.5.3 (Sep 17, 2013)
------------------------------
-
-* Fixes for Slash lexer
-* Improve highlighting for Slash lexer
-* Upgrade to latest pygments (1.7, changes summary follows.  See pygments changelog for details)
-  * Add Clay lexer
-  * Add Perl 6 lexer
-  * Add Swig lexer
-  * Add nesC lexer
-  * Add BlitzBasic lexer
-  * Add EBNF lexer
-  * Add Igor Pro lexer
-  * Add Rexx lexer
-  * Add Agda lexer
-  * Recognize vim modelines
-  * Improve Python 3 lexer
-  * Improve Opa lexer
-  * Improve Julia lexer
-  * Improve Lasso lexer
-  * Improve Objective C/C++ lexer
-  * Improve Ruby lexer
-  * Improve Stan lexer
-  * Improve JavaScript lexer
-  * Improve HTTP lexer
-  * Improve Koka lexer
-  * Improve Haxe lexer
-  * Improve Prolog lexer
-  * Improve F# lexer
-
-Version 0.5.2 (July 17, 2013)
------------------------------
-
-* Add Slash lexer
-
-Version 0.5.1 (June 25, 2013)
------------------------------
-
-* Ensure compatability across distros by detecting if `python2` is available
-
-Version 0.5.0 (Apr 13, 2013)
------------------------------
-
-* Use #rstrip to fix table mode bug
-
-Version 0.4.2 (Feb 25, 2013)
------------------------------
-
-* Add new lexers, including custom lexers
-
-Version 0.3.7 (Jan 2, 2013)
------------------------------
-
-* Fixed missing custom lexers
-* Added syntax highlighting for Hxml
-
-Version 0.3.4 (Dec 28, 2012)
------------------------------
-
-* Add support for Windows
-* Add MIT license
-
-

data/README.md

@@ -1,121 +0,0 @@
-# pygments.rb [![CircleCI](https://circleci.com/gh/tmm1/pygments.rb.svg?style=svg)](https://circleci.com/gh/tmm1/pygments.rb) [![Gem Version](https://badge.fury.io/rb/pygments.rb.svg)](https://badge.fury.io/rb/pygments.rb)
-
-A Ruby wrapper for the Python [pygments syntax highlighter](http://pygments.org/).
-
-pygments.rb works by talking over a simple pipe to a long-lived
-Python child process. This library replaces [github/albino](https://github.com/github/albino),
-as well as a version of pygments.rb that used an embedded Python
-interpreter.
-
-Each Ruby process that runs has its own 'personal Python';
-for example, 4 Unicorn workers will have one Python process each.
-If a Python process dies, a new one will be spawned on the next
-pygments.rb request.
-
-## system requirements
-
-- Python 2.5, Python 2.6, or Python 2.7. You can always use Python 2.x from a `virtualenv` if
-  your default Python install is 3.x.
-
-## usage
-
-``` ruby 
-require 'pygments'
-``` 
-
-``` ruby
-Pygments.highlight(File.read(__FILE__), :lexer => 'ruby')
-```
-
-Encoding and other lexer/formatter options can be passed in via an
-options hash:
-
-``` ruby
-Pygments.highlight('code', :options => {:encoding => 'utf-8'})
-```
-
-pygments.rb defaults to using an HTML formatter. 
-To use a formatter other than `html`, specify it explicitly
-like so:
-
-``` ruby
-Pygments.highlight('code', :formatter => 'bbcode')
-Pygments.highlight('code', :formatter => 'terminal')
-```
-
-To generate CSS for HTML formatted code, use the `#css` method:
-
-``` ruby
-Pygments.css
-Pygments.css('.highlight')
-```
-
-To use a specific pygments style, pass the `:style` option to the `#css` method:
-
-``` ruby
-Pygments.css(:style => "monokai")
-```
-
-Other Pygments high-level API methods are also available.
-These methods return arrays detailing all the available lexers, formatters, 
-and styles.
-
-``` ruby
-Pygments.lexers
-Pygments.formatters
-Pygments.styles
-```
-
-To use a custom pygments installation, specify the path to
-`Pygments#start`:
-
-``` ruby
-Pygments.start("/path/to/pygments")
-```
-
-If you'd like logging, set the environmental variable `MENTOS_LOG` to a file path for your logfile.
-
-By default pygments.rb will timeout calls to pygments that take over 8 seconds. You can change this
-by setting the environmental variable `MENTOS_TIMEOUT` to a different positive integer value.
-
-## benchmarks
-
-
-    $ ruby bench.rb 50
-       Benchmarking....
-       Size: 698 bytes
-       Iterations: 50
-                                                      user     system      total        real
-       pygments popen                                0.010000   0.010000   0.020000 (  0.460370)
-       pygments popen (process already started)      0.010000   0.000000   0.010000 (  0.272975)
-       pygments popen (process already started 2)    0.000000   0.000000   0.000000 (  0.273589)
-
-    $ ruby bench.rb 10
-       Benchmarking....
-       Size: 15523 bytes
-       Iterations: 10
-                                                      user     system      total        real
-       pygments popen                               0.000000   0.000000   0.000000 (  0.819419)
-       pygments popen (process already started)     0.010000   0.000000   0.010000 (  0.676515)
-       pygments popen (process already started 2)   0.000000   0.010000   0.010000 (  0.674189)
-
-## license
-
-The MIT License (MIT)
-
-Copyright (c) Ted Nyman and Aman Gupta, 2012-2013
-
-Permission is hereby granted, free of charge, to any person obtaining a copy of this software and 
-associated documentation files (the "Software"), to deal in the Software without restriction, 
-including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, 
-and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, 
-subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in all copies or substantial 
-portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT 
-LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. 
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, 
-WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/circle.yml

@@ -1,20 +0,0 @@
-machine:
-  ruby:
-    version: 2.3.1
-  java:
-    version: oraclejdk8
-  environment:
-    JRUBY_OPTS: '--dev --debug'
-    JRUBY: jruby-9.1.6.0
-
-dependencies:
-  pre:
-    - echo rvm_autoupdate_flag=0 >> ~/.rvmrc
-    - rvm install $JRUBY
-  post:
-    - rvm-exec $JRUBY bundle install
-
-test:
-  override:
-    - rvm-exec $JRUBY bundle exec rake test
-    - bundle exec rake test

data/test/test_data.py

@@ -1,514 +0,0 @@
-# -*- coding: utf-8 -
-#
-# This file is part of gunicorn released under the MIT license.
-# See the NOTICE for more information.
-
-from __future__ import with_statement
-
-import errno
-import os
-import select
-import signal
-import sys
-import time
-import traceback
-
-
-from gunicorn.errors import HaltServer
-from gunicorn.pidfile import Pidfile
-from gunicorn.sock import create_socket
-from gunicorn import util
-
-from gunicorn import __version__, SERVER_SOFTWARE
-
-class Arbiter(object):
-    """
-    Arbiter maintain the workers processes alive. It launches or
-    kills them if needed. It also manages application reloading
-    via SIGHUP/USR2.
-    """
-
-    # A flag indicating if a worker failed to
-    # to boot. If a worker process exist with
-    # this error code, the arbiter will terminate.
-    WORKER_BOOT_ERROR = 3
-
-    START_CTX = {}
-
-    LISTENER = None
-    WORKERS = {}
-    PIPE = []
-
-    # I love dynamic languages
-    SIG_QUEUE = []
-    SIGNALS = map(
-        lambda x: getattr(signal, "SIG%s" % x),
-        "HUP QUIT INT TERM TTIN TTOU USR1 USR2 WINCH".split()
-    )
-    SIG_NAMES = dict(
-        (getattr(signal, name), name[3:].lower()) for name in dir(signal)
-        if name[:3] == "SIG" and name[3] != "_"
-    )
-
-    def __init__(self, app):
-        os.environ["SERVER_SOFTWARE"] = SERVER_SOFTWARE
-
-        self.setup(app)
-
-        self.pidfile = None
-        self.worker_age = 0
-        self.reexec_pid = 0
-        self.master_name = "Master"
-
-        # get current path, try to use PWD env first
-        try:
-            a = os.stat(os.environ['PWD'])
-            b = os.stat(os.getcwd())
-            if a.ino == b.ino and a.dev == b.dev:
-                cwd = os.environ['PWD']
-            else:
-                cwd = os.getcwd()
-        except:
-            cwd = os.getcwd()
-
-        args = sys.argv[:]
-        args.insert(0, sys.executable)
-
-        # init start context
-        self.START_CTX = {
-            "args": args,
-            "cwd": cwd,
-            0: sys.executable
-        }
-
-    def setup(self, app):
-        self.app = app
-        self.cfg = app.cfg
-        self.log = self.cfg.logger_class(app.cfg)
-
-        # reopen files
-        if 'GUNICORN_FD' in os.environ:
-            self.log.reopen_files()
-
-        self.address = self.cfg.address
-        self.num_workers = self.cfg.workers
-        self.debug = self.cfg.debug
-        self.timeout = self.cfg.timeout
-        self.proc_name = self.cfg.proc_name
-        self.worker_class = self.cfg.worker_class
-
-        if self.cfg.debug:
-            self.log.debug("Current configuration:")
-            for config, value in sorted(self.cfg.settings.iteritems()):
-                self.log.debug("  %s: %s", config, value.value)
-
-        if self.cfg.preload_app:
-            if not self.cfg.debug:
-                self.app.wsgi()
-            else:
-                self.log.warning("debug mode: app isn't preloaded.")
-
-    def start(self):
-        """\
-        Initialize the arbiter. Start listening and set pidfile if needed.
-        """
-        self.log.info("Starting gunicorn %s", __version__)
-        self.cfg.on_starting(self)
-        self.pid = os.getpid()
-        self.init_signals()
-        if not self.LISTENER:
-            self.LISTENER = create_socket(self.cfg, self.log)
-
-        if self.cfg.pidfile is not None:
-            self.pidfile = Pidfile(self.cfg.pidfile)
-            self.pidfile.create(self.pid)
-        self.log.debug("Arbiter booted")
-        self.log.info("Listening at: %s (%s)", self.LISTENER,
-            self.pid)
-        self.log.info("Using worker: %s",
-                self.cfg.settings['worker_class'].get())
-
-        self.cfg.when_ready(self)
-
-    def init_signals(self):
-        """\
-        Initialize master signal handling. Most of the signals
-        are queued. Child signals only wake up the master.
-        """
-        if self.PIPE:
-            map(os.close, self.PIPE)
-        self.PIPE = pair = os.pipe()
-        map(util.set_non_blocking, pair)
-        map(util.close_on_exec, pair)
-        self.log.close_on_exec()
-        map(lambda s: signal.signal(s, self.signal), self.SIGNALS)
-        signal.signal(signal.SIGCHLD, self.handle_chld)
-
-    def signal(self, sig, frame):
-        if len(self.SIG_QUEUE) < 5:
-            self.SIG_QUEUE.append(sig)
-            self.wakeup()
-
-    def run(self):
-        "Main master loop."
-        self.start()
-        util._setproctitle("master [%s]" % self.proc_name)
-
-        self.manage_workers()
-        while True:
-            try:
-                self.reap_workers()
-                sig = self.SIG_QUEUE.pop(0) if len(self.SIG_QUEUE) else None
-                if sig is None:
-                    self.sleep()
-                    self.murder_workers()
-                    self.manage_workers()
-                    continue
-
-                if sig not in self.SIG_NAMES:
-                    self.log.info("Ignoring unknown signal: %s", sig)
-                    continue
-
-                signame = self.SIG_NAMES.get(sig)
-                handler = getattr(self, "handle_%s" % signame, None)
-                if not handler:
-                    self.log.error("Unhandled signal: %s", signame)
-                    continue
-                self.log.info("Handling signal: %s", signame)
-                handler()
-                self.wakeup()
-            except StopIteration:
-                self.halt()
-            except KeyboardInterrupt:
-                self.halt()
-            except HaltServer, inst:
-                self.halt(reason=inst.reason, exit_status=inst.exit_status)
-            except SystemExit:
-                raise
-            except Exception:
-                self.log.info("Unhandled exception in main loop:\n%s",
-                            traceback.format_exc())
-                self.stop(False)
-                if self.pidfile is not None:
-                    self.pidfile.unlink()
-                sys.exit(-1)
-
-    def handle_chld(self, sig, frame):
-        "SIGCHLD handling"
-        self.wakeup()
-
-    def handle_hup(self):
-        """\
-        HUP handling.
-        - Reload configuration
-        - Start the new worker processes with a new configuration
-        - Gracefully shutdown the old worker processes
-        """
-        self.log.info("Hang up: %s", self.master_name)
-        self.reload()
-
-    def handle_quit(self):
-        "SIGQUIT handling"
-        raise StopIteration
-
-    def handle_int(self):
-        "SIGINT handling"
-        self.stop(False)
-        raise StopIteration
-
-    def handle_term(self):
-        "SIGTERM handling"
-        self.stop(False)
-        raise StopIteration
-
-    def handle_ttin(self):
-        """\
-        SIGTTIN handling.
-        Increases the number of workers by one.
-        """
-        self.num_workers += 1
-        self.manage_workers()
-
-    def handle_ttou(self):
-        """\
-        SIGTTOU handling.
-        Decreases the number of workers by one.
-        """
-        if self.num_workers <= 1:
-            return
-        self.num_workers -= 1
-        self.manage_workers()
-
-    def handle_usr1(self):
-        """\
-        SIGUSR1 handling.
-        Kill all workers by sending them a SIGUSR1
-        """
-        self.kill_workers(signal.SIGUSR1)
-        self.log.reopen_files()
-
-    def handle_usr2(self):
-        """\
-        SIGUSR2 handling.
-        Creates a new master/worker set as a slave of the current
-        master without affecting old workers. Use this to do live
-        deployment with the ability to backout a change.
-        """
-        self.reexec()
-
-    def handle_winch(self):
-        "SIGWINCH handling"
-        if os.getppid() == 1 or os.getpgrp() != os.getpid():
-            self.log.info("graceful stop of workers")
-            self.num_workers = 0
-            self.kill_workers(signal.SIGQUIT)
-        else:
-            self.log.info("SIGWINCH ignored. Not daemonized")
-
-    def wakeup(self):
-        """\
-        Wake up the arbiter by writing to the PIPE
-        """
-        try:
-            os.write(self.PIPE[1], '.')
-        except IOError, e:
-            if e.errno not in [errno.EAGAIN, errno.EINTR]:
-                raise
-
-    def halt(self, reason=None, exit_status=0):
-        """ halt arbiter """
-        self.stop()
-        self.log.info("Shutting down: %s", self.master_name)
-        if reason is not None:
-            self.log.info("Reason: %s", reason)
-        if self.pidfile is not None:
-            self.pidfile.unlink()
-        sys.exit(exit_status)
-
-    def sleep(self):
-        """\
-        Sleep until PIPE is readable or we timeout.
-        A readable PIPE means a signal occurred.
-        """
-        try:
-            ready = select.select([self.PIPE[0]], [], [], 1.0)
-            if not ready[0]:
-                return
-            while os.read(self.PIPE[0], 1):
-                pass
-        except select.error, e:
-            if e[0] not in [errno.EAGAIN, errno.EINTR]:
-                raise
-        except OSError, e:
-            if e.errno not in [errno.EAGAIN, errno.EINTR]:
-                raise
-        except KeyboardInterrupt:
-            sys.exit()
-
-
-    def stop(self, graceful=True):
-        """\
-        Stop workers
-
-        :attr graceful: boolean, If True (the default) workers will be
-        killed gracefully  (ie. trying to wait for the current connection)
-        """
-        try:
-            self.LISTENER.close()
-        except Exception:
-            pass
-        self.LISTENER = None
-        sig = signal.SIGQUIT
-        if not graceful:
-            sig = signal.SIGTERM
-        limit = time.time() + self.cfg.graceful_timeout
-        while self.WORKERS and time.time() < limit:
-            self.kill_workers(sig)
-            time.sleep(0.1)
-            self.reap_workers()
-        self.kill_workers(signal.SIGKILL)
-
-    def reexec(self):
-        """\
-        Relaunch the master and workers.
-        """
-        if self.pidfile is not None:
-            self.pidfile.rename("%s.oldbin" % self.pidfile.fname)
-
-        self.reexec_pid = os.fork()
-        if self.reexec_pid != 0:
-            self.master_name = "Old Master"
-            return
-
-        os.environ['GUNICORN_FD'] = str(self.LISTENER.fileno())
-        os.chdir(self.START_CTX['cwd'])
-        self.cfg.pre_exec(self)
-        util.closerange(3, self.LISTENER.fileno())
-        util.closerange(self.LISTENER.fileno()+1, util.get_maxfd())
-        os.execvpe(self.START_CTX[0], self.START_CTX['args'], os.environ)
-
-    def reload(self):
-        old_address = self.cfg.address
-
-        # reload conf
-        self.app.reload()
-        self.setup(self.app)
-
-        # reopen log files
-        self.log.reopen_files()
-
-        # do we need to change listener ?
-        if old_address != self.cfg.address:
-            self.LISTENER.close()
-            self.LISTENER = create_socket(self.cfg, self.log)
-            self.log.info("Listening at: %s", self.LISTENER)
-
-        # do some actions on reload
-        self.cfg.on_reload(self)
-
-        # unlink pidfile
-        if self.pidfile is not None:
-            self.pidfile.unlink()
-
-        # create new pidfile
-        if self.cfg.pidfile is not None:
-            self.pidfile = Pidfile(self.cfg.pidfile)
-            self.pidfile.create(self.pid)
-
-        # set new proc_name
-        util._setproctitle("master [%s]" % self.proc_name)
-
-        # spawn new workers
-        for i in range(self.cfg.workers):
-            self.spawn_worker()
-
-        # manage workers
-        self.manage_workers()
-
-    def murder_workers(self):
-        """\
-        Kill unused/idle workers
-        """
-        for (pid, worker) in self.WORKERS.items():
-            try:
-                if time.time() - worker.tmp.last_update() <= self.timeout:
-                    continue
-            except ValueError:
-                continue
-
-            self.log.critical("WORKER TIMEOUT (pid:%s)", pid)
-            self.kill_worker(pid, signal.SIGKILL)
-
-    def reap_workers(self):
-        """\
-        Reap workers to avoid zombie processes
-        """
-        try:
-            while True:
-                wpid, status = os.waitpid(-1, os.WNOHANG)
-                if not wpid:
-                    break
-                if self.reexec_pid == wpid:
-                    self.reexec_pid = 0
-                else:
-                    # A worker said it cannot boot. We'll shutdown
-                    # to avoid infinite start/stop cycles.
-                    exitcode = status >> 8
-                    if exitcode == self.WORKER_BOOT_ERROR:
-                        reason = "Worker failed to boot."
-                        raise HaltServer(reason, self.WORKER_BOOT_ERROR)
-                    worker = self.WORKERS.pop(wpid, None)
-                    if not worker:
-                        continue
-                    worker.tmp.close()
-        except OSError, e:
-            if e.errno == errno.ECHILD:
-                pass
-
-    def manage_workers(self):
-        """\
-        Maintain the number of workers by spawning or killing
-        as required.
-        """
-        if len(self.WORKERS.keys()) < self.num_workers:
-            self.spawn_workers()
-
-        workers = self.WORKERS.items()
-        workers.sort(key=lambda w: w[1].age)
-        while len(workers) > self.num_workers:
-            (pid, _) = workers.pop(0)
-            self.kill_worker(pid, signal.SIGQUIT)
-
-    def spawn_worker(self):
-        self.worker_age += 1
-        worker = self.worker_class(self.worker_age, self.pid, self.LISTENER,
-                                    self.app, self.timeout/2.0,
-                                    self.cfg, self.log)
-        self.cfg.pre_fork(self, worker)
-        pid = os.fork()
-        if pid != 0:
-            self.WORKERS[pid] = worker
-            return pid
-
-        # Process Child
-        worker_pid = os.getpid()
-        try:
-            util._setproctitle("worker [%s]" % self.proc_name)
-            self.log.info("Booting worker with pid: %s", worker_pid)
-            self.cfg.post_fork(self, worker)
-            worker.init_process()
-            sys.exit(0)
-        except SystemExit:
-            raise
-        except:
-            self.log.debug("Exception in worker process:\n%s",
-                    traceback.format_exc())
-            if not worker.booted:
-                sys.exit(self.WORKER_BOOT_ERROR)
-            sys.exit(-1)
-        finally:
-            self.log.info("Worker exiting (pid: %s)", worker_pid)
-            try:
-                worker.tmp.close()
-                self.cfg.worker_exit(self, worker)
-            except:
-                pass
-
-    def spawn_workers(self):
-        """\
-        Spawn new workers as needed.
-
-        This is where a worker process leaves the main loop
-        of the master process.
-        """
-
-        for i in range(self.num_workers - len(self.WORKERS.keys())):
-            self.spawn_worker()
-
-    def kill_workers(self, sig):
-        """\
-        Kill all workers with the signal `sig`
-        :attr sig: `signal.SIG*` value
-        """
-        for pid in self.WORKERS.keys():
-            self.kill_worker(pid, sig)
-
-    def kill_worker(self, pid, sig):
-        """\
-        Kill a worker
-
-        :attr pid: int, worker pid
-        :attr sig: `signal.SIG*` value
-         """
-        try:
-            os.kill(pid, sig)
-        except OSError, e:
-            if e.errno == errno.ESRCH:
-                try:
-                    worker = self.WORKERS.pop(pid)
-                    worker.tmp.close()
-                    self.cfg.worker_exit(self, worker)
-                    return
-                except (KeyError, OSError):
-                    return
-            raise