pycparser 2.22__tar.gz → 3.0__tar.gz

{pycparser-2.22 → pycparser-3.0}/MANIFEST.in

@@ -6,7 +6,3 @@ include README.*
 include LICENSE
 include CHANGES
 include setup.*
-exclude setup.pyc
-
-recursive-exclude tests yacctab.* lextab.*
-recursive-exclude examples yacctab.* lextab.*

pycparser-3.0/PKG-INFO

@@ -0,0 +1,244 @@
+Metadata-Version: 2.4
+Name: pycparser
+Version: 3.0
+Summary: C parser in Python
+Author-email: Eli Bendersky <eliben@gmail.com>
+Maintainer-email: Eli Bendersky <eliben@gmail.com>
+License-Expression: BSD-3-Clause
+Project-URL: Homepage, https://github.com/eliben/pycparser
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.10
+Description-Content-Type: text/x-rst
+License-File: LICENSE
+Dynamic: license-file
+
+===============
+pycparser v3.00
+===============
+
+
+.. image:: https://github.com/eliben/pycparser/workflows/pycparser-tests/badge.svg
+  :align: center
+  :target: https://github.com/eliben/pycparser/actions
+
+----
+
+.. contents::
+    :backlinks: none
+
+.. sectnum::
+
+Introduction
+============
+
+What is pycparser?
+------------------
+
+**pycparser** is a parser for the C language, written in pure Python. It is a
+module designed to be easily integrated into applications that need to parse
+C source code.
+
+What is it good for?
+--------------------
+
+Anything that needs C code to be parsed. The following are some uses for
+**pycparser**, taken from real user reports:
+
+* C code obfuscator
+* Front-end for various specialized C compilers
+* Static code checker
+* Automatic unit-test discovery
+* Adding specialized extensions to the C language
+
+One of the most popular uses of **pycparser** is in the `cffi
+<https://cffi.readthedocs.io/en/latest/>`_ library, which uses it to parse the
+declarations of C functions and types in order to auto-generate FFIs.
+
+**pycparser** is unique in the sense that it's written in pure Python - a very
+high level language that's easy to experiment with and tweak. To people familiar
+with Lex and Yacc, **pycparser**'s code will be simple to understand. It also
+has no external dependencies (except for a Python interpreter), making it very
+simple to install and deploy.
+
+Which version of C does pycparser support?
+------------------------------------------
+
+**pycparser** aims to support the full C99 language (according to the standard
+ISO/IEC 9899). Some features from C11 are also supported, and patches to support
+more are welcome.
+
+**pycparser** supports very few GCC extensions, but it's fairly easy to set
+things up so that it parses code with a lot of GCC-isms successfully. See the
+`FAQ <https://github.com/eliben/pycparser/wiki/FAQ>`_ for more details.
+
+What grammar does pycparser follow?
+-----------------------------------
+
+**pycparser** very closely follows the C grammar provided in Annex A of the C99
+standard (ISO/IEC 9899).
+
+How is pycparser licensed?
+--------------------------
+
+`BSD license <https://github.com/eliben/pycparser/blob/master/LICENSE>`_.
+
+Contact details
+---------------
+
+For reporting problems with **pycparser** or submitting feature requests, please
+open an `issue <https://github.com/eliben/pycparser/issues>`_, or submit a
+pull request.
+
+
+Installing
+==========
+
+Prerequisites
+-------------
+
+**pycparser** is being tested with modern versions of Python on
+Linux, macOS and Windows. See `the CI dashboard <https://github.com/eliben/pycparser/actions/workflows/ci.yml>`__
+for details.
+
+**pycparser** has no external dependencies.
+
+Installation process
+--------------------
+
+The recommended way to install **pycparser** is with ``pip``::
+
+    > pip install pycparser
+
+Using
+=====
+
+Interaction with the C preprocessor
+-----------------------------------
+
+In order to be compilable, C code must be preprocessed by the C preprocessor -
+``cpp``. A compatible ``cpp`` handles preprocessing directives like ``#include`` and
+``#define``, removes comments, and performs other minor tasks that prepare the C
+code for compilation.
+
+For all but the most trivial snippets of C code **pycparser**, like a C
+compiler, must receive preprocessed C code in order to function correctly. If
+you import the top-level ``parse_file`` function from the **pycparser** package,
+it will interact with ``cpp`` for you, as long as it's in your PATH, or you
+provide a path to it.
+
+Note also that you can use ``gcc -E`` or ``clang -E`` instead of ``cpp``. See
+the ``using_gcc_E_libc.py`` example for more details. Windows users can download
+and install a binary build of Clang for Windows `from this website
+<http://llvm.org/releases/download.html>`_.
+
+What about the standard C library headers?
+------------------------------------------
+
+C code almost always ``#include``\s various header files from the standard C
+library, like ``stdio.h``. While (with some effort) **pycparser** can be made to
+parse the standard headers from any C compiler, it's much simpler to use the
+provided "fake" standard includes for C11 in ``utils/fake_libc_include``. These
+are standard C header files that contain only the bare necessities to allow
+valid parsing of the files that use them. As a bonus, since they're minimal, it
+can significantly improve the performance of parsing large C files.
+
+The key point to understand here is that **pycparser** doesn't really care about
+the semantics of types. It only needs to know whether some token encountered in
+the source is a previously defined type. This is essential in order to be able
+to parse C correctly.
+
+See `this blog post
+<https://eli.thegreenplace.net/2015/on-parsing-c-type-declarations-and-fake-headers>`_
+for more details.
+
+Note that the fake headers are not included in the ``pip`` package nor installed
+via the package build (`#224 <https://github.com/eliben/pycparser/issues/224>`_).
+
+Basic usage
+-----------
+
+Take a look at the |examples|_ directory of the distribution for a few examples
+of using **pycparser**. These should be enough to get you started. Please note
+that most realistic C code samples would require running the C preprocessor
+before passing the code to **pycparser**; see the previous sections for more
+details.
+
+.. |examples| replace:: ``examples``
+.. _examples: examples
+
+
+Advanced usage
+--------------
+
+The public interface of **pycparser** is well documented with comments in
+``pycparser/c_parser.py``. For a detailed overview of the various AST nodes
+created by the parser, see ``pycparser/_c_ast.cfg``.
+
+There's also a `FAQ available here <https://github.com/eliben/pycparser/wiki/FAQ>`_.
+In any case, you can always drop me an `email <eliben@gmail.com>`_ for help.
+
+
+Modifying
+=========
+
+There are a few points to keep in mind when modifying **pycparser**:
+
+* The code for **pycparser**'s AST nodes is automatically generated from a
+  configuration file - ``_c_ast.cfg``, by ``_ast_gen.py``. If you modify the AST
+  configuration, make sure to re-generate the code. This can be done by running
+  the ``_ast_gen.py`` script (from the repository root or the
+  ``pycparser`` directory).
+* Read the docstring in the constructor of the ``CParser`` class for details
+  on configuration and compatibility arguments.
+
+
+Package contents
+================
+
+Once you unzip the ``pycparser`` package, you'll see the following files and
+directories:
+
+README.rst:
+  This README file.
+
+LICENSE:
+  The pycparser license
+
+setup.py:
+  Legacy installation script (build metadata lives in ``pyproject.toml``).
+
+pyproject.toml:
+  Package metadata and build configuration.
+
+examples/:
+  A directory with some examples of using **pycparser**
+
+pycparser/:
+  The **pycparser** module source code.
+
+tests/:
+  Unit tests.
+
+utils/fake_libc_include:
+  Minimal standard C library include files that should allow to parse any C code.
+  Note that these headers now include C11 code, so they may not work when the
+  preprocessor is configured to an earlier C standard (like ``-std=c99``).
+
+utils/internal/:
+  Internal utilities for my own use. You probably don't need them.
+
+
+Contributors
+============
+
+Some people have contributed to **pycparser** by opening issues on bugs they've
+found and/or submitting patches. The list of contributors is in the CONTRIBUTORS
+file in the source distribution. After **pycparser** moved to Github I stopped
+updating this list because Github does a much better job at tracking
+contributions.

{pycparser-2.22 → pycparser-3.0}/README.rst

@@ -1,5 +1,5 @@
 ===============
-pycparser v2.22
+pycparser v3.00
 ===============
 
 
@@ -14,7 +14,6 @@ pycparser v2.22
 
 .. sectnum::
 
-
 Introduction
 ============
 
@@ -83,18 +82,11 @@ Installing
 Prerequisites
 -------------
 
-* **pycparser** was tested with Python 3.8+ on Linux, macOS and Windows.
-
-* **pycparser** has no external dependencies. The only non-stdlib library it
-  uses is PLY, which is bundled in ``pycparser/ply``. The current PLY version is
-  3.10, retrieved from `<http://www.dabeaz.com/ply/>`_
+**pycparser** is being tested with modern versions of Python on
+Linux, macOS and Windows. See `the CI dashboard <https://github.com/eliben/pycparser/actions/workflows/ci.yml>`__
+for details.
 
-Note that **pycparser** (and PLY) uses docstrings for grammar specifications.
-Python installations that strip docstrings (such as when using the Python
-``-OO`` option) will fail to instantiate and use **pycparser**. You can try to
-work around this problem by making sure the PLY parsing tables are pre-generated
-in normal mode; this isn't an officially supported/tested mode of operation,
-though.
+**pycparser** has no external dependencies.
 
 Installation process
 --------------------
@@ -110,7 +102,7 @@ Interaction with the C preprocessor
 -----------------------------------
 
 In order to be compilable, C code must be preprocessed by the C preprocessor -
-``cpp``. ``cpp`` handles preprocessing directives like ``#include`` and
+``cpp``. A compatible ``cpp`` handles preprocessing directives like ``#include`` and
 ``#define``, removes comments, and performs other minor tasks that prepare the C
 code for compilation.
 
@@ -146,7 +138,7 @@ See `this blog post
 for more details.
 
 Note that the fake headers are not included in the ``pip`` package nor installed
-via ``setup.py`` (`#224 <https://github.com/eliben/pycparser/issues/224>`_).
+via the package build (`#224 <https://github.com/eliben/pycparser/issues/224>`_).
 
 Basic usage
 -----------
@@ -180,11 +172,10 @@ There are a few points to keep in mind when modifying **pycparser**:
 * The code for **pycparser**'s AST nodes is automatically generated from a
   configuration file - ``_c_ast.cfg``, by ``_ast_gen.py``. If you modify the AST
   configuration, make sure to re-generate the code. This can be done by running
-  the ``_build_tables.py`` script from the ``pycparser`` directory.
-* Make sure you understand the optimized mode of **pycparser** - for that you
-  must read the docstring in the constructor of the ``CParser`` class. For
-  development you should create the parser without optimizations, so that it
-  will regenerate the Yacc and Lex tables when you change the grammar.
+  the ``_ast_gen.py`` script (from the repository root or the
+  ``pycparser`` directory).
+* Read the docstring in the constructor of the ``CParser`` class for details
+  on configuration and compatibility arguments.
 
 
 Package contents
@@ -200,7 +191,10 @@ LICENSE:
   The pycparser license
 
 setup.py:
-  Installation script
+  Legacy installation script (build metadata lives in ``pyproject.toml``).
+
+pyproject.toml:
+  Package metadata and build configuration.
 
 examples/:
   A directory with some examples of using **pycparser**
@@ -228,5 +222,3 @@ found and/or submitting patches. The list of contributors is in the CONTRIBUTORS
 file in the source distribution. After **pycparser** moved to Github I stopped
 updating this list because Github does a much better job at tracking
 contributions.
-
-

{pycparser-2.22 → pycparser-3.0}/examples/c-to-c.py

@@ -1,4 +1,4 @@
-#------------------------------------------------------------------------------
+# ------------------------------------------------------------------------------
 # pycparser: c-to-c.py
 #
 # Example of using pycparser.c_generator, serving as a simplistic translator
@@ -6,19 +6,18 @@
 #
 # Eli Bendersky [https://eli.thegreenplace.net/]
 # License: BSD
-#------------------------------------------------------------------------------
+# ------------------------------------------------------------------------------
 import sys
 
 # This is not required if you've installed pycparser into
 # your site-packages/ with setup.py
-sys.path.extend(['.', '..'])
+sys.path.extend([".", ".."])
 
 from pycparser import parse_file, c_generator
 
 
-def translate_to_c(filename):
-    """ Simply use the c_generator module to emit a parsed AST.
-    """
+def translate_to_c(filename: str) -> None:
+    """Simply use the c_generator module to emit a parsed AST."""
     ast = parse_file(filename, use_cpp=True)
     generator = c_generator.CGenerator()
     print(generator.visit(ast))

{pycparser-2.22 → pycparser-3.0}/examples/c_files/funky.c

@@ -12,9 +12,4 @@ int maxout_in(int paste, char** matrix)
 int main()
 {
     auto char* multi = "a multi";
-    
-    
 }
-
-
-

{pycparser-2.22 → pycparser-3.0}/examples/c_files/hash.c

@@ -24,7 +24,7 @@ typedef struct tagHash
 {
     unsigned int table_size;
 
-    Node** heads; 
+    Node** heads;
 
 } Hash;
 
@@ -137,7 +137,7 @@ ReturnCode HashRemove(Hash* hash, const char* key)
 
             return SUCCESS;
         }
-        
+
         temp2 = temp1;
         temp1 = temp1->next;
     }
@@ -187,10 +187,10 @@ void HashDestroy(Hash* hash)
             free(temp->entry);
 
             temp = temp->next;
-            
+
             free(temp2);
         }
-    }    
+    }
 
     free(hash->heads);
     hash->heads = NULL;

{pycparser-2.22 → pycparser-3.0}/examples/c_json.py

@@ -1,4 +1,4 @@
-#------------------------------------------------------------------------------
+# ------------------------------------------------------------------------------
 # pycparser: c_json.py
 #
 # by Michael White (@mypalmike)
@@ -33,39 +33,48 @@
 #             ]
 #         }
 #     }
-#------------------------------------------------------------------------------
+# ------------------------------------------------------------------------------
 import json
 import sys
 import re
+from typing import Any, Callable, Dict, Optional, Set, TypeVar
 
 # This is not required if you've installed pycparser into
 # your site-packages/ with setup.py
-#
-sys.path.extend(['.', '..'])
+sys.path.extend([".", ".."])
 
 from pycparser import parse_file, c_ast
-from pycparser.plyparser import Coord
+from pycparser.c_parser import Coord
 
 
-RE_CHILD_ARRAY = re.compile(r'(.*)\[(.*)\]')
-RE_INTERNAL_ATTR = re.compile('__.*__')
+RE_CHILD_ARRAY = re.compile(r"(.*)\[(.*)\]")
+RE_INTERNAL_ATTR = re.compile("__.*__")
 
 
 class CJsonError(Exception):
     pass
 
 
-def memodict(fn):
-    """ Fast memoization decorator for a function taking a single argument """
-    class memodict(dict):
-        def __missing__(self, key):
-            ret = self[key] = fn(key)
-            return ret
-    return memodict().__getitem__
+_T = TypeVar("_T")
+_R = TypeVar("_R")
+
+
+def memodict(fn: Callable[[_T], _R]) -> Callable[[_T], _R]:
+    """Fast memoization decorator for a function taking a single argument"""
+    cache: Dict[_T, _R] = {}
+
+    def memoized(arg: _T) -> _R:
+        if arg in cache:
+            return cache[arg]
+        result = fn(arg)
+        cache[arg] = result
+        return result
+
+    return memoized
 
 
 @memodict
-def child_attrs_of(klass):
+def child_attrs_of(klass: type[c_ast.Node]) -> Set[str]:
     """
     Given a Node class, get a set of child attrs.
     Memoized to avoid highly repetitive string manipulation
@@ -76,14 +85,14 @@ def child_attrs_of(klass):
     return all_attrs - non_child_attrs
 
 
-def to_dict(node):
-    """ Recursively convert an ast into dict representation. """
+def to_dict(node: c_ast.Node) -> Dict[str, Any]:
+    """Recursively convert an ast into dict representation."""
     klass = node.__class__
 
-    result = {}
+    result: Dict[str, Any] = {}
 
     # Metadata
-    result['_nodetype'] = klass.__name__
+    result["_nodetype"] = klass.__name__
 
     # Local node attributes
     for attr in klass.attr_names:
@@ -91,9 +100,9 @@ def to_dict(node):
 
     # Coord object
     if node.coord:
-        result['coord'] = str(node.coord)
+        result["coord"] = str(node.coord)
     else:
-        result['coord'] = None
+        result["coord"] = None
 
     # Child attributes
     for child_name, child in node.children():
@@ -105,9 +114,12 @@ def to_dict(node):
             # arrays come in order, so we verify and append.
             result[array_name] = result.get(array_name, [])
             if array_index != len(result[array_name]):
-                raise CJsonError('Internal ast error. Array {} out of order. '
-                    'Expected index {}, got {}'.format(
-                    array_name, len(result[array_name]), array_index))
+                raise CJsonError(
+                    "Internal ast error. Array {} out of order. "
+                    "Expected index {}, got {}".format(
+                        array_name, len(result[array_name]), array_index
+                    )
+                )
             result[array_name].append(to_dict(child))
         else:
             result[child_name] = to_dict(child)
@@ -120,53 +132,55 @@ def to_dict(node):
     return result
 
 
-def to_json(node, **kwargs):
-    """ Convert ast node to json string """
+def to_json(node: c_ast.Node, **kwargs: Any) -> str:
+    """Convert ast node to json string"""
     return json.dumps(to_dict(node), **kwargs)
 
 
-def file_to_dict(filename):
-    """ Load C file into dict representation of ast """
+def file_to_dict(filename: str) -> Dict[str, Any]:
+    """Load C file into dict representation of ast"""
     ast = parse_file(filename, use_cpp=True)
     return to_dict(ast)
 
 
-def file_to_json(filename, **kwargs):
-    """ Load C file into json string representation of ast """
+def file_to_json(filename: str, **kwargs: Any) -> str:
+    """Load C file into json string representation of ast"""
     ast = parse_file(filename, use_cpp=True)
     return to_json(ast, **kwargs)
 
 
-def _parse_coord(coord_str):
-    """ Parse coord string (file:line[:column]) into Coord object. """
+def _parse_coord(coord_str: Optional[str]) -> Optional[Coord]:
+    """Parse coord string (file:line[:column]) into Coord object."""
     if coord_str is None:
         return None
 
-    vals = coord_str.split(':')
-    vals.extend([None] * 3)
+    vals = coord_str.split(":")
+    vals.extend(["", "", ""])
     filename, line, column = vals[:3]
-    return Coord(filename, line, column)
+    line_num = int(line) if line else 0
+    column_num = int(column) if column else None
+    return Coord(filename, line_num, column_num)
 
 
-def _convert_to_obj(value):
+def _convert_to_obj(value: Any) -> Any:
     """
     Convert an object in the dict representation into an object.
     Note: Mutually recursive with from_dict.
 
     """
-    value_type = type(value)
-    if value_type == dict:
-        return from_dict(value)
-    elif value_type == list:
-        return [_convert_to_obj(item) for item in value]
-    else:
-        # String
-        return value
+    match value:
+        case dict():
+            return from_dict(value)
+        case list():
+            return [_convert_to_obj(item) for item in value]
+        case _:
+            # String
+            return value
 
 
-def from_dict(node_dict):
-    """ Recursively build an ast from dict representation """
-    class_name = node_dict.pop('_nodetype')
+def from_dict(node_dict: Dict[str, Any]) -> c_ast.Node:
+    """Recursively build an ast from dict representation"""
+    class_name = node_dict.pop("_nodetype")
 
     klass = getattr(c_ast, class_name)
 
@@ -174,7 +188,7 @@ def from_dict(node_dict):
     # to node constructors.
     objs = {}
     for key, value in node_dict.items():
-        if key == 'coord':
+        if key == "coord":
             objs[key] = _parse_coord(value)
         else:
             objs[key] = _convert_to_obj(value)
@@ -184,12 +198,12 @@ def from_dict(node_dict):
     return klass(**objs)
 
 
-def from_json(ast_json):
-    """ Build an ast from json string representation """
+def from_json(ast_json: str) -> c_ast.Node:
+    """Build an ast from json string representation"""
     return from_dict(json.loads(ast_json))
 
 
-#------------------------------------------------------------------------------
+# ------------------------------------------------------------------------------
 if __name__ == "__main__":
     if len(sys.argv) > 1:
         # Some test code...