Sign Up
Log In
Log In
or
Sign Up
Places
All Projects
Status Monitor
Collapse sidebar
SUSE:SLE-15-SP1:Update
python3.33975
CVE-2020-10735-DoS-no-limit-int-size.patch
Overview
Repositories
Revisions
Requests
Users
Attributes
Meta
File CVE-2020-10735-DoS-no-limit-int-size.patch of Package python3.33975
From e091acddc1315ac2533c0698f51c676ee80d003b Mon Sep 17 00:00:00 2001 From: Victor Stinner <vstinner@python.org> Date: Thu, 15 Sep 2022 17:35:24 +0200 Subject: [PATCH] [CVE-2020-10735] prevent DoS by very large int() Unlimited size of integers allows DoS by excessively long processing of large numbers. >> n = 10**(10**7) ; s = str(n) Converting between `int` and `str` in bases other than 2 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now raises a `ValueError` if the number of digits in string form is above a limit to avoid potential denial of service attacks due to the algorithmic complexity. This is a mitigation for CVE-2020-10735 (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735). This new limit can be configured or disabled by environment variable, command line flag, or :mod:`sys` APIs. See the `Integer String Conversion Length Limitation` documentation. The default limit is 4300 digits in string form. Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. Notes on the backport to Python 3.6: * Use "Python 3.6.15-13" version in the documentation, whereas this version will never be released * Only add _Py_global_config_int_max_str_digits global variable: Python 3.6 doesn't have PyConfig API (PEP 597) nor _PyRuntime. * sys.flags.int_max_str_digits cannot be -1 on Python 3.6: it is set to the default limit. Adapt test_int_max_str_digits() for that. * Declare _PY_LONG_DEFAULT_MAX_STR_DIGITS and _PY_LONG_MAX_STR_DIGITS_THRESHOLD macros in longobject.h but only if the Py_BUILD_CORE macro is defined. * Declare _Py_global_config_int_max_str_digits in pydebug.h. (cherry picked from commit 511ca9452033ef95bc7d7fc404b8161068226002) gh#95778: Mention sys.set_int_max_str_digits() in error message (gh#96874) When ValueError is raised if an integer is larger than the limit, mention sys.set_int_max_str_digits() in the error message. (cherry picked from commit e841ffc915e82e5ea6e3b473205417d63494808d) gh-96848: Fix -X int_max_str_digits option parsing (gh#96988) Fix command line parsing: reject "-X int_max_str_digits" option with no value (invalid) when the PYTHONINTMAXSTRDIGITS environment variable is set to a valid limit. (cherry picked from commit 41351662bcd21672d8ccfa62fe44d72027e6bcf8) Originally by Victor Stinner of Red Hat gh#fedora-python/cpython@31cfb692dc5d Fixes: bsc#1203125 Fixes: gh#fedora-python/cpython#96504 Patch: CVE-2020-10735-DoS-no-limit-int-size.patch --- Doc/library/functions.rst | 8 + Doc/library/json.rst | 11 + Doc/library/stdtypes.rst | 159 ++++++++++++++ Doc/library/sys.rst | 53 ++++- Doc/library/test.rst | 10 + Doc/using/cmdline.rst | 14 ++ Doc/whatsnew/3.6.rst | 15 ++ Include/longobject.h | 38 ++++ Include/pydebug.h | 2 + Lib/test/support/__init__.py | 10 + Lib/test/test_ast.py | 8 + Lib/test/test_cmd_line.py | 35 ++++ Lib/test/test_compile.py | 13 ++ Lib/test/test_decimal.py | 18 ++ Lib/test/test_int.py | 196 ++++++++++++++++++ Lib/test/test_json/test_decode.py | 8 + Lib/test/test_sys.py | 11 +- Lib/test/test_xmlrpc.py | 10 + .../2019-05-15-18-28-43.bpo-27987.FaxuLy.rst | 2 + ...2-09-16-19-02-40.gh-issue-95778.cJmnst.rst | 3 + ...08-07-16-53-38.gh-issue-95778.ch010gps.rst | 14 ++ Modules/main.c | 7 + Objects/longobject.c | 168 ++++++++++++++- Python/ast.c | 26 ++- Python/pylifecycle.c | 2 + Python/sysmodule.c | 46 +++- 26 files changed, 872 insertions(+), 15 deletions(-) create mode 100644 Misc/NEWS.d/next/Core and Builtins/2019-05-15-18-28-43.bpo-27987.FaxuLy.rst create mode 100644 Misc/NEWS.d/next/Core and Builtins/2022-09-16-19-02-40.gh-issue-95778.cJmnst.rst create mode 100644 Misc/NEWS.d/next/Security/2022-08-07-16-53-38.gh-issue-95778.ch010gps.rst diff --git a/Doc/library/functions.rst b/Doc/library/functions.rst index bc528dd1711..72eaf17f645 100644 --- a/Doc/library/functions.rst +++ b/Doc/library/functions.rst @@ -748,6 +748,14 @@ are always available. They are listed here in alphabetical order. .. versionchanged:: 3.6 Grouping digits with underscores as in code literals is allowed. + .. versionchanged:: 3.6.15-13 + :class:`int` string inputs and string representations can be limited to + help avoid denial of service attacks. A :exc:`ValueError` is raised when + the limit is exceeded while converting a string *x* to an :class:`int` or + when converting an :class:`int` into a string would exceed the limit. + See the :ref:`integer string conversion length limitation + <int_max_str_digits>` documentation. + .. function:: isinstance(object, classinfo) diff --git a/Doc/library/json.rst b/Doc/library/json.rst index 98ca86e4b9a..51adb4a2ffd 100644 --- a/Doc/library/json.rst +++ b/Doc/library/json.rst @@ -18,6 +18,11 @@ is a lightweight data interchange format inspired by `JavaScript <https://en.wikipedia.org/wiki/JavaScript>`_ object literal syntax (although it is not a strict subset of JavaScript [#rfc-errata]_ ). +.. warning:: + Be cautious when parsing JSON data from untrusted sources. A malicious + JSON string may cause the decoder to consume considerable CPU and memory + resources. Limiting the size of data to be parsed is recommended. + :mod:`json` exposes an API familiar to users of the standard library :mod:`marshal` and :mod:`pickle` modules. @@ -245,6 +250,12 @@ Basic Usage be used to use another datatype or parser for JSON integers (e.g. :class:`float`). + .. versionchanged:: 3.6.15-13 + The default *parse_int* of :func:`int` now limits the maximum length of + the integer string via the interpreter's :ref:`integer string + conversion length limitation <int_max_str_digits>` to help avoid denial + of service attacks. + *parse_constant*, if specified, will be called with one of the following strings: ``'-Infinity'``, ``'Infinity'``, ``'NaN'``. This can be used to raise an exception if invalid JSON numbers diff --git a/Doc/library/stdtypes.rst b/Doc/library/stdtypes.rst index 00e1f4c1bf0..7ae0f58c44a 100644 --- a/Doc/library/stdtypes.rst +++ b/Doc/library/stdtypes.rst @@ -4671,6 +4671,165 @@ types, where they are relevant. Some of these are not reported by the [<class 'bool'>] +.. _int_max_str_digits: + +Integer string conversion length limitation +=========================================== + +CPython has a global limit for converting between :class:`int` and :class:`str` +to mitigate denial of service attacks. This limit *only* applies to decimal or +other non-power-of-two number bases. Hexadecimal, octal, and binary conversions +are unlimited. The limit can be configured. + +The :class:`int` type in CPython is an abitrary length number stored in binary +form (commonly known as a "bignum"). There exists no algorithm that can convert +a string to a binary integer or a binary integer to a string in linear time, +*unless* the base is a power of 2. Even the best known algorithms for base 10 +have sub-quadratic complexity. Converting a large value such as ``int('1' * +500_000)`` can take over a second on a fast CPU. + +Limiting conversion size offers a practical way to avoid `CVE-2020-10735 +<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_. + +The limit is applied to the number of digit characters in the input or output +string when a non-linear conversion algorithm would be involved. Underscores +and the sign are not counted towards the limit. + +When an operation would exceed the limit, a :exc:`ValueError` is raised: + +.. doctest:: + + >>> import sys + >>> sys.set_int_max_str_digits(4300) # Illustrative, this is the default. + >>> _ = int('2' * 5432) + Traceback (most recent call last): + ... + ValueError: Exceeds the limit (4300) for integer string conversion: value has 5432 digits; use sys.set_int_max_str_digits() to increase the limit. + >>> i = int('2' * 4300) + >>> len(str(i)) + 4300 + >>> i_squared = i*i + >>> len(str(i_squared)) + Traceback (most recent call last): + ... + ValueError: Exceeds the limit (4300) for integer string conversion: value has 8599 digits; use sys.set_int_max_str_digits() to increase the limit. + >>> len(hex(i_squared)) + 7144 + >>> assert int(hex(i_squared), base=16) == i*i # Hexadecimal is unlimited. + +The default limit is 4300 digits as provided in +:data:`sys.int_info.default_max_str_digits <sys.int_info>`. +The lowest limit that can be configured is 640 digits as provided in +:data:`sys.int_info.str_digits_check_threshold <sys.int_info>`. + +Verification: + +.. doctest:: + + >>> import sys + >>> assert sys.int_info.default_max_str_digits == 4300, sys.int_info + >>> assert sys.int_info.str_digits_check_threshold == 640, sys.int_info + >>> msg = int('578966293710682886880994035146873798396722250538762761564' + ... '9252925514383915483333812743580549779436104706260696366600' + ... '571186405732').to_bytes(53, 'big') + ... + +.. versionadded:: 3.6.15-13 + +Affected APIs +------------- + +The limitation only applies to potentially slow conversions between :class:`int` +and :class:`str` or :class:`bytes`: + +* ``int(string)`` with default base 10. +* ``int(string, base)`` for all bases that are not a power of 2. +* ``str(integer)``. +* ``repr(integer)`` +* any other string conversion to base 10, for example ``f"{integer}"``, + ``"{}".format(integer)``, or ``b"%d" % integer``. + +The limitations do not apply to functions with a linear algorithm: + +* ``int(string, base)`` with base 2, 4, 8, 16, or 32. +* :func:`int.from_bytes` and :func:`int.to_bytes`. +* :func:`hex`, :func:`oct`, :func:`bin`. +* :ref:`formatspec` for hex, octal, and binary numbers. +* :class:`str` to :class:`float`. +* :class:`str` to :class:`decimal.Decimal`. + +Configuring the limit +--------------------- + +Before Python starts up you can use an environment variable or an interpreter +command line flag to configure the limit: + +* :envvar:`PYTHONINTMAXSTRDIGITS`, e.g. + ``PYTHONINTMAXSTRDIGITS=640 python3`` to set the limit to 640 or + ``PYTHONINTMAXSTRDIGITS=0 python3`` to disable the limitation. +* :option:`-X int_max_str_digits <-X>`, e.g. + ``python3 -X int_max_str_digits=640`` +* :data:`sys.flags.int_max_str_digits` contains the value of + :envvar:`PYTHONINTMAXSTRDIGITS` or :option:`-X int_max_str_digits <-X>`. + If both the env var and the ``-X`` option are set, the ``-X`` option takes + precedence. A value of *-1* indicates that both were unset, thus a value of + :data:`sys.int_info.default_max_str_digits` was used during initilization. + +From code, you can inspect the current limit and set a new one using these +:mod:`sys` APIs: + +* :func:`sys.get_int_max_str_digits` and :func:`sys.set_int_max_str_digits` are + a getter and setter for the interpreter-wide limit. Subinterpreters have + their own limit. + +Information about the default and minimum can be found in :attr:`sys.int_info`: + +* :data:`sys.int_info.default_max_str_digits <sys.int_info>` is the compiled-in + default limit. +* :data:`sys.int_info.str_digits_check_threshold <sys.int_info>` is the lowest + accepted value for the limit (other than 0 which disables it). + +.. versionadded:: 3.6.15-13 + +.. caution:: + + Setting a low limit *can* lead to problems. While rare, code exists that + contains integer constants in decimal in their source that exceed the + minimum threshold. A consequence of setting the limit is that Python source + code containing decimal integer literals longer than the limit will + encounter an error during parsing, usually at startup time or import time or + even at installation time - anytime an up to date ``.pyc`` does not already + exist for the code. A workaround for source that contains such large + constants is to convert them to ``0x`` hexadecimal form as it has no limit. + + Test your application thoroughly if you use a low limit. Ensure your tests + run with the limit set early via the environment or flag so that it applies + during startup and even during any installation step that may invoke Python + to precompile ``.py`` sources to ``.pyc`` files. + +Recommended configuration +------------------------- + +The default :data:`sys.int_info.default_max_str_digits` is expected to be +reasonable for most applications. If your application requires a different +limit, set it from your main entry point using Python version agnostic code as +these APIs were added in security patch releases in versions before 3.11. + +Example:: + + >>> import sys + >>> if hasattr(sys, "set_int_max_str_digits"): + ... upper_bound = 68000 + ... lower_bound = 4004 + ... current_limit = sys.get_int_max_str_digits() + ... if current_limit == 0 or current_limit > upper_bound: + ... sys.set_int_max_str_digits(upper_bound) + ... elif current_limit < lower_bound: + ... sys.set_int_max_str_digits(lower_bound) + +If you need to disable it entirely, set it to ``0``. + + .. rubric:: Footnotes .. [1] Additional information on these special methods may be found in the Python diff --git a/Doc/library/sys.rst b/Doc/library/sys.rst index 91984667bf2..f1e6f3bcf08 100644 --- a/Doc/library/sys.rst +++ b/Doc/library/sys.rst @@ -296,6 +296,7 @@ always available. :const:`bytes_warning` :option:`-b` :const:`quiet` :option:`-q` :const:`hash_randomization` :option:`-R` + :const:`int_max_str_digits` :option:`-X int_max_str_digits <-X>` (:ref:`integer string conversion length limitation <int_max_str_digits>`) ============================= ============================= .. versionchanged:: 3.2 @@ -310,6 +311,9 @@ always available. .. versionchanged:: 3.4 Added ``isolated`` attribute for :option:`-I` ``isolated`` flag. + .. versionchanged:: 3.6.15-13 + Added the ``int_max_str_digits`` attribute. + .. data:: float_info A :term:`struct sequence` holding information about the float type. It @@ -468,6 +472,15 @@ always available. .. versionadded:: 3.6 + +.. function:: get_int_max_str_digits() + + Returns the current value for the :ref:`integer string conversion length + limitation <int_max_str_digits>`. See also :func:`set_int_max_str_digits`. + + .. versionadded:: 3.6.15-13 + + .. function:: getrefcount(object) Return the reference count of the *object*. The count returned is generally one @@ -730,19 +743,31 @@ always available. .. tabularcolumns:: |l|L| - +-------------------------+----------------------------------------------+ - | Attribute | Explanation | - +=========================+==============================================+ - | :const:`bits_per_digit` | number of bits held in each digit. Python | - | | integers are stored internally in base | - | | ``2**int_info.bits_per_digit`` | - +-------------------------+----------------------------------------------+ - | :const:`sizeof_digit` | size in bytes of the C type used to | - | | represent a digit | - +-------------------------+----------------------------------------------+ + +----------------------------------------+-----------------------------------------------+ + | Attribute | Explanation | + +========================================+===============================================+ + | :const:`bits_per_digit` | number of bits held in each digit. Python | + | | integers are stored internally in base | + | | ``2**int_info.bits_per_digit`` | + +----------------------------------------+-----------------------------------------------+ + | :const:`sizeof_digit` | size in bytes of the C type used to | + | | represent a digit | + +----------------------------------------+-----------------------------------------------+ + | :const:`default_max_str_digits` | default value for | + | | :func:`sys.get_int_max_str_digits` when it | + | | is not otherwise explicitly configured. | + +----------------------------------------+-----------------------------------------------+ + | :const:`str_digits_check_threshold` | minimum non-zero value for | + | | :func:`sys.set_int_max_str_digits`, | + | | :envvar:`PYTHONINTMAXSTRDIGITS`, or | + | | :option:`-X int_max_str_digits <-X>`. | + +----------------------------------------+-----------------------------------------------+ .. versionadded:: 3.1 + .. versionchanged:: 3.6.15-13 + Added ``default_max_str_digits`` and ``str_digits_check_threshold``. + .. data:: __interactivehook__ @@ -1001,6 +1026,14 @@ always available. Availability: Unix. +.. function:: set_int_max_str_digits(n) + + Set the :ref:`integer string conversion length limitation + <int_max_str_digits>` used by this interpreter. See also + :func:`get_int_max_str_digits`. + + .. versionadded:: 3.6.15-13 + .. function:: setprofile(profilefunc) .. index:: diff --git a/Doc/library/test.rst b/Doc/library/test.rst index 04d6cd87eac..a15e7768bbd 100644 --- a/Doc/library/test.rst +++ b/Doc/library/test.rst @@ -625,6 +625,16 @@ The :mod:`test.support` module defines the following functions: .. versionadded:: 3.6 +.. function:: adjust_int_max_str_digits(max_digits) + + This function returns a context manager that will change the global + :func:`sys.set_int_max_str_digits` setting for the duration of the + context to allow execution of test code that needs a different limit + on the number of digits when converting between an integer and string. + + .. versionadded:: 3.6.15-13 + + The :mod:`test.support` module defines the following classes: .. class:: TransientResource(exc, **kwargs) diff --git a/Doc/using/cmdline.rst b/Doc/using/cmdline.rst index 65aa3ad0f52..fe7a926ba15 100644 --- a/Doc/using/cmdline.rst +++ b/Doc/using/cmdline.rst @@ -422,6 +422,9 @@ Miscellaneous options * ``-X showalloccount`` to output the total count of allocated objects for each type when the program finishes. This only works when Python was built with ``COUNT_ALLOCS`` defined. + * ``-X int_max_str_digits`` configures the :ref:`integer string conversion + length limitation <int_max_str_digits>`. See also + :envvar:`PYTHONINTMAXSTRDIGITS`. It also allows passing arbitrary values and retrieving them through the :data:`sys._xoptions` dictionary. @@ -438,6 +441,9 @@ Miscellaneous options .. versionadded:: 3.6 The ``-X showalloccount`` option. + .. versionadded:: 3.6.15-13 + The ``-X int_max_str_digits`` option. + Options you shouldn't use ~~~~~~~~~~~~~~~~~~~~~~~~~ @@ -571,6 +577,14 @@ conflict. .. versionadded:: 3.2.3 +.. envvar:: PYTHONINTMAXSTRDIGITS + + If this variable is set to an integer, it is used to configure the + interpreter's global :ref:`integer string conversion length limitation + <int_max_str_digits>`. + + .. versionadded:: 3.6.15-13 + .. envvar:: PYTHONIOENCODING If this is set before running the interpreter, it overrides the encoding used diff --git a/Doc/whatsnew/3.6.rst b/Doc/whatsnew/3.6.rst index 5f8f478eb37..19811f4a819 100644 --- a/Doc/whatsnew/3.6.rst +++ b/Doc/whatsnew/3.6.rst @@ -2488,3 +2488,18 @@ ASCII newline ``\n``, ``\r`` and tab ``\t`` characters are stripped from the URL by the parser :func:`urllib.parse` preventing such attacks. The removal characters are controlled by a new module level variable ``urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE``. (See :issue:`43882`) + + +Notable security feature in 3.6.15-13 +===================================== + +Converting between :class:`int` and :class:`str` in bases other than 2 +(binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) +now raises a :exc:`ValueError` if the number of digits in string form is +above a limit to avoid potential denial of service attacks due to the +algorithmic complexity. This is a mitigation for `CVE-2020-10735 +<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_. +This limit can be configured or disabled by environment variable, command +line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion +length limitation <int_max_str_digits>` documentation. The default limit +is 4300 digits in string form. diff --git a/Include/longobject.h b/Include/longobject.h index efd409c75a8..d8a080a3bc2 100644 --- a/Include/longobject.h +++ b/Include/longobject.h @@ -209,6 +209,44 @@ PyAPI_FUNC(long) PyOS_strtol(const char *, char **, int); PyAPI_FUNC(PyObject *) _PyLong_GCD(PyObject *, PyObject *); #endif /* !Py_LIMITED_API */ +#ifdef Py_BUILD_CORE +/* + * Default int base conversion size limitation: Denial of Service prevention. + * + * Chosen such that this isn't wildly slow on modern hardware and so that + * everyone's existing deployed numpy test suite passes before + * https://github.com/numpy/numpy/issues/22098 is widely available. + * + * $ python -m timeit -s 's = "1"*4300' 'int(s)' + * 2000 loops, best of 5: 125 usec per loop + * $ python -m timeit -s 's = "1"*4300; v = int(s)' 'str(v)' + * 1000 loops, best of 5: 311 usec per loop + * (zen2 cloud VM) + * + * 4300 decimal digits fits a ~14284 bit number. + */ +#define _PY_LONG_DEFAULT_MAX_STR_DIGITS 4300 +/* + * Threshold for max digits check. For performance reasons int() and + * int.__str__() don't checks values that are smaller than this + * threshold. Acts as a guaranteed minimum size limit for bignums that + * applications can expect from CPython. + * + * % python -m timeit -s 's = "1"*640; v = int(s)' 'str(int(s))' + * 20000 loops, best of 5: 12 usec per loop + * + * "640 digits should be enough for anyone." - gps + * fits a ~2126 bit decimal number. + */ +#define _PY_LONG_MAX_STR_DIGITS_THRESHOLD 640 + +#if ((_PY_LONG_DEFAULT_MAX_STR_DIGITS != 0) && \ + (_PY_LONG_DEFAULT_MAX_STR_DIGITS < _PY_LONG_MAX_STR_DIGITS_THRESHOLD)) +# error "_PY_LONG_DEFAULT_MAX_STR_DIGITS smaller than threshold." +#endif + +#endif /* Py_BUILD_CORE */ + #ifdef __cplusplus } #endif diff --git a/Include/pydebug.h b/Include/pydebug.h index 6e23a896c3d..657c55266a2 100644 --- a/Include/pydebug.h +++ b/Include/pydebug.h @@ -28,6 +28,8 @@ PyAPI_DATA(int) Py_IsolatedFlag; PyAPI_DATA(int) Py_LegacyWindowsStdioFlag; #endif +PyAPI_DATA(int) _Py_global_config_int_max_str_digits; + /* this is a wrapper around getenv() that pays attention to Py_IgnoreEnvironmentFlag. It should be used for getting variables like PYTHONPATH and PYTHONHOME from the environment */ diff --git a/Lib/test/support/__init__.py b/Lib/test/support/__init__.py index 66c0fed8411..9ca9ae79e68 100644 --- a/Lib/test/support/__init__.py +++ b/Lib/test/support/__init__.py @@ -2868,3 +2868,13 @@ class FakePath: raise self.path else: return self.path + +@contextlib.contextmanager +def adjust_int_max_str_digits(max_digits): + """Temporarily change the integer string conversion length limit.""" + current = sys.get_int_max_str_digits() + try: + sys.set_int_max_str_digits(max_digits) + yield + finally: + sys.set_int_max_str_digits(current) diff --git a/Lib/test/test_ast.py b/Lib/test/test_ast.py index e68d0de2f8b..7f2d7936343 100644 --- a/Lib/test/test_ast.py +++ b/Lib/test/test_ast.py @@ -571,6 +571,14 @@ class ASTHelpers_Test(unittest.TestCase): exec(code, ns) self.assertIn('sleep', ns) + def test_literal_eval_str_int_limit(self): + with support.adjust_int_max_str_digits(4000): + ast.literal_eval('3'*4000) # no error + with self.assertRaises(SyntaxError) as err_ctx: + ast.literal_eval('3'*4001) + self.assertIn('Exceeds the limit ', str(err_ctx.exception)) + self.assertIn(' Consider hexadecimal ', str(err_ctx.exception)) + class ASTValidatorTests(unittest.TestCase): diff --git a/Lib/test/test_cmd_line.py b/Lib/test/test_cmd_line.py index 5922ed917c9..fe456eece89 100644 --- a/Lib/test/test_cmd_line.py +++ b/Lib/test/test_cmd_line.py @@ -502,6 +502,41 @@ class CmdLineTest(unittest.TestCase): self.assertEqual(proc.returncode, 0, proc) self.assertEqual(proc.stdout.strip(), b'0') + def test_int_max_str_digits(self): + code = "import sys; print(sys.flags.int_max_str_digits, sys.get_int_max_str_digits())" + + assert_python_failure('-X', 'int_max_str_digits', '-c', code) + assert_python_failure('-X', 'int_max_str_digits=foo', '-c', code) + assert_python_failure('-X', 'int_max_str_digits=100', '-c', code) + assert_python_failure('-X', 'int_max_str_digits', '-c', code, + PYTHONINTMAXSTRDIGITS='4000') + + assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='foo') + assert_python_failure('-c', code, PYTHONINTMAXSTRDIGITS='100') + + def res2int(res): + out = res.out.strip().decode("utf-8") + return tuple(int(i) for i in out.split()) + + res = assert_python_ok('-c', code) + self.assertEqual(res2int(res), (sys.get_int_max_str_digits(), sys.get_int_max_str_digits())) + res = assert_python_ok('-X', 'int_max_str_digits=0', '-c', code) + self.assertEqual(res2int(res), (0, 0)) + res = assert_python_ok('-X', 'int_max_str_digits=4000', '-c', code) + self.assertEqual(res2int(res), (4000, 4000)) + res = assert_python_ok('-X', 'int_max_str_digits=100000', '-c', code) + self.assertEqual(res2int(res), (100000, 100000)) + + res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='0') + self.assertEqual(res2int(res), (0, 0)) + res = assert_python_ok('-c', code, PYTHONINTMAXSTRDIGITS='4000') + self.assertEqual(res2int(res), (4000, 4000)) + res = assert_python_ok( + '-X', 'int_max_str_digits=6000', '-c', code, + PYTHONINTMAXSTRDIGITS='4000' + ) + self.assertEqual(res2int(res), (6000, 6000)) + def test_main(): test.support.run_unittest(CmdLineTest) diff --git a/Lib/test/test_compile.py b/Lib/test/test_compile.py index 13cc8821b08..00e64681413 100644 --- a/Lib/test/test_compile.py +++ b/Lib/test/test_compile.py @@ -189,6 +189,19 @@ if 1: self.assertEqual(eval("0o777"), 511) self.assertEqual(eval("-0o0000010"), -8) + def test_int_literals_too_long(self): + n = 3000 + source = f"a = 1\nb = 2\nc = {'3'*n}\nd = 4" + with support.adjust_int_max_str_digits(n): + compile(source, "<long_int_pass>", "exec") # no errors. + with support.adjust_int_max_str_digits(n-1): + with self.assertRaises(SyntaxError) as err_ctx: + compile(source, "<long_int_fail>", "exec") + exc = err_ctx.exception + self.assertEqual(exc.lineno, 3) + self.assertIn('Exceeds the limit ', str(exc)) + self.assertIn(' Consider hexadecimal ', str(exc)) + def test_unary_minus(self): # Verify treatment of unary minus on negative numbers SF bug #660455 if sys.maxsize == 2147483647: diff --git a/Lib/test/test_decimal.py b/Lib/test/test_decimal.py index 8808a670f93..4459b82cffd 100644 --- a/Lib/test/test_decimal.py +++ b/Lib/test/test_decimal.py @@ -2442,6 +2442,15 @@ class CUsabilityTest(UsabilityTest): class PyUsabilityTest(UsabilityTest): decimal = P + def setUp(self): + super().setUp() + self._previous_int_limit = sys.get_int_max_str_digits() + sys.set_int_max_str_digits(7000) + + def tearDown(self): + sys.set_int_max_str_digits(self._previous_int_limit) + super().tearDown() + class PythonAPItests(unittest.TestCase): def test_abc(self): @@ -4499,6 +4508,15 @@ class CCoverage(Coverage): class PyCoverage(Coverage): decimal = P + def setUp(self): + super().setUp() + self._previous_int_limit = sys.get_int_max_str_digits() + sys.set_int_max_str_digits(7000) + + def tearDown(self): + sys.set_int_max_str_digits(self._previous_int_limit) + super().tearDown() + class PyFunctionality(unittest.TestCase): """Extra functionality in decimal.py""" diff --git a/Lib/test/test_int.py b/Lib/test/test_int.py index a36076e0104..6cd9a5ed893 100644 --- a/Lib/test/test_int.py +++ b/Lib/test/test_int.py @@ -1,4 +1,5 @@ import sys +import time import unittest from test import support @@ -514,5 +515,200 @@ class IntTestCases(unittest.TestCase): self.assertEqual(int('1_2_3_4_5_6_7', 32), 1144132807) +class IntStrDigitLimitsTests(unittest.TestCase): + + int_class = int # Override this in subclasses to reuse the suite. + + def setUp(self): + super().setUp() + self._previous_limit = sys.get_int_max_str_digits() + sys.set_int_max_str_digits(2048) + + def tearDown(self): + sys.set_int_max_str_digits(self._previous_limit) + super().tearDown() + + def test_disabled_limit(self): + self.assertGreater(sys.get_int_max_str_digits(), 0) + self.assertLess(sys.get_int_max_str_digits(), 20_000) + with support.adjust_int_max_str_digits(0): + self.assertEqual(sys.get_int_max_str_digits(), 0) + i = self.int_class('1' * 20_000) + str(i) + self.assertGreater(sys.get_int_max_str_digits(), 0) + + def test_max_str_digits_edge_cases(self): + """Ignore the +/- sign and space padding.""" + int_class = self.int_class + maxdigits = sys.get_int_max_str_digits() + + int_class('1' * maxdigits) + int_class(' ' + '1' * maxdigits) + int_class('1' * maxdigits + ' ') + int_class('+' + '1' * maxdigits) + int_class('-' + '1' * maxdigits) + self.assertEqual(len(str(10 ** (maxdigits - 1))), maxdigits) + + def check(self, i, base=None): + with self.assertRaises(ValueError): + if base is None: + self.int_class(i) + else: + self.int_class(i, base) + + def test_max_str_digits(self): + maxdigits = sys.get_int_max_str_digits() + + self.check('1' * (maxdigits + 1)) + self.check(' ' + '1' * (maxdigits + 1)) + self.check('1' * (maxdigits + 1) + ' ') + self.check('+' + '1' * (maxdigits + 1)) + self.check('-' + '1' * (maxdigits + 1)) + self.check('1' * (maxdigits + 1)) + + i = 10 ** maxdigits + with self.assertRaises(ValueError): + str(i) + + def test_denial_of_service_prevented_int_to_str(self): + """Regression test: ensure we fail before performing O(N**2) work.""" + maxdigits = sys.get_int_max_str_digits() + assert maxdigits < 50_000, maxdigits # A test prerequisite. + get_time = time.process_time + if get_time() <= 0: # some platforms like WASM lack process_time() + get_time = time.monotonic + + huge_int = int(f'0x{"c"*65_000}', base=16) # 78268 decimal digits. + digits = 78_268 + with support.adjust_int_max_str_digits(digits): + start = get_time() + huge_decimal = str(huge_int) + seconds_to_convert = get_time() - start + self.assertEqual(len(huge_decimal), digits) + # Ensuring that we chose a slow enough conversion to measure. + # It takes 0.1 seconds on a Zen based cloud VM in an opt build. + if seconds_to_convert < 0.005: + raise unittest.SkipTest('"slow" conversion took only ' + f'{seconds_to_convert} seconds.') + + # We test with the limit almost at the size needed to check performance. + # The performant limit check is slightly fuzzy, give it a some room. + with support.adjust_int_max_str_digits(int(.995 * digits)): + with self.assertRaises(ValueError) as err: + start = get_time() + str(huge_int) + seconds_to_fail_huge = get_time() - start + self.assertIn('conversion', str(err.exception)) + self.assertLess(seconds_to_fail_huge, seconds_to_convert/8) + + # Now we test that a conversion that would take 30x as long also fails + # in a similarly fast fashion. + extra_huge_int = int(f'0x{"c"*500_000}', base=16) # 602060 digits. + with self.assertRaises(ValueError) as err: + start = get_time() + # If not limited, 8 seconds said Zen based cloud VM. + str(extra_huge_int) + seconds_to_fail_extra_huge = get_time() - start + self.assertIn('conversion', str(err.exception)) + self.assertLess(seconds_to_fail_extra_huge, seconds_to_convert/8) + + def test_denial_of_service_prevented_str_to_int(self): + """Regression test: ensure we fail before performing O(N**2) work.""" + maxdigits = sys.get_int_max_str_digits() + assert maxdigits < 100_000, maxdigits # A test prerequisite. + get_time = time.process_time + if get_time() <= 0: # some platforms like WASM lack process_time() + get_time = time.monotonic + + digits = 133700 + huge = '8'*digits + with support.adjust_int_max_str_digits(digits): + start = get_time() + int(huge) + seconds_to_convert = get_time() - start + # Ensuring that we chose a slow enough conversion to measure. + # It takes 0.1 seconds on a Zen based cloud VM in an opt build. + if seconds_to_convert < 0.005: + raise unittest.SkipTest('"slow" conversion took only ' + f'{seconds_to_convert} seconds.') + + with support.adjust_int_max_str_digits(digits - 1): + with self.assertRaises(ValueError) as err: + start = get_time() + int(huge) + seconds_to_fail_huge = get_time() - start + self.assertIn('conversion', str(err.exception)) + self.assertLess(seconds_to_fail_huge, seconds_to_convert/8) + + # Now we test that a conversion that would take 30x as long also fails + # in a similarly fast fashion. + extra_huge = '7'*1_200_000 + with self.assertRaises(ValueError) as err: + start = get_time() + # If not limited, 8 seconds in the Zen based cloud VM. + int(extra_huge) + seconds_to_fail_extra_huge = get_time() - start + self.assertIn('conversion', str(err.exception)) + self.assertLess(seconds_to_fail_extra_huge, seconds_to_convert/8) + + def test_power_of_two_bases_unlimited(self): + """The limit does not apply to power of 2 bases.""" + maxdigits = sys.get_int_max_str_digits() + + for base in (2, 4, 8, 16, 32): + with self.subTest(base=base): + self.int_class('1' * (maxdigits + 1), base) + assert maxdigits < 100_000 + self.int_class('1' * 100_000, base) + + def test_underscores_ignored(self): + maxdigits = sys.get_int_max_str_digits() + + triples = maxdigits // 3 + s = '111' * triples + s_ = '1_11' * triples + self.int_class(s) # succeeds + self.int_class(s_) # succeeds + self.check(f'{s}111') + self.check(f'{s_}_111') + + def test_sign_not_counted(self): + int_class = self.int_class + max_digits = sys.get_int_max_str_digits() + s = '5' * max_digits + i = int_class(s) + pos_i = int_class(f'+{s}') + assert i == pos_i + neg_i = int_class(f'-{s}') + assert -pos_i == neg_i + str(pos_i) + str(neg_i) + + def _other_base_helper(self, base): + int_class = self.int_class + max_digits = sys.get_int_max_str_digits() + s = '2' * max_digits + i = int_class(s, base) + if base > 10: + with self.assertRaises(ValueError): + str(i) + elif base < 10: + str(i) + with self.assertRaises(ValueError) as err: + int_class(f'{s}1', base) + + def test_int_from_other_bases(self): + base = 3 + with self.subTest(base=base): + self._other_base_helper(base) + base = 36 + with self.subTest(base=base): + self._other_base_helper(base) + + +class IntSubclassStrDigitLimitsTests(IntStrDigitLimitsTests): + int_class = IntSubclass + + if __name__ == "__main__": unittest.main() diff --git a/Lib/test/test_json/test_decode.py b/Lib/test/test_json/test_decode.py index 738f109fe68..b5fd4f22914 100644 --- a/Lib/test/test_json/test_decode.py +++ b/Lib/test/test_json/test_decode.py @@ -2,6 +2,7 @@ import decimal from io import StringIO, BytesIO from collections import OrderedDict from test.test_json import PyTest, CTest +from test import support class TestDecode: @@ -95,5 +96,12 @@ class TestDecode: d = self.json.JSONDecoder() self.assertRaises(ValueError, d.raw_decode, 'a'*42, -50000) + def test_limit_int(self): + maxdigits = 5000 + with support.adjust_int_max_str_digits(maxdigits): + self.loads('1' * maxdigits) + with self.assertRaises(ValueError): + self.loads('1' * (maxdigits + 1)) + class TestPyDecode(TestDecode, PyTest): pass class TestCDecode(TestDecode, CTest): pass diff --git a/Lib/test/test_sys.py b/Lib/test/test_sys.py index b41239ab7a2..f2b9e08017b 100644 --- a/Lib/test/test_sys.py +++ b/Lib/test/test_sys.py @@ -444,11 +444,17 @@ class SysModuleTest(unittest.TestCase): self.assertIsInstance(sys.executable, str) self.assertEqual(len(sys.float_info), 11) self.assertEqual(sys.float_info.radix, 2) - self.assertEqual(len(sys.int_info), 2) + self.assertEqual(len(sys.int_info), 4) self.assertTrue(sys.int_info.bits_per_digit % 5 == 0) self.assertTrue(sys.int_info.sizeof_digit >= 1) + self.assertGreaterEqual(sys.int_info.default_max_str_digits, 500) + self.assertGreaterEqual(sys.int_info.str_digits_check_threshold, 100) + self.assertGreater(sys.int_info.default_max_str_digits, + sys.int_info.str_digits_check_threshold) self.assertEqual(type(sys.int_info.bits_per_digit), int) self.assertEqual(type(sys.int_info.sizeof_digit), int) + self.assertIsInstance(sys.int_info.default_max_str_digits, int) + self.assertIsInstance(sys.int_info.str_digits_check_threshold, int) self.assertIsInstance(sys.hexversion, int) self.assertEqual(len(sys.hash_info), 9) @@ -552,7 +558,8 @@ class SysModuleTest(unittest.TestCase): attrs = ("debug", "inspect", "interactive", "optimize", "dont_write_bytecode", "no_user_site", "no_site", "ignore_environment", "verbose", - "bytes_warning", "quiet", "hash_randomization", "isolated") + "bytes_warning", "quiet", "hash_randomization", "isolated", + "int_max_str_digits") for attr in attrs: self.assertTrue(hasattr(sys.flags, attr), attr) self.assertEqual(type(getattr(sys.flags, attr)), int, attr) diff --git a/Lib/test/test_xmlrpc.py b/Lib/test/test_xmlrpc.py index fc601d45522..58553dd91a1 100644 --- a/Lib/test/test_xmlrpc.py +++ b/Lib/test/test_xmlrpc.py @@ -286,6 +286,16 @@ class XMLRPCTestCase(unittest.TestCase): check('<bigdecimal>9876543210.0123456789</bigdecimal>', decimal.Decimal('9876543210.0123456789')) + def test_limit_int(self): + check = self.check_loads + maxdigits = 5000 + with support.adjust_int_max_str_digits(maxdigits): + s = '1' * (maxdigits + 1) + with self.assertRaises(ValueError): + check(f'<int>{s}</int>', None) + with self.assertRaises(ValueError): + check(f'<biginteger>{s}</biginteger>', None) + def test_get_host_info(self): # see bug #3613, this raised a TypeError transp = xmlrpc.client.Transport() diff --git a/Misc/NEWS.d/next/Core and Builtins/2019-05-15-18-28-43.bpo-27987.FaxuLy.rst b/Misc/NEWS.d/next/Core and Builtins/2019-05-15-18-28-43.bpo-27987.FaxuLy.rst new file mode 100644 index 00000000000..98073471ca7 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2019-05-15-18-28-43.bpo-27987.FaxuLy.rst @@ -0,0 +1,2 @@ +``PyGC_Head`` structure is aligned to ``long double``. This is needed to +ensure GC-ed objects are aligned properly. Patch by Inada Naoki. diff --git a/Misc/NEWS.d/next/Core and Builtins/2022-09-16-19-02-40.gh-issue-95778.cJmnst.rst b/Misc/NEWS.d/next/Core and Builtins/2022-09-16-19-02-40.gh-issue-95778.cJmnst.rst new file mode 100644 index 00000000000..ebf63778a60 --- /dev/null +++ b/Misc/NEWS.d/next/Core and Builtins/2022-09-16-19-02-40.gh-issue-95778.cJmnst.rst @@ -0,0 +1,3 @@ +When :exc:`ValueError` is raised if an integer is larger than the limit, +mention the :func:`sys.set_int_max_str_digits` function in the error message. +Patch by Victor Stinner. diff --git a/Misc/NEWS.d/next/Security/2022-08-07-16-53-38.gh-issue-95778.ch010gps.rst b/Misc/NEWS.d/next/Security/2022-08-07-16-53-38.gh-issue-95778.ch010gps.rst new file mode 100644 index 00000000000..8eb8a34884d --- /dev/null +++ b/Misc/NEWS.d/next/Security/2022-08-07-16-53-38.gh-issue-95778.ch010gps.rst @@ -0,0 +1,14 @@ +Converting between :class:`int` and :class:`str` in bases other than 2 +(binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) now +raises a :exc:`ValueError` if the number of digits in string form is above a +limit to avoid potential denial of service attacks due to the algorithmic +complexity. This is a mitigation for `CVE-2020-10735 +<https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2020-10735>`_. + +This new limit can be configured or disabled by environment variable, command +line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion length +limitation <int_max_str_digits>` documentation. The default limit is 4300 +digits in string form. + +Patch by Gregory P. Smith [Google] and Christian Heimes [Red Hat] with feedback +from Victor Stinner, Thomas Wouters, Steve Dower, Ned Deily, and Mark Dickinson. diff --git a/Modules/main.c b/Modules/main.c index 96d8be46850..405e8838c79 100644 --- a/Modules/main.c +++ b/Modules/main.c @@ -84,6 +84,9 @@ static const char usage_3[] = "\ also PYTHONWARNINGS=arg\n\ -x : skip first line of source, allowing use of non-Unix forms of #!cmd\n\ -X opt : set implementation-specific option\n\ +-X int_max_str_digits=number: limit the size of int<->str conversions.\n\ + This helps avoid denial of service attacks when parsing untrusted data.\n\ + The default is sys.int_info.default_max_str_digits. 0 disables.\n\ "; static const char usage_4[] = "\ file : program read from script file\n\ @@ -105,6 +108,10 @@ static const char usage_6[] = " to seed the hashes of str, bytes and datetime objects. It can also be\n" " set to an integer in the range [0,4294967295] to get hash values with a\n" " predictable seed.\n" +"PYTHONINTMAXSTRDIGITS: limits the maximum digit characters in an int value\n" +" when converting from a string and when converting an int back to a str.\n" +" A value of 0 disables the limit. Conversions to or from bases 2, 4, 8,\n" +" 16, and 32 are never limited.\n" "PYTHONMALLOC: set the Python memory allocators and/or install debug hooks\n" " on Python memory allocators. Use PYTHONMALLOC=debug to install debug\n" " hooks.\n" diff --git a/Objects/longobject.c b/Objects/longobject.c index 3864cec97a0..ea0116908f5 100644 --- a/Objects/longobject.c +++ b/Objects/longobject.c @@ -33,6 +33,9 @@ static PyLongObject small_ints[NSMALLNEGINTS + NSMALLPOSINTS]; Py_ssize_t quick_int_allocs, quick_neg_int_allocs; #endif +#define _MAX_STR_DIGITS_ERROR_FMT_TO_INT "Exceeds the limit (%d) for integer string conversion: value has %zd digits; use sys.set_int_max_str_digits() to increase the limit" +#define _MAX_STR_DIGITS_ERROR_FMT_TO_STR "Exceeds the limit (%d) for integer string conversion; use sys.set_int_max_str_digits() to increase the limit" + static PyObject * get_small_int(sdigit ival) { @@ -1593,6 +1596,22 @@ long_to_decimal_string_internal(PyObject *aa, size_a = Py_ABS(Py_SIZE(a)); negative = Py_SIZE(a) < 0; + /* quick and dirty pre-check for overflowing the decimal digit limit, + based on the inequality 10/3 >= log2(10) + + explanation in https://github.com/python/cpython/pull/96537 + */ + if (size_a >= 10 * _PY_LONG_MAX_STR_DIGITS_THRESHOLD + / (3 * PyLong_SHIFT) + 2) { + int max_str_digits = _Py_global_config_int_max_str_digits ; + if ((max_str_digits > 0) && + (max_str_digits / (3 * PyLong_SHIFT) <= (size_a - 11) / 10)) { + PyErr_Format(PyExc_ValueError, _MAX_STR_DIGITS_ERROR_FMT_TO_STR, + max_str_digits); + return -1; + } + } + /* quick and dirty upper bound for the number of digits required to express a in base _PyLong_DECIMAL_BASE: @@ -1652,6 +1671,16 @@ long_to_decimal_string_internal(PyObject *aa, tenpow *= 10; strlen++; } + if (strlen > _PY_LONG_MAX_STR_DIGITS_THRESHOLD) { + int max_str_digits = _Py_global_config_int_max_str_digits ; + Py_ssize_t strlen_nosign = strlen - negative; + if ((max_str_digits > 0) && (strlen_nosign > max_str_digits)) { + Py_DECREF(scratch); + PyErr_Format(PyExc_ValueError, _MAX_STR_DIGITS_ERROR_FMT_TO_STR, + max_str_digits); + return -1; + } + } if (writer) { if (_PyUnicodeWriter_Prepare(writer, strlen, '9') == -1) { Py_DECREF(scratch); @@ -2166,6 +2195,7 @@ PyLong_FromString(const char *str, char **pend, int base) start = str; if ((base & (base - 1)) == 0) { + /* binary bases are not limited by int_max_str_digits */ int res = long_from_binary_base(&str, base, &z); if (res < 0) { /* Syntax error. */ @@ -2318,6 +2348,16 @@ digit beyond the first. goto onError; } + /* Limit the size to avoid excessive computation attacks. */ + if (digits > _PY_LONG_MAX_STR_DIGITS_THRESHOLD) { + int max_str_digits = _Py_global_config_int_max_str_digits ; + if ((max_str_digits > 0) && (digits > max_str_digits)) { + PyErr_Format(PyExc_ValueError, _MAX_STR_DIGITS_ERROR_FMT_TO_INT, + max_str_digits, digits); + return NULL; + } + } + /* Create an int object that can contain the largest possible * integer with this base and length. Note that there's no * need to initialize z->ob_digit -- no slot is read up before @@ -4820,6 +4860,7 @@ long_new(PyTypeObject *type, PyObject *args, PyObject *kwds) } return PyLong_FromLong(0L); } + /* default base and limit, forward to standard implementation */ if (obase == NULL) return PyNumber_Long(x); @@ -5482,6 +5523,8 @@ internal representation of integers. The attributes are read only."); static PyStructSequence_Field int_info_fields[] = { {"bits_per_digit", "size of a digit in bits"}, {"sizeof_digit", "size in bytes of the C type used to represent a digit"}, + {"default_max_str_digits", "maximum string conversion digits limitation"}, + {"str_digits_check_threshold", "minimum positive value for int_max_str_digits"}, {NULL, NULL} }; @@ -5489,7 +5532,7 @@ static PyStructSequence_Desc int_info_desc = { "sys.int_info", /* name */ int_info__doc__, /* doc */ int_info_fields, /* fields */ - 2 /* number of fields */ + 4 /* number of fields */ }; PyObject * @@ -5504,6 +5547,17 @@ PyLong_GetInfo(void) PyLong_FromLong(PyLong_SHIFT)); PyStructSequence_SET_ITEM(int_info, field++, PyLong_FromLong(sizeof(digit))); + /* + * The following two fields were added after investigating uses of + * sys.int_info in the wild: Exceedingly rarely used. The ONLY use found was + * numba using sys.int_info.bits_per_digit as attribute access rather than + * sequence unpacking. Cython and sympy also refer to sys.int_info but only + * as info for debugging. No concern about adding these in a backport. + */ + PyStructSequence_SET_ITEM(int_info, field++, + PyLong_FromLong(_PY_LONG_DEFAULT_MAX_STR_DIGITS)); + PyStructSequence_SET_ITEM(int_info, field++, + PyLong_FromLong(_PY_LONG_MAX_STR_DIGITS_THRESHOLD)); if (PyErr_Occurred()) { Py_CLEAR(int_info); return NULL; @@ -5511,6 +5565,116 @@ PyLong_GetInfo(void) return int_info; } + +static int +pymain_str_to_int(const char *str, int *result) +{ + errno = 0; + const char *endptr = str; + long value = strtol(str, (char **)&endptr, 10); + if (*endptr != '\0' || errno == ERANGE) { + return -1; + } + if (value < INT_MIN || value > INT_MAX) { + return -1; + } + + *result = (int)value; + return 0; +} + + +static int +long_get_max_str_digits_xoption(int *pmaxdigits) +{ + PyObject *xoptions = PySys_GetXOptions(); + if (xoptions == NULL) { + return -1; + } + + PyObject *key = PyUnicode_FromString("int_max_str_digits"); + if (key == NULL) { + return -1; + } + + PyObject *value = PyDict_GetItemWithError(xoptions, key); /* borrowed */ + Py_DECREF(key); + if (value == NULL) { + if (PyErr_Occurred()) { + return -1; + } + return 0; + } + + if (!PyUnicode_Check(value)) { + return -1; + } + + PyObject *valuelong = PyLong_FromUnicodeObject(value, 10); + if (valuelong == NULL) { + return -1; + } + + int maxdigits = _PyLong_AsInt(valuelong); + Py_DECREF(valuelong); + if (maxdigits == -1 && PyErr_Occurred()) { + return -1; + } + + *pmaxdigits = maxdigits; + return 1; +} + + +static void +long_init_max_str_digits(void) +{ + // PYTHONINTMAXSTRDIGITS env var + char *opt = Py_GETENV("PYTHONINTMAXSTRDIGITS"); + int maxdigits; + if (opt) { + int valid = 0; + if (!pymain_str_to_int(opt, &maxdigits)) { + valid = ((maxdigits == 0) || (maxdigits >= _PY_LONG_MAX_STR_DIGITS_THRESHOLD)); + } + if (!valid) { +#define STRINGIFY(VAL) _STRINGIFY(VAL) +#define _STRINGIFY(VAL) #VAL + fprintf(stderr, "Error in PYTHONINTMAXSTRDIGITS: " + "invalid limit; must be >= " + STRINGIFY(_PY_LONG_MAX_STR_DIGITS_THRESHOLD) + " or 0 for unlimited.\n"); + exit(1); + } + _Py_global_config_int_max_str_digits = maxdigits; + } + + // -X int_max_str_digits command line option + int res = long_get_max_str_digits_xoption(&maxdigits); + if (res == 1) { + int valid = ((maxdigits == 0) || (maxdigits >= _PY_LONG_MAX_STR_DIGITS_THRESHOLD)); + if (!valid) { + res = -1; + } + } + if (res < 0) { + fprintf(stderr, "Error in -X int_max_str_digits: " + "invalid limit; must be >= " + STRINGIFY(_PY_LONG_MAX_STR_DIGITS_THRESHOLD) + " or 0 for unlimited.\n"); + exit(1); + } + if (res == 1) { + _Py_global_config_int_max_str_digits = maxdigits; + } + + // Default value + if (_Py_global_config_int_max_str_digits == -1) { + _Py_global_config_int_max_str_digits = _PY_LONG_DEFAULT_MAX_STR_DIGITS; + } +} + + int _PyLong_Init(void) { @@ -5549,6 +5713,8 @@ _PyLong_Init(void) return 0; } + long_init_max_str_digits(); + return 1; } diff --git a/Python/ast.c b/Python/ast.c index 675063eff06..4b69d8664e2 100644 --- a/Python/ast.c +++ b/Python/ast.c @@ -2149,8 +2149,32 @@ ast_for_atom(struct compiling *c, const node *n) } case NUMBER: { PyObject *pynum = parsenumber(c, STR(ch)); - if (!pynum) + if (!pynum) { + PyThreadState *tstate = PyThreadState_GET(); + // The only way a ValueError should happen in _this_ code is via + // PyLong_FromString hitting a length limit. + if (tstate->curexc_type == PyExc_ValueError && + tstate->curexc_value != NULL) { + PyObject *type, *value, *tb; + // This acts as PyErr_Clear() as we're replacing curexc. + PyErr_Fetch(&type, &value, &tb); + Py_XDECREF(tb); + Py_DECREF(type); + PyObject *helpful_msg = PyUnicode_FromFormat( + "%S - Consider hexadecimal for huge integer literals " + "to avoid decimal conversion limits.", + value); + if (helpful_msg) { + const char* error_msg = PyUnicode_AsUTF8(helpful_msg); + if (error_msg) { + ast_error(c, ch, error_msg); + } + Py_DECREF(helpful_msg); + } + Py_DECREF(value); + } return NULL; + } if (PyArena_AddPyObject(c->c_arena, pynum) < 0) { Py_DECREF(pynum); diff --git a/Python/pylifecycle.c b/Python/pylifecycle.c index 0b985bf78b9..095d1ef84fd 100644 --- a/Python/pylifecycle.c +++ b/Python/pylifecycle.c @@ -97,6 +97,8 @@ int Py_IsolatedFlag = 0; /* for -I, isolate from user's env */ int Py_LegacyWindowsFSEncodingFlag = 0; /* Uses mbcs instead of utf-8 */ int Py_LegacyWindowsStdioFlag = 0; /* Uses FileIO instead of WindowsConsoleIO */ #endif +/* Unusual name compared to the above for backporting from 3.12 reasons. */ +int _Py_global_config_int_max_str_digits = -1; /* -X int_max_str_digits or PYTHONINTMAXSTRDIGITS */ PyThreadState *_Py_Finalizing = NULL; diff --git a/Python/sysmodule.c b/Python/sysmodule.c index 7d1493cbe14..ecdb62cd1a1 100644 --- a/Python/sysmodule.c +++ b/Python/sysmodule.c @@ -1079,6 +1079,46 @@ sys_mdebug(PyObject *self, PyObject *args) } #endif /* USE_MALLOPT */ +static PyObject * +sys_get_int_max_str_digits(PyObject *module, PyObject *Py_UNUSED(ignored)) +{ + return PyLong_FromSsize_t(_Py_global_config_int_max_str_digits); +} + +PyDoc_STRVAR(sys_get_int_max_str_digits__doc__, +"get_int_max_str_digits($module, /)\n" +"--\n" +"\n" +"Set the maximum string digits limit for non-binary int<->str conversions."); + +static PyObject * +sys_set_int_max_str_digits(PyObject *module, PyObject *args, PyObject *kwds) +{ + static char *kwlist[] = {"maxdigits", NULL}; + int maxdigits; + + if (!PyArg_ParseTupleAndKeywords(args, kwds, "i:set_int_max_str_digits", + kwlist, &maxdigits)) + return NULL; + + if ((!maxdigits) || (maxdigits >= _PY_LONG_MAX_STR_DIGITS_THRESHOLD)) { + _Py_global_config_int_max_str_digits = maxdigits; + Py_RETURN_NONE; + } else { + PyErr_Format( + PyExc_ValueError, "maxdigits must be 0 or larger than %d", + _PY_LONG_MAX_STR_DIGITS_THRESHOLD); + return NULL; + } +} + +PyDoc_STRVAR(sys_set_int_max_str_digits__doc__, +"set_int_max_str_digits($module, /, maxdigits)\n" +"--\n" +"\n" +"Set the maximum string digits limit for non-binary int<->str conversions."); + + size_t _PySys_GetSizeOf(PyObject *o) { @@ -1434,6 +1474,8 @@ static PyMethodDef sys_methods[] = { METH_VARARGS | METH_KEYWORDS, set_asyncgen_hooks_doc}, {"get_asyncgen_hooks", sys_get_asyncgen_hooks, METH_NOARGS, get_asyncgen_hooks_doc}, + {"get_int_max_str_digits", sys_get_int_max_str_digits, METH_NOARGS, sys_get_int_max_str_digits__doc__}, + {"set_int_max_str_digits", (PyCFunction)sys_set_int_max_str_digits, METH_VARARGS|METH_KEYWORDS, sys_set_int_max_str_digits__doc__}, {NULL, NULL} /* sentinel */ }; @@ -1681,6 +1723,7 @@ static PyStructSequence_Field flags_fields[] = { {"quiet", "-q"}, {"hash_randomization", "-R"}, {"isolated", "-I"}, + {"int_max_str_digits", "-X int_max_str_digits"}, {0} }; @@ -1688,7 +1731,7 @@ static PyStructSequence_Desc flags_desc = { "sys.flags", /* name */ flags__doc__, /* doc */ flags_fields, /* fields */ - 13 + 14 }; static PyObject* @@ -1719,6 +1762,7 @@ make_flags(void) SetFlag(Py_QuietFlag); SetFlag(Py_HashRandomizationFlag); SetFlag(Py_IsolatedFlag); + SetFlag(_Py_global_config_int_max_str_digits); #undef SetFlag if (PyErr_Occurred()) { -- 2.45.0
Locations
Projects
Search
Status Monitor
Help
OpenBuildService.org
Documentation
API Documentation
Code of Conduct
Contact
Support
@OBShq
Terms
openSUSE Build Service is sponsored by
The Open Build Service is an
openSUSE project
.
Sign Up
Log In
Places
Places
All Projects
Status Monitor
