pybind11/tests/test_exceptions.py

393 lines
13 KiB
Python
Raw Normal View History

import sys
import pytest
import env
Fix builtin exception handlers to work across modules The builtin exception handler currently doesn't work across modules under clang/libc++ for builtin pybind exceptions like `pybind11::error_already_set` or `pybind11::stop_iteration`: under RTLD_LOCAL module loading clang considers each module's exception classes distinct types. This then means that the base exception translator fails to catch the exceptions and the fall through to the generic `std::exception` handler, which completely breaks things like `stop_iteration`: only the `stop_iteration` of the first module loaded actually works properly; later modules raise a RuntimeError with no message when trying to invoke their iterators. For example, two modules defined like this exhibit the behaviour under clang++/libc++: z1.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z1, m) { py::bind_vector<std::vector<long>>(m, "IntVector"); } z2.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z2, m) { py::bind_vector<std::vector<double>>(m, "FloatVector"); } Python: import z1, z2 for i in z2.FloatVector(): pass results in: Traceback (most recent call last): File "zs.py", line 2, in <module> for i in z2.FloatVector(): RuntimeError This commit fixes the issue by adding a new exception translator each time the internals pointer is initialized from python builtins: this generally means the internals data was initialized by some other module. (The extra translator(s) are skipped under libstdc++).
2017-07-29 01:38:23 +00:00
import pybind11_cross_module_tests as cm
import pybind11_tests # noqa: F401
from pybind11_tests import exceptions as m
def test_std_exception(msg):
with pytest.raises(RuntimeError) as excinfo:
m.throw_std_exception()
assert msg(excinfo.value) == "This exception was intentionally thrown."
def test_error_already_set(msg):
with pytest.raises(RuntimeError) as excinfo:
m.throw_already_set(False)
assert (
msg(excinfo.value)
== "Internal error: pybind11::error_already_set called while Python error indicator not set."
)
with pytest.raises(ValueError) as excinfo:
m.throw_already_set(True)
assert msg(excinfo.value) == "foo"
def test_raise_from(msg):
with pytest.raises(ValueError) as excinfo:
m.raise_from()
assert msg(excinfo.value) == "outer"
assert msg(excinfo.value.__cause__) == "inner"
def test_raise_from_already_set(msg):
with pytest.raises(ValueError) as excinfo:
m.raise_from_already_set()
assert msg(excinfo.value) == "outer"
assert msg(excinfo.value.__cause__) == "inner"
Feature/local exception translator (#2650) * Create a module_internals struct Since we now have two things that are going to be module local, it felt correct to add a struct to manage them. * Add local exception translators These are added via the register_local_exception_translator function and are then applied before the global translators * Add unit tests to show the local exception translator works * Fix a bug in the unit test with the string value of KeyError * Fix a formatting issue * Rename registered_local_types_cpp() Rename it to get_registered_local_types_cpp() to disambiguate from the new member of module_internals * Add additional comments to new local exception code path * Add a register_local_exception function * Add additional unit tests for register_local_exception * Use get_local_internals like get_internals * Update documentation for new local exception feature * Add back a missing space * Clean-up some issues in the docs * Remove the code duplication when translating exceptions Separated out the exception processing into a standalone function in the details namespace. Clean-up some comments as per PR notes as well * Remove the code duplication in register_exception * Cleanup some formatting things caught by clang-format * Remove the templates from exception translators But I added a using declaration to alias the type. * Remove the extra local from local_internals variable names * Add an extra explanatory comment to local_internals * Fix a typo in the code
2021-07-21 12:22:18 +00:00
def test_cross_module_exceptions(msg):
Fix builtin exception handlers to work across modules The builtin exception handler currently doesn't work across modules under clang/libc++ for builtin pybind exceptions like `pybind11::error_already_set` or `pybind11::stop_iteration`: under RTLD_LOCAL module loading clang considers each module's exception classes distinct types. This then means that the base exception translator fails to catch the exceptions and the fall through to the generic `std::exception` handler, which completely breaks things like `stop_iteration`: only the `stop_iteration` of the first module loaded actually works properly; later modules raise a RuntimeError with no message when trying to invoke their iterators. For example, two modules defined like this exhibit the behaviour under clang++/libc++: z1.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z1, m) { py::bind_vector<std::vector<long>>(m, "IntVector"); } z2.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z2, m) { py::bind_vector<std::vector<double>>(m, "FloatVector"); } Python: import z1, z2 for i in z2.FloatVector(): pass results in: Traceback (most recent call last): File "zs.py", line 2, in <module> for i in z2.FloatVector(): RuntimeError This commit fixes the issue by adding a new exception translator each time the internals pointer is initialized from python builtins: this generally means the internals data was initialized by some other module. (The extra translator(s) are skipped under libstdc++).
2017-07-29 01:38:23 +00:00
with pytest.raises(RuntimeError) as excinfo:
cm.raise_runtime_error()
assert str(excinfo.value) == "My runtime error"
with pytest.raises(ValueError) as excinfo:
cm.raise_value_error()
assert str(excinfo.value) == "My value error"
with pytest.raises(ValueError) as excinfo:
cm.throw_pybind_value_error()
assert str(excinfo.value) == "pybind11 value error"
with pytest.raises(TypeError) as excinfo:
cm.throw_pybind_type_error()
assert str(excinfo.value) == "pybind11 type error"
with pytest.raises(StopIteration) as excinfo:
cm.throw_stop_iteration()
Feature/local exception translator (#2650) * Create a module_internals struct Since we now have two things that are going to be module local, it felt correct to add a struct to manage them. * Add local exception translators These are added via the register_local_exception_translator function and are then applied before the global translators * Add unit tests to show the local exception translator works * Fix a bug in the unit test with the string value of KeyError * Fix a formatting issue * Rename registered_local_types_cpp() Rename it to get_registered_local_types_cpp() to disambiguate from the new member of module_internals * Add additional comments to new local exception code path * Add a register_local_exception function * Add additional unit tests for register_local_exception * Use get_local_internals like get_internals * Update documentation for new local exception feature * Add back a missing space * Clean-up some issues in the docs * Remove the code duplication when translating exceptions Separated out the exception processing into a standalone function in the details namespace. Clean-up some comments as per PR notes as well * Remove the code duplication in register_exception * Cleanup some formatting things caught by clang-format * Remove the templates from exception translators But I added a using declaration to alias the type. * Remove the extra local from local_internals variable names * Add an extra explanatory comment to local_internals * Fix a typo in the code
2021-07-21 12:22:18 +00:00
with pytest.raises(cm.LocalSimpleException) as excinfo:
cm.throw_local_simple_error()
assert msg(excinfo.value) == "external mod"
with pytest.raises(KeyError) as excinfo:
cm.throw_local_error()
# KeyError is a repr of the key, so it has an extra set of quotes
assert str(excinfo.value) == "'just local'"
Fix builtin exception handlers to work across modules The builtin exception handler currently doesn't work across modules under clang/libc++ for builtin pybind exceptions like `pybind11::error_already_set` or `pybind11::stop_iteration`: under RTLD_LOCAL module loading clang considers each module's exception classes distinct types. This then means that the base exception translator fails to catch the exceptions and the fall through to the generic `std::exception` handler, which completely breaks things like `stop_iteration`: only the `stop_iteration` of the first module loaded actually works properly; later modules raise a RuntimeError with no message when trying to invoke their iterators. For example, two modules defined like this exhibit the behaviour under clang++/libc++: z1.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z1, m) { py::bind_vector<std::vector<long>>(m, "IntVector"); } z2.cpp: #include <pybind11/pybind11.h> #include <pybind11/stl_bind.h> namespace py = pybind11; PYBIND11_MODULE(z2, m) { py::bind_vector<std::vector<double>>(m, "FloatVector"); } Python: import z1, z2 for i in z2.FloatVector(): pass results in: Traceback (most recent call last): File "zs.py", line 2, in <module> for i in z2.FloatVector(): RuntimeError This commit fixes the issue by adding a new exception translator each time the internals pointer is initialized from python builtins: this generally means the internals data was initialized by some other module. (The extra translator(s) are skipped under libstdc++).
2017-07-29 01:38:23 +00:00
# TODO: FIXME
@pytest.mark.xfail(
"env.MACOS and (env.PYPY or pybind11_tests.compiler_info.startswith('Homebrew Clang'))",
raises=RuntimeError,
reason="See Issue #2847, PR #2999, PR #4324",
)
def test_cross_module_exception_translator():
with pytest.raises(KeyError):
# translator registered in cross_module_tests
m.throw_should_be_translated_to_key_error()
def test_python_call_in_catch():
d = {}
assert m.python_call_in_destructor(d) is True
assert d["good"] is True
def ignore_pytest_unraisable_warning(f):
unraisable = "PytestUnraisableExceptionWarning"
if hasattr(pytest, unraisable): # Python >= 3.8 and pytest >= 6
dec = pytest.mark.filterwarnings(f"ignore::pytest.{unraisable}")
return dec(f)
else:
return f
# TODO: find out why this fails on PyPy, https://foss.heptapod.net/pypy/pypy/-/issues/3583
@pytest.mark.xfail(env.PYPY, reason="Failure on PyPy 3.8 (7.3.7)", strict=False)
@ignore_pytest_unraisable_warning
def test_python_alreadyset_in_destructor(monkeypatch, capsys):
hooked = False
triggered = False
if hasattr(sys, "unraisablehook"): # Python 3.8+
hooked = True
# Don't take `sys.unraisablehook`, as that's overwritten by pytest
default_hook = sys.__unraisablehook__
def hook(unraisable_hook_args):
exc_type, exc_value, exc_tb, err_msg, obj = unraisable_hook_args
if obj == "already_set demo":
nonlocal triggered
triggered = True
default_hook(unraisable_hook_args)
return
# Use monkeypatch so pytest can apply and remove the patch as appropriate
monkeypatch.setattr(sys, "unraisablehook", hook)
assert m.python_alreadyset_in_destructor("already_set demo") is True
if hooked:
assert triggered is True
_, captured_stderr = capsys.readouterr()
assert captured_stderr.startswith("Exception ignored in: 'already_set demo'")
assert captured_stderr.rstrip().endswith("KeyError: 'bar'")
def test_exception_matches():
assert m.exception_matches()
assert m.exception_matches_base()
assert m.modulenotfound_exception_matches_base()
def test_custom(msg):
# Can we catch a MyException?
with pytest.raises(m.MyException) as excinfo:
m.throws1()
assert msg(excinfo.value) == "this error should go to a custom type"
# Can we translate to standard Python exceptions?
with pytest.raises(RuntimeError) as excinfo:
m.throws2()
assert msg(excinfo.value) == "this error should go to a standard Python exception"
# Can we handle unknown exceptions?
with pytest.raises(RuntimeError) as excinfo:
m.throws3()
assert msg(excinfo.value) == "Caught an unknown exception!"
# Can we delegate to another handler by rethrowing?
with pytest.raises(m.MyException) as excinfo:
m.throws4()
assert msg(excinfo.value) == "this error is rethrown"
# Can we fall-through to the default handler?
with pytest.raises(RuntimeError) as excinfo:
m.throws_logic_error()
assert (
msg(excinfo.value) == "this error should fall through to the standard handler"
)
# OverFlow error translation.
with pytest.raises(OverflowError) as excinfo:
m.throws_overflow_error()
# Can we handle a helper-declared exception?
with pytest.raises(m.MyException5) as excinfo:
m.throws5()
assert msg(excinfo.value) == "this is a helper-defined translated exception"
# Exception subclassing:
with pytest.raises(m.MyException5) as excinfo:
m.throws5_1()
assert msg(excinfo.value) == "MyException5 subclass"
assert isinstance(excinfo.value, m.MyException5_1)
with pytest.raises(m.MyException5_1) as excinfo:
m.throws5_1()
assert msg(excinfo.value) == "MyException5 subclass"
with pytest.raises(m.MyException5) as excinfo:
try:
m.throws5()
except m.MyException5_1 as err:
raise RuntimeError("Exception error: caught child from parent") from err
assert msg(excinfo.value) == "this is a helper-defined translated exception"
Simplify error_already_set `error_already_set` is more complicated than it needs to be, partly because it manages reference counts itself rather than using `py::object`, and partly because it tries to do more exception clearing than is needed. This commit greatly simplifies it, and fixes #927. Using `py::object` instead of `PyObject *` means we can rely on implicit copy/move constructors. The current logic did both a `PyErr_Clear` on deletion *and* a `PyErr_Fetch` on creation. I can't see how the `PyErr_Clear` on deletion is ever useful: the `Fetch` on creation itself clears the error, so the only way doing a `PyErr_Clear` on deletion could do anything if is some *other* exception was raised while the `error_already_set` object was alive--but in that case, clearing some other exception seems wrong. (Code that is worried about an exception handler raising another exception would already catch a second `error_already_set` from exception code). The destructor itself called `clear()`, but `clear()` was a little bit more paranoid that needed: it called `restore()` to restore the currently captured error, but then immediately cleared it, using the `PyErr_Restore` to release the references. That's unnecessary: it's valid for us to release the references manually. This updates the code to simply release the references on the three objects (preserving the gil acquire). `clear()`, however, also had the side effect of clearing the current error, even if the current `error_already_set` didn't have a current error (e.g. because of a previous `restore()` or `clear()` call). I don't really see how clearing the error here can ever actually be useful: the only way the current error could be set is if you called `restore()` (in which case the current stored error-related members have already been released), or if some *other* code raised the error, in which case `clear()` on *this* object is clearing an error for which it shouldn't be responsible. Neither of those seem like intentional or desirable features, and manually requesting deletion of the stored references similarly seems pointless, so I've just made `clear()` an empty method and marked it deprecated. This also fixes a minor potential issue with the destruction: it is technically possible for `value` to be null (though this seems likely to be rare in practice); this updates the check to look at `type` which will always be non-null for a `Fetch`ed exception. This also adds error_already_set round-trip throw tests to the test suite.
2017-07-21 03:14:33 +00:00
def test_nested_throws(capture):
"""Tests nested (e.g. C++ -> Python -> C++) exception handling"""
def throw_myex():
raise m.MyException("nested error")
def throw_myex5():
raise m.MyException5("nested error 5")
# In the comments below, the exception is caught in the first step, thrown in the last step
# C++ -> Python
with capture:
m.try_catch(m.MyException5, throw_myex5)
assert str(capture).startswith("MyException5: nested error 5")
# Python -> C++ -> Python
with pytest.raises(m.MyException) as excinfo:
m.try_catch(m.MyException5, throw_myex)
assert str(excinfo.value) == "nested error"
def pycatch(exctype, f, *args):
try:
f(*args)
except m.MyException as e:
print(e)
# C++ -> Python -> C++ -> Python
with capture:
m.try_catch(
m.MyException5,
pycatch,
m.MyException,
m.try_catch,
m.MyException,
throw_myex5,
)
Simplify error_already_set `error_already_set` is more complicated than it needs to be, partly because it manages reference counts itself rather than using `py::object`, and partly because it tries to do more exception clearing than is needed. This commit greatly simplifies it, and fixes #927. Using `py::object` instead of `PyObject *` means we can rely on implicit copy/move constructors. The current logic did both a `PyErr_Clear` on deletion *and* a `PyErr_Fetch` on creation. I can't see how the `PyErr_Clear` on deletion is ever useful: the `Fetch` on creation itself clears the error, so the only way doing a `PyErr_Clear` on deletion could do anything if is some *other* exception was raised while the `error_already_set` object was alive--but in that case, clearing some other exception seems wrong. (Code that is worried about an exception handler raising another exception would already catch a second `error_already_set` from exception code). The destructor itself called `clear()`, but `clear()` was a little bit more paranoid that needed: it called `restore()` to restore the currently captured error, but then immediately cleared it, using the `PyErr_Restore` to release the references. That's unnecessary: it's valid for us to release the references manually. This updates the code to simply release the references on the three objects (preserving the gil acquire). `clear()`, however, also had the side effect of clearing the current error, even if the current `error_already_set` didn't have a current error (e.g. because of a previous `restore()` or `clear()` call). I don't really see how clearing the error here can ever actually be useful: the only way the current error could be set is if you called `restore()` (in which case the current stored error-related members have already been released), or if some *other* code raised the error, in which case `clear()` on *this* object is clearing an error for which it shouldn't be responsible. Neither of those seem like intentional or desirable features, and manually requesting deletion of the stored references similarly seems pointless, so I've just made `clear()` an empty method and marked it deprecated. This also fixes a minor potential issue with the destruction: it is technically possible for `value` to be null (though this seems likely to be rare in practice); this updates the check to look at `type` which will always be non-null for a `Fetch`ed exception. This also adds error_already_set round-trip throw tests to the test suite.
2017-07-21 03:14:33 +00:00
assert str(capture).startswith("MyException5: nested error 5")
# C++ -> Python -> C++
with capture:
m.try_catch(m.MyException, pycatch, m.MyException5, m.throws4)
assert capture == "this error is rethrown"
# Python -> C++ -> Python -> C++
with pytest.raises(m.MyException5) as excinfo:
m.try_catch(m.MyException, pycatch, m.MyException, m.throws5)
assert str(excinfo.value) == "this is a helper-defined translated exception"
def test_throw_nested_exception():
with pytest.raises(RuntimeError) as excinfo:
m.throw_nested_exception()
assert str(excinfo.value) == "Outer Exception"
assert str(excinfo.value.__cause__) == "Inner Exception"
# This can often happen if you wrap a pybind11 class in a Python wrapper
def test_invalid_repr():
class MyRepr:
def __repr__(self):
raise AttributeError("Example error")
with pytest.raises(TypeError):
m.simple_bool_passthrough(MyRepr())
Feature/local exception translator (#2650) * Create a module_internals struct Since we now have two things that are going to be module local, it felt correct to add a struct to manage them. * Add local exception translators These are added via the register_local_exception_translator function and are then applied before the global translators * Add unit tests to show the local exception translator works * Fix a bug in the unit test with the string value of KeyError * Fix a formatting issue * Rename registered_local_types_cpp() Rename it to get_registered_local_types_cpp() to disambiguate from the new member of module_internals * Add additional comments to new local exception code path * Add a register_local_exception function * Add additional unit tests for register_local_exception * Use get_local_internals like get_internals * Update documentation for new local exception feature * Add back a missing space * Clean-up some issues in the docs * Remove the code duplication when translating exceptions Separated out the exception processing into a standalone function in the details namespace. Clean-up some comments as per PR notes as well * Remove the code duplication in register_exception * Cleanup some formatting things caught by clang-format * Remove the templates from exception translators But I added a using declaration to alias the type. * Remove the extra local from local_internals variable names * Add an extra explanatory comment to local_internals * Fix a typo in the code
2021-07-21 12:22:18 +00:00
def test_local_translator(msg):
"""Tests that a local translator works and that the local translator from
the cross module is not applied"""
with pytest.raises(RuntimeError) as excinfo:
m.throws6()
assert msg(excinfo.value) == "MyException6 only handled in this module"
with pytest.raises(RuntimeError) as excinfo:
m.throws_local_error()
assert not isinstance(excinfo.value, KeyError)
assert msg(excinfo.value) == "never caught"
with pytest.raises(Exception) as excinfo:
m.throws_local_simple_error()
assert not isinstance(excinfo.value, cm.LocalSimpleException)
assert msg(excinfo.value) == "this mod"
def test_error_already_set_message_with_unicode_surrogate(): # Issue #4288
assert m.error_already_set_what(RuntimeError, "\ud927") == (
"RuntimeError: \\ud927",
False,
)
def test_error_already_set_message_with_malformed_utf8():
assert m.error_already_set_what(RuntimeError, b"\x80") == (
"RuntimeError: b'\\x80'",
False,
)
class FlakyException(Exception):
def __init__(self, failure_point):
if failure_point == "failure_point_init":
raise ValueError("triggered_failure_point_init")
self.failure_point = failure_point
def __str__(self):
if self.failure_point == "failure_point_str":
raise ValueError("triggered_failure_point_str")
return "FlakyException.__str__"
@pytest.mark.parametrize(
"exc_type, exc_value, expected_what",
(
(ValueError, "plain_str", "ValueError: plain_str"),
(ValueError, ("tuple_elem",), "ValueError: tuple_elem"),
(FlakyException, ("happy",), "FlakyException: FlakyException.__str__"),
),
)
def test_error_already_set_what_with_happy_exceptions(
exc_type, exc_value, expected_what
):
what, py_err_set_after_what = m.error_already_set_what(exc_type, exc_value)
assert not py_err_set_after_what
assert what == expected_what
@pytest.mark.skipif("env.PYPY", reason="PyErr_NormalizeException Segmentation fault")
def test_flaky_exception_failure_point_init():
error_already_set::what() is now constructed lazily (#1895) * error_already_set::what() is now constructed lazily Prior to this commit throwing error_already_set was expensive due to the eager construction of the error string (which required traversing the Python stack). See #1853 for more context and an alternative take on the issue. Note that error_already_set no longer inherits from std::runtime_error because the latter has no default constructor. * Do not attempt to normalize if no exception occurred This is not supported on PyPy-2.7 5.8.0. * Extract exception name via tp_name This is faster than dynamically looking up __name__ via GetAttrString. Note though that the runtime of the code throwing an error_already_set will be dominated by stack unwinding so the improvement will not be noticeable. Before: 396 ns ± 0.913 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) After: 277 ns ± 0.549 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) Benchmark: const std::string foo() { PyErr_SetString(PyExc_KeyError, ""); const std::string &s = py::detail::error_string(); PyErr_Clear(); return s; } PYBIND11_MODULE(foo, m) { m.def("foo", &::foo); } * Reverted error_already_set to subclass std::runtime_error * Revert "Extract exception name via tp_name" The implementation of __name__ is slightly more complex than that. It handles the module name prefix, and heap-allocated types. We could port it to pybind11 later on but for now it seems like an overkill. This reverts commit f1435c7e6b068a1ed13ebd3db597ea3bb15aa398. * Cosmit following @YannickJadoul's comments Note that detail::error_string() no longer calls PyException_SetTraceback as it is unncessary for pretty-printing the exception. * Fixed PyPy build * Moved normalization to error_already_set ctor * Fix merge bugs * Fix more merge errors * Improve formatting * Improve error message in rare case * Revert back if statements * Fix clang-tidy * Try removing mutable * Does build_mode release fix it * Set to Debug to expose segfault * Fix remove set error string * Do not run error_string() more than once * Trying setting the tracebackk to the value * guard if m_type is null * Try to debug PGI * One last try for PGI * Does reverting this fix PyPy * Reviewer suggestions * Remove unnecessary initialization * Add noexcept move and explicit fail throw * Optimize error_string creation * Fix typo * Revert noexcept * Fix merge conflict error * Abuse assignment operator * Revert operator abuse * See if we still need debug * Remove unnecessary mutable * Report "FATAL failure building pybind11::error_already_set error_string" and terminate process. * Try specifying noexcept again * Try explicit ctor * default ctor is noexcept too * Apply reviewer suggestions, simplify code, and make helper method private * Remove unnecessary include * Clang-Tidy fix * detail::obj_class_name(), fprintf with [STDERR], [STDOUT] tags, polish comments * consistently check m_lazy_what.empty() also in production builds * Make a comment slightly less ambiguous. * Bug fix: Remove `what();` from `restore()`. It sure would need to be guarded by `if (m_type)`, otherwise `what()` fails and masks that no error was set (see update unit test). But since `error_already_set` is copyable, there is no point in releasing m_type, m_value, m_trace, therefore we can just as well avoid the runtime overhead of force-building `m_lazy_what`, it may never be used. * Replace extremely opaque (unhelpful) error message with a truthful reflection of what we know. * Fix clang-tidy error [performance-move-constructor-init]. * Make expected error message less specific. * Various changes. * bug fix: error_string(PyObject **, ...) * Putting back the two eager PyErr_NormalizeException() calls. * Change error_already_set() to call pybind11_fail() if the Python error indicator not set. The net result is that a std::runtime_error is thrown instead of error_already_set, but all tests pass as is. * Remove mutable (fixes oversight in the previous commit). * Normalize the exception only locally in error_string(). Python 3.6 & 3.7 test failures expected. This is meant for benchmarking, to determine if it is worth the trouble looking into the failures. * clang-tidy: use auto * Use `gil_scoped_acquire_local` in `error_already_set` destructor. See long comment. * For Python < 3.8: `PyErr_NormalizeException` before `PyErr_WriteUnraisable` * Go back to replacing the held Python exception with then normalized exception, if & when needed. Consistently document the side-effect. * Slightly rewording comment. (There were also other failures.) * Add 1-line comment for obj_class_name() * Benchmark code, with results in this commit message. function #calls test time [s] μs / call master pure_unwind 729540 1.061 14.539876 err_set_unwind_err_clear 681476 1.040 15.260282 err_set_error_already_set 508038 1.049 20.640525 error_already_set_restore 555578 1.052 18.933288 pr1895_original_foo 244113 1.050 43.018168 PR / master PR #1895 pure_unwind 736981 1.054 14.295685 98.32% err_set_unwind_err_clear 685820 1.045 15.237399 99.85% err_set_error_already_set 661374 1.046 15.811879 76.61% error_already_set_restore 669881 1.048 15.645176 82.63% pr1895_original_foo 318243 1.059 33.290806 77.39% master @ commit ad146b2a1877e8ba3803f94a7837969835a297a7 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,729540,1.061,14.539876 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,681476,1.040,15.260282 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,508038,1.049,20.640525 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,555578,1.052,18.933288 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,244113,1.050,43.018168 PASSED ============================== 5 passed in 12.38s ============================== pr1895 @ commit 8dff51d12e4af11aff415ee966070368fe606664 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,736981,1.054,14.295685 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,685820,1.045,15.237399 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,661374,1.046,15.811879 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,669881,1.048,15.645176 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,318243,1.059,33.290806 PASSED ============================== 5 passed in 12.40s ============================== clang++ -o pybind11/tests/test_perf_error_already_set.os -c -std=c++17 -fPIC -fvisibility=hidden -Os -flto -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wnon-virtual-dtor -Wunused-result -isystem /usr/include/python3.9 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_perf_error_already_set.cpp clang++ -o lib/pybind11_tests.so -shared -fPIC -Os -flto -shared ... Debian clang version 13.0.1-3+build2 Target: x86_64-pc-linux-gnu Thread model: posix * Changing call_repetitions_target_elapsed_secs to 0.1 for regular unit testing. * Adding in `recursion_depth` * Optimized ctor * Fix silly bug in recurse_first_then_call() * Add tests that have equivalent PyErr_Fetch(), PyErr_Restore() but no try-catch. * Add call_error_string to tests. Sample only recursion_depth 0, 100. * Show lazy-what speed-up in percent. * Include real_work in benchmarks. * Replace all PyErr_SetString() with generate_python_exception_with_traceback() * Better organization of test loops. * Add test_error_already_set_copy_move * Fix bug in newly added test (discovered by clang-tidy): actually use move ctor * MSVC detects the unreachable return * change test_perf_error_already_set.py back to quick mode * Inherit from std::exception (instead of std::runtime_error, which does not make sense anymore with the lazy what) * Special handling under Windows. * print with leading newline * Removing test_perf_error_already_set (copies are under https://github.com/rwgk/rwgk_tbx/commit/7765113fbb659e1ea004c5ba24fb578244bc6cfd). * Avoid gil and scope overhead if there is nothing to release. * Restore default move ctor. "member function" instead of "function" (note that "method" is Python terminology). * Delete error_already_set copy ctor. * Make restore() non-const again to resolve clang-tidy failure (still experimenting). * Bring back error_already_set copy ctor, to see if that resolves the 4 MSVC test failures. * Add noexcept to error_already_set copy & move ctors (as suggested by @skylion007 IIUC). * Trying one-by-one noexcept copy ctor for old compilers. * Add back test covering copy ctor. Add another simple test that exercises the copy ctor. * Exclude more older compilers from using the noexcept = default ctors. (The tests in the previous commit exposed that those are broken.) * Factor out & reuse gil_scoped_acquire_local as gil_scoped_acquire_simple * Guard gil_scoped_acquire_simple by _Py_IsFinalizing() check. * what() GIL safety * clang-tidy & Python 3.6 fixes * Use `gil_scoped_acquire` in dtor, copy ctor, `what()`. Remove `_Py_IsFinalizing()` checks (they are racy: https://github.com/python/cpython/pull/28525). * Remove error_scope from copy ctor. * Add `error_scope` to `get_internals()`, to cover the situation that `get_internals()` is called from the `error_already_set` dtor while a new Python error is in flight already. Also backing out `gil_scoped_acquire_simple` change. * Add `FlakyException` tests with failure triggers in `__init__` and `__str__` THIS IS STILL A WORK IN PROGRESS. This commit is only an important resting point. This commit is a first attempt at addressing the observation that `PyErr_NormalizeException()` completely replaces the original exception if `__init__` fails. This can be very confusing even in small applications, and extremely confusing in large ones. * Tweaks to resolve Py 3.6 and PyPy CI failures. * Normalize Python exception immediately in error_already_set ctor. For background see: https://github.com/pybind/pybind11/pull/1895#issuecomment-1135304081 * Fix oversights based on CI failures (copy & move ctor initialization). * Move @pytest.mark.xfail("env.PYPY") after @pytest.mark.parametrize(...) * Use @pytest.mark.skipif (xfail does not work for segfaults, of course). * Remove unused obj_class_name_or() function (it was added only under this PR). * Remove already obsolete C++ comments and code that were added only under this PR. * Slightly better (newly added) comments. * Factor out detail::error_fetch_and_normalize. Preparation for producing identical results from error_already_set::what() and detail::error_string(). Note that this is a very conservative refactoring. It would be much better to first move detail::error_string into detail/error_string.h * Copy most of error_string() code to new error_fetch_and_normalize::complete_lazy_error_string() * Remove all error_string() code from detail/type_caster_base.h. Note that this commit includes a subtle bug fix: previously error_string() restored the Python error, which will upset pybind11_fail(). This never was a problem in practice because the two PyType_Ready() calls in detail/class.h do not usually fail. * Return const std::string& instead of const char * and move error_string() to pytypes.h * Remove gil_scope_acquire from error_fetch_and_normalize, add back to error_already_set * Better handling of FlakyException __str__ failure. * Move error_fetch_and_normalize::complete_lazy_error_string() implementation from pybind11.h to pytypes.h * Add error_fetch_and_normalize::release_py_object_references() and use from error_already_set dtor. * Use shared_ptr for m_fetched_error => 1. non-racy, copy ctor that does not need the GIL; 2. enables guard against duplicate restore() calls. * Add comments. * Trivial renaming of a newly introduced member function. * Workaround for PyPy * Bug fix (oversight). Only valgrind got this one. * Use shared_ptr custom deleter for m_fetched_error in error_already_set. This enables removing the dtor, copy ctor, move ctor completely. * Further small simplification. With the GIL held, simply deleting the raw_ptr takes care of everything. * IWYU cleanup ``` iwyu version: include-what-you-use 0.17 based on Debian clang version 13.0.1-3+build2 ``` Command used: ``` iwyu -c -std=c++17 -DPYBIND11_TEST_BOOST -Iinclude/pybind11 -I/usr/include/python3.9 -I/usr/include/eigen3 include/pybind11/pytypes.cpp ``` pytypes.cpp is a temporary file: `#include "pytypes.h"` The raw output is very long and noisy. I decided to use `#include <cstddef>` instead of `#include <cstdio>` for `std::size_t` (iwyu sticks to the manual choice). I ignored all iwyu suggestions that are indirectly covered by `#include <Python.h>`. I manually verified that all added includes are actually needed. Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
2022-06-02 23:17:38 +00:00
with pytest.raises(RuntimeError) as excinfo:
m.error_already_set_what(FlakyException, ("failure_point_init",))
lines = str(excinfo.value).splitlines()
# PyErr_NormalizeException replaces the original FlakyException with ValueError:
error_already_set::what() is now constructed lazily (#1895) * error_already_set::what() is now constructed lazily Prior to this commit throwing error_already_set was expensive due to the eager construction of the error string (which required traversing the Python stack). See #1853 for more context and an alternative take on the issue. Note that error_already_set no longer inherits from std::runtime_error because the latter has no default constructor. * Do not attempt to normalize if no exception occurred This is not supported on PyPy-2.7 5.8.0. * Extract exception name via tp_name This is faster than dynamically looking up __name__ via GetAttrString. Note though that the runtime of the code throwing an error_already_set will be dominated by stack unwinding so the improvement will not be noticeable. Before: 396 ns ± 0.913 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) After: 277 ns ± 0.549 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) Benchmark: const std::string foo() { PyErr_SetString(PyExc_KeyError, ""); const std::string &s = py::detail::error_string(); PyErr_Clear(); return s; } PYBIND11_MODULE(foo, m) { m.def("foo", &::foo); } * Reverted error_already_set to subclass std::runtime_error * Revert "Extract exception name via tp_name" The implementation of __name__ is slightly more complex than that. It handles the module name prefix, and heap-allocated types. We could port it to pybind11 later on but for now it seems like an overkill. This reverts commit f1435c7e6b068a1ed13ebd3db597ea3bb15aa398. * Cosmit following @YannickJadoul's comments Note that detail::error_string() no longer calls PyException_SetTraceback as it is unncessary for pretty-printing the exception. * Fixed PyPy build * Moved normalization to error_already_set ctor * Fix merge bugs * Fix more merge errors * Improve formatting * Improve error message in rare case * Revert back if statements * Fix clang-tidy * Try removing mutable * Does build_mode release fix it * Set to Debug to expose segfault * Fix remove set error string * Do not run error_string() more than once * Trying setting the tracebackk to the value * guard if m_type is null * Try to debug PGI * One last try for PGI * Does reverting this fix PyPy * Reviewer suggestions * Remove unnecessary initialization * Add noexcept move and explicit fail throw * Optimize error_string creation * Fix typo * Revert noexcept * Fix merge conflict error * Abuse assignment operator * Revert operator abuse * See if we still need debug * Remove unnecessary mutable * Report "FATAL failure building pybind11::error_already_set error_string" and terminate process. * Try specifying noexcept again * Try explicit ctor * default ctor is noexcept too * Apply reviewer suggestions, simplify code, and make helper method private * Remove unnecessary include * Clang-Tidy fix * detail::obj_class_name(), fprintf with [STDERR], [STDOUT] tags, polish comments * consistently check m_lazy_what.empty() also in production builds * Make a comment slightly less ambiguous. * Bug fix: Remove `what();` from `restore()`. It sure would need to be guarded by `if (m_type)`, otherwise `what()` fails and masks that no error was set (see update unit test). But since `error_already_set` is copyable, there is no point in releasing m_type, m_value, m_trace, therefore we can just as well avoid the runtime overhead of force-building `m_lazy_what`, it may never be used. * Replace extremely opaque (unhelpful) error message with a truthful reflection of what we know. * Fix clang-tidy error [performance-move-constructor-init]. * Make expected error message less specific. * Various changes. * bug fix: error_string(PyObject **, ...) * Putting back the two eager PyErr_NormalizeException() calls. * Change error_already_set() to call pybind11_fail() if the Python error indicator not set. The net result is that a std::runtime_error is thrown instead of error_already_set, but all tests pass as is. * Remove mutable (fixes oversight in the previous commit). * Normalize the exception only locally in error_string(). Python 3.6 & 3.7 test failures expected. This is meant for benchmarking, to determine if it is worth the trouble looking into the failures. * clang-tidy: use auto * Use `gil_scoped_acquire_local` in `error_already_set` destructor. See long comment. * For Python < 3.8: `PyErr_NormalizeException` before `PyErr_WriteUnraisable` * Go back to replacing the held Python exception with then normalized exception, if & when needed. Consistently document the side-effect. * Slightly rewording comment. (There were also other failures.) * Add 1-line comment for obj_class_name() * Benchmark code, with results in this commit message. function #calls test time [s] μs / call master pure_unwind 729540 1.061 14.539876 err_set_unwind_err_clear 681476 1.040 15.260282 err_set_error_already_set 508038 1.049 20.640525 error_already_set_restore 555578 1.052 18.933288 pr1895_original_foo 244113 1.050 43.018168 PR / master PR #1895 pure_unwind 736981 1.054 14.295685 98.32% err_set_unwind_err_clear 685820 1.045 15.237399 99.85% err_set_error_already_set 661374 1.046 15.811879 76.61% error_already_set_restore 669881 1.048 15.645176 82.63% pr1895_original_foo 318243 1.059 33.290806 77.39% master @ commit ad146b2a1877e8ba3803f94a7837969835a297a7 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,729540,1.061,14.539876 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,681476,1.040,15.260282 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,508038,1.049,20.640525 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,555578,1.052,18.933288 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,244113,1.050,43.018168 PASSED ============================== 5 passed in 12.38s ============================== pr1895 @ commit 8dff51d12e4af11aff415ee966070368fe606664 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,736981,1.054,14.295685 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,685820,1.045,15.237399 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,661374,1.046,15.811879 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,669881,1.048,15.645176 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,318243,1.059,33.290806 PASSED ============================== 5 passed in 12.40s ============================== clang++ -o pybind11/tests/test_perf_error_already_set.os -c -std=c++17 -fPIC -fvisibility=hidden -Os -flto -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wnon-virtual-dtor -Wunused-result -isystem /usr/include/python3.9 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_perf_error_already_set.cpp clang++ -o lib/pybind11_tests.so -shared -fPIC -Os -flto -shared ... Debian clang version 13.0.1-3+build2 Target: x86_64-pc-linux-gnu Thread model: posix * Changing call_repetitions_target_elapsed_secs to 0.1 for regular unit testing. * Adding in `recursion_depth` * Optimized ctor * Fix silly bug in recurse_first_then_call() * Add tests that have equivalent PyErr_Fetch(), PyErr_Restore() but no try-catch. * Add call_error_string to tests. Sample only recursion_depth 0, 100. * Show lazy-what speed-up in percent. * Include real_work in benchmarks. * Replace all PyErr_SetString() with generate_python_exception_with_traceback() * Better organization of test loops. * Add test_error_already_set_copy_move * Fix bug in newly added test (discovered by clang-tidy): actually use move ctor * MSVC detects the unreachable return * change test_perf_error_already_set.py back to quick mode * Inherit from std::exception (instead of std::runtime_error, which does not make sense anymore with the lazy what) * Special handling under Windows. * print with leading newline * Removing test_perf_error_already_set (copies are under https://github.com/rwgk/rwgk_tbx/commit/7765113fbb659e1ea004c5ba24fb578244bc6cfd). * Avoid gil and scope overhead if there is nothing to release. * Restore default move ctor. "member function" instead of "function" (note that "method" is Python terminology). * Delete error_already_set copy ctor. * Make restore() non-const again to resolve clang-tidy failure (still experimenting). * Bring back error_already_set copy ctor, to see if that resolves the 4 MSVC test failures. * Add noexcept to error_already_set copy & move ctors (as suggested by @skylion007 IIUC). * Trying one-by-one noexcept copy ctor for old compilers. * Add back test covering copy ctor. Add another simple test that exercises the copy ctor. * Exclude more older compilers from using the noexcept = default ctors. (The tests in the previous commit exposed that those are broken.) * Factor out & reuse gil_scoped_acquire_local as gil_scoped_acquire_simple * Guard gil_scoped_acquire_simple by _Py_IsFinalizing() check. * what() GIL safety * clang-tidy & Python 3.6 fixes * Use `gil_scoped_acquire` in dtor, copy ctor, `what()`. Remove `_Py_IsFinalizing()` checks (they are racy: https://github.com/python/cpython/pull/28525). * Remove error_scope from copy ctor. * Add `error_scope` to `get_internals()`, to cover the situation that `get_internals()` is called from the `error_already_set` dtor while a new Python error is in flight already. Also backing out `gil_scoped_acquire_simple` change. * Add `FlakyException` tests with failure triggers in `__init__` and `__str__` THIS IS STILL A WORK IN PROGRESS. This commit is only an important resting point. This commit is a first attempt at addressing the observation that `PyErr_NormalizeException()` completely replaces the original exception if `__init__` fails. This can be very confusing even in small applications, and extremely confusing in large ones. * Tweaks to resolve Py 3.6 and PyPy CI failures. * Normalize Python exception immediately in error_already_set ctor. For background see: https://github.com/pybind/pybind11/pull/1895#issuecomment-1135304081 * Fix oversights based on CI failures (copy & move ctor initialization). * Move @pytest.mark.xfail("env.PYPY") after @pytest.mark.parametrize(...) * Use @pytest.mark.skipif (xfail does not work for segfaults, of course). * Remove unused obj_class_name_or() function (it was added only under this PR). * Remove already obsolete C++ comments and code that were added only under this PR. * Slightly better (newly added) comments. * Factor out detail::error_fetch_and_normalize. Preparation for producing identical results from error_already_set::what() and detail::error_string(). Note that this is a very conservative refactoring. It would be much better to first move detail::error_string into detail/error_string.h * Copy most of error_string() code to new error_fetch_and_normalize::complete_lazy_error_string() * Remove all error_string() code from detail/type_caster_base.h. Note that this commit includes a subtle bug fix: previously error_string() restored the Python error, which will upset pybind11_fail(). This never was a problem in practice because the two PyType_Ready() calls in detail/class.h do not usually fail. * Return const std::string& instead of const char * and move error_string() to pytypes.h * Remove gil_scope_acquire from error_fetch_and_normalize, add back to error_already_set * Better handling of FlakyException __str__ failure. * Move error_fetch_and_normalize::complete_lazy_error_string() implementation from pybind11.h to pytypes.h * Add error_fetch_and_normalize::release_py_object_references() and use from error_already_set dtor. * Use shared_ptr for m_fetched_error => 1. non-racy, copy ctor that does not need the GIL; 2. enables guard against duplicate restore() calls. * Add comments. * Trivial renaming of a newly introduced member function. * Workaround for PyPy * Bug fix (oversight). Only valgrind got this one. * Use shared_ptr custom deleter for m_fetched_error in error_already_set. This enables removing the dtor, copy ctor, move ctor completely. * Further small simplification. With the GIL held, simply deleting the raw_ptr takes care of everything. * IWYU cleanup ``` iwyu version: include-what-you-use 0.17 based on Debian clang version 13.0.1-3+build2 ``` Command used: ``` iwyu -c -std=c++17 -DPYBIND11_TEST_BOOST -Iinclude/pybind11 -I/usr/include/python3.9 -I/usr/include/eigen3 include/pybind11/pytypes.cpp ``` pytypes.cpp is a temporary file: `#include "pytypes.h"` The raw output is very long and noisy. I decided to use `#include <cstddef>` instead of `#include <cstdio>` for `std::size_t` (iwyu sticks to the manual choice). I ignored all iwyu suggestions that are indirectly covered by `#include <Python.h>`. I manually verified that all added includes are actually needed. Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
2022-06-02 23:17:38 +00:00
assert lines[:3] == [
"pybind11::error_already_set: MISMATCH of original and normalized active exception types:"
" ORIGINAL FlakyException REPLACED BY ValueError: triggered_failure_point_init",
"",
"At:",
]
# Checking the first two lines of the traceback as formatted in error_string():
assert "test_exceptions.py(" in lines[3]
assert lines[3].endswith("): __init__")
assert lines[4].endswith("): test_flaky_exception_failure_point_init")
def test_flaky_exception_failure_point_str():
error_already_set::what() is now constructed lazily (#1895) * error_already_set::what() is now constructed lazily Prior to this commit throwing error_already_set was expensive due to the eager construction of the error string (which required traversing the Python stack). See #1853 for more context and an alternative take on the issue. Note that error_already_set no longer inherits from std::runtime_error because the latter has no default constructor. * Do not attempt to normalize if no exception occurred This is not supported on PyPy-2.7 5.8.0. * Extract exception name via tp_name This is faster than dynamically looking up __name__ via GetAttrString. Note though that the runtime of the code throwing an error_already_set will be dominated by stack unwinding so the improvement will not be noticeable. Before: 396 ns ± 0.913 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) After: 277 ns ± 0.549 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) Benchmark: const std::string foo() { PyErr_SetString(PyExc_KeyError, ""); const std::string &s = py::detail::error_string(); PyErr_Clear(); return s; } PYBIND11_MODULE(foo, m) { m.def("foo", &::foo); } * Reverted error_already_set to subclass std::runtime_error * Revert "Extract exception name via tp_name" The implementation of __name__ is slightly more complex than that. It handles the module name prefix, and heap-allocated types. We could port it to pybind11 later on but for now it seems like an overkill. This reverts commit f1435c7e6b068a1ed13ebd3db597ea3bb15aa398. * Cosmit following @YannickJadoul's comments Note that detail::error_string() no longer calls PyException_SetTraceback as it is unncessary for pretty-printing the exception. * Fixed PyPy build * Moved normalization to error_already_set ctor * Fix merge bugs * Fix more merge errors * Improve formatting * Improve error message in rare case * Revert back if statements * Fix clang-tidy * Try removing mutable * Does build_mode release fix it * Set to Debug to expose segfault * Fix remove set error string * Do not run error_string() more than once * Trying setting the tracebackk to the value * guard if m_type is null * Try to debug PGI * One last try for PGI * Does reverting this fix PyPy * Reviewer suggestions * Remove unnecessary initialization * Add noexcept move and explicit fail throw * Optimize error_string creation * Fix typo * Revert noexcept * Fix merge conflict error * Abuse assignment operator * Revert operator abuse * See if we still need debug * Remove unnecessary mutable * Report "FATAL failure building pybind11::error_already_set error_string" and terminate process. * Try specifying noexcept again * Try explicit ctor * default ctor is noexcept too * Apply reviewer suggestions, simplify code, and make helper method private * Remove unnecessary include * Clang-Tidy fix * detail::obj_class_name(), fprintf with [STDERR], [STDOUT] tags, polish comments * consistently check m_lazy_what.empty() also in production builds * Make a comment slightly less ambiguous. * Bug fix: Remove `what();` from `restore()`. It sure would need to be guarded by `if (m_type)`, otherwise `what()` fails and masks that no error was set (see update unit test). But since `error_already_set` is copyable, there is no point in releasing m_type, m_value, m_trace, therefore we can just as well avoid the runtime overhead of force-building `m_lazy_what`, it may never be used. * Replace extremely opaque (unhelpful) error message with a truthful reflection of what we know. * Fix clang-tidy error [performance-move-constructor-init]. * Make expected error message less specific. * Various changes. * bug fix: error_string(PyObject **, ...) * Putting back the two eager PyErr_NormalizeException() calls. * Change error_already_set() to call pybind11_fail() if the Python error indicator not set. The net result is that a std::runtime_error is thrown instead of error_already_set, but all tests pass as is. * Remove mutable (fixes oversight in the previous commit). * Normalize the exception only locally in error_string(). Python 3.6 & 3.7 test failures expected. This is meant for benchmarking, to determine if it is worth the trouble looking into the failures. * clang-tidy: use auto * Use `gil_scoped_acquire_local` in `error_already_set` destructor. See long comment. * For Python < 3.8: `PyErr_NormalizeException` before `PyErr_WriteUnraisable` * Go back to replacing the held Python exception with then normalized exception, if & when needed. Consistently document the side-effect. * Slightly rewording comment. (There were also other failures.) * Add 1-line comment for obj_class_name() * Benchmark code, with results in this commit message. function #calls test time [s] μs / call master pure_unwind 729540 1.061 14.539876 err_set_unwind_err_clear 681476 1.040 15.260282 err_set_error_already_set 508038 1.049 20.640525 error_already_set_restore 555578 1.052 18.933288 pr1895_original_foo 244113 1.050 43.018168 PR / master PR #1895 pure_unwind 736981 1.054 14.295685 98.32% err_set_unwind_err_clear 685820 1.045 15.237399 99.85% err_set_error_already_set 661374 1.046 15.811879 76.61% error_already_set_restore 669881 1.048 15.645176 82.63% pr1895_original_foo 318243 1.059 33.290806 77.39% master @ commit ad146b2a1877e8ba3803f94a7837969835a297a7 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,729540,1.061,14.539876 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,681476,1.040,15.260282 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,508038,1.049,20.640525 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,555578,1.052,18.933288 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,244113,1.050,43.018168 PASSED ============================== 5 passed in 12.38s ============================== pr1895 @ commit 8dff51d12e4af11aff415ee966070368fe606664 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,736981,1.054,14.295685 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,685820,1.045,15.237399 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,661374,1.046,15.811879 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,669881,1.048,15.645176 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,318243,1.059,33.290806 PASSED ============================== 5 passed in 12.40s ============================== clang++ -o pybind11/tests/test_perf_error_already_set.os -c -std=c++17 -fPIC -fvisibility=hidden -Os -flto -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wnon-virtual-dtor -Wunused-result -isystem /usr/include/python3.9 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_perf_error_already_set.cpp clang++ -o lib/pybind11_tests.so -shared -fPIC -Os -flto -shared ... Debian clang version 13.0.1-3+build2 Target: x86_64-pc-linux-gnu Thread model: posix * Changing call_repetitions_target_elapsed_secs to 0.1 for regular unit testing. * Adding in `recursion_depth` * Optimized ctor * Fix silly bug in recurse_first_then_call() * Add tests that have equivalent PyErr_Fetch(), PyErr_Restore() but no try-catch. * Add call_error_string to tests. Sample only recursion_depth 0, 100. * Show lazy-what speed-up in percent. * Include real_work in benchmarks. * Replace all PyErr_SetString() with generate_python_exception_with_traceback() * Better organization of test loops. * Add test_error_already_set_copy_move * Fix bug in newly added test (discovered by clang-tidy): actually use move ctor * MSVC detects the unreachable return * change test_perf_error_already_set.py back to quick mode * Inherit from std::exception (instead of std::runtime_error, which does not make sense anymore with the lazy what) * Special handling under Windows. * print with leading newline * Removing test_perf_error_already_set (copies are under https://github.com/rwgk/rwgk_tbx/commit/7765113fbb659e1ea004c5ba24fb578244bc6cfd). * Avoid gil and scope overhead if there is nothing to release. * Restore default move ctor. "member function" instead of "function" (note that "method" is Python terminology). * Delete error_already_set copy ctor. * Make restore() non-const again to resolve clang-tidy failure (still experimenting). * Bring back error_already_set copy ctor, to see if that resolves the 4 MSVC test failures. * Add noexcept to error_already_set copy & move ctors (as suggested by @skylion007 IIUC). * Trying one-by-one noexcept copy ctor for old compilers. * Add back test covering copy ctor. Add another simple test that exercises the copy ctor. * Exclude more older compilers from using the noexcept = default ctors. (The tests in the previous commit exposed that those are broken.) * Factor out & reuse gil_scoped_acquire_local as gil_scoped_acquire_simple * Guard gil_scoped_acquire_simple by _Py_IsFinalizing() check. * what() GIL safety * clang-tidy & Python 3.6 fixes * Use `gil_scoped_acquire` in dtor, copy ctor, `what()`. Remove `_Py_IsFinalizing()` checks (they are racy: https://github.com/python/cpython/pull/28525). * Remove error_scope from copy ctor. * Add `error_scope` to `get_internals()`, to cover the situation that `get_internals()` is called from the `error_already_set` dtor while a new Python error is in flight already. Also backing out `gil_scoped_acquire_simple` change. * Add `FlakyException` tests with failure triggers in `__init__` and `__str__` THIS IS STILL A WORK IN PROGRESS. This commit is only an important resting point. This commit is a first attempt at addressing the observation that `PyErr_NormalizeException()` completely replaces the original exception if `__init__` fails. This can be very confusing even in small applications, and extremely confusing in large ones. * Tweaks to resolve Py 3.6 and PyPy CI failures. * Normalize Python exception immediately in error_already_set ctor. For background see: https://github.com/pybind/pybind11/pull/1895#issuecomment-1135304081 * Fix oversights based on CI failures (copy & move ctor initialization). * Move @pytest.mark.xfail("env.PYPY") after @pytest.mark.parametrize(...) * Use @pytest.mark.skipif (xfail does not work for segfaults, of course). * Remove unused obj_class_name_or() function (it was added only under this PR). * Remove already obsolete C++ comments and code that were added only under this PR. * Slightly better (newly added) comments. * Factor out detail::error_fetch_and_normalize. Preparation for producing identical results from error_already_set::what() and detail::error_string(). Note that this is a very conservative refactoring. It would be much better to first move detail::error_string into detail/error_string.h * Copy most of error_string() code to new error_fetch_and_normalize::complete_lazy_error_string() * Remove all error_string() code from detail/type_caster_base.h. Note that this commit includes a subtle bug fix: previously error_string() restored the Python error, which will upset pybind11_fail(). This never was a problem in practice because the two PyType_Ready() calls in detail/class.h do not usually fail. * Return const std::string& instead of const char * and move error_string() to pytypes.h * Remove gil_scope_acquire from error_fetch_and_normalize, add back to error_already_set * Better handling of FlakyException __str__ failure. * Move error_fetch_and_normalize::complete_lazy_error_string() implementation from pybind11.h to pytypes.h * Add error_fetch_and_normalize::release_py_object_references() and use from error_already_set dtor. * Use shared_ptr for m_fetched_error => 1. non-racy, copy ctor that does not need the GIL; 2. enables guard against duplicate restore() calls. * Add comments. * Trivial renaming of a newly introduced member function. * Workaround for PyPy * Bug fix (oversight). Only valgrind got this one. * Use shared_ptr custom deleter for m_fetched_error in error_already_set. This enables removing the dtor, copy ctor, move ctor completely. * Further small simplification. With the GIL held, simply deleting the raw_ptr takes care of everything. * IWYU cleanup ``` iwyu version: include-what-you-use 0.17 based on Debian clang version 13.0.1-3+build2 ``` Command used: ``` iwyu -c -std=c++17 -DPYBIND11_TEST_BOOST -Iinclude/pybind11 -I/usr/include/python3.9 -I/usr/include/eigen3 include/pybind11/pytypes.cpp ``` pytypes.cpp is a temporary file: `#include "pytypes.h"` The raw output is very long and noisy. I decided to use `#include <cstddef>` instead of `#include <cstdio>` for `std::size_t` (iwyu sticks to the manual choice). I ignored all iwyu suggestions that are indirectly covered by `#include <Python.h>`. I manually verified that all added includes are actually needed. Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
2022-06-02 23:17:38 +00:00
what, py_err_set_after_what = m.error_already_set_what(
FlakyException, ("failure_point_str",)
)
assert not py_err_set_after_what
lines = what.splitlines()
if env.PYPY and len(lines) == 3:
n = 3 # Traceback is missing.
else:
n = 5
assert (
lines[:n]
== [
"FlakyException: <MESSAGE UNAVAILABLE DUE TO ANOTHER EXCEPTION>",
"",
"MESSAGE UNAVAILABLE DUE TO EXCEPTION: ValueError: triggered_failure_point_str",
"",
"At:",
][:n]
)
def test_cross_module_interleaved_error_already_set():
with pytest.raises(RuntimeError) as excinfo:
m.test_cross_module_interleaved_error_already_set()
assert str(excinfo.value) in (
"2nd error.", # Almost all platforms.
"RuntimeError: 2nd error.", # Some PyPy builds (seen under macOS).
)
error_already_set::what() is now constructed lazily (#1895) * error_already_set::what() is now constructed lazily Prior to this commit throwing error_already_set was expensive due to the eager construction of the error string (which required traversing the Python stack). See #1853 for more context and an alternative take on the issue. Note that error_already_set no longer inherits from std::runtime_error because the latter has no default constructor. * Do not attempt to normalize if no exception occurred This is not supported on PyPy-2.7 5.8.0. * Extract exception name via tp_name This is faster than dynamically looking up __name__ via GetAttrString. Note though that the runtime of the code throwing an error_already_set will be dominated by stack unwinding so the improvement will not be noticeable. Before: 396 ns ± 0.913 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) After: 277 ns ± 0.549 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each) Benchmark: const std::string foo() { PyErr_SetString(PyExc_KeyError, ""); const std::string &s = py::detail::error_string(); PyErr_Clear(); return s; } PYBIND11_MODULE(foo, m) { m.def("foo", &::foo); } * Reverted error_already_set to subclass std::runtime_error * Revert "Extract exception name via tp_name" The implementation of __name__ is slightly more complex than that. It handles the module name prefix, and heap-allocated types. We could port it to pybind11 later on but for now it seems like an overkill. This reverts commit f1435c7e6b068a1ed13ebd3db597ea3bb15aa398. * Cosmit following @YannickJadoul's comments Note that detail::error_string() no longer calls PyException_SetTraceback as it is unncessary for pretty-printing the exception. * Fixed PyPy build * Moved normalization to error_already_set ctor * Fix merge bugs * Fix more merge errors * Improve formatting * Improve error message in rare case * Revert back if statements * Fix clang-tidy * Try removing mutable * Does build_mode release fix it * Set to Debug to expose segfault * Fix remove set error string * Do not run error_string() more than once * Trying setting the tracebackk to the value * guard if m_type is null * Try to debug PGI * One last try for PGI * Does reverting this fix PyPy * Reviewer suggestions * Remove unnecessary initialization * Add noexcept move and explicit fail throw * Optimize error_string creation * Fix typo * Revert noexcept * Fix merge conflict error * Abuse assignment operator * Revert operator abuse * See if we still need debug * Remove unnecessary mutable * Report "FATAL failure building pybind11::error_already_set error_string" and terminate process. * Try specifying noexcept again * Try explicit ctor * default ctor is noexcept too * Apply reviewer suggestions, simplify code, and make helper method private * Remove unnecessary include * Clang-Tidy fix * detail::obj_class_name(), fprintf with [STDERR], [STDOUT] tags, polish comments * consistently check m_lazy_what.empty() also in production builds * Make a comment slightly less ambiguous. * Bug fix: Remove `what();` from `restore()`. It sure would need to be guarded by `if (m_type)`, otherwise `what()` fails and masks that no error was set (see update unit test). But since `error_already_set` is copyable, there is no point in releasing m_type, m_value, m_trace, therefore we can just as well avoid the runtime overhead of force-building `m_lazy_what`, it may never be used. * Replace extremely opaque (unhelpful) error message with a truthful reflection of what we know. * Fix clang-tidy error [performance-move-constructor-init]. * Make expected error message less specific. * Various changes. * bug fix: error_string(PyObject **, ...) * Putting back the two eager PyErr_NormalizeException() calls. * Change error_already_set() to call pybind11_fail() if the Python error indicator not set. The net result is that a std::runtime_error is thrown instead of error_already_set, but all tests pass as is. * Remove mutable (fixes oversight in the previous commit). * Normalize the exception only locally in error_string(). Python 3.6 & 3.7 test failures expected. This is meant for benchmarking, to determine if it is worth the trouble looking into the failures. * clang-tidy: use auto * Use `gil_scoped_acquire_local` in `error_already_set` destructor. See long comment. * For Python < 3.8: `PyErr_NormalizeException` before `PyErr_WriteUnraisable` * Go back to replacing the held Python exception with then normalized exception, if & when needed. Consistently document the side-effect. * Slightly rewording comment. (There were also other failures.) * Add 1-line comment for obj_class_name() * Benchmark code, with results in this commit message. function #calls test time [s] μs / call master pure_unwind 729540 1.061 14.539876 err_set_unwind_err_clear 681476 1.040 15.260282 err_set_error_already_set 508038 1.049 20.640525 error_already_set_restore 555578 1.052 18.933288 pr1895_original_foo 244113 1.050 43.018168 PR / master PR #1895 pure_unwind 736981 1.054 14.295685 98.32% err_set_unwind_err_clear 685820 1.045 15.237399 99.85% err_set_error_already_set 661374 1.046 15.811879 76.61% error_already_set_restore 669881 1.048 15.645176 82.63% pr1895_original_foo 318243 1.059 33.290806 77.39% master @ commit ad146b2a1877e8ba3803f94a7837969835a297a7 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,729540,1.061,14.539876 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,681476,1.040,15.260282 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,508038,1.049,20.640525 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,555578,1.052,18.933288 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,244113,1.050,43.018168 PASSED ============================== 5 passed in 12.38s ============================== pr1895 @ commit 8dff51d12e4af11aff415ee966070368fe606664 Running tests in directory "/usr/local/google/home/rwgk/forked/pybind11/tests": ============================= test session starts ============================== platform linux -- Python 3.9.10, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini collecting ... collected 5 items test_perf_error_already_set.py::test_perf[pure_unwind] PERF pure_unwind,736981,1.054,14.295685 PASSED test_perf_error_already_set.py::test_perf[err_set_unwind_err_clear] PERF err_set_unwind_err_clear,685820,1.045,15.237399 PASSED test_perf_error_already_set.py::test_perf[err_set_error_already_set] PERF err_set_error_already_set,661374,1.046,15.811879 PASSED test_perf_error_already_set.py::test_perf[error_already_set_restore] PERF error_already_set_restore,669881,1.048,15.645176 PASSED test_perf_error_already_set.py::test_perf[pr1895_original_foo] PERF pr1895_original_foo,318243,1.059,33.290806 PASSED ============================== 5 passed in 12.40s ============================== clang++ -o pybind11/tests/test_perf_error_already_set.os -c -std=c++17 -fPIC -fvisibility=hidden -Os -flto -Wall -Wextra -Wconversion -Wcast-qual -Wdeprecated -Wnon-virtual-dtor -Wunused-result -isystem /usr/include/python3.9 -isystem /usr/include/eigen3 -DPYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX -DPYBIND11_TEST_BOOST -Ipybind11/include -I/usr/local/google/home/rwgk/forked/pybind11/include -I/usr/local/google/home/rwgk/clone/pybind11/include /usr/local/google/home/rwgk/forked/pybind11/tests/test_perf_error_already_set.cpp clang++ -o lib/pybind11_tests.so -shared -fPIC -Os -flto -shared ... Debian clang version 13.0.1-3+build2 Target: x86_64-pc-linux-gnu Thread model: posix * Changing call_repetitions_target_elapsed_secs to 0.1 for regular unit testing. * Adding in `recursion_depth` * Optimized ctor * Fix silly bug in recurse_first_then_call() * Add tests that have equivalent PyErr_Fetch(), PyErr_Restore() but no try-catch. * Add call_error_string to tests. Sample only recursion_depth 0, 100. * Show lazy-what speed-up in percent. * Include real_work in benchmarks. * Replace all PyErr_SetString() with generate_python_exception_with_traceback() * Better organization of test loops. * Add test_error_already_set_copy_move * Fix bug in newly added test (discovered by clang-tidy): actually use move ctor * MSVC detects the unreachable return * change test_perf_error_already_set.py back to quick mode * Inherit from std::exception (instead of std::runtime_error, which does not make sense anymore with the lazy what) * Special handling under Windows. * print with leading newline * Removing test_perf_error_already_set (copies are under https://github.com/rwgk/rwgk_tbx/commit/7765113fbb659e1ea004c5ba24fb578244bc6cfd). * Avoid gil and scope overhead if there is nothing to release. * Restore default move ctor. "member function" instead of "function" (note that "method" is Python terminology). * Delete error_already_set copy ctor. * Make restore() non-const again to resolve clang-tidy failure (still experimenting). * Bring back error_already_set copy ctor, to see if that resolves the 4 MSVC test failures. * Add noexcept to error_already_set copy & move ctors (as suggested by @skylion007 IIUC). * Trying one-by-one noexcept copy ctor for old compilers. * Add back test covering copy ctor. Add another simple test that exercises the copy ctor. * Exclude more older compilers from using the noexcept = default ctors. (The tests in the previous commit exposed that those are broken.) * Factor out & reuse gil_scoped_acquire_local as gil_scoped_acquire_simple * Guard gil_scoped_acquire_simple by _Py_IsFinalizing() check. * what() GIL safety * clang-tidy & Python 3.6 fixes * Use `gil_scoped_acquire` in dtor, copy ctor, `what()`. Remove `_Py_IsFinalizing()` checks (they are racy: https://github.com/python/cpython/pull/28525). * Remove error_scope from copy ctor. * Add `error_scope` to `get_internals()`, to cover the situation that `get_internals()` is called from the `error_already_set` dtor while a new Python error is in flight already. Also backing out `gil_scoped_acquire_simple` change. * Add `FlakyException` tests with failure triggers in `__init__` and `__str__` THIS IS STILL A WORK IN PROGRESS. This commit is only an important resting point. This commit is a first attempt at addressing the observation that `PyErr_NormalizeException()` completely replaces the original exception if `__init__` fails. This can be very confusing even in small applications, and extremely confusing in large ones. * Tweaks to resolve Py 3.6 and PyPy CI failures. * Normalize Python exception immediately in error_already_set ctor. For background see: https://github.com/pybind/pybind11/pull/1895#issuecomment-1135304081 * Fix oversights based on CI failures (copy & move ctor initialization). * Move @pytest.mark.xfail("env.PYPY") after @pytest.mark.parametrize(...) * Use @pytest.mark.skipif (xfail does not work for segfaults, of course). * Remove unused obj_class_name_or() function (it was added only under this PR). * Remove already obsolete C++ comments and code that were added only under this PR. * Slightly better (newly added) comments. * Factor out detail::error_fetch_and_normalize. Preparation for producing identical results from error_already_set::what() and detail::error_string(). Note that this is a very conservative refactoring. It would be much better to first move detail::error_string into detail/error_string.h * Copy most of error_string() code to new error_fetch_and_normalize::complete_lazy_error_string() * Remove all error_string() code from detail/type_caster_base.h. Note that this commit includes a subtle bug fix: previously error_string() restored the Python error, which will upset pybind11_fail(). This never was a problem in practice because the two PyType_Ready() calls in detail/class.h do not usually fail. * Return const std::string& instead of const char * and move error_string() to pytypes.h * Remove gil_scope_acquire from error_fetch_and_normalize, add back to error_already_set * Better handling of FlakyException __str__ failure. * Move error_fetch_and_normalize::complete_lazy_error_string() implementation from pybind11.h to pytypes.h * Add error_fetch_and_normalize::release_py_object_references() and use from error_already_set dtor. * Use shared_ptr for m_fetched_error => 1. non-racy, copy ctor that does not need the GIL; 2. enables guard against duplicate restore() calls. * Add comments. * Trivial renaming of a newly introduced member function. * Workaround for PyPy * Bug fix (oversight). Only valgrind got this one. * Use shared_ptr custom deleter for m_fetched_error in error_already_set. This enables removing the dtor, copy ctor, move ctor completely. * Further small simplification. With the GIL held, simply deleting the raw_ptr takes care of everything. * IWYU cleanup ``` iwyu version: include-what-you-use 0.17 based on Debian clang version 13.0.1-3+build2 ``` Command used: ``` iwyu -c -std=c++17 -DPYBIND11_TEST_BOOST -Iinclude/pybind11 -I/usr/include/python3.9 -I/usr/include/eigen3 include/pybind11/pytypes.cpp ``` pytypes.cpp is a temporary file: `#include "pytypes.h"` The raw output is very long and noisy. I decided to use `#include <cstddef>` instead of `#include <cstdio>` for `std::size_t` (iwyu sticks to the manual choice). I ignored all iwyu suggestions that are indirectly covered by `#include <Python.h>`. I manually verified that all added includes are actually needed. Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
2022-06-02 23:17:38 +00:00
def test_error_already_set_double_restore():
m.test_error_already_set_double_restore(True) # dry_run
with pytest.raises(RuntimeError) as excinfo:
m.test_error_already_set_double_restore(False)
assert str(excinfo.value) == (
"Internal error: pybind11::detail::error_fetch_and_normalize::restore()"
" called a second time. ORIGINAL ERROR: ValueError: Random error."
)
For PyPy only, re-enable old behavior (runs the risk of masking bugs) (#4079) * For PyPy only, re-enable old behavior (likely to mask bugs), to avoid segfault with unknown root cause. Change prompted by https://github.com/pybind/pybind11/issues/4075 * Undo the change in tests/test_exceptions.py I turns out (I forgot) that PyPy segfaults in `test_flaky_exception_failure_point_init` already before the `MISMATCH` code path is reached: https://github.com/pybind/pybind11/runs/7383663596 ``` RPython traceback: test_exceptions.py .......X.........Error in cpyext, CPython compatibility layer: File "pypy_module_cpyext.c", line 14052, in wrapper_second_level__star_3_1 File "pypy_module_cpyext_1.c", line 35750, in not_supposed_to_fail Fatal Python error: Segmentation fault Stack (most recent call first, approximate line numbers): File "/home/runner/work/pybind11/pybind11/tests/test_exceptions.py", line 306 in test_flaky_exception_failure_point_init The function PyErr_NormalizeException was not supposed to fail File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/python.py", line 185 in pytest_pyfunc_call File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_callers.py", line 9 in _multicall File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_manager.py", line 77 in _hookexec File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_hooks.py", line 244 in __call__ File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/python.py", line 1716 in runtest File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 159 in pytest_runtest_call File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_callers.py", line 9 in _multicall File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_manager.py", line 77 in _hookexec Fatal error in cpyext, CPython compatibility layer, calling PyErr_NormalizeException Either report a bug or consider not using this particular extension <SystemError object at 0x7fcc8cea6868> File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_hooks.py", line 244 in __call__ File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 261 in <lambda> File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 317 in from_call File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 246 in call_runtest_hook File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 218 in call_and_report File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 118 in runtestprotocol File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/runner.py", line 110 in pytest_runtest_protocol File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_callers.py", line 9 in _multicall File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_manager.py", line 77 in _hookexec File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_hooks.py", line 244 in __call__ File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/main.py", line 335 in pytest_runtestloop File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_callers.py", line 9 in _multicall File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_manager.py", line 77 in _hookexec File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_hooks.py", line 244 in __call__ File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/main.py", line 318 in _main File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/main.py", line 255 in wrap_session File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/main.py", line 314 in pytest_cmdline_main File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_callers.py", line 9 in _multicall File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_manager.py", line 77 in _hookexec File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pluggy/_hooks.py", line 244 in __call__ File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/config/__init__.py", line 133 in main File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/_pytest/config/__init__.py", line 181 in console_main File "/opt/hostedtoolcache/PyPy/3.7.13/x64/site-packages/pytest/__main__.py", line 1 in <module> File "/opt/hostedtoolcache/PyPy/3.7.13/x64/lib-python/3/runpy.py", line 62 in _run_code File "/opt/hostedtoolcache/PyPy/3.7.13/x64/lib-python/3/runpy.py", line 170 in _run_module_as_main File "<builtin>/app_main.py", line 109 in run_toplevel File "<builtin>/app_main.py", line 652 in run_command_line File "<builtin>/app_main.py", line 996 in entry_point Segmentation fault (core dumped) ``` * Add test_pypy_oserror_normalization * Disable new `PYPY_VERSION` `#if`, to verify that the new test actually fails. * Restore PYPY_VERSION workaround and update comment to reflect what was learned. * [ci skip] Fix trivial oversight in comment.
2022-07-21 13:38:00 +00:00
def test_pypy_oserror_normalization():
# https://github.com/pybind/pybind11/issues/4075
what = m.test_pypy_oserror_normalization()
assert "this_filename_must_not_exist" in what
def test_fn_cast_int_exception():
with pytest.raises(RuntimeError) as excinfo:
m.test_fn_cast_int(lambda: None)
assert str(excinfo.value).startswith(
"Unable to cast Python instance of type <class 'NoneType'> to C++ type"
)