Add and document py::error_already_set::discard_as_unraisable()

To deal with exceptions that hit destructors or other noexcept functions.

Includes fixes to support Python 2.7 and extends documentation on
error handling.

@virtuald and @YannickJadoul both contributed to this PR.
This commit is contained in:
James R. Barlow 2020-08-08 03:07:14 -07:00 committed by Ralf W. Grosse-Kunstleve
parent a876aac2cf
commit 3618bea2aa
5 changed files with 154 additions and 3 deletions

View File

@ -559,6 +559,44 @@ crucial that instances are deallocated on the C++ side to avoid memory leaks.
py::class_<MyClass, std::unique_ptr<MyClass, py::nodelete>>(m, "MyClass")
.def(py::init<>())
.. _destructors_that_call_python:
Destructors that call Python
============================
If a Python function is invoked from a C++ destructor, an exception may be thrown
of type :class:`error_already_set`. If this error is thrown out of a class destructor,
``std::terminate()`` will be called, terminating the process. Class destructors
must catch all exceptions of type :class:`error_already_set` to discard the Python
exception using :func:`error_already_set::discard_as_unraisable`.
Every Python function should be treated as *possibly throwing*. When a Python generator
stops yielding items, Python will throw a ``StopIteration`` exception, which can pass
though C++ destructors if the generator's stack frame holds the last reference to C++
objects.
For more information, see :ref:`the documentation on exceptions <unraisable_exceptions>`.
.. code-block:: cpp
class MyClass {
public:
~MyClass() {
try {
py::print("Even printing is dangerous in a destructor");
py::exec("raise ValueError('This is an unraisable exception')");
} catch (py::error_already_set &e) {
// error_context should be information about where/why the occurred,
// e.g. use __func__ to get the name of the current function
e.discard_as_unraisable(__func__);
}
}
};
.. note::
pybind11 does not support C++ destructors marked ``noexcept(false)``.
.. _implicit_conversions:
Implicit conversions

View File

@ -53,9 +53,15 @@ exceptions:
| | a Python exception back to Python. |
+--------------------------------------+--------------------------------------+
When a Python function invoked from C++ throws an exception, it is converted
into a C++ exception of type :class:`error_already_set` whose string payload
contains a textual summary.
When a Python function invoked from C++ throws an exception, pybind11 will convert
it into a C++ exception of type :class:`error_already_set` whose string payload
contains a textual summary. If you call the Python C-API directly, and it
returns an error, you should ``throw py::error_already_set();``, which allows
pybind11 to deal with the exception and pass it back to the Python interpreter.
(Another option is to call ``PyErr_Clear`` in the
`Python C-API <https://docs.python.org/3/c-api/exceptions.html#c.PyErr_Clear>`_
to clear the error. The Python error must be thrown or cleared, or Python/pybind11
will be left in an invalid state.)
There is also a special exception :class:`cast_error` that is thrown by
:func:`handle::call` when the input arguments cannot be converted to Python
@ -142,3 +148,43 @@ section.
Exceptions that you do not plan to handle should simply not be caught, or
may be explicitly (re-)thrown to delegate it to the other,
previously-declared existing exception translators.
.. _unraisable_exceptions:
Handling unraisable exceptions
==============================
If a Python function invoked from a C++ destructor or any function marked
``noexcept(true)`` (collectively, "noexcept functions") throws an exception, there
is no way to propagate the exception, as such functions may not throw at
run-time.
Neither Python nor C++ allow exceptions raised in a noexcept function to propagate. In
Python, an exception raised in a class's ``__del__`` method is logged as an
unraisable error. In Python 3.8+, a system hook is triggered and an auditing
event is logged. In C++, ``std::terminate()`` is called to abort immediately.
Any noexcept function should have a try-catch block that traps
class:`error_already_set` (or any other exception that can occur). Note that pybind11
wrappers around Python exceptions such as :class:`pybind11::value_error` are *not*
Python exceptions; they are C++ exceptions that pybind11 catches and converts to
Python exceptions. Noexcept functions cannot propagate these exceptions either.
You can convert them to Python exceptions and then discard as unraisable.
.. code-block:: cpp
void nonthrowing_func() noexcept(true) {
try {
// ...
} catch (py::error_already_set &eas) {
// Discard the Python error using Python APIs, using the C++ magic
// variable __func__. Python already knows the type and value and of the
// exception object.
eas.discard_as_unraisable(__func__);
} catch (const std::exception &e) {
// Log and discard C++ exceptions.
// (We cannot use discard_as_unraisable, since we have a generic C++
// exception, not an exception that originated from Python.)
third_party::log(e);
}
}

View File

@ -337,6 +337,20 @@ public:
/// error variables (but the `.what()` string is still available).
void restore() { PyErr_Restore(m_type.release().ptr(), m_value.release().ptr(), m_trace.release().ptr()); }
/// If it is impossible to raise the currently-held error, such as in destructor, we can write
/// it out using Python's unraisable hook (sys.unraisablehook). The error context should be
/// some object whose repr() helps identify the location of the error. Python already knows the
/// type and value of the error, so there is no need to repeat that. For example, __func__ could
/// be helpful. After this call, the current object no longer stores the error variables,
/// and neither does Python.
void discard_as_unraisable(object err_context) {
restore();
PyErr_WriteUnraisable(err_context.ptr());
}
void discard_as_unraisable(const char *err_context) {
discard_as_unraisable(reinterpret_steal<object>(PYBIND11_FROM_STRING(err_context)));
}
// Does nothing; provided for backwards compatibility.
PYBIND11_DEPRECATED("Use of error_already_set.clear() is deprecated")
void clear() {}

View File

@ -65,6 +65,25 @@ struct PythonCallInDestructor {
py::dict d;
};
struct PythonAlreadySetInDestructor {
PythonAlreadySetInDestructor(const py::str &s) : s(s) {}
~PythonAlreadySetInDestructor() {
py::dict foo;
try {
// Assign to a py::object to force read access of nonexistent dict entry
py::object o = foo["bar"];
}
catch (py::error_already_set& ex) {
ex.discard_as_unraisable(s);
}
}
py::str s;
};
TEST_SUBMODULE(exceptions, m) {
m.def("throw_std_exception", []() {
throw std::runtime_error("This exception was intentionally thrown.");
@ -183,6 +202,11 @@ TEST_SUBMODULE(exceptions, m) {
return false;
});
m.def("python_alreadyset_in_destructor", [](py::str s) {
PythonAlreadySetInDestructor alreadyset_in_destructor(s);
return true;
});
// test_nested_throws
m.def("try_catch", [m](py::object exc_type, py::function f, py::args args) {
try { f(*args); }

View File

@ -1,4 +1,6 @@
# -*- coding: utf-8 -*-
import sys
import pytest
from pybind11_tests import exceptions as m
@ -48,6 +50,33 @@ def test_python_call_in_catch():
assert d["good"] is True
def test_python_alreadyset_in_destructor(monkeypatch, capsys):
hooked = False
triggered = [False] # mutable, so Python 2.7 closure can modify it
if hasattr(sys, 'unraisablehook'): # Python 3.8+
hooked = True
default_hook = sys.unraisablehook
def hook(unraisable_hook_args):
exc_type, exc_value, exc_tb, err_msg, obj = unraisable_hook_args
if obj == 'already_set demo':
triggered[0] = True
default_hook(unraisable_hook_args)
return
# Use monkeypatch so pytest can apply and remove the patch as appropriate
monkeypatch.setattr(sys, 'unraisablehook', hook)
assert m.python_alreadyset_in_destructor('already_set demo') is True
if hooked:
assert triggered[0] is True
_, captured_stderr = capsys.readouterr()
# Error message is different in Python 2 and 3, check for words that appear in both
assert 'ignored' in captured_stderr and 'already_set demo' in captured_stderr
def test_exception_matches():
assert m.exception_matches()
assert m.exception_matches_base()