Add py::set_error(), use in updated py::exception<> documentation (#4772)

* Copy clang 17 compatibility fixes from PR #4762 to a separate PR.

* static py::exception<> -> static py::handle

* Add `py::set_error()` but also try the suggestion of @malfet (https://github.com/pytorch/pytorch/pull/106401#pullrequestreview-1559961407).

* clang 17 compatibility fixes (#4767)

* Copy clang 17 compatibility fixes from PR #4762 to a separate PR.

* Add gcc:13 C++20

* Add silkeh/clang:16-bullseye C++20

* chore(deps): update pre-commit hooks (#4770)

updates:
- [github.com/psf/black: 23.3.0 → 23.7.0](https://github.com/psf/black/compare/23.3.0...23.7.0)
- [github.com/astral-sh/ruff-pre-commit: v0.0.276 → v0.0.281](https://github.com/astral-sh/ruff-pre-commit/compare/v0.0.276...v0.0.281)
- [github.com/asottile/blacken-docs: 1.14.0 → 1.15.0](https://github.com/asottile/blacken-docs/compare/1.14.0...1.15.0)

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* docs: Remove upper bound on pybind11 in example pyproject.toml for setuptools (#4774)

* docs: Remove upper bound on pybind11 in example pyproject.toml for setuptools

* Update docs/compiling.rst

---------

Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>

* Provide better type hints for a variety of generic types (#4259)

* Provide better type hints for a variety of generic types

* Makes better documentation
* tuple, dict, list, set, function

* Move to py::typing

* style: pre-commit fixes

* Update copyright line with correct year and actual author. The author information was copy-pasted from the git log output.

---------

Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>

* Use `py::set_error()` everywhere possible (only one special case, in common.h).
Overload `py::set_error(py::handle, py::handle)`.
Change back to `static py::handle exc = ... .release();`
Deprecate `py::exception<>::operator()`

* Add `PYBIND11_WARNING_DISABLE` for INTEL and MSVC (and sort alphabetically).

* `PYBIND11_WARNING_DISABLE_INTEL(10441)` does not work.

For ICC only, falling back to the recommended `py::set_error()` to keep the testing simple.

It is troublesome to add `--diag-disable=10441` specifically for test_exceptions.cpp, even that is non-ideal because it covers the entire file, not just the one line we need it for, and the value of exercising the trivial deprecated `operator()` on this one extra platform is practically zero.

* Fix silly oversight.

* NVHPC 23.5.0 generates deprecation warnings. They are currently not treated as errors, but falling back to using `py::set_error()` to not have to deal with that distraction.

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Keto D. Zhang <keto.zhang@gmail.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Dustin Spicuzza <dustin@virtualroadside.com>

docs/advanced/exceptions.rst

@@ -127,8 +127,7 @@ before a global translator is tried.
 Inside the translator, ``std::rethrow_exception`` should be used within
 a try block to re-throw the exception.  One or more catch clauses to catch
 the appropriate exceptions should then be used with each clause using
-``PyErr_SetString`` to set a Python exception or ``ex(string)`` to set
+``py::set_error()`` (see below).
-the python exception to a custom exception type (see below).
 
 To declare a custom Python exception type, declare a ``py::exception`` variable
 and use this in the associated exception translator (note: it is often useful
@@ -142,14 +141,17 @@ standard python RuntimeError:
 
 .. code-block:: cpp
 
-    static py::exception<MyCustomException> exc(m, "MyCustomError");
+    // This is a static object, so we must leak the Python reference:
+    // It is undefined when the destructor will run, possibly only after the
+    // Python interpreter is finalized already.
+    static py::handle exc = py::exception<MyCustomException>(m, "MyCustomError").release();
     py::register_exception_translator([](std::exception_ptr p) {
         try {
             if (p) std::rethrow_exception(p);
         } catch (const MyCustomException &e) {
-            exc(e.what());
+            py::set_error(exc, e.what());
         } catch (const OtherException &e) {
-            PyErr_SetString(PyExc_RuntimeError, e.what());
+            py::set_error(PyExc_RuntimeError, e.what());
         }
     });
 
@@ -168,8 +170,7 @@ section.
 
 .. note::
 
-    Call either ``PyErr_SetString`` or a custom exception's call
+    Call ``py::set_error()`` for every exception caught in a custom exception
-    operator (``exc(string)``) for every exception caught in a custom exception
     translator.  Failure to do so will cause Python to crash with ``SystemError:
     error return without exception set``.
 
@@ -200,7 +201,7 @@ If module1 has the following translator:
         try {
             if (p) std::rethrow_exception(p);
         } catch (const std::invalid_argument &e) {
-            PyErr_SetString("module1 handled this")
+            py::set_error(PyExc_ArgumentError, "module1 handled this");
         }
       }
 
@@ -212,7 +213,7 @@ and module2 has the following similar translator:
         try {
             if (p) std::rethrow_exception(p);
         } catch (const std::invalid_argument &e) {
-            PyErr_SetString("module2 handled this")
+            py::set_error(PyExc_ArgumentError, "module2 handled this");
         }
       }
 
@@ -312,11 +313,11 @@ error protocol, which is outlined here.
 After calling the Python C API, if Python returns an error,
 ``throw py::error_already_set();``, which allows pybind11 to deal with the
 exception and pass it back to the Python interpreter. This includes calls to
-the error setting functions such as ``PyErr_SetString``.
+the error setting functions such as ``py::set_error()``.
 
 .. code-block:: cpp
 
-    PyErr_SetString(PyExc_TypeError, "C API type error demo");
+    py::set_error(PyExc_TypeError, "C API type error demo");
     throw py::error_already_set();
 
     // But it would be easier to simply...

include/pybind11/detail/class.h

@@ -375,7 +375,7 @@ extern "C" inline PyObject *pybind11_object_new(PyTypeObject *type, PyObject *,
 extern "C" inline int pybind11_object_init(PyObject *self, PyObject *, PyObject *) {
     PyTypeObject *type = Py_TYPE(self);
     std::string msg = get_fully_qualified_tp_name(type) + ": No constructor defined!";
-    PyErr_SetString(PyExc_TypeError, msg.c_str());
+    set_error(PyExc_TypeError, msg.c_str());
     return -1;
 }
 
@@ -579,7 +579,7 @@ extern "C" inline int pybind11_getbuffer(PyObject *obj, Py_buffer *view, int fla
         if (view) {
             view->obj = nullptr;
         }
-        PyErr_SetString(PyExc_BufferError, "pybind11_getbuffer(): Internal error");
+        set_error(PyExc_BufferError, "pybind11_getbuffer(): Internal error");
         return -1;
     }
     std::memset(view, 0, sizeof(Py_buffer));
@@ -587,7 +587,7 @@ extern "C" inline int pybind11_getbuffer(PyObject *obj, Py_buffer *view, int fla
     if ((flags & PyBUF_WRITABLE) == PyBUF_WRITABLE && info->readonly) {
         delete info;
         // view->obj = nullptr;  // Was just memset to 0, so not necessary
-        PyErr_SetString(PyExc_BufferError, "Writable buffer requested for readonly storage");
+        set_error(PyExc_BufferError, "Writable buffer requested for readonly storage");
         return -1;
     }
     view->obj = obj;

include/pybind11/detail/common.h

@@ -399,7 +399,7 @@ PYBIND11_WARNING_POP
         return nullptr;                                                                           \
     }                                                                                             \
     catch (const std::exception &e) {                                                             \
-        PyErr_SetString(PyExc_ImportError, e.what());                                             \
+        ::pybind11::set_error(PyExc_ImportError, e.what());                                       \
         return nullptr;                                                                           \
     }
 

include/pybind11/detail/internals.h

@@ -352,7 +352,7 @@ inline bool raise_err(PyObject *exc_type, const char *msg) {
         raise_from(exc_type, msg);
         return true;
     }
-    PyErr_SetString(exc_type, msg);
+    set_error(exc_type, msg);
     return false;
 }
 

include/pybind11/detail/type_caster_base.h

@@ -786,7 +786,7 @@ public:
         std::string tname = rtti_type ? rtti_type->name() : cast_type.name();
         detail::clean_type_id(tname);
         std::string msg = "Unregistered type : " + tname;
-        PyErr_SetString(PyExc_TypeError, msg.c_str());
+        set_error(PyExc_TypeError, msg.c_str());
         return {nullptr, nullptr};
     }
 

include/pybind11/numpy.h

@@ -1008,7 +1008,7 @@ protected:
     /// Create array from any object -- always returns a new reference
     static PyObject *raw_array(PyObject *ptr, int ExtraFlags = 0) {
         if (ptr == nullptr) {
-            PyErr_SetString(PyExc_ValueError, "cannot create a pybind11::array from a nullptr");
+            set_error(PyExc_ValueError, "cannot create a pybind11::array from a nullptr");
             return nullptr;
         }
         return detail::npy_api::get().PyArray_FromAny_(
@@ -1155,7 +1155,7 @@ protected:
     /// Create array from any object -- always returns a new reference
     static PyObject *raw_array_t(PyObject *ptr) {
         if (ptr == nullptr) {
-            PyErr_SetString(PyExc_ValueError, "cannot create a pybind11::array_t from a nullptr");
+            set_error(PyExc_ValueError, "cannot create a pybind11::array_t from a nullptr");
             return nullptr;
         }
         return detail::npy_api::get().PyArray_FromAny_(ptr,

include/pybind11/pybind11.h

@@ -694,8 +694,7 @@ protected:
         if (overloads->is_constructor) {
             if (!parent
                 || !PyObject_TypeCheck(parent.ptr(), (PyTypeObject *) overloads->scope.ptr())) {
-                PyErr_SetString(
+                set_error(PyExc_TypeError,
-                    PyExc_TypeError,
                           "__init__(self, ...) called with invalid or missing `self` argument");
                 return nullptr;
             }
@@ -1007,7 +1006,7 @@ protected:
 
                A translator may choose to do one of the following:
 
-                - catch the exception and call PyErr_SetString or PyErr_SetObject
+                - catch the exception and call py::set_error()
                   to set a standard (or custom) Python exception, or
                 - do nothing and let the exception fall through to the next translator, or
                 - delegate translation to the next translator by throwing a new type of exception.
@@ -1023,8 +1022,7 @@ protected:
                 return nullptr;
             }
 
-            PyErr_SetString(PyExc_SystemError,
+            set_error(PyExc_SystemError, "Exception escaped from default exception translator!");
-                            "Exception escaped from default exception translator!");
             return nullptr;
         }
 
@@ -1125,7 +1123,7 @@ protected:
                 raise_from(PyExc_TypeError, msg.c_str());
                 return nullptr;
             }
-            PyErr_SetString(PyExc_TypeError, msg.c_str());
+            set_error(PyExc_TypeError, msg.c_str());
             return nullptr;
         }
         if (!result) {
@@ -1138,7 +1136,7 @@ protected:
                 raise_from(PyExc_TypeError, msg.c_str());
                 return nullptr;
             }
-            PyErr_SetString(PyExc_TypeError, msg.c_str());
+            set_error(PyExc_TypeError, msg.c_str());
             return nullptr;
         }
         if (overloads->is_constructor && !self_value_and_holder.holder_constructed()) {
@@ -2528,7 +2526,7 @@ inline void register_local_exception_translator(ExceptionTranslator &&translator
 /**
  * Wrapper to generate a new Python exception type.
  *
- * This should only be used with PyErr_SetString for now.
+ * This should only be used with py::set_error() for now.
  * It is not (yet) possible to use as a py::base.
  * Template type argument is reserved for future use.
  */
@@ -2549,7 +2547,9 @@ public:
     }
 
     // Sets the current python exception to this exception object with the given message
-    void operator()(const char *message) { PyErr_SetString(m_ptr, message); }
+    PYBIND11_DEPRECATED("Please use py::set_error() instead "
+                        "(https://github.com/pybind/pybind11/pull/4772)")
+    void operator()(const char *message) const { set_error(*this, message); }
 };
 
 PYBIND11_NAMESPACE_BEGIN(detail)
@@ -2581,7 +2581,7 @@ register_exception_impl(handle scope, const char *name, handle base, bool isLoca
         try {
             std::rethrow_exception(p);
         } catch (const CppException &e) {
-            detail::get_exception_object<CppException>()(e.what());
+            set_error(detail::get_exception_object<CppException>(), e.what());
         }
     });
     return ex;

include/pybind11/pytypes.h

@@ -334,6 +334,14 @@ public:
 #endif
 };
 
+inline void set_error(const handle &type, const char *message) {
+    PyErr_SetString(type.ptr(), message);
+}
+
+inline void set_error(const handle &type, const handle &value) {
+    PyErr_SetObject(type.ptr(), value.ptr());
+}
+
 /** \rst
     Holds a reference to a Python object (with reference counting)
 

tests/cross_module_interleaved_error_already_set.cpp

@@ -16,12 +16,12 @@ namespace {
 namespace py = pybind11;
 
 void interleaved_error_already_set() {
-    PyErr_SetString(PyExc_RuntimeError, "1st error.");
+    py::set_error(PyExc_RuntimeError, "1st error.");
     try {
         throw py::error_already_set();
     } catch (const py::error_already_set &) {
         // The 2nd error could be conditional in a real application.
-        PyErr_SetString(PyExc_RuntimeError, "2nd error.");
+        py::set_error(PyExc_RuntimeError, "2nd error.");
     } // Here the 1st error is destroyed before the 2nd error is fetched.
     // The error_already_set dtor triggers a pybind11::detail::get_internals()
     // call via pybind11::gil_scoped_acquire.

tests/pybind11_cross_module_tests.cpp

@@ -31,11 +31,11 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
     // test_exceptions.py
     py::register_local_exception<LocalSimpleException>(m, "LocalSimpleException");
     m.def("raise_runtime_error", []() {
-        PyErr_SetString(PyExc_RuntimeError, "My runtime error");
+        py::set_error(PyExc_RuntimeError, "My runtime error");
         throw py::error_already_set();
     });
     m.def("raise_value_error", []() {
-        PyErr_SetString(PyExc_ValueError, "My value error");
+        py::set_error(PyExc_ValueError, "My value error");
         throw py::error_already_set();
     });
     m.def("throw_pybind_value_error", []() { throw py::value_error("pybind11 value error"); });
@@ -49,7 +49,7 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
                 std::rethrow_exception(p);
             }
         } catch (const shared_exception &e) {
-            PyErr_SetString(PyExc_KeyError, e.what());
+            py::set_error(PyExc_KeyError, e.what());
         }
     });
 
@@ -60,7 +60,7 @@ PYBIND11_MODULE(pybind11_cross_module_tests, m) {
                 std::rethrow_exception(p);
             }
         } catch (const LocalException &e) {
-            PyErr_SetString(PyExc_KeyError, e.what());
+            py::set_error(PyExc_KeyError, e.what());
         }
     });
 

tests/test_exceptions.cpp

@@ -25,6 +25,10 @@ private:
     std::string message = "";
 };
 
+class MyExceptionUseDeprecatedOperatorCall : public MyException {
+    using MyException::MyException;
+};
+
 // A type that should be translated to a standard Python exception
 class MyException2 : public std::exception {
 public:
@@ -109,8 +113,8 @@ TEST_SUBMODULE(exceptions, m) {
     m.def("throw_std_exception",
           []() { throw std::runtime_error("This exception was intentionally thrown."); });
 
-    // make a new custom exception and use it as a translation target
+    // PLEASE KEEP IN SYNC with docs/advanced/exceptions.rst
-    static py::exception<MyException> ex(m, "MyException");
+    static py::handle ex = py::exception<MyException>(m, "MyException").release();
     py::register_exception_translator([](std::exception_ptr p) {
         try {
             if (p) {
@@ -118,7 +122,32 @@ TEST_SUBMODULE(exceptions, m) {
             }
         } catch (const MyException &e) {
             // Set MyException as the active python error
-            ex(e.what());
+            py::set_error(ex, e.what());
+        }
+    });
+
+    // Same as above, but using the deprecated `py::exception<>::operator()`
+    // We want to be sure it still works, until it's removed.
+    static const auto *const exd = new py::exception<MyExceptionUseDeprecatedOperatorCall>(
+        m, "MyExceptionUseDeprecatedOperatorCall");
+    py::register_exception_translator([](std::exception_ptr p) {
+        try {
+            if (p) {
+                std::rethrow_exception(p);
+            }
+        } catch (const MyExceptionUseDeprecatedOperatorCall &e) {
+#if defined(__INTEL_COMPILER) || defined(__NVCOMPILER)
+            // It is not worth the trouble dealing with warning suppressions for these compilers.
+            // Falling back to the recommended approach to keep the test code simple.
+            py::set_error(*exd, e.what());
+#else
+            PYBIND11_WARNING_PUSH
+            PYBIND11_WARNING_DISABLE_CLANG("-Wdeprecated-declarations")
+            PYBIND11_WARNING_DISABLE_GCC("-Wdeprecated-declarations")
+            PYBIND11_WARNING_DISABLE_MSVC(4996)
+            (*exd)(e.what());
+            PYBIND11_WARNING_POP
+#endif
         }
     });
 
@@ -132,7 +161,7 @@ TEST_SUBMODULE(exceptions, m) {
             }
         } catch (const MyException2 &e) {
             // Translate this exception to a standard RuntimeError
-            PyErr_SetString(PyExc_RuntimeError, e.what());
+            py::set_error(PyExc_RuntimeError, e.what());
         }
     });
 
@@ -162,11 +191,16 @@ TEST_SUBMODULE(exceptions, m) {
                 std::rethrow_exception(p);
             }
         } catch (const MyException6 &e) {
-            PyErr_SetString(PyExc_RuntimeError, e.what());
+            py::set_error(PyExc_RuntimeError, e.what());
         }
     });
 
-    m.def("throws1", []() { throw MyException("this error should go to a custom type"); });
+    m.def("throws1",
+          []() { throw MyException("this error should go to py::exception<MyException>"); });
+    m.def("throws1d", []() {
+        throw MyExceptionUseDeprecatedOperatorCall(
+            "this error should go to py::exception<MyExceptionUseDeprecatedOperatorCall>");
+    });
     m.def("throws2",
           []() { throw MyException2("this error should go to a standard Python exception"); });
     m.def("throws3", []() { throw MyException3("this error cannot be translated"); });
@@ -222,7 +256,7 @@ TEST_SUBMODULE(exceptions, m) {
 
     m.def("throw_already_set", [](bool err) {
         if (err) {
-            PyErr_SetString(PyExc_ValueError, "foo");
+            py::set_error(PyExc_ValueError, "foo");
         }
         try {
             throw py::error_already_set();
@@ -238,7 +272,7 @@ TEST_SUBMODULE(exceptions, m) {
         }
         PyErr_Clear();
         if (err) {
-            PyErr_SetString(PyExc_ValueError, "foo");
+            py::set_error(PyExc_ValueError, "foo");
         }
         throw py::error_already_set();
     });
@@ -247,7 +281,7 @@ TEST_SUBMODULE(exceptions, m) {
         bool retval = false;
         try {
             PythonCallInDestructor set_dict_in_destructor(d);
-            PyErr_SetString(PyExc_ValueError, "foo");
+            py::set_error(PyExc_ValueError, "foo");
             throw py::error_already_set();
         } catch (const py::error_already_set &) {
             retval = true;
@@ -282,14 +316,14 @@ TEST_SUBMODULE(exceptions, m) {
     m.def("throw_should_be_translated_to_key_error", []() { throw shared_exception(); });
 
     m.def("raise_from", []() {
-        PyErr_SetString(PyExc_ValueError, "inner");
+        py::set_error(PyExc_ValueError, "inner");
         py::raise_from(PyExc_ValueError, "outer");
         throw py::error_already_set();
     });
 
     m.def("raise_from_already_set", []() {
         try {
-            PyErr_SetString(PyExc_ValueError, "inner");
+            py::set_error(PyExc_ValueError, "inner");
             throw py::error_already_set();
         } catch (py::error_already_set &e) {
             py::raise_from(e, PyExc_ValueError, "outer");
@@ -306,7 +340,7 @@ TEST_SUBMODULE(exceptions, m) {
     });
 
     m.def("error_already_set_what", [](const py::object &exc_type, const py::object &exc_value) {
-        PyErr_SetObject(exc_type.ptr(), exc_value.ptr());
+        py::set_error(exc_type, exc_value);
         std::string what = py::error_already_set().what();
         bool py_err_set_after_what = (PyErr_Occurred() != nullptr);
         PyErr_Clear();
@@ -321,7 +355,7 @@ TEST_SUBMODULE(exceptions, m) {
     });
 
     m.def("test_error_already_set_double_restore", [](bool dry_run) {
-        PyErr_SetString(PyExc_ValueError, "Random error.");
+        py::set_error(PyExc_ValueError, "Random error.");
         py::error_already_set e;
         e.restore();
         PyErr_Clear();

tests/test_exceptions.h

@@ -9,5 +9,5 @@ class PYBIND11_EXPORT_EXCEPTION shared_exception : public pybind11::builtin_exce
 public:
     using builtin_exception::builtin_exception;
     explicit shared_exception() : shared_exception("") {}
-    void set_error() const override { PyErr_SetString(PyExc_RuntimeError, what()); }
+    void set_error() const override { py::set_error(PyExc_RuntimeError, what()); }
 };

tests/test_exceptions.py

@@ -139,7 +139,15 @@ def test_custom(msg):
     # Can we catch a MyException?
     with pytest.raises(m.MyException) as excinfo:
         m.throws1()
-    assert msg(excinfo.value) == "this error should go to a custom type"
+    assert msg(excinfo.value) == "this error should go to py::exception<MyException>"
+
+    # Can we catch a MyExceptionUseDeprecatedOperatorCall?
+    with pytest.raises(m.MyExceptionUseDeprecatedOperatorCall) as excinfo:
+        m.throws1d()
+    assert (
+        msg(excinfo.value)
+        == "this error should go to py::exception<MyExceptionUseDeprecatedOperatorCall>"
+    )
 
     # Can we translate to standard Python exceptions?
     with pytest.raises(RuntimeError) as excinfo:

tests/test_pytypes.cpp

@@ -25,7 +25,7 @@ PyObject *conv(PyObject *o) {
             ret = PyFloat_FromDouble(v);
         }
     } else {
-        PyErr_SetString(PyExc_TypeError, "Unexpected type");
+        py::set_error(PyExc_TypeError, "Unexpected type");
     }
     return ret;
 }

tests/test_type_caster_pyobject_ptr.cpp

@@ -78,14 +78,14 @@ TEST_SUBMODULE(type_caster_pyobject_ptr, m) {
 
     m.def("cast_to_pyobject_ptr_nullptr", [](bool set_error) {
         if (set_error) {
-            PyErr_SetString(PyExc_RuntimeError, "Reflective of healthy error handling.");
+            py::set_error(PyExc_RuntimeError, "Reflective of healthy error handling.");
         }
         PyObject *ptr = nullptr;
         py::cast(ptr);
     });
 
     m.def("cast_to_pyobject_ptr_non_nullptr_with_error_set", []() {
-        PyErr_SetString(PyExc_RuntimeError, "Reflective of unhealthy error handling.");
+        py::set_error(PyExc_RuntimeError, "Reflective of unhealthy error handling.");
         py::cast(Py_None);
     });