From fab1eebe2c4c52e0abac249f3d058787bc83b5ec Mon Sep 17 00:00:00 2001 From: Lalaland Date: Tue, 18 Oct 2022 16:54:16 -0700 Subject: [PATCH 01/53] First draft of Eigen::Tensor support (#4201) * First draft of Eigen::Tensor support * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix build errors * Weird allocator stuff? * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unused + additional allocator junk * Disable warning * Use constexpr * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * clang tidy fixes * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Resolve comments * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove auto constexpr function * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Try again for older C++ * Oops forgot constexpr * Move to new files as suggested * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix weird tests * Fix nits * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Oops, forgot import * Fix clang 3.6 bug * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * More comprehensive test suite * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Refactor allocators to make things more clear * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Switch to std::copy * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Switch to DSizes instead of array * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Address feedback * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Fix python + dummy c++ change to trigger build * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Alignment * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Add include guard * Forgot inline * Fix compiler warning * Remove bad test * Better type signatures * Add guards to make compiler requirements more explicit * style: pre-commit fixes * Force rerun of tests due to flake * style: pre-commit fixes * Keep pragmas & all related comments together, add PLEASE KEEP IN SYNC * Move headers out of detail * style: pre-commit fixes * Fix cmake * Improve casting * style: pre-commit fixes * Add a ton more tests + refactor * Improve names * style: pre-commit fixes * Update include/pybind11/eigen/tensor.h Co-authored-by: Aaron Gokaslan * Fix tests * style: pre-commit fixes * Update * Add a test to verify that strange numpy arrays work * Fix dumb compiler warning * Better tests * Better tests * Fix tests * style: pre-commit fixes * More test fixes * style: pre-commit fixes * A ton more test coverage * Fix tests * style: pre-commit fixes * style: pre-commit fixes * Add back constexpr * Another test * style: pre-commit fixes * Improve tests * Whoops * Less magic numbers * Update tests/test_eigen_tensor.py Co-authored-by: Sergiu Deitsch * Update tests/test_eigen_tensor.py Co-authored-by: Sergiu Deitsch * style: pre-commit fixes * Fix tests * style: pre-commit fixes * Fix memory leak * style: pre-commit fixes * Fix order * style: pre-commit fixes * Add test to make sure unsafe casts fail * Minor bug fix to work on 32 bit machines * Implement convert flag * style: pre-commit fixes * Switch to correct TensorMap const use * style: pre-commit fixes * Support older versions of eigen * Weird c++ compilers * Fix Eigen bug * Fix another eigen bug * Yet another eigen bug * Potential flakes? * style: pre-commit fixes * Rerun tests with dummy exception to find out what is going on * Another dummy test run * Ablate more * Found the broken test? * One step closer * one step further * Double check * one thing at a time * Give up and disable the test * Clang lies about being gcc * Oops, fix matrix test * style: pre-commit fixes * Add tests to verify scalar conversions * style: pre-commit fixes * Fix nits * Support no_array * Fix tests * style: pre-commit fixes * Silence compiler warning * Improve build system for ancient compilers * Make clang happy * Make gcc happy * Implement Skylion's suggestions * Fix warning * Inline const pointer check * Implement suggestions * style: pre-commit fixes * Improve tests * Typo * style: pre-commit fixes * Support Google's build environment * style: pre-commit fixes * Update include/pybind11/eigen/tensor.h Co-authored-by: Aaron Gokaslan * style: pre-commit fixes * Test cleanup per Skylion * Switch to remvove_cv_t * Cleaner test * style: pre-commit fixes * Remove tensor from eigen.h, update tests * style: pre-commit fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf W. Grosse-Kunstleve Co-authored-by: Aaron Gokaslan Co-authored-by: Aaron Gokaslan Co-authored-by: Sergiu Deitsch --- .gitignore | 1 + CMakeLists.txt | 2 + include/pybind11/eigen.h | 703 +---------------- include/pybind11/eigen/matrix.h | 713 ++++++++++++++++++ include/pybind11/eigen/tensor.h | 518 +++++++++++++ tests/CMakeLists.txt | 36 +- tests/extra_python_package/test_files.py | 8 +- .../{test_eigen.cpp => test_eigen_matrix.cpp} | 4 +- tests/{test_eigen.py => test_eigen_matrix.py} | 2 +- tests/test_eigen_tensor.cpp | 16 + tests/test_eigen_tensor.inl | 333 ++++++++ tests/test_eigen_tensor.py | 288 +++++++ tests/test_eigen_tensor_avoid_stl_array.cpp | 16 + tests/test_numpy_array.cpp | 2 + tests/test_numpy_array.py | 6 + tools/setup_global.py.in | 4 +- tools/setup_main.py.in | 2 + 17 files changed, 1944 insertions(+), 710 deletions(-) create mode 100644 include/pybind11/eigen/matrix.h create mode 100644 include/pybind11/eigen/tensor.h rename tests/{test_eigen.cpp => test_eigen_matrix.cpp} (99%) rename tests/{test_eigen.py => test_eigen_matrix.py} (99%) create mode 100644 tests/test_eigen_tensor.cpp create mode 100644 tests/test_eigen_tensor.inl create mode 100644 tests/test_eigen_tensor.py create mode 100644 tests/test_eigen_tensor_avoid_stl_array.cpp diff --git a/.gitignore b/.gitignore index 3cf4fbbda..43d5094c9 100644 --- a/.gitignore +++ b/.gitignore @@ -43,3 +43,4 @@ pybind11Targets.cmake /pybind11/share/* /docs/_build/* .ipynb_checkpoints/ +tests/main.cpp diff --git a/CMakeLists.txt b/CMakeLists.txt index ee0975bc1..3284e21eb 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -120,6 +120,8 @@ set(PYBIND11_HEADERS include/pybind11/complex.h include/pybind11/options.h include/pybind11/eigen.h + include/pybind11/eigen/matrix.h + include/pybind11/eigen/tensor.h include/pybind11/embed.h include/pybind11/eval.h include/pybind11/gil.h diff --git a/include/pybind11/eigen.h b/include/pybind11/eigen.h index 831625229..273b9c930 100644 --- a/include/pybind11/eigen.h +++ b/include/pybind11/eigen.h @@ -9,705 +9,4 @@ #pragma once -/* HINT: To suppress warnings originating from the Eigen headers, use -isystem. - See also: - https://stackoverflow.com/questions/2579576/i-dir-vs-isystem-dir - https://stackoverflow.com/questions/1741816/isystem-for-ms-visual-studio-c-compiler -*/ - -#include "numpy.h" - -// The C4127 suppression was introduced for Eigen 3.4.0. In theory we could -// make it version specific, or even remove it later, but considering that -// 1. C4127 is generally far more distracting than useful for modern template code, and -// 2. we definitely want to ignore any MSVC warnings originating from Eigen code, -// it is probably best to keep this around indefinitely. -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable : 4127) // C4127: conditional expression is constant -# pragma warning(disable : 5054) // https://github.com/pybind/pybind11/pull/3741 -// C5054: operator '&': deprecated between enumerations of different types -#elif defined(__MINGW32__) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wmaybe-uninitialized" -#endif - -#include -#include - -#if defined(_MSC_VER) -# pragma warning(pop) -#elif defined(__MINGW32__) -# pragma GCC diagnostic pop -#endif - -// Eigen prior to 3.2.7 doesn't have proper move constructors--but worse, some classes get implicit -// move constructors that break things. We could detect this an explicitly copy, but an extra copy -// of matrices seems highly undesirable. -static_assert(EIGEN_VERSION_AT_LEAST(3, 2, 7), - "Eigen support in pybind11 requires Eigen >= 3.2.7"); - -PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) - -// Provide a convenience alias for easier pass-by-ref usage with fully dynamic strides: -using EigenDStride = Eigen::Stride; -template -using EigenDRef = Eigen::Ref; -template -using EigenDMap = Eigen::Map; - -PYBIND11_NAMESPACE_BEGIN(detail) - -#if EIGEN_VERSION_AT_LEAST(3, 3, 0) -using EigenIndex = Eigen::Index; -template -using EigenMapSparseMatrix = Eigen::Map>; -#else -using EigenIndex = EIGEN_DEFAULT_DENSE_INDEX_TYPE; -template -using EigenMapSparseMatrix = Eigen::MappedSparseMatrix; -#endif - -// Matches Eigen::Map, Eigen::Ref, blocks, etc: -template -using is_eigen_dense_map = all_of, - std::is_base_of, T>>; -template -using is_eigen_mutable_map = std::is_base_of, T>; -template -using is_eigen_dense_plain - = all_of>, is_template_base_of>; -template -using is_eigen_sparse = is_template_base_of; -// Test for objects inheriting from EigenBase that aren't captured by the above. This -// basically covers anything that can be assigned to a dense matrix but that don't have a typical -// matrix data layout that can be copied from their .data(). For example, DiagonalMatrix and -// SelfAdjointView fall into this category. -template -using is_eigen_other - = all_of, - negation, is_eigen_dense_plain, is_eigen_sparse>>>; - -// Captures numpy/eigen conformability status (returned by EigenProps::conformable()): -template -struct EigenConformable { - bool conformable = false; - EigenIndex rows = 0, cols = 0; - EigenDStride stride{0, 0}; // Only valid if negativestrides is false! - bool negativestrides = false; // If true, do not use stride! - - // NOLINTNEXTLINE(google-explicit-constructor) - EigenConformable(bool fits = false) : conformable{fits} {} - // Matrix type: - EigenConformable(EigenIndex r, EigenIndex c, EigenIndex rstride, EigenIndex cstride) - : conformable{true}, rows{r}, cols{c}, - // TODO: when Eigen bug #747 is fixed, remove the tests for non-negativity. - // http://eigen.tuxfamily.org/bz/show_bug.cgi?id=747 - stride{EigenRowMajor ? (rstride > 0 ? rstride : 0) - : (cstride > 0 ? cstride : 0) /* outer stride */, - EigenRowMajor ? (cstride > 0 ? cstride : 0) - : (rstride > 0 ? rstride : 0) /* inner stride */}, - negativestrides{rstride < 0 || cstride < 0} {} - // Vector type: - EigenConformable(EigenIndex r, EigenIndex c, EigenIndex stride) - : EigenConformable(r, c, r == 1 ? c * stride : stride, c == 1 ? r : r * stride) {} - - template - bool stride_compatible() const { - // To have compatible strides, we need (on both dimensions) one of fully dynamic strides, - // matching strides, or a dimension size of 1 (in which case the stride value is - // irrelevant). Alternatively, if any dimension size is 0, the strides are not relevant - // (and numpy ≥ 1.23 sets the strides to 0 in that case, so we need to check explicitly). - if (negativestrides) { - return false; - } - if (rows == 0 || cols == 0) { - return true; - } - return (props::inner_stride == Eigen::Dynamic || props::inner_stride == stride.inner() - || (EigenRowMajor ? cols : rows) == 1) - && (props::outer_stride == Eigen::Dynamic || props::outer_stride == stride.outer() - || (EigenRowMajor ? rows : cols) == 1); - } - // NOLINTNEXTLINE(google-explicit-constructor) - operator bool() const { return conformable; } -}; - -template -struct eigen_extract_stride { - using type = Type; -}; -template -struct eigen_extract_stride> { - using type = StrideType; -}; -template -struct eigen_extract_stride> { - using type = StrideType; -}; - -// Helper struct for extracting information from an Eigen type -template -struct EigenProps { - using Type = Type_; - using Scalar = typename Type::Scalar; - using StrideType = typename eigen_extract_stride::type; - static constexpr EigenIndex rows = Type::RowsAtCompileTime, cols = Type::ColsAtCompileTime, - size = Type::SizeAtCompileTime; - static constexpr bool row_major = Type::IsRowMajor, - vector - = Type::IsVectorAtCompileTime, // At least one dimension has fixed size 1 - fixed_rows = rows != Eigen::Dynamic, fixed_cols = cols != Eigen::Dynamic, - fixed = size != Eigen::Dynamic, // Fully-fixed size - dynamic = !fixed_rows && !fixed_cols; // Fully-dynamic size - - template - using if_zero = std::integral_constant; - static constexpr EigenIndex inner_stride - = if_zero::value, - outer_stride = if_zero < StrideType::OuterStrideAtCompileTime, - vector ? size - : row_major ? cols - : rows > ::value; - static constexpr bool dynamic_stride - = inner_stride == Eigen::Dynamic && outer_stride == Eigen::Dynamic; - static constexpr bool requires_row_major - = !dynamic_stride && !vector && (row_major ? inner_stride : outer_stride) == 1; - static constexpr bool requires_col_major - = !dynamic_stride && !vector && (row_major ? outer_stride : inner_stride) == 1; - - // Takes an input array and determines whether we can make it fit into the Eigen type. If - // the array is a vector, we attempt to fit it into either an Eigen 1xN or Nx1 vector - // (preferring the latter if it will fit in either, i.e. for a fully dynamic matrix type). - static EigenConformable conformable(const array &a) { - const auto dims = a.ndim(); - if (dims < 1 || dims > 2) { - return false; - } - - if (dims == 2) { // Matrix type: require exact match (or dynamic) - - EigenIndex np_rows = a.shape(0), np_cols = a.shape(1), - np_rstride = a.strides(0) / static_cast(sizeof(Scalar)), - np_cstride = a.strides(1) / static_cast(sizeof(Scalar)); - if ((PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && np_rows != rows) - || (PYBIND11_SILENCE_MSVC_C4127(fixed_cols) && np_cols != cols)) { - return false; - } - - return {np_rows, np_cols, np_rstride, np_cstride}; - } - - // Otherwise we're storing an n-vector. Only one of the strides will be used, but - // whichever is used, we want the (single) numpy stride value. - const EigenIndex n = a.shape(0), - stride = a.strides(0) / static_cast(sizeof(Scalar)); - - if (vector) { // Eigen type is a compile-time vector - if (PYBIND11_SILENCE_MSVC_C4127(fixed) && size != n) { - return false; // Vector size mismatch - } - return {rows == 1 ? 1 : n, cols == 1 ? 1 : n, stride}; - } - if (fixed) { - // The type has a fixed size, but is not a vector: abort - return false; - } - if (fixed_cols) { - // Since this isn't a vector, cols must be != 1. We allow this only if it exactly - // equals the number of elements (rows is Dynamic, and so 1 row is allowed). - if (cols != n) { - return false; - } - return {1, n, stride}; - } // Otherwise it's either fully dynamic, or column dynamic; both become a column vector - if (PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && rows != n) { - return false; - } - return {n, 1, stride}; - } - - static constexpr bool show_writeable - = is_eigen_dense_map::value && is_eigen_mutable_map::value; - static constexpr bool show_order = is_eigen_dense_map::value; - static constexpr bool show_c_contiguous = show_order && requires_row_major; - static constexpr bool show_f_contiguous - = !show_c_contiguous && show_order && requires_col_major; - - static constexpr auto descriptor - = const_name("numpy.ndarray[") + npy_format_descriptor::name + const_name("[") - + const_name(const_name<(size_t) rows>(), const_name("m")) + const_name(", ") - + const_name(const_name<(size_t) cols>(), const_name("n")) + const_name("]") - + - // For a reference type (e.g. Ref) we have other constraints that might need to - // be satisfied: writeable=True (for a mutable reference), and, depending on the map's - // stride options, possibly f_contiguous or c_contiguous. We include them in the - // descriptor output to provide some hint as to why a TypeError is occurring (otherwise - // it can be confusing to see that a function accepts a 'numpy.ndarray[float64[3,2]]' and - // an error message that you *gave* a numpy.ndarray of the right type and dimensions. - const_name(", flags.writeable", "") - + const_name(", flags.c_contiguous", "") - + const_name(", flags.f_contiguous", "") + const_name("]"); -}; - -// Casts an Eigen type to numpy array. If given a base, the numpy array references the src data, -// otherwise it'll make a copy. writeable lets you turn off the writeable flag for the array. -template -handle -eigen_array_cast(typename props::Type const &src, handle base = handle(), bool writeable = true) { - constexpr ssize_t elem_size = sizeof(typename props::Scalar); - array a; - if (props::vector) { - a = array({src.size()}, {elem_size * src.innerStride()}, src.data(), base); - } else { - a = array({src.rows(), src.cols()}, - {elem_size * src.rowStride(), elem_size * src.colStride()}, - src.data(), - base); - } - - if (!writeable) { - array_proxy(a.ptr())->flags &= ~detail::npy_api::NPY_ARRAY_WRITEABLE_; - } - - return a.release(); -} - -// Takes an lvalue ref to some Eigen type and a (python) base object, creating a numpy array that -// reference the Eigen object's data with `base` as the python-registered base class (if omitted, -// the base will be set to None, and lifetime management is up to the caller). The numpy array is -// non-writeable if the given type is const. -template -handle eigen_ref_array(Type &src, handle parent = none()) { - // none here is to get past array's should-we-copy detection, which currently always - // copies when there is no base. Setting the base to None should be harmless. - return eigen_array_cast(src, parent, !std::is_const::value); -} - -// Takes a pointer to some dense, plain Eigen type, builds a capsule around it, then returns a -// numpy array that references the encapsulated data with a python-side reference to the capsule to -// tie its destruction to that of any dependent python objects. Const-ness is determined by -// whether or not the Type of the pointer given is const. -template ::value>> -handle eigen_encapsulate(Type *src) { - capsule base(src, [](void *o) { delete static_cast(o); }); - return eigen_ref_array(*src, base); -} - -// Type caster for regular, dense matrix types (e.g. MatrixXd), but not maps/refs/etc. of dense -// types. -template -struct type_caster::value>> { - using Scalar = typename Type::Scalar; - using props = EigenProps; - - bool load(handle src, bool convert) { - // If we're in no-convert mode, only load if given an array of the correct type - if (!convert && !isinstance>(src)) { - return false; - } - - // Coerce into an array, but don't do type conversion yet; the copy below handles it. - auto buf = array::ensure(src); - - if (!buf) { - return false; - } - - auto dims = buf.ndim(); - if (dims < 1 || dims > 2) { - return false; - } - - auto fits = props::conformable(buf); - if (!fits) { - return false; - } - - // Allocate the new type, then build a numpy reference into it - value = Type(fits.rows, fits.cols); - auto ref = reinterpret_steal(eigen_ref_array(value)); - if (dims == 1) { - ref = ref.squeeze(); - } else if (ref.ndim() == 1) { - buf = buf.squeeze(); - } - - int result = detail::npy_api::get().PyArray_CopyInto_(ref.ptr(), buf.ptr()); - - if (result < 0) { // Copy failed! - PyErr_Clear(); - return false; - } - - return true; - } - -private: - // Cast implementation - template - static handle cast_impl(CType *src, return_value_policy policy, handle parent) { - switch (policy) { - case return_value_policy::take_ownership: - case return_value_policy::automatic: - return eigen_encapsulate(src); - case return_value_policy::move: - return eigen_encapsulate(new CType(std::move(*src))); - case return_value_policy::copy: - return eigen_array_cast(*src); - case return_value_policy::reference: - case return_value_policy::automatic_reference: - return eigen_ref_array(*src); - case return_value_policy::reference_internal: - return eigen_ref_array(*src, parent); - default: - throw cast_error("unhandled return_value_policy: should not happen!"); - }; - } - -public: - // Normal returned non-reference, non-const value: - static handle cast(Type &&src, return_value_policy /* policy */, handle parent) { - return cast_impl(&src, return_value_policy::move, parent); - } - // If you return a non-reference const, we mark the numpy array readonly: - static handle cast(const Type &&src, return_value_policy /* policy */, handle parent) { - return cast_impl(&src, return_value_policy::move, parent); - } - // lvalue reference return; default (automatic) becomes copy - static handle cast(Type &src, return_value_policy policy, handle parent) { - if (policy == return_value_policy::automatic - || policy == return_value_policy::automatic_reference) { - policy = return_value_policy::copy; - } - return cast_impl(&src, policy, parent); - } - // const lvalue reference return; default (automatic) becomes copy - static handle cast(const Type &src, return_value_policy policy, handle parent) { - if (policy == return_value_policy::automatic - || policy == return_value_policy::automatic_reference) { - policy = return_value_policy::copy; - } - return cast(&src, policy, parent); - } - // non-const pointer return - static handle cast(Type *src, return_value_policy policy, handle parent) { - return cast_impl(src, policy, parent); - } - // const pointer return - static handle cast(const Type *src, return_value_policy policy, handle parent) { - return cast_impl(src, policy, parent); - } - - static constexpr auto name = props::descriptor; - - // NOLINTNEXTLINE(google-explicit-constructor) - operator Type *() { return &value; } - // NOLINTNEXTLINE(google-explicit-constructor) - operator Type &() { return value; } - // NOLINTNEXTLINE(google-explicit-constructor) - operator Type &&() && { return std::move(value); } - template - using cast_op_type = movable_cast_op_type; - -private: - Type value; -}; - -// Base class for casting reference/map/block/etc. objects back to python. -template -struct eigen_map_caster { -private: - using props = EigenProps; - -public: - // Directly referencing a ref/map's data is a bit dangerous (whatever the map/ref points to has - // to stay around), but we'll allow it under the assumption that you know what you're doing - // (and have an appropriate keep_alive in place). We return a numpy array pointing directly at - // the ref's data (The numpy array ends up read-only if the ref was to a const matrix type.) - // Note that this means you need to ensure you don't destroy the object in some other way (e.g. - // with an appropriate keep_alive, or with a reference to a statically allocated matrix). - static handle cast(const MapType &src, return_value_policy policy, handle parent) { - switch (policy) { - case return_value_policy::copy: - return eigen_array_cast(src); - case return_value_policy::reference_internal: - return eigen_array_cast(src, parent, is_eigen_mutable_map::value); - case return_value_policy::reference: - case return_value_policy::automatic: - case return_value_policy::automatic_reference: - return eigen_array_cast(src, none(), is_eigen_mutable_map::value); - default: - // move, take_ownership don't make any sense for a ref/map: - pybind11_fail("Invalid return_value_policy for Eigen Map/Ref/Block type"); - } - } - - static constexpr auto name = props::descriptor; - - // Explicitly delete these: support python -> C++ conversion on these (i.e. these can be return - // types but not bound arguments). We still provide them (with an explicitly delete) so that - // you end up here if you try anyway. - bool load(handle, bool) = delete; - operator MapType() = delete; - template - using cast_op_type = MapType; -}; - -// We can return any map-like object (but can only load Refs, specialized next): -template -struct type_caster::value>> : eigen_map_caster {}; - -// Loader for Ref<...> arguments. See the documentation for info on how to make this work without -// copying (it requires some extra effort in many cases). -template -struct type_caster< - Eigen::Ref, - enable_if_t>::value>> - : public eigen_map_caster> { -private: - using Type = Eigen::Ref; - using props = EigenProps; - using Scalar = typename props::Scalar; - using MapType = Eigen::Map; - using Array - = array_t; - static constexpr bool need_writeable = is_eigen_mutable_map::value; - // Delay construction (these have no default constructor) - std::unique_ptr map; - std::unique_ptr ref; - // Our array. When possible, this is just a numpy array pointing to the source data, but - // sometimes we can't avoid copying (e.g. input is not a numpy array at all, has an - // incompatible layout, or is an array of a type that needs to be converted). Using a numpy - // temporary (rather than an Eigen temporary) saves an extra copy when we need both type - // conversion and storage order conversion. (Note that we refuse to use this temporary copy - // when loading an argument for a Ref with M non-const, i.e. a read-write reference). - Array copy_or_ref; - -public: - bool load(handle src, bool convert) { - // First check whether what we have is already an array of the right type. If not, we - // can't avoid a copy (because the copy is also going to do type conversion). - bool need_copy = !isinstance(src); - - EigenConformable fits; - if (!need_copy) { - // We don't need a converting copy, but we also need to check whether the strides are - // compatible with the Ref's stride requirements - auto aref = reinterpret_borrow(src); - - if (aref && (!need_writeable || aref.writeable())) { - fits = props::conformable(aref); - if (!fits) { - return false; // Incompatible dimensions - } - if (!fits.template stride_compatible()) { - need_copy = true; - } else { - copy_or_ref = std::move(aref); - } - } else { - need_copy = true; - } - } - - if (need_copy) { - // We need to copy: If we need a mutable reference, or we're not supposed to convert - // (either because we're in the no-convert overload pass, or because we're explicitly - // instructed not to copy (via `py::arg().noconvert()`) we have to fail loading. - if (!convert || need_writeable) { - return false; - } - - Array copy = Array::ensure(src); - if (!copy) { - return false; - } - fits = props::conformable(copy); - if (!fits || !fits.template stride_compatible()) { - return false; - } - copy_or_ref = std::move(copy); - loader_life_support::add_patient(copy_or_ref); - } - - ref.reset(); - map.reset(new MapType(data(copy_or_ref), - fits.rows, - fits.cols, - make_stride(fits.stride.outer(), fits.stride.inner()))); - ref.reset(new Type(*map)); - - return true; - } - - // NOLINTNEXTLINE(google-explicit-constructor) - operator Type *() { return ref.get(); } - // NOLINTNEXTLINE(google-explicit-constructor) - operator Type &() { return *ref; } - template - using cast_op_type = pybind11::detail::cast_op_type<_T>; - -private: - template ::value, int> = 0> - Scalar *data(Array &a) { - return a.mutable_data(); - } - - template ::value, int> = 0> - const Scalar *data(Array &a) { - return a.data(); - } - - // Attempt to figure out a constructor of `Stride` that will work. - // If both strides are fixed, use a default constructor: - template - using stride_ctor_default = bool_constant::value>; - // Otherwise, if there is a two-index constructor, assume it is (outer,inner) like - // Eigen::Stride, and use it: - template - using stride_ctor_dual - = bool_constant::value - && std::is_constructible::value>; - // Otherwise, if there is a one-index constructor, and just one of the strides is dynamic, use - // it (passing whichever stride is dynamic). - template - using stride_ctor_outer - = bool_constant, stride_ctor_dual>::value - && S::OuterStrideAtCompileTime == Eigen::Dynamic - && S::InnerStrideAtCompileTime != Eigen::Dynamic - && std::is_constructible::value>; - template - using stride_ctor_inner - = bool_constant, stride_ctor_dual>::value - && S::InnerStrideAtCompileTime == Eigen::Dynamic - && S::OuterStrideAtCompileTime != Eigen::Dynamic - && std::is_constructible::value>; - - template ::value, int> = 0> - static S make_stride(EigenIndex, EigenIndex) { - return S(); - } - template ::value, int> = 0> - static S make_stride(EigenIndex outer, EigenIndex inner) { - return S(outer, inner); - } - template ::value, int> = 0> - static S make_stride(EigenIndex outer, EigenIndex) { - return S(outer); - } - template ::value, int> = 0> - static S make_stride(EigenIndex, EigenIndex inner) { - return S(inner); - } -}; - -// type_caster for special matrix types (e.g. DiagonalMatrix), which are EigenBase, but not -// EigenDense (i.e. they don't have a data(), at least not with the usual matrix layout). -// load() is not supported, but we can cast them into the python domain by first copying to a -// regular Eigen::Matrix, then casting that. -template -struct type_caster::value>> { -protected: - using Matrix - = Eigen::Matrix; - using props = EigenProps; - -public: - static handle cast(const Type &src, return_value_policy /* policy */, handle /* parent */) { - handle h = eigen_encapsulate(new Matrix(src)); - return h; - } - static handle cast(const Type *src, return_value_policy policy, handle parent) { - return cast(*src, policy, parent); - } - - static constexpr auto name = props::descriptor; - - // Explicitly delete these: support python -> C++ conversion on these (i.e. these can be return - // types but not bound arguments). We still provide them (with an explicitly delete) so that - // you end up here if you try anyway. - bool load(handle, bool) = delete; - operator Type() = delete; - template - using cast_op_type = Type; -}; - -template -struct type_caster::value>> { - using Scalar = typename Type::Scalar; - using StorageIndex = remove_reference_t().outerIndexPtr())>; - using Index = typename Type::Index; - static constexpr bool rowMajor = Type::IsRowMajor; - - bool load(handle src, bool) { - if (!src) { - return false; - } - - auto obj = reinterpret_borrow(src); - object sparse_module = module_::import("scipy.sparse"); - object matrix_type = sparse_module.attr(rowMajor ? "csr_matrix" : "csc_matrix"); - - if (!type::handle_of(obj).is(matrix_type)) { - try { - obj = matrix_type(obj); - } catch (const error_already_set &) { - return false; - } - } - - auto values = array_t((object) obj.attr("data")); - auto innerIndices = array_t((object) obj.attr("indices")); - auto outerIndices = array_t((object) obj.attr("indptr")); - auto shape = pybind11::tuple((pybind11::object) obj.attr("shape")); - auto nnz = obj.attr("nnz").cast(); - - if (!values || !innerIndices || !outerIndices) { - return false; - } - - value = EigenMapSparseMatrix(shape[0].cast(), - shape[1].cast(), - std::move(nnz), - outerIndices.mutable_data(), - innerIndices.mutable_data(), - values.mutable_data()); - - return true; - } - - static handle cast(const Type &src, return_value_policy /* policy */, handle /* parent */) { - const_cast(src).makeCompressed(); - - object matrix_type - = module_::import("scipy.sparse").attr(rowMajor ? "csr_matrix" : "csc_matrix"); - - array data(src.nonZeros(), src.valuePtr()); - array outerIndices((rowMajor ? src.rows() : src.cols()) + 1, src.outerIndexPtr()); - array innerIndices(src.nonZeros(), src.innerIndexPtr()); - - return matrix_type(pybind11::make_tuple( - std::move(data), std::move(innerIndices), std::move(outerIndices)), - pybind11::make_tuple(src.rows(), src.cols())) - .release(); - } - - PYBIND11_TYPE_CASTER(Type, - const_name<(Type::IsRowMajor) != 0>("scipy.sparse.csr_matrix[", - "scipy.sparse.csc_matrix[") - + npy_format_descriptor::name + const_name("]")); -}; - -PYBIND11_NAMESPACE_END(detail) -PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) +#include "eigen/matrix.h" diff --git a/include/pybind11/eigen/matrix.h b/include/pybind11/eigen/matrix.h new file mode 100644 index 000000000..5f5ad3867 --- /dev/null +++ b/include/pybind11/eigen/matrix.h @@ -0,0 +1,713 @@ +/* + pybind11/eigen/matrix.h: Transparent conversion for dense and sparse Eigen matrices + + Copyright (c) 2016 Wenzel Jakob + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#pragma once + +#include "../numpy.h" + +// Similar to comments & pragma block in eigen_tensor.h. PLEASE KEEP IN SYNC. +/* HINT: To suppress warnings originating from the Eigen headers, use -isystem. + See also: + https://stackoverflow.com/questions/2579576/i-dir-vs-isystem-dir + https://stackoverflow.com/questions/1741816/isystem-for-ms-visual-studio-c-compiler +*/ +// The C4127 suppression was introduced for Eigen 3.4.0. In theory we could +// make it version specific, or even remove it later, but considering that +// 1. C4127 is generally far more distracting than useful for modern template code, and +// 2. we definitely want to ignore any MSVC warnings originating from Eigen code, +// it is probably best to keep this around indefinitely. +#if defined(_MSC_VER) +# pragma warning(push) +# pragma warning(disable : 4127) // C4127: conditional expression is constant +# pragma warning(disable : 5054) // https://github.com/pybind/pybind11/pull/3741 +// C5054: operator '&': deprecated between enumerations of different types +#elif defined(__MINGW32__) +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wmaybe-uninitialized" +#endif + +#include +#include + +#if defined(_MSC_VER) +# pragma warning(pop) +#elif defined(__MINGW32__) +# pragma GCC diagnostic pop +#endif + +// Eigen prior to 3.2.7 doesn't have proper move constructors--but worse, some classes get implicit +// move constructors that break things. We could detect this an explicitly copy, but an extra copy +// of matrices seems highly undesirable. +static_assert(EIGEN_VERSION_AT_LEAST(3, 2, 7), + "Eigen matrix support in pybind11 requires Eigen >= 3.2.7"); + +PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + +// Provide a convenience alias for easier pass-by-ref usage with fully dynamic strides: +using EigenDStride = Eigen::Stride; +template +using EigenDRef = Eigen::Ref; +template +using EigenDMap = Eigen::Map; + +PYBIND11_NAMESPACE_BEGIN(detail) + +#if EIGEN_VERSION_AT_LEAST(3, 3, 0) +using EigenIndex = Eigen::Index; +template +using EigenMapSparseMatrix = Eigen::Map>; +#else +using EigenIndex = EIGEN_DEFAULT_DENSE_INDEX_TYPE; +template +using EigenMapSparseMatrix = Eigen::MappedSparseMatrix; +#endif + +// Matches Eigen::Map, Eigen::Ref, blocks, etc: +template +using is_eigen_dense_map = all_of, + std::is_base_of, T>>; +template +using is_eigen_mutable_map = std::is_base_of, T>; +template +using is_eigen_dense_plain + = all_of>, is_template_base_of>; +template +using is_eigen_sparse = is_template_base_of; +// Test for objects inheriting from EigenBase that aren't captured by the above. This +// basically covers anything that can be assigned to a dense matrix but that don't have a typical +// matrix data layout that can be copied from their .data(). For example, DiagonalMatrix and +// SelfAdjointView fall into this category. +template +using is_eigen_other + = all_of, + negation, is_eigen_dense_plain, is_eigen_sparse>>>; + +// Captures numpy/eigen conformability status (returned by EigenProps::conformable()): +template +struct EigenConformable { + bool conformable = false; + EigenIndex rows = 0, cols = 0; + EigenDStride stride{0, 0}; // Only valid if negativestrides is false! + bool negativestrides = false; // If true, do not use stride! + + // NOLINTNEXTLINE(google-explicit-constructor) + EigenConformable(bool fits = false) : conformable{fits} {} + // Matrix type: + EigenConformable(EigenIndex r, EigenIndex c, EigenIndex rstride, EigenIndex cstride) + : conformable{true}, rows{r}, cols{c}, + // TODO: when Eigen bug #747 is fixed, remove the tests for non-negativity. + // http://eigen.tuxfamily.org/bz/show_bug.cgi?id=747 + stride{EigenRowMajor ? (rstride > 0 ? rstride : 0) + : (cstride > 0 ? cstride : 0) /* outer stride */, + EigenRowMajor ? (cstride > 0 ? cstride : 0) + : (rstride > 0 ? rstride : 0) /* inner stride */}, + negativestrides{rstride < 0 || cstride < 0} {} + // Vector type: + EigenConformable(EigenIndex r, EigenIndex c, EigenIndex stride) + : EigenConformable(r, c, r == 1 ? c * stride : stride, c == 1 ? r : r * stride) {} + + template + bool stride_compatible() const { + // To have compatible strides, we need (on both dimensions) one of fully dynamic strides, + // matching strides, or a dimension size of 1 (in which case the stride value is + // irrelevant). Alternatively, if any dimension size is 0, the strides are not relevant + // (and numpy ≥ 1.23 sets the strides to 0 in that case, so we need to check explicitly). + if (negativestrides) { + return false; + } + if (rows == 0 || cols == 0) { + return true; + } + return (props::inner_stride == Eigen::Dynamic || props::inner_stride == stride.inner() + || (EigenRowMajor ? cols : rows) == 1) + && (props::outer_stride == Eigen::Dynamic || props::outer_stride == stride.outer() + || (EigenRowMajor ? rows : cols) == 1); + } + // NOLINTNEXTLINE(google-explicit-constructor) + operator bool() const { return conformable; } +}; + +template +struct eigen_extract_stride { + using type = Type; +}; +template +struct eigen_extract_stride> { + using type = StrideType; +}; +template +struct eigen_extract_stride> { + using type = StrideType; +}; + +// Helper struct for extracting information from an Eigen type +template +struct EigenProps { + using Type = Type_; + using Scalar = typename Type::Scalar; + using StrideType = typename eigen_extract_stride::type; + static constexpr EigenIndex rows = Type::RowsAtCompileTime, cols = Type::ColsAtCompileTime, + size = Type::SizeAtCompileTime; + static constexpr bool row_major = Type::IsRowMajor, + vector + = Type::IsVectorAtCompileTime, // At least one dimension has fixed size 1 + fixed_rows = rows != Eigen::Dynamic, fixed_cols = cols != Eigen::Dynamic, + fixed = size != Eigen::Dynamic, // Fully-fixed size + dynamic = !fixed_rows && !fixed_cols; // Fully-dynamic size + + template + using if_zero = std::integral_constant; + static constexpr EigenIndex inner_stride + = if_zero::value, + outer_stride = if_zero < StrideType::OuterStrideAtCompileTime, + vector ? size + : row_major ? cols + : rows > ::value; + static constexpr bool dynamic_stride + = inner_stride == Eigen::Dynamic && outer_stride == Eigen::Dynamic; + static constexpr bool requires_row_major + = !dynamic_stride && !vector && (row_major ? inner_stride : outer_stride) == 1; + static constexpr bool requires_col_major + = !dynamic_stride && !vector && (row_major ? outer_stride : inner_stride) == 1; + + // Takes an input array and determines whether we can make it fit into the Eigen type. If + // the array is a vector, we attempt to fit it into either an Eigen 1xN or Nx1 vector + // (preferring the latter if it will fit in either, i.e. for a fully dynamic matrix type). + static EigenConformable conformable(const array &a) { + const auto dims = a.ndim(); + if (dims < 1 || dims > 2) { + return false; + } + + if (dims == 2) { // Matrix type: require exact match (or dynamic) + + EigenIndex np_rows = a.shape(0), np_cols = a.shape(1), + np_rstride = a.strides(0) / static_cast(sizeof(Scalar)), + np_cstride = a.strides(1) / static_cast(sizeof(Scalar)); + if ((PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && np_rows != rows) + || (PYBIND11_SILENCE_MSVC_C4127(fixed_cols) && np_cols != cols)) { + return false; + } + + return {np_rows, np_cols, np_rstride, np_cstride}; + } + + // Otherwise we're storing an n-vector. Only one of the strides will be used, but + // whichever is used, we want the (single) numpy stride value. + const EigenIndex n = a.shape(0), + stride = a.strides(0) / static_cast(sizeof(Scalar)); + + if (vector) { // Eigen type is a compile-time vector + if (PYBIND11_SILENCE_MSVC_C4127(fixed) && size != n) { + return false; // Vector size mismatch + } + return {rows == 1 ? 1 : n, cols == 1 ? 1 : n, stride}; + } + if (fixed) { + // The type has a fixed size, but is not a vector: abort + return false; + } + if (fixed_cols) { + // Since this isn't a vector, cols must be != 1. We allow this only if it exactly + // equals the number of elements (rows is Dynamic, and so 1 row is allowed). + if (cols != n) { + return false; + } + return {1, n, stride}; + } // Otherwise it's either fully dynamic, or column dynamic; both become a column vector + if (PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && rows != n) { + return false; + } + return {n, 1, stride}; + } + + static constexpr bool show_writeable + = is_eigen_dense_map::value && is_eigen_mutable_map::value; + static constexpr bool show_order = is_eigen_dense_map::value; + static constexpr bool show_c_contiguous = show_order && requires_row_major; + static constexpr bool show_f_contiguous + = !show_c_contiguous && show_order && requires_col_major; + + static constexpr auto descriptor + = const_name("numpy.ndarray[") + npy_format_descriptor::name + const_name("[") + + const_name(const_name<(size_t) rows>(), const_name("m")) + const_name(", ") + + const_name(const_name<(size_t) cols>(), const_name("n")) + const_name("]") + + + // For a reference type (e.g. Ref) we have other constraints that might need to + // be satisfied: writeable=True (for a mutable reference), and, depending on the map's + // stride options, possibly f_contiguous or c_contiguous. We include them in the + // descriptor output to provide some hint as to why a TypeError is occurring (otherwise + // it can be confusing to see that a function accepts a 'numpy.ndarray[float64[3,2]]' and + // an error message that you *gave* a numpy.ndarray of the right type and dimensions. + const_name(", flags.writeable", "") + + const_name(", flags.c_contiguous", "") + + const_name(", flags.f_contiguous", "") + const_name("]"); +}; + +// Casts an Eigen type to numpy array. If given a base, the numpy array references the src data, +// otherwise it'll make a copy. writeable lets you turn off the writeable flag for the array. +template +handle +eigen_array_cast(typename props::Type const &src, handle base = handle(), bool writeable = true) { + constexpr ssize_t elem_size = sizeof(typename props::Scalar); + array a; + if (props::vector) { + a = array({src.size()}, {elem_size * src.innerStride()}, src.data(), base); + } else { + a = array({src.rows(), src.cols()}, + {elem_size * src.rowStride(), elem_size * src.colStride()}, + src.data(), + base); + } + + if (!writeable) { + array_proxy(a.ptr())->flags &= ~detail::npy_api::NPY_ARRAY_WRITEABLE_; + } + + return a.release(); +} + +// Takes an lvalue ref to some Eigen type and a (python) base object, creating a numpy array that +// reference the Eigen object's data with `base` as the python-registered base class (if omitted, +// the base will be set to None, and lifetime management is up to the caller). The numpy array is +// non-writeable if the given type is const. +template +handle eigen_ref_array(Type &src, handle parent = none()) { + // none here is to get past array's should-we-copy detection, which currently always + // copies when there is no base. Setting the base to None should be harmless. + return eigen_array_cast(src, parent, !std::is_const::value); +} + +// Takes a pointer to some dense, plain Eigen type, builds a capsule around it, then returns a +// numpy array that references the encapsulated data with a python-side reference to the capsule to +// tie its destruction to that of any dependent python objects. Const-ness is determined by +// whether or not the Type of the pointer given is const. +template ::value>> +handle eigen_encapsulate(Type *src) { + capsule base(src, [](void *o) { delete static_cast(o); }); + return eigen_ref_array(*src, base); +} + +// Type caster for regular, dense matrix types (e.g. MatrixXd), but not maps/refs/etc. of dense +// types. +template +struct type_caster::value>> { + using Scalar = typename Type::Scalar; + using props = EigenProps; + + bool load(handle src, bool convert) { + // If we're in no-convert mode, only load if given an array of the correct type + if (!convert && !isinstance>(src)) { + return false; + } + + // Coerce into an array, but don't do type conversion yet; the copy below handles it. + auto buf = array::ensure(src); + + if (!buf) { + return false; + } + + auto dims = buf.ndim(); + if (dims < 1 || dims > 2) { + return false; + } + + auto fits = props::conformable(buf); + if (!fits) { + return false; + } + + // Allocate the new type, then build a numpy reference into it + value = Type(fits.rows, fits.cols); + auto ref = reinterpret_steal(eigen_ref_array(value)); + if (dims == 1) { + ref = ref.squeeze(); + } else if (ref.ndim() == 1) { + buf = buf.squeeze(); + } + + int result = detail::npy_api::get().PyArray_CopyInto_(ref.ptr(), buf.ptr()); + + if (result < 0) { // Copy failed! + PyErr_Clear(); + return false; + } + + return true; + } + +private: + // Cast implementation + template + static handle cast_impl(CType *src, return_value_policy policy, handle parent) { + switch (policy) { + case return_value_policy::take_ownership: + case return_value_policy::automatic: + return eigen_encapsulate(src); + case return_value_policy::move: + return eigen_encapsulate(new CType(std::move(*src))); + case return_value_policy::copy: + return eigen_array_cast(*src); + case return_value_policy::reference: + case return_value_policy::automatic_reference: + return eigen_ref_array(*src); + case return_value_policy::reference_internal: + return eigen_ref_array(*src, parent); + default: + throw cast_error("unhandled return_value_policy: should not happen!"); + }; + } + +public: + // Normal returned non-reference, non-const value: + static handle cast(Type &&src, return_value_policy /* policy */, handle parent) { + return cast_impl(&src, return_value_policy::move, parent); + } + // If you return a non-reference const, we mark the numpy array readonly: + static handle cast(const Type &&src, return_value_policy /* policy */, handle parent) { + return cast_impl(&src, return_value_policy::move, parent); + } + // lvalue reference return; default (automatic) becomes copy + static handle cast(Type &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast_impl(&src, policy, parent); + } + // const lvalue reference return; default (automatic) becomes copy + static handle cast(const Type &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast(&src, policy, parent); + } + // non-const pointer return + static handle cast(Type *src, return_value_policy policy, handle parent) { + return cast_impl(src, policy, parent); + } + // const pointer return + static handle cast(const Type *src, return_value_policy policy, handle parent) { + return cast_impl(src, policy, parent); + } + + static constexpr auto name = props::descriptor; + + // NOLINTNEXTLINE(google-explicit-constructor) + operator Type *() { return &value; } + // NOLINTNEXTLINE(google-explicit-constructor) + operator Type &() { return value; } + // NOLINTNEXTLINE(google-explicit-constructor) + operator Type &&() && { return std::move(value); } + template + using cast_op_type = movable_cast_op_type; + +private: + Type value; +}; + +// Base class for casting reference/map/block/etc. objects back to python. +template +struct eigen_map_caster { +private: + using props = EigenProps; + +public: + // Directly referencing a ref/map's data is a bit dangerous (whatever the map/ref points to has + // to stay around), but we'll allow it under the assumption that you know what you're doing + // (and have an appropriate keep_alive in place). We return a numpy array pointing directly at + // the ref's data (The numpy array ends up read-only if the ref was to a const matrix type.) + // Note that this means you need to ensure you don't destroy the object in some other way (e.g. + // with an appropriate keep_alive, or with a reference to a statically allocated matrix). + static handle cast(const MapType &src, return_value_policy policy, handle parent) { + switch (policy) { + case return_value_policy::copy: + return eigen_array_cast(src); + case return_value_policy::reference_internal: + return eigen_array_cast(src, parent, is_eigen_mutable_map::value); + case return_value_policy::reference: + case return_value_policy::automatic: + case return_value_policy::automatic_reference: + return eigen_array_cast(src, none(), is_eigen_mutable_map::value); + default: + // move, take_ownership don't make any sense for a ref/map: + pybind11_fail("Invalid return_value_policy for Eigen Map/Ref/Block type"); + } + } + + static constexpr auto name = props::descriptor; + + // Explicitly delete these: support python -> C++ conversion on these (i.e. these can be return + // types but not bound arguments). We still provide them (with an explicitly delete) so that + // you end up here if you try anyway. + bool load(handle, bool) = delete; + operator MapType() = delete; + template + using cast_op_type = MapType; +}; + +// We can return any map-like object (but can only load Refs, specialized next): +template +struct type_caster::value>> : eigen_map_caster {}; + +// Loader for Ref<...> arguments. See the documentation for info on how to make this work without +// copying (it requires some extra effort in many cases). +template +struct type_caster< + Eigen::Ref, + enable_if_t>::value>> + : public eigen_map_caster> { +private: + using Type = Eigen::Ref; + using props = EigenProps; + using Scalar = typename props::Scalar; + using MapType = Eigen::Map; + using Array + = array_t; + static constexpr bool need_writeable = is_eigen_mutable_map::value; + // Delay construction (these have no default constructor) + std::unique_ptr map; + std::unique_ptr ref; + // Our array. When possible, this is just a numpy array pointing to the source data, but + // sometimes we can't avoid copying (e.g. input is not a numpy array at all, has an + // incompatible layout, or is an array of a type that needs to be converted). Using a numpy + // temporary (rather than an Eigen temporary) saves an extra copy when we need both type + // conversion and storage order conversion. (Note that we refuse to use this temporary copy + // when loading an argument for a Ref with M non-const, i.e. a read-write reference). + Array copy_or_ref; + +public: + bool load(handle src, bool convert) { + // First check whether what we have is already an array of the right type. If not, we + // can't avoid a copy (because the copy is also going to do type conversion). + bool need_copy = !isinstance(src); + + EigenConformable fits; + if (!need_copy) { + // We don't need a converting copy, but we also need to check whether the strides are + // compatible with the Ref's stride requirements + auto aref = reinterpret_borrow(src); + + if (aref && (!need_writeable || aref.writeable())) { + fits = props::conformable(aref); + if (!fits) { + return false; // Incompatible dimensions + } + if (!fits.template stride_compatible()) { + need_copy = true; + } else { + copy_or_ref = std::move(aref); + } + } else { + need_copy = true; + } + } + + if (need_copy) { + // We need to copy: If we need a mutable reference, or we're not supposed to convert + // (either because we're in the no-convert overload pass, or because we're explicitly + // instructed not to copy (via `py::arg().noconvert()`) we have to fail loading. + if (!convert || need_writeable) { + return false; + } + + Array copy = Array::ensure(src); + if (!copy) { + return false; + } + fits = props::conformable(copy); + if (!fits || !fits.template stride_compatible()) { + return false; + } + copy_or_ref = std::move(copy); + loader_life_support::add_patient(copy_or_ref); + } + + ref.reset(); + map.reset(new MapType(data(copy_or_ref), + fits.rows, + fits.cols, + make_stride(fits.stride.outer(), fits.stride.inner()))); + ref.reset(new Type(*map)); + + return true; + } + + // NOLINTNEXTLINE(google-explicit-constructor) + operator Type *() { return ref.get(); } + // NOLINTNEXTLINE(google-explicit-constructor) + operator Type &() { return *ref; } + template + using cast_op_type = pybind11::detail::cast_op_type<_T>; + +private: + template ::value, int> = 0> + Scalar *data(Array &a) { + return a.mutable_data(); + } + + template ::value, int> = 0> + const Scalar *data(Array &a) { + return a.data(); + } + + // Attempt to figure out a constructor of `Stride` that will work. + // If both strides are fixed, use a default constructor: + template + using stride_ctor_default = bool_constant::value>; + // Otherwise, if there is a two-index constructor, assume it is (outer,inner) like + // Eigen::Stride, and use it: + template + using stride_ctor_dual + = bool_constant::value + && std::is_constructible::value>; + // Otherwise, if there is a one-index constructor, and just one of the strides is dynamic, use + // it (passing whichever stride is dynamic). + template + using stride_ctor_outer + = bool_constant, stride_ctor_dual>::value + && S::OuterStrideAtCompileTime == Eigen::Dynamic + && S::InnerStrideAtCompileTime != Eigen::Dynamic + && std::is_constructible::value>; + template + using stride_ctor_inner + = bool_constant, stride_ctor_dual>::value + && S::InnerStrideAtCompileTime == Eigen::Dynamic + && S::OuterStrideAtCompileTime != Eigen::Dynamic + && std::is_constructible::value>; + + template ::value, int> = 0> + static S make_stride(EigenIndex, EigenIndex) { + return S(); + } + template ::value, int> = 0> + static S make_stride(EigenIndex outer, EigenIndex inner) { + return S(outer, inner); + } + template ::value, int> = 0> + static S make_stride(EigenIndex outer, EigenIndex) { + return S(outer); + } + template ::value, int> = 0> + static S make_stride(EigenIndex, EigenIndex inner) { + return S(inner); + } +}; + +// type_caster for special matrix types (e.g. DiagonalMatrix), which are EigenBase, but not +// EigenDense (i.e. they don't have a data(), at least not with the usual matrix layout). +// load() is not supported, but we can cast them into the python domain by first copying to a +// regular Eigen::Matrix, then casting that. +template +struct type_caster::value>> { +protected: + using Matrix + = Eigen::Matrix; + using props = EigenProps; + +public: + static handle cast(const Type &src, return_value_policy /* policy */, handle /* parent */) { + handle h = eigen_encapsulate(new Matrix(src)); + return h; + } + static handle cast(const Type *src, return_value_policy policy, handle parent) { + return cast(*src, policy, parent); + } + + static constexpr auto name = props::descriptor; + + // Explicitly delete these: support python -> C++ conversion on these (i.e. these can be return + // types but not bound arguments). We still provide them (with an explicitly delete) so that + // you end up here if you try anyway. + bool load(handle, bool) = delete; + operator Type() = delete; + template + using cast_op_type = Type; +}; + +template +struct type_caster::value>> { + using Scalar = typename Type::Scalar; + using StorageIndex = remove_reference_t().outerIndexPtr())>; + using Index = typename Type::Index; + static constexpr bool rowMajor = Type::IsRowMajor; + + bool load(handle src, bool) { + if (!src) { + return false; + } + + auto obj = reinterpret_borrow(src); + object sparse_module = module_::import("scipy.sparse"); + object matrix_type = sparse_module.attr(rowMajor ? "csr_matrix" : "csc_matrix"); + + if (!type::handle_of(obj).is(matrix_type)) { + try { + obj = matrix_type(obj); + } catch (const error_already_set &) { + return false; + } + } + + auto values = array_t((object) obj.attr("data")); + auto innerIndices = array_t((object) obj.attr("indices")); + auto outerIndices = array_t((object) obj.attr("indptr")); + auto shape = pybind11::tuple((pybind11::object) obj.attr("shape")); + auto nnz = obj.attr("nnz").cast(); + + if (!values || !innerIndices || !outerIndices) { + return false; + } + + value = EigenMapSparseMatrix(shape[0].cast(), + shape[1].cast(), + std::move(nnz), + outerIndices.mutable_data(), + innerIndices.mutable_data(), + values.mutable_data()); + + return true; + } + + static handle cast(const Type &src, return_value_policy /* policy */, handle /* parent */) { + const_cast(src).makeCompressed(); + + object matrix_type + = module_::import("scipy.sparse").attr(rowMajor ? "csr_matrix" : "csc_matrix"); + + array data(src.nonZeros(), src.valuePtr()); + array outerIndices((rowMajor ? src.rows() : src.cols()) + 1, src.outerIndexPtr()); + array innerIndices(src.nonZeros(), src.innerIndexPtr()); + + return matrix_type(pybind11::make_tuple( + std::move(data), std::move(innerIndices), std::move(outerIndices)), + pybind11::make_tuple(src.rows(), src.cols())) + .release(); + } + + PYBIND11_TYPE_CASTER(Type, + const_name<(Type::IsRowMajor) != 0>("scipy.sparse.csr_matrix[", + "scipy.sparse.csc_matrix[") + + npy_format_descriptor::name + const_name("]")); +}; + +PYBIND11_NAMESPACE_END(detail) +PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/include/pybind11/eigen/tensor.h b/include/pybind11/eigen/tensor.h new file mode 100644 index 000000000..a823c0f39 --- /dev/null +++ b/include/pybind11/eigen/tensor.h @@ -0,0 +1,518 @@ +/* + pybind11/eigen/tensor.h: Transparent conversion for Eigen tensors + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#pragma once + +#include "../numpy.h" + +#if defined(__GNUC__) && !defined(__clang__) && !defined(__INTEL_COMPILER) +static_assert(__GNUC__ > 5, "Eigen Tensor support in pybind11 requires GCC > 5.0"); +#endif + +#if defined(_MSC_VER) +# pragma warning(push) +# pragma warning(disable : 4554) // Tensor.h warning +# pragma warning(disable : 4127) // Tensor.h warning +#elif defined(__MINGW32__) +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wmaybe-uninitialized" +#endif + +#include + +#if defined(_MSC_VER) +# pragma warning(pop) +#elif defined(__MINGW32__) +# pragma GCC diagnostic pop +#endif + +static_assert(EIGEN_VERSION_AT_LEAST(3, 3, 0), + "Eigen Tensor support in pybind11 requires Eigen >= 3.3.0"); + +PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + +PYBIND11_NAMESPACE_BEGIN(detail) + +inline bool is_tensor_aligned(const void *data) { + return (reinterpret_cast(data) % EIGEN_DEFAULT_ALIGN_BYTES) == 0; +} + +template +constexpr int compute_array_flag_from_tensor() { + static_assert((static_cast(T::Layout) == static_cast(Eigen::RowMajor)) + || (static_cast(T::Layout) == static_cast(Eigen::ColMajor)), + "Layout must be row or column major"); + return (static_cast(T::Layout) == static_cast(Eigen::RowMajor)) ? array::c_style + : array::f_style; +} + +template +struct eigen_tensor_helper {}; + +template +struct eigen_tensor_helper> { + using Type = Eigen::Tensor; + using ValidType = void; + + static Eigen::DSizes get_shape(const Type &f) { + return f.dimensions(); + } + + static constexpr bool + is_correct_shape(const Eigen::DSizes & /*shape*/) { + return true; + } + + template + struct helper {}; + + template + struct helper> { + static constexpr auto value = concat(const_name(((void) Is, "?"))...); + }; + + static constexpr auto dimensions_descriptor + = helper())>::value; + + template + static Type *alloc(Args &&...args) { + return new Type(std::forward(args)...); + } + + static void free(Type *tensor) { delete tensor; } +}; + +template +struct eigen_tensor_helper< + Eigen::TensorFixedSize, Options_, IndexType>> { + using Type = Eigen::TensorFixedSize, Options_, IndexType>; + using ValidType = void; + + static constexpr Eigen::DSizes + get_shape(const Type & /*f*/) { + return get_shape(); + } + + static constexpr Eigen::DSizes get_shape() { + return Eigen::DSizes(Indices...); + } + + static bool + is_correct_shape(const Eigen::DSizes &shape) { + return get_shape() == shape; + } + + static constexpr auto dimensions_descriptor = concat(const_name()...); + + template + static Type *alloc(Args &&...args) { + Eigen::aligned_allocator allocator; + return ::new (allocator.allocate(1)) Type(std::forward(args)...); + } + + static void free(Type *tensor) { + Eigen::aligned_allocator allocator; + tensor->~Type(); + allocator.deallocate(tensor, 1); + } +}; + +template +struct get_tensor_descriptor { + static constexpr auto details + = const_name(", flags.writeable", "") + + const_name(Type::Layout) == static_cast(Eigen::RowMajor)>( + ", flags.c_contiguous", ", flags.f_contiguous"); + static constexpr auto value + = const_name("numpy.ndarray[") + npy_format_descriptor::name + + const_name("[") + eigen_tensor_helper>::dimensions_descriptor + + const_name("]") + const_name(details, const_name("")) + const_name("]"); +}; + +// When EIGEN_AVOID_STL_ARRAY is defined, Eigen::DSizes does not have the begin() member +// function. Falling back to a simple loop works around this issue. +// +// We need to disable the type-limits warning for the inner loop when size = 0. + +#if defined(__GNUC__) +# pragma GCC diagnostic push +# pragma GCC diagnostic ignored "-Wtype-limits" +#endif + +template +std::vector convert_dsizes_to_vector(const Eigen::DSizes &arr) { + std::vector result(size); + + for (size_t i = 0; i < size; i++) { + result[i] = arr[i]; + } + + return result; +} + +template +Eigen::DSizes get_shape_for_array(const array &arr) { + Eigen::DSizes result; + const T *shape = arr.shape(); + for (size_t i = 0; i < size; i++) { + result[i] = shape[i]; + } + + return result; +} + +#if defined(__GNUC__) +# pragma GCC diagnostic pop +#endif + +template +struct type_caster::ValidType> { + using Helper = eigen_tensor_helper; + static constexpr auto temp_name = get_tensor_descriptor::value; + PYBIND11_TYPE_CASTER(Type, temp_name); + + bool load(handle src, bool convert) { + if (!convert) { + if (!isinstance(src)) { + return false; + } + array temp = array::ensure(src); + if (!temp) { + return false; + } + + if (!convert && !temp.dtype().is(dtype::of())) { + return false; + } + } + + array_t()> arr( + reinterpret_borrow(src)); + + if (arr.ndim() != Type::NumIndices) { + return false; + } + auto shape = get_shape_for_array(arr); + + if (!Helper::is_correct_shape(shape)) { + return false; + } + +#if EIGEN_VERSION_AT_LEAST(3, 4, 0) + auto data_pointer = arr.data(); +#else + // Handle Eigen bug + auto data_pointer = const_cast(arr.data()); +#endif + + if (is_tensor_aligned(arr.data())) { + value = Eigen::TensorMap(data_pointer, shape); + } else { + value = Eigen::TensorMap(data_pointer, shape); + } + + return true; + } + + static handle cast(Type &&src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::reference + || policy == return_value_policy::reference_internal) { + pybind11_fail("Cannot use a reference return value policy for an rvalue"); + } + return cast_impl(&src, return_value_policy::move, parent); + } + + static handle cast(const Type &&src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::reference + || policy == return_value_policy::reference_internal) { + pybind11_fail("Cannot use a reference return value policy for an rvalue"); + } + return cast_impl(&src, return_value_policy::move, parent); + } + + static handle cast(Type &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast_impl(&src, policy, parent); + } + + static handle cast(const Type &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast(&src, policy, parent); + } + + static handle cast(Type *src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic) { + policy = return_value_policy::take_ownership; + } else if (policy == return_value_policy::automatic_reference) { + policy = return_value_policy::reference; + } + return cast_impl(src, policy, parent); + } + + static handle cast(const Type *src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic) { + policy = return_value_policy::take_ownership; + } else if (policy == return_value_policy::automatic_reference) { + policy = return_value_policy::reference; + } + return cast_impl(src, policy, parent); + } + + template + static handle cast_impl(C *src, return_value_policy policy, handle parent) { + object parent_object; + bool writeable = false; + switch (policy) { + case return_value_policy::move: + if (std::is_const::value) { + pybind11_fail("Cannot move from a constant reference"); + } + + src = Helper::alloc(std::move(*src)); + + parent_object + = capsule(src, [](void *ptr) { Helper::free(reinterpret_cast(ptr)); }); + writeable = true; + break; + + case return_value_policy::take_ownership: + if (std::is_const::value) { + // This cast is ugly, and might be UB in some cases, but we don't have an + // alterantive here as we must free that memory + Helper::free(const_cast(src)); + pybind11_fail("Cannot take ownership of a const reference"); + } + + parent_object + = capsule(src, [](void *ptr) { Helper::free(reinterpret_cast(ptr)); }); + writeable = true; + break; + + case return_value_policy::copy: + writeable = true; + break; + + case return_value_policy::reference: + parent_object = none(); + writeable = !std::is_const::value; + break; + + case return_value_policy::reference_internal: + // Default should do the right thing + if (!parent) { + pybind11_fail("Cannot use reference internal when there is no parent"); + } + parent_object = reinterpret_borrow(parent); + writeable = !std::is_const::value; + break; + + default: + pybind11_fail("pybind11 bug in eigen.h, please file a bug report"); + } + + auto result = array_t()>( + convert_dsizes_to_vector(Helper::get_shape(*src)), src->data(), parent_object); + + if (!writeable) { + array_proxy(result.ptr())->flags &= ~detail::npy_api::NPY_ARRAY_WRITEABLE_; + } + + return result.release(); + } +}; + +template = true> +StoragePointerType get_array_data_for_type(array &arr) { +#if EIGEN_VERSION_AT_LEAST(3, 4, 0) + return reinterpret_cast(arr.data()); +#else + // Handle Eigen bug + return reinterpret_cast(const_cast(arr.data())); +#endif +} + +template = true> +StoragePointerType get_array_data_for_type(array &arr) { + return reinterpret_cast(arr.mutable_data()); +} + +template +struct get_storage_pointer_type; + +template +struct get_storage_pointer_type> { + using SPT = typename MapType::StoragePointerType; +}; + +template +struct get_storage_pointer_type> { + using SPT = typename MapType::PointerArgType; +}; + +template +struct type_caster, + typename eigen_tensor_helper>::ValidType> { + using MapType = Eigen::TensorMap; + using Helper = eigen_tensor_helper>; + + bool load(handle src, bool /*convert*/) { + // Note that we have a lot more checks here as we want to make sure to avoid copies + if (!isinstance(src)) { + return false; + } + auto arr = reinterpret_borrow(src); + if ((arr.flags() & compute_array_flag_from_tensor()) == 0) { + return false; + } + + if (!arr.dtype().is(dtype::of())) { + return false; + } + + if (arr.ndim() != Type::NumIndices) { + return false; + } + + constexpr bool is_aligned = (Options & Eigen::Aligned) != 0; + + if (PYBIND11_SILENCE_MSVC_C4127(is_aligned) && !is_tensor_aligned(arr.data())) { + return false; + } + + auto shape = get_shape_for_array(arr); + + if (!Helper::is_correct_shape(shape)) { + return false; + } + + if (PYBIND11_SILENCE_MSVC_C4127(needs_writeable) && !arr.writeable()) { + return false; + } + + auto result = get_array_data_for_type::SPT, + needs_writeable>(arr); + + value.reset(new MapType(std::move(result), std::move(shape))); + + return true; + } + + static handle cast(MapType &&src, return_value_policy policy, handle parent) { + return cast_impl(&src, policy, parent); + } + + static handle cast(const MapType &&src, return_value_policy policy, handle parent) { + return cast_impl(&src, policy, parent); + } + + static handle cast(MapType &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast_impl(&src, policy, parent); + } + + static handle cast(const MapType &src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic + || policy == return_value_policy::automatic_reference) { + policy = return_value_policy::copy; + } + return cast(&src, policy, parent); + } + + static handle cast(MapType *src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic) { + policy = return_value_policy::take_ownership; + } else if (policy == return_value_policy::automatic_reference) { + policy = return_value_policy::reference; + } + return cast_impl(src, policy, parent); + } + + static handle cast(const MapType *src, return_value_policy policy, handle parent) { + if (policy == return_value_policy::automatic) { + policy = return_value_policy::take_ownership; + } else if (policy == return_value_policy::automatic_reference) { + policy = return_value_policy::reference; + } + return cast_impl(src, policy, parent); + } + + template + static handle cast_impl(C *src, return_value_policy policy, handle parent) { + object parent_object; + constexpr bool writeable = !std::is_const::value; + switch (policy) { + case return_value_policy::reference: + parent_object = none(); + break; + + case return_value_policy::reference_internal: + // Default should do the right thing + if (!parent) { + pybind11_fail("Cannot use reference internal when there is no parent"); + } + parent_object = reinterpret_borrow(parent); + break; + + case return_value_policy::take_ownership: + delete src; + // fallthrough + default: + // move, take_ownership don't make any sense for a ref/map: + pybind11_fail("Invalid return_value_policy for Eigen Map type, must be either " + "reference or reference_internal"); + } + + auto result = array_t()>( + convert_dsizes_to_vector(Helper::get_shape(*src)), + src->data(), + std::move(parent_object)); + + if (!writeable) { + array_proxy(result.ptr())->flags &= ~detail::npy_api::NPY_ARRAY_WRITEABLE_; + } + + return result.release(); + } + +#if EIGEN_VERSION_AT_LEAST(3, 4, 0) + + static constexpr bool needs_writeable = !std::is_const::SPT>::type>::value; +#else + // Handle Eigen bug + static constexpr bool needs_writeable = !std::is_const::value; +#endif + +protected: + // TODO: Move to std::optional once std::optional has more support + std::unique_ptr value; + +public: + static constexpr auto name = get_tensor_descriptor::value; + explicit operator MapType *() { return value.get(); } + explicit operator MapType &() { return *value; } + explicit operator MapType &&() && { return std::move(*value); } + + template + using cast_op_type = ::pybind11::detail::movable_cast_op_type; +}; + +PYBIND11_NAMESPACE_END(detail) +PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 7296cd1b8..491f215ce 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -128,7 +128,9 @@ set(PYBIND11_TEST_FILES test_custom_type_casters test_custom_type_setup test_docstring_options - test_eigen + test_eigen_matrix + test_eigen_tensor + test_eigen_tensor_avoid_stl_array.cpp test_enum test_eval test_exceptions @@ -233,7 +235,10 @@ list(GET PYBIND11_EIGEN_VERSION_AND_HASH 1 PYBIND11_EIGEN_VERSION_HASH) # Check if Eigen is available; if not, remove from PYBIND11_TEST_FILES (but # keep it in PYBIND11_PYTEST_FILES, so that we get the "eigen is not installed" # skip message). -list(FIND PYBIND11_TEST_FILES test_eigen.cpp PYBIND11_TEST_FILES_EIGEN_I) +list(FIND PYBIND11_TEST_FILES test_eigen_matrix.cpp PYBIND11_TEST_FILES_EIGEN_I) +if(PYBIND11_TEST_FILES_EIGEN_I EQUAL -1) + list(FIND PYBIND11_TEST_FILES test_eigen_tensor.cpp PYBIND11_TEST_FILES_EIGEN_I) +endif() if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) # Try loading via newer Eigen's Eigen3Config first (bypassing tools/FindEigen3.cmake). # Eigen 3.3.1+ exports a cmake 3.0+ target for handling dependency requirements, but also @@ -289,12 +294,37 @@ if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) endif() message(STATUS "Building tests with Eigen v${EIGEN3_VERSION}") else() - list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + list(FIND PYBIND11_TEST_FILES test_eigen_matrix.cpp PYBIND11_TEST_FILES_EIGEN_I) + if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) + list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + endif() + + list(FIND PYBIND11_TEST_FILES test_eigen_tensor.cpp PYBIND11_TEST_FILES_EIGEN_I) + if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) + list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + endif() + list(FIND PYBIND11_TEST_FILES test_eigen_tensor_avoid_stl_array.cpp + PYBIND11_TEST_FILES_EIGEN_I) + if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) + list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + endif() message( STATUS "Building tests WITHOUT Eigen, use -DDOWNLOAD_EIGEN=ON on CMake 3.11+ to download") endif() endif() +# Some code doesn't support gcc 4 +if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 5.0) + list(FIND PYBIND11_TEST_FILES test_eigen_tensor.cpp PYBIND11_TEST_FILES_EIGEN_I) + if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) + list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + endif() + list(FIND PYBIND11_TEST_FILES test_eigen_tensor_avoid_stl_array.cpp PYBIND11_TEST_FILES_EIGEN_I) + if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) + list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) + endif() +endif() + # Optional dependency for some tests (boost::variant is only supported with version >= 1.56) find_package(Boost 1.56) diff --git a/tests/extra_python_package/test_files.py b/tests/extra_python_package/test_files.py index 8e1ddd850..9a9bb1556 100644 --- a/tests/extra_python_package/test_files.py +++ b/tests/extra_python_package/test_files.py @@ -55,6 +55,11 @@ detail_headers = { "include/pybind11/detail/typeid.h", } +eigen_headers = { + "include/pybind11/eigen/matrix.h", + "include/pybind11/eigen/tensor.h", +} + stl_headers = { "include/pybind11/stl/filesystem.h", } @@ -82,7 +87,7 @@ py_files = { "setup_helpers.py", } -headers = main_headers | detail_headers | stl_headers +headers = main_headers | detail_headers | eigen_headers | stl_headers src_files = headers | cmake_files | pkgconfig_files all_files = src_files | py_files @@ -92,6 +97,7 @@ sdist_files = { "pybind11/include", "pybind11/include/pybind11", "pybind11/include/pybind11/detail", + "pybind11/include/pybind11/eigen", "pybind11/include/pybind11/stl", "pybind11/share", "pybind11/share/cmake", diff --git a/tests/test_eigen.cpp b/tests/test_eigen_matrix.cpp similarity index 99% rename from tests/test_eigen.cpp rename to tests/test_eigen_matrix.cpp index b0c7bdb48..71e41d198 100644 --- a/tests/test_eigen.cpp +++ b/tests/test_eigen_matrix.cpp @@ -7,7 +7,7 @@ BSD-style license that can be found in the LICENSE file. */ -#include +#include #include #include "constructor_stats.h" @@ -81,7 +81,7 @@ struct CustomOperatorNew { EIGEN_MAKE_ALIGNED_OPERATOR_NEW; }; -TEST_SUBMODULE(eigen, m) { +TEST_SUBMODULE(eigen_matrix, m) { using FixedMatrixR = Eigen::Matrix; using FixedMatrixC = Eigen::Matrix; using DenseMatrixR = Eigen::Matrix; diff --git a/tests/test_eigen.py b/tests/test_eigen_matrix.py similarity index 99% rename from tests/test_eigen.py rename to tests/test_eigen_matrix.py index 713432a81..4407fa6ae 100644 --- a/tests/test_eigen.py +++ b/tests/test_eigen_matrix.py @@ -3,7 +3,7 @@ import pytest from pybind11_tests import ConstructorStats np = pytest.importorskip("numpy") -m = pytest.importorskip("pybind11_tests.eigen") +m = pytest.importorskip("pybind11_tests.eigen_matrix") ref = np.array( diff --git a/tests/test_eigen_tensor.cpp b/tests/test_eigen_tensor.cpp new file mode 100644 index 000000000..40b494005 --- /dev/null +++ b/tests/test_eigen_tensor.cpp @@ -0,0 +1,16 @@ +/* + tests/eigen_tensor.cpp -- automatic conversion of Eigen Tensor + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +constexpr const char *test_eigen_tensor_module_name = "eigen_tensor"; + +#define PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE eigen_tensor + +#ifdef EIGEN_AVOID_STL_ARRAY +# undef EIGEN_AVOID_STL_ARRAY +#endif + +#include "test_eigen_tensor.inl" diff --git a/tests/test_eigen_tensor.inl b/tests/test_eigen_tensor.inl new file mode 100644 index 000000000..09b35fa13 --- /dev/null +++ b/tests/test_eigen_tensor.inl @@ -0,0 +1,333 @@ +/* + tests/eigen_tensor.cpp -- automatic conversion of Eigen Tensor + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +#include + +#include "pybind11_tests.h" + +namespace PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE { + +template +void reset_tensor(M &x) { + for (int i = 0; i < x.dimension(0); i++) { + for (int j = 0; j < x.dimension(1); j++) { + for (int k = 0; k < x.dimension(2); k++) { + x(i, j, k) = i * (5 * 2) + j * 2 + k; + } + } + } +} + +template +bool check_tensor(M &x) { + for (int i = 0; i < x.dimension(0); i++) { + for (int j = 0; j < x.dimension(1); j++) { + for (int k = 0; k < x.dimension(2); k++) { + if (x(i, j, k) != (i * (5 * 2) + j * 2 + k)) { + return false; + } + } + } + } + return true; +} + +template +Eigen::Tensor &get_tensor() { + static Eigen::Tensor *x; + + if (!x) { + x = new Eigen::Tensor(3, 5, 2); + reset_tensor(*x); + } + + return *x; +} + +template +Eigen::TensorMap> &get_tensor_map() { + static Eigen::TensorMap> *x; + + if (!x) { + x = new Eigen::TensorMap>(get_tensor()); + } + + return *x; +} + +template +Eigen::TensorFixedSize, Options> &get_fixed_tensor() { + static Eigen::TensorFixedSize, Options> *x; + + if (!x) { + Eigen::aligned_allocator, Options>> + allocator; + x = new (allocator.allocate(1)) + Eigen::TensorFixedSize, Options>(); + reset_tensor(*x); + } + + return *x; +} + +template +const Eigen::Tensor &get_const_tensor() { + return get_tensor(); +} + +template +struct CustomExample { + CustomExample() : member(get_tensor()), view_member(member) {} + + Eigen::Tensor member; + Eigen::TensorMap> view_member; +}; + +template +void init_tensor_module(pybind11::module &m) { + const char *needed_options = ""; + if (PYBIND11_SILENCE_MSVC_C4127(Options == Eigen::ColMajor)) { + needed_options = "F"; + } else { + needed_options = "C"; + } + m.attr("needed_options") = needed_options; + + m.def("setup", []() { + reset_tensor(get_tensor()); + reset_tensor(get_fixed_tensor()); + }); + + m.def("is_ok", []() { + return check_tensor(get_tensor()) && check_tensor(get_fixed_tensor()); + }); + + py::class_>(m, "CustomExample") + .def(py::init<>()) + .def_readonly( + "member", &CustomExample::member, py::return_value_policy::reference_internal) + .def_readonly("member_view", + &CustomExample::view_member, + py::return_value_policy::reference_internal); + + m.def( + "copy_fixed_tensor", + []() { return &get_fixed_tensor(); }, + py::return_value_policy::copy); + + m.def( + "copy_tensor", []() { return &get_tensor(); }, py::return_value_policy::copy); + + m.def( + "copy_const_tensor", + []() { return &get_const_tensor(); }, + py::return_value_policy::copy); + + m.def( + "move_fixed_tensor_copy", + []() -> Eigen::TensorFixedSize, Options> { + return get_fixed_tensor(); + }, + py::return_value_policy::move); + + m.def( + "move_tensor_copy", + []() -> Eigen::Tensor { return get_tensor(); }, + py::return_value_policy::move); + + m.def( + "move_const_tensor", + []() -> const Eigen::Tensor & { return get_const_tensor(); }, + py::return_value_policy::move); + + m.def( + "take_fixed_tensor", + + []() { + Eigen::aligned_allocator< + Eigen::TensorFixedSize, Options>> + allocator; + return new (allocator.allocate(1)) + Eigen::TensorFixedSize, Options>( + get_fixed_tensor()); + }, + py::return_value_policy::take_ownership); + + m.def( + "take_tensor", + []() { return new Eigen::Tensor(get_tensor()); }, + py::return_value_policy::take_ownership); + + m.def( + "take_const_tensor", + []() -> const Eigen::Tensor * { + return new Eigen::Tensor(get_tensor()); + }, + py::return_value_policy::take_ownership); + + m.def( + "take_view_tensor", + []() -> const Eigen::TensorMap> * { + return new Eigen::TensorMap>(get_tensor()); + }, + py::return_value_policy::take_ownership); + + m.def( + "reference_tensor", + []() { return &get_tensor(); }, + py::return_value_policy::reference); + + m.def( + "reference_tensor_v2", + []() -> Eigen::Tensor & { return get_tensor(); }, + py::return_value_policy::reference); + + m.def( + "reference_tensor_internal", + []() { return &get_tensor(); }, + py::return_value_policy::reference_internal); + + m.def( + "reference_fixed_tensor", + []() { return &get_tensor(); }, + py::return_value_policy::reference); + + m.def( + "reference_const_tensor", + []() { return &get_const_tensor(); }, + py::return_value_policy::reference); + + m.def( + "reference_const_tensor_v2", + []() -> const Eigen::Tensor & { return get_const_tensor(); }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor", + []() -> Eigen::TensorMap> { + return get_tensor_map(); + }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor_v2", + // NOLINTNEXTLINE(readability-const-return-type) + []() -> const Eigen::TensorMap> { + return get_tensor_map(); // NOLINT(readability-const-return-type) + }, // NOLINT(readability-const-return-type) + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor_v3", + []() -> Eigen::TensorMap> * { + return &get_tensor_map(); + }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor_v4", + []() -> const Eigen::TensorMap> * { + return &get_tensor_map(); + }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor_v5", + []() -> Eigen::TensorMap> & { + return get_tensor_map(); + }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_tensor_v6", + []() -> const Eigen::TensorMap> & { + return get_tensor_map(); + }, + py::return_value_policy::reference); + + m.def( + "reference_view_of_fixed_tensor", + []() { + return Eigen::TensorMap< + Eigen::TensorFixedSize, Options>>( + get_fixed_tensor()); + }, + py::return_value_policy::reference); + + m.def("round_trip_tensor", + [](const Eigen::Tensor &tensor) { return tensor; }); + + m.def( + "round_trip_tensor_noconvert", + [](const Eigen::Tensor &tensor) { return tensor; }, + py::arg("tensor").noconvert()); + + m.def("round_trip_tensor2", + [](const Eigen::Tensor &tensor) { return tensor; }); + + m.def("round_trip_fixed_tensor", + [](const Eigen::TensorFixedSize, Options> &tensor) { + return tensor; + }); + + m.def( + "round_trip_view_tensor", + [](Eigen::TensorMap> view) { return view; }, + py::return_value_policy::reference); + + m.def( + "round_trip_view_tensor_ref", + [](Eigen::TensorMap> &view) { return view; }, + py::return_value_policy::reference); + + m.def( + "round_trip_view_tensor_ptr", + [](Eigen::TensorMap> *view) { return view; }, + py::return_value_policy::reference); + + m.def( + "round_trip_aligned_view_tensor", + [](Eigen::TensorMap, Eigen::Aligned> view) { + return view; + }, + py::return_value_policy::reference); + + m.def( + "round_trip_const_view_tensor", + [](Eigen::TensorMap> view) { + return Eigen::Tensor(view); + }, + py::return_value_policy::move); + + m.def( + "round_trip_rank_0", + [](const Eigen::Tensor &tensor) { return tensor; }, + py::return_value_policy::move); + + m.def( + "round_trip_rank_0_noconvert", + [](const Eigen::Tensor &tensor) { return tensor; }, + py::arg("tensor").noconvert(), + py::return_value_policy::move); + + m.def( + "round_trip_rank_0_view", + [](Eigen::TensorMap> &tensor) { return tensor; }, + py::return_value_policy::reference); +} + +void test_module(py::module_ &); +test_initializer name(test_eigen_tensor_module_name, test_module); +void test_module(py::module_ &m) { + auto f_style = m.def_submodule("f_style"); + auto c_style = m.def_submodule("c_style"); + + init_tensor_module(f_style); + init_tensor_module(c_style); +} + +} // namespace PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE diff --git a/tests/test_eigen_tensor.py b/tests/test_eigen_tensor.py new file mode 100644 index 000000000..5ee5fa01b --- /dev/null +++ b/tests/test_eigen_tensor.py @@ -0,0 +1,288 @@ +import sys + +import pytest + +np = pytest.importorskip("numpy") +eigen_tensor = pytest.importorskip("pybind11_tests.eigen_tensor") +submodules = [eigen_tensor.c_style, eigen_tensor.f_style] +try: + from pybind11_tests import eigen_tensor_avoid_stl_array as avoid + + submodules += [avoid.c_style, avoid.f_style] +except ImportError: + pass + +tensor_ref = np.empty((3, 5, 2), dtype=np.int64) + +for i in range(tensor_ref.shape[0]): + for j in range(tensor_ref.shape[1]): + for k in range(tensor_ref.shape[2]): + tensor_ref[i, j, k] = i * (5 * 2) + j * 2 + k + +indices = (2, 3, 1) + + +@pytest.fixture(autouse=True) +def cleanup(): + for module in submodules: + module.setup() + + yield + + for module in submodules: + assert module.is_ok() + + +def test_import_avoid_stl_array(): + pytest.importorskip("pybind11_tests.eigen_tensor_avoid_stl_array") + assert len(submodules) == 4 + + +def assert_equal_tensor_ref(mat, writeable=True, modified=None): + assert mat.flags.writeable == writeable + + copy = np.array(tensor_ref) + if modified is not None: + copy[indices] = modified + + np.testing.assert_array_equal(mat, copy) + + +@pytest.mark.parametrize("m", submodules) +@pytest.mark.parametrize("member_name", ["member", "member_view"]) +def test_reference_internal(m, member_name): + + if not hasattr(sys, "getrefcount"): + pytest.skip("No reference counting") + foo = m.CustomExample() + counts = sys.getrefcount(foo) + mem = getattr(foo, member_name) + assert_equal_tensor_ref(mem, writeable=False) + new_counts = sys.getrefcount(foo) + assert new_counts == counts + 1 + assert_equal_tensor_ref(mem, writeable=False) + del mem + assert sys.getrefcount(foo) == counts + + +assert_equal_funcs = [ + "copy_tensor", + "copy_fixed_tensor", + "copy_const_tensor", + "move_tensor_copy", + "move_fixed_tensor_copy", + "take_tensor", + "take_fixed_tensor", + "reference_tensor", + "reference_tensor_v2", + "reference_fixed_tensor", + "reference_view_of_tensor", + "reference_view_of_tensor_v3", + "reference_view_of_tensor_v5", + "reference_view_of_fixed_tensor", +] + +assert_equal_const_funcs = [ + "reference_view_of_tensor_v2", + "reference_view_of_tensor_v4", + "reference_view_of_tensor_v6", + "reference_const_tensor", + "reference_const_tensor_v2", +] + + +@pytest.mark.parametrize("m", submodules) +@pytest.mark.parametrize("func_name", assert_equal_funcs + assert_equal_const_funcs) +def test_convert_tensor_to_py(m, func_name): + writeable = func_name in assert_equal_funcs + assert_equal_tensor_ref(getattr(m, func_name)(), writeable=writeable) + + +@pytest.mark.parametrize("m", submodules) +def test_bad_cpp_to_python_casts(m): + + with pytest.raises( + RuntimeError, match="Cannot use reference internal when there is no parent" + ): + m.reference_tensor_internal() + + with pytest.raises(RuntimeError, match="Cannot move from a constant reference"): + m.move_const_tensor() + + with pytest.raises( + RuntimeError, match="Cannot take ownership of a const reference" + ): + m.take_const_tensor() + + with pytest.raises( + RuntimeError, + match="Invalid return_value_policy for Eigen Map type, must be either reference or reference_internal", + ): + m.take_view_tensor() + + +@pytest.mark.parametrize("m", submodules) +def test_bad_python_to_cpp_casts(m): + + with pytest.raises( + TypeError, match=r"^round_trip_tensor\(\): incompatible function arguments" + ): + m.round_trip_tensor(np.zeros((2, 3))) + + with pytest.raises(TypeError, match=r"^Cannot cast array data from dtype"): + m.round_trip_tensor(np.zeros(dtype=np.str_, shape=(2, 3, 1))) + + with pytest.raises( + TypeError, + match=r"^round_trip_tensor_noconvert\(\): incompatible function arguments", + ): + m.round_trip_tensor_noconvert(tensor_ref) + + assert_equal_tensor_ref( + m.round_trip_tensor_noconvert(tensor_ref.astype(np.float64)) + ) + + if m.needed_options == "F": + bad_options = "C" + else: + bad_options = "F" + # Shape, dtype and the order need to be correct for a TensorMap cast + with pytest.raises( + TypeError, match=r"^round_trip_view_tensor\(\): incompatible function arguments" + ): + m.round_trip_view_tensor( + np.zeros((3, 5, 2), dtype=np.float64, order=bad_options) + ) + + with pytest.raises( + TypeError, match=r"^round_trip_view_tensor\(\): incompatible function arguments" + ): + m.round_trip_view_tensor( + np.zeros((3, 5, 2), dtype=np.float32, order=m.needed_options) + ) + + with pytest.raises( + TypeError, match=r"^round_trip_view_tensor\(\): incompatible function arguments" + ): + m.round_trip_view_tensor( + np.zeros((3, 5), dtype=np.float64, order=m.needed_options) + ) + + with pytest.raises( + TypeError, match=r"^round_trip_view_tensor\(\): incompatible function arguments" + ): + temp = np.zeros((3, 5, 2), dtype=np.float64, order=m.needed_options) + m.round_trip_view_tensor( + temp[:, ::-1, :], + ) + + with pytest.raises( + TypeError, match=r"^round_trip_view_tensor\(\): incompatible function arguments" + ): + temp = np.zeros((3, 5, 2), dtype=np.float64, order=m.needed_options) + temp.setflags(write=False) + m.round_trip_view_tensor(temp) + + +@pytest.mark.parametrize("m", submodules) +def test_references_actually_refer(m): + + a = m.reference_tensor() + temp = a[indices] + a[indices] = 100 + assert_equal_tensor_ref(m.copy_const_tensor(), modified=100) + a[indices] = temp + assert_equal_tensor_ref(m.copy_const_tensor()) + + a = m.reference_view_of_tensor() + a[indices] = 100 + assert_equal_tensor_ref(m.copy_const_tensor(), modified=100) + a[indices] = temp + assert_equal_tensor_ref(m.copy_const_tensor()) + + +@pytest.mark.parametrize("m", submodules) +def test_round_trip(m): + + assert_equal_tensor_ref(m.round_trip_tensor(tensor_ref)) + + with pytest.raises(TypeError, match="^Cannot cast array data from"): + assert_equal_tensor_ref(m.round_trip_tensor2(tensor_ref)) + + assert_equal_tensor_ref(m.round_trip_tensor2(np.array(tensor_ref, dtype=np.int32))) + assert_equal_tensor_ref(m.round_trip_fixed_tensor(tensor_ref)) + assert_equal_tensor_ref(m.round_trip_aligned_view_tensor(m.reference_tensor())) + + copy = np.array(tensor_ref, dtype=np.float64, order=m.needed_options) + assert_equal_tensor_ref(m.round_trip_view_tensor(copy)) + assert_equal_tensor_ref(m.round_trip_view_tensor_ref(copy)) + assert_equal_tensor_ref(m.round_trip_view_tensor_ptr(copy)) + copy.setflags(write=False) + assert_equal_tensor_ref(m.round_trip_const_view_tensor(copy)) + + np.testing.assert_array_equal( + tensor_ref[:, ::-1, :], m.round_trip_tensor(tensor_ref[:, ::-1, :]) + ) + + assert m.round_trip_rank_0(np.float64(3.5)) == 3.5 + assert m.round_trip_rank_0(3.5) == 3.5 + + with pytest.raises( + TypeError, + match=r"^round_trip_rank_0_noconvert\(\): incompatible function arguments", + ): + m.round_trip_rank_0_noconvert(np.float64(3.5)) + + with pytest.raises( + TypeError, + match=r"^round_trip_rank_0_noconvert\(\): incompatible function arguments", + ): + m.round_trip_rank_0_noconvert(3.5) + + with pytest.raises( + TypeError, match=r"^round_trip_rank_0_view\(\): incompatible function arguments" + ): + m.round_trip_rank_0_view(np.float64(3.5)) + + with pytest.raises( + TypeError, match=r"^round_trip_rank_0_view\(\): incompatible function arguments" + ): + m.round_trip_rank_0_view(3.5) + + +@pytest.mark.parametrize("m", submodules) +def test_round_trip_references_actually_refer(m): + + # Need to create a copy that matches the type on the C side + copy = np.array(tensor_ref, dtype=np.float64, order=m.needed_options) + a = m.round_trip_view_tensor(copy) + temp = a[indices] + a[indices] = 100 + assert_equal_tensor_ref(copy, modified=100) + a[indices] = temp + assert_equal_tensor_ref(copy) + + +@pytest.mark.parametrize("m", submodules) +def test_doc_string(m, doc): + assert ( + doc(m.copy_tensor) == "copy_tensor() -> numpy.ndarray[numpy.float64[?, ?, ?]]" + ) + assert ( + doc(m.copy_fixed_tensor) + == "copy_fixed_tensor() -> numpy.ndarray[numpy.float64[3, 5, 2]]" + ) + assert ( + doc(m.reference_const_tensor) + == "reference_const_tensor() -> numpy.ndarray[numpy.float64[?, ?, ?]]" + ) + + order_flag = f"flags.{m.needed_options.lower()}_contiguous" + assert doc(m.round_trip_view_tensor) == ( + f"round_trip_view_tensor(arg0: numpy.ndarray[numpy.float64[?, ?, ?], flags.writeable, {order_flag}])" + + f" -> numpy.ndarray[numpy.float64[?, ?, ?], flags.writeable, {order_flag}]" + ) + assert doc(m.round_trip_const_view_tensor) == ( + f"round_trip_const_view_tensor(arg0: numpy.ndarray[numpy.float64[?, ?, ?], {order_flag}])" + + " -> numpy.ndarray[numpy.float64[?, ?, ?]]" + ) diff --git a/tests/test_eigen_tensor_avoid_stl_array.cpp b/tests/test_eigen_tensor_avoid_stl_array.cpp new file mode 100644 index 000000000..58bedf62d --- /dev/null +++ b/tests/test_eigen_tensor_avoid_stl_array.cpp @@ -0,0 +1,16 @@ +/* + tests/eigen_tensor.cpp -- automatic conversion of Eigen Tensor + + All rights reserved. Use of this source code is governed by a + BSD-style license that can be found in the LICENSE file. +*/ + +constexpr const char *test_eigen_tensor_module_name = "eigen_tensor_avoid_stl_array"; + +#ifndef EIGEN_AVOID_STL_ARRAY +# define EIGEN_AVOID_STL_ARRAY +#endif + +#define PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE eigen_tensor_avoid_stl_array + +#include "test_eigen_tensor.inl" diff --git a/tests/test_numpy_array.cpp b/tests/test_numpy_array.cpp index 69ddbe1ef..b118e2c6c 100644 --- a/tests/test_numpy_array.cpp +++ b/tests/test_numpy_array.cpp @@ -521,4 +521,6 @@ TEST_SUBMODULE(numpy_array, sm) { sm.def("test_fmt_desc_double", [](const py::array_t &) {}); sm.def("test_fmt_desc_const_float", [](const py::array_t &) {}); sm.def("test_fmt_desc_const_double", [](const py::array_t &) {}); + + sm.def("round_trip_float", [](double d) { return d; }); } diff --git a/tests/test_numpy_array.py b/tests/test_numpy_array.py index 504963b16..cdec9ad60 100644 --- a/tests/test_numpy_array.py +++ b/tests/test_numpy_array.py @@ -585,3 +585,9 @@ def test_dtype_refcount_leak(): m.ndim(a) after = getrefcount(dtype) assert after == before + + +def test_round_trip_float(): + arr = np.zeros((), np.float64) + arr[()] = 37.2 + assert m.round_trip_float(arr) == 37.2 diff --git a/tools/setup_global.py.in b/tools/setup_global.py.in index d91468c10..885ac5c72 100644 --- a/tools/setup_global.py.in +++ b/tools/setup_global.py.in @@ -27,10 +27,11 @@ class InstallHeadersNested(install_headers): main_headers = glob.glob("pybind11/include/pybind11/*.h") detail_headers = glob.glob("pybind11/include/pybind11/detail/*.h") +eigen_headers = glob.glob("pybind11/include/pybind11/eigen/*.h") stl_headers = glob.glob("pybind11/include/pybind11/stl/*.h") cmake_files = glob.glob("pybind11/share/cmake/pybind11/*.cmake") pkgconfig_files = glob.glob("pybind11/share/pkgconfig/*.pc") -headers = main_headers + detail_headers + stl_headers +headers = main_headers + detail_headers + stl_headers + eigen_headers cmdclass = {"install_headers": InstallHeadersNested} $extra_cmd @@ -55,6 +56,7 @@ setup( (base + "share/pkgconfig", pkgconfig_files), (base + "include/pybind11", main_headers), (base + "include/pybind11/detail", detail_headers), + (base + "include/pybind11/eigen", eigen_headers), (base + "include/pybind11/stl", stl_headers), ], cmdclass=cmdclass, diff --git a/tools/setup_main.py.in b/tools/setup_main.py.in index 65198bdb6..6358cc7b9 100644 --- a/tools/setup_main.py.in +++ b/tools/setup_main.py.in @@ -15,6 +15,7 @@ setup( "pybind11", "pybind11.include.pybind11", "pybind11.include.pybind11.detail", + "pybind11.include.pybind11.eigen", "pybind11.include.pybind11.stl", "pybind11.share.cmake.pybind11", "pybind11.share.pkgconfig", @@ -23,6 +24,7 @@ setup( "pybind11": ["py.typed"], "pybind11.include.pybind11": ["*.h"], "pybind11.include.pybind11.detail": ["*.h"], + "pybind11.include.pybind11.eigen": ["*.h"], "pybind11.include.pybind11.stl": ["*.h"], "pybind11.share.cmake.pybind11": ["*.cmake"], "pybind11.share.pkgconfig": ["*.pc"], From 17c68091658f036eb6aed05cebabfc9265407a5b Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 20 Oct 2022 05:49:52 -0700 Subject: [PATCH 02/53] ci: update PGI build (old one no longer signed) (#4260) * Simply replace "22.3" with "22.9" to see what happens. * Remove PYBIND11_TEST_FILTER to see what happens. * Revert "Remove PYBIND11_TEST_FILTER to see what happens." This reverts commit 0cba2cef0c5c5bcbf57cad219c7d5dfb0cda241c. * Remove only test_smart_ptr.cpp to see what happens. * Revert "Remove only test_smart_ptr.cpp to see what happens." This reverts commit 8e9df22c85292003abcab50260393e547567fc18. * Remove only test_virtual_functions.cpp to see what happens. --- .github/workflows/ci.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bc7175d82..2951a0ad1 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -391,7 +391,7 @@ jobs: # Testing on CentOS 7 + PGI compilers, which seems to require more workarounds centos-nvhpc7: runs-on: ubuntu-latest - name: "🐍 3 • CentOS7 / PGI 22.3 • x64" + name: "🐍 3 • CentOS7 / PGI 22.9 • x64" container: centos:7 steps: @@ -401,7 +401,7 @@ jobs: run: yum update -y && yum install -y epel-release && yum install -y git python3-devel make environment-modules cmake3 yum-utils - name: Install NVidia HPC SDK - run: yum-config-manager --add-repo https://developer.download.nvidia.com/hpc-sdk/rhel/nvhpc.repo && yum -y install nvhpc-22.3 + run: yum-config-manager --add-repo https://developer.download.nvidia.com/hpc-sdk/rhel/nvhpc.repo && yum -y install nvhpc-22.9 # On CentOS 7, we have to filter a few tests (compiler internal error) # and allow deeper template recursion (not needed on CentOS 8 with a newer @@ -411,12 +411,12 @@ jobs: shell: bash run: | source /etc/profile.d/modules.sh - module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/22.3 + module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/22.9 cmake3 -S . -B build -DDOWNLOAD_CATCH=ON \ -DCMAKE_CXX_STANDARD=11 \ -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") \ -DCMAKE_CXX_FLAGS="-Wc,--pending_instantiations=0" \ - -DPYBIND11_TEST_FILTER="test_smart_ptr.cpp;test_virtual_functions.cpp" + -DPYBIND11_TEST_FILTER="test_smart_ptr.cpp" # Building before installing Pip should produce a warning but not an error - name: Build From c3854682e6644c60ff4f99f81595636f582b04f5 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 20 Oct 2022 09:39:19 -0400 Subject: [PATCH 03/53] ci(fix): don't label weekly dep updates & ci fixes (#4264) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- .github/workflows/labeler.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml index d2b597968..5152cebca 100644 --- a/.github/workflows/labeler.yml +++ b/.github/workflows/labeler.yml @@ -10,7 +10,10 @@ jobs: steps: - uses: actions/labeler@main - if: github.event.pull_request.merged == true + if: > + github.event.pull_request.merged == true && + !startsWith(github.event.pull_request.title, 'chore(deps):') && + !startsWith(github.event.plul_request.title, 'ci(fix):' with: repo-token: ${{ secrets.GITHUB_TOKEN }} configuration-path: .github/labeler_merged.yml From 1d4a65e2f145a25b66e44e9d5edae9ec11abbc73 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 20 Oct 2022 10:35:18 -0400 Subject: [PATCH 04/53] feat: add entrypoint for cmake modules dir (#4258) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- setup.py | 2 ++ tests/extra_python_package/test_files.py | 2 ++ tools/setup_main.py.in | 3 +++ 3 files changed, 7 insertions(+) diff --git a/setup.py b/setup.py index 68573519c..9097440bf 100644 --- a/setup.py +++ b/setup.py @@ -144,6 +144,8 @@ with remove_output("pybind11/include", "pybind11/share"): stdout=sys.stdout, stderr=sys.stderr, ) + if not global_sdist: + Path("pybind11/share/cmake/pybind11/__init__.py").touch() txt = get_and_replace(setup_py, version=version, extra_cmd=extra_cmd) code = compile(txt, setup_py, "exec") diff --git a/tests/extra_python_package/test_files.py b/tests/extra_python_package/test_files.py index 9a9bb1556..ae69e56d3 100644 --- a/tests/extra_python_package/test_files.py +++ b/tests/extra_python_package/test_files.py @@ -166,6 +166,7 @@ def test_build_sdist(monkeypatch, tmpdir): files |= {f"pybind11{n}" for n in local_sdist_files} files.add("pybind11.egg-info/entry_points.txt") files.add("pybind11.egg-info/requires.txt") + files.add("pybind11/share/cmake/pybind11/__init__.py") assert simpler == files with open(os.path.join(MAIN_DIR, "tools", "setup_main.py.in"), "rb") as f: @@ -252,6 +253,7 @@ def tests_build_wheel(monkeypatch, tmpdir): "dist-info/entry_points.txt", "dist-info/top_level.txt", } + files.add("pybind11/share/cmake/pybind11/__init__.py") with zipfile.ZipFile(str(wheel)) as z: names = z.namelist() diff --git a/tools/setup_main.py.in b/tools/setup_main.py.in index 6358cc7b9..b3681bba8 100644 --- a/tools/setup_main.py.in +++ b/tools/setup_main.py.in @@ -38,6 +38,9 @@ setup( ], "pipx.run": [ "pybind11 = pybind11.__main__:main", + ], + "cmake.modules": [ + "pybind11 = pybind11.share.cmake.pybind11", ] }, cmdclass=cmdclass From 128d988ef183cfb6334714f0bcf8b8e7b7e2e95b Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 20 Oct 2022 10:37:12 -0400 Subject: [PATCH 05/53] ci: fix labeler --- .github/workflows/labeler.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml index 5152cebca..165a2fd87 100644 --- a/.github/workflows/labeler.yml +++ b/.github/workflows/labeler.yml @@ -13,7 +13,8 @@ jobs: if: > github.event.pull_request.merged == true && !startsWith(github.event.pull_request.title, 'chore(deps):') && - !startsWith(github.event.plul_request.title, 'ci(fix):' + !startsWith(github.event.pull_request.title, 'ci(fix):') && + !startsWith(github.event.pull_request.title, 'docs(changelog):') with: repo-token: ${{ secrets.GITHUB_TOKEN }} configuration-path: .github/labeler_merged.yml From 36ccb08b0d8f007f674ce373095b4b278689fc10 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Thu, 20 Oct 2022 10:58:04 -0400 Subject: [PATCH 06/53] docs: update changelog (#4265) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- docs/changelog.rst | 101 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) diff --git a/docs/changelog.rst b/docs/changelog.rst index 9c8ff423f..1199ecded 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -16,6 +16,107 @@ IN DEVELOPMENT Changes will be summarized here periodically. +Version 2.10.1 (Oct 2?, 2022) +----------------------------- + + +Changes: + + +* Allow ``pybind11::capsule`` constructor to take null destructor pointers. + `#4221 `_ + +* ``embed.h`` was changed so that ``PYTHONPATH`` is used also with Python 3.11 + (established behavior). + `#4119 `_ + +Bug fixes: + +* Fix MSVC 2019 v.1924 & C++14 mode error for ``overload_cast``. + `#4188 `_ + +* Make augmented assignment operators non-const for the object-api. Behavior + was previously broken for augmented assignment operators. + `#4065 `_ + +* Add proper error checking to C++ bindings for Python list append and insert. + `#4208 `_ + +* Work-around for Nvidia's CUDA nvcc compiler in versions 11.4.0 - 11.8.0. + `#4220 `_ + +* A workaround for PyPy was added in the ``py::error_already_set`` + implementation, related to PR `#1895 `_ + released with v2.10.0. + `#4079 `_ + +* Fixed compiler errors when C++23 ``std::forward_like`` is available. + `#4136 `_ + +* Properly raise exceptions in contains methods (like when an object in unhashable). + `#4209 `_ + +* Further improve another error in exception handling. + `#4232 `_ + +* ``get_local_internals()`` was made compatible with + ``finalize_interpreter()``, fixing potential freezes during interpreter + finalization. + `#4192 `_ + + +Performance and style: + +* Reserve space in set and STL map casters if possible. This will prevent + unnecessary rehashing / resizing by knowing the number of keys ahead of time + for Python to C++ casting. This improvement will greatly speed up the casting + of large unordered maps and sets. + `#4194 `_ + +* GIL RAII scopes are non-copyable to avoid potential bugs. + `#4183 `_ + +* Explicitly default all relevant ctors for pytypes in the ``PYBIND11_OBJECT`` + macros and enforce the clang-tidy checks ``modernize-use-equals-default`` in + macros as well. + `#4017 `_ + +* Optimize iterator advancement in C++ bindings. + `#4237 `_ + +* Use the modern ``PyObject_GenericGetDict`` and ``PyObject_GenericSetDict`` + for handling dynamic attribute dictionaries. + `#4106 `_ + +* Document that users should use ``PYBIND11_NAMESPACE`` instead of using ``pybind11`` when + opening namespaces. Using namespace declarations and namespace qualification + remain the same as ``pybind11``. This is done to ensure consistent symbol + visibility. + `#4098 `_ + +* Mark ``detail::forward_like`` as constexpr. + `#4147 `_ + +* Optimize unpacking_collector when processing ``arg_v`` arguments. + `#4219 `_ + + +Build system improvements: + +* Experimental support for ``cmake.modules`` entrypoint. + `#4258 `_ + +* Include a pkg-config file when installing pybind11, such as in the Python + package. + `#4077 `_ + +* Avoid stripping debug symbols when ``CMAKE_BUILD_TYPE`` is set to ``DEBUG`` + instead of ``Debug``. + `#4078 `_ + +* Followup to `#3948 `_, fixing vcpkg again. + `#4123 `_ + Version 2.10.0 (Jul 15, 2022) ----------------------------- From 2ce76f783373b1dd34e94c8283598af3051ef3d7 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 21 Oct 2022 12:51:26 -0400 Subject: [PATCH 07/53] Cleanup casters to release none() to avoid ref counting (#4269) --- include/pybind11/cast.h | 6 +++--- include/pybind11/functional.h | 2 +- include/pybind11/stl.h | 2 +- 3 files changed, 5 insertions(+), 5 deletions(-) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index b89e4946a..5699d4fb0 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -248,7 +248,7 @@ public: return false; } static handle cast(T, return_value_policy /* policy */, handle /* parent */) { - return none().inc_ref(); + return none().release(); } PYBIND11_TYPE_CASTER(T, const_name("None")); }; @@ -291,7 +291,7 @@ public: if (ptr) { return capsule(ptr).release(); } - return none().inc_ref(); + return none().release(); } template @@ -537,7 +537,7 @@ public: static handle cast(const CharT *src, return_value_policy policy, handle parent) { if (src == nullptr) { - return pybind11::none().inc_ref(); + return pybind11::none().release(); } return StringCaster::cast(StringType(src), policy, parent); } diff --git a/include/pybind11/functional.h b/include/pybind11/functional.h index 4034990d8..102d1a938 100644 --- a/include/pybind11/functional.h +++ b/include/pybind11/functional.h @@ -110,7 +110,7 @@ public: template static handle cast(Func &&f_, return_value_policy policy, handle /* parent */) { if (!f_) { - return none().inc_ref(); + return none().release(); } auto result = f_.template target(); diff --git a/include/pybind11/stl.h b/include/pybind11/stl.h index 8f243502e..2d144b598 100644 --- a/include/pybind11/stl.h +++ b/include/pybind11/stl.h @@ -311,7 +311,7 @@ struct optional_caster { template static handle cast(T &&src, return_value_policy policy, handle parent) { if (!src) { - return none().inc_ref(); + return none().release(); } if (!std::is_lvalue_reference::value) { policy = return_value_policy_override::policy(policy); From 91cfb7702280d3176e56a6667d8a091d9731a779 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 21 Oct 2022 17:25:53 -0400 Subject: [PATCH 08/53] Revert "feat: add entrypoint for cmake modules dir" (#4270) * Revert "feat: add entrypoint for cmake modules dir (#4258)" This reverts commit 1d4a65e2f145a25b66e44e9d5edae9ec11abbc73. * docs: revert changelog mention too --- docs/changelog.rst | 3 --- setup.py | 2 -- tests/extra_python_package/test_files.py | 2 -- tools/setup_main.py.in | 3 --- 4 files changed, 10 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 1199ecded..df3181c52 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -103,9 +103,6 @@ Performance and style: Build system improvements: -* Experimental support for ``cmake.modules`` entrypoint. - `#4258 `_ - * Include a pkg-config file when installing pybind11, such as in the Python package. `#4077 `_ diff --git a/setup.py b/setup.py index 9097440bf..68573519c 100644 --- a/setup.py +++ b/setup.py @@ -144,8 +144,6 @@ with remove_output("pybind11/include", "pybind11/share"): stdout=sys.stdout, stderr=sys.stderr, ) - if not global_sdist: - Path("pybind11/share/cmake/pybind11/__init__.py").touch() txt = get_and_replace(setup_py, version=version, extra_cmd=extra_cmd) code = compile(txt, setup_py, "exec") diff --git a/tests/extra_python_package/test_files.py b/tests/extra_python_package/test_files.py index ae69e56d3..9a9bb1556 100644 --- a/tests/extra_python_package/test_files.py +++ b/tests/extra_python_package/test_files.py @@ -166,7 +166,6 @@ def test_build_sdist(monkeypatch, tmpdir): files |= {f"pybind11{n}" for n in local_sdist_files} files.add("pybind11.egg-info/entry_points.txt") files.add("pybind11.egg-info/requires.txt") - files.add("pybind11/share/cmake/pybind11/__init__.py") assert simpler == files with open(os.path.join(MAIN_DIR, "tools", "setup_main.py.in"), "rb") as f: @@ -253,7 +252,6 @@ def tests_build_wheel(monkeypatch, tmpdir): "dist-info/entry_points.txt", "dist-info/top_level.txt", } - files.add("pybind11/share/cmake/pybind11/__init__.py") with zipfile.ZipFile(str(wheel)) as z: names = z.namelist() diff --git a/tools/setup_main.py.in b/tools/setup_main.py.in index b3681bba8..6358cc7b9 100644 --- a/tools/setup_main.py.in +++ b/tools/setup_main.py.in @@ -38,9 +38,6 @@ setup( ], "pipx.run": [ "pybind11 = pybind11.__main__:main", - ], - "cmake.modules": [ - "pybind11 = pybind11.share.cmake.pybind11", ] }, cmdclass=cmdclass From 17c1e27b3d202aaab6d5443b1800124bbf24d9ca Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Fri, 21 Oct 2022 18:04:01 -0400 Subject: [PATCH 09/53] fix: Revert pfect args make iterator (#4234) * Revert "chore: perfectly forward all make_iterator args (#3980)" This reverts commit 8da58da53996e9aa40d051b3b84cb3220fdbbb58. * Redo unrelated optimization in commit * Add tests for ambiguous overloads --- include/pybind11/pybind11.h | 22 ++++++++-------------- tests/test_sequences_and_iterators.cpp | 19 +++++++++++++++++++ tests/test_sequences_and_iterators.py | 8 ++++++++ 3 files changed, 35 insertions(+), 14 deletions(-) diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index fb4b7578d..e662236d0 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -2333,7 +2333,7 @@ template -iterator make_iterator_impl(Iterator &&first, Sentinel &&last, Extra &&...extra) { +iterator make_iterator_impl(Iterator first, Sentinel last, Extra &&...extra) { using state = detail::iterator_state; // TODO: state captures only the types of Extra, not the values @@ -2359,7 +2359,7 @@ iterator make_iterator_impl(Iterator &&first, Sentinel &&last, Extra &&...extra) Policy); } - return cast(state{std::forward(first), std::forward(last), true}); + return cast(state{first, last, true}); } PYBIND11_NAMESPACE_END(detail) @@ -2370,15 +2370,13 @@ template ::result_type, typename... Extra> -iterator make_iterator(Iterator &&first, Sentinel &&last, Extra &&...extra) { +iterator make_iterator(Iterator first, Sentinel last, Extra &&...extra) { return detail::make_iterator_impl, Policy, Iterator, Sentinel, ValueType, - Extra...>(std::forward(first), - std::forward(last), - std::forward(extra)...); + Extra...>(first, last, std::forward(extra)...); } /// Makes a python iterator over the keys (`.first`) of a iterator over pairs from a @@ -2388,15 +2386,13 @@ template ::result_type, typename... Extra> -iterator make_key_iterator(Iterator &&first, Sentinel &&last, Extra &&...extra) { +iterator make_key_iterator(Iterator first, Sentinel last, Extra &&...extra) { return detail::make_iterator_impl, Policy, Iterator, Sentinel, KeyType, - Extra...>(std::forward(first), - std::forward(last), - std::forward(extra)...); + Extra...>(first, last, std::forward(extra)...); } /// Makes a python iterator over the values (`.second`) of a iterator over pairs from a @@ -2406,15 +2402,13 @@ template ::result_type, typename... Extra> -iterator make_value_iterator(Iterator &&first, Sentinel &&last, Extra &&...extra) { +iterator make_value_iterator(Iterator first, Sentinel last, Extra &&...extra) { return detail::make_iterator_impl, Policy, Iterator, Sentinel, ValueType, - Extra...>(std::forward(first), - std::forward(last), - std::forward(extra)...); + Extra...>(first, last, std::forward(extra)...); } /// Makes an iterator over values of an stl container or other container supporting diff --git a/tests/test_sequences_and_iterators.cpp b/tests/test_sequences_and_iterators.cpp index b867f49a2..1de65edbf 100644 --- a/tests/test_sequences_and_iterators.cpp +++ b/tests/test_sequences_and_iterators.cpp @@ -559,4 +559,23 @@ TEST_SUBMODULE(sequences_and_iterators, m) { []() { return py::make_iterator(list); }); m.def("make_iterator_2", []() { return py::make_iterator(list); }); + + // test_iterator on c arrays + // #4100: ensure lvalue required as increment operand + class CArrayHolder { + public: + CArrayHolder(double x, double y, double z) { + values[0] = x; + values[1] = y; + values[2] = z; + }; + double values[3]; + }; + + py::class_(m, "CArrayHolder") + .def(py::init()) + .def( + "__iter__", + [](const CArrayHolder &v) { return py::make_iterator(v.values, v.values + 3); }, + py::keep_alive<0, 1>()); } diff --git a/tests/test_sequences_and_iterators.py b/tests/test_sequences_and_iterators.py index 062e3b3d3..de486e3e8 100644 --- a/tests/test_sequences_and_iterators.py +++ b/tests/test_sequences_and_iterators.py @@ -241,3 +241,11 @@ def test_iterator_rvp(): assert list(m.make_iterator_1()) == [1, 2, 3] assert list(m.make_iterator_2()) == [1, 2, 3] assert not isinstance(m.make_iterator_1(), type(m.make_iterator_2())) + + +def test_carray_iterator(): + """#4100: Check for proper iterator overload with C-Arrays""" + args_gt = list(float(i) for i in range(3)) + arr_h = m.CArrayHolder(*args_gt) + args = list(arr_h) + assert args_gt == args From 8ea75ab4d754305676d1bff5e324df2d9470dec7 Mon Sep 17 00:00:00 2001 From: Lalaland Date: Sat, 22 Oct 2022 16:52:35 -0700 Subject: [PATCH 10/53] Fix casts to void* (#4275) * Fix casts to void* * Improve tests * style: pre-commit fixes * remove c style cast Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan --- include/pybind11/cast.h | 6 ++---- tests/test_class.cpp | 18 +++++++++++++++++- tests/test_class.py | 2 ++ 3 files changed, 21 insertions(+), 5 deletions(-) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 5699d4fb0..430c62f35 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -1179,11 +1179,9 @@ enable_if_t::value, T> cast_safe(object &&) pybind11_fail("Internal error: cast_safe fallback invoked"); } template -enable_if_t>::value, void> cast_safe(object &&) {} +enable_if_t::value, void> cast_safe(object &&) {} template -enable_if_t, - std::is_same>>::value, - T> +enable_if_t, std::is_void>::value, T> cast_safe(object &&o) { return pybind11::cast(std::move(o)); } diff --git a/tests/test_class.cpp b/tests/test_class.cpp index 18c8d358b..fc1c17b17 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -384,6 +384,8 @@ TEST_SUBMODULE(class_, m) { protected: virtual int foo() const { return value; } + virtual void *void_foo() { return static_cast(&value); } + virtual void *get_self() { return static_cast(this); } private: int value = 42; @@ -392,6 +394,8 @@ TEST_SUBMODULE(class_, m) { class TrampolineB : public ProtectedB { public: int foo() const override { PYBIND11_OVERRIDE(int, ProtectedB, foo, ); } + void *void_foo() override { PYBIND11_OVERRIDE(void *, ProtectedB, void_foo, ); } + void *get_self() override { PYBIND11_OVERRIDE(void *, ProtectedB, get_self, ); } }; class PublicistB : public ProtectedB { @@ -401,11 +405,23 @@ TEST_SUBMODULE(class_, m) { // (in Debug builds only, tested with icpc (ICC) 2021.1 Beta 20200827) ~PublicistB() override{}; // NOLINT(modernize-use-equals-default) using ProtectedB::foo; + using ProtectedB::get_self; + using ProtectedB::void_foo; }; + m.def("read_foo", [](const void *original) { + const int *ptr = reinterpret_cast(original); + return *ptr; + }); + + m.def("pointers_equal", + [](const void *original, const void *comparison) { return original == comparison; }); + py::class_(m, "ProtectedB") .def(py::init<>()) - .def("foo", &PublicistB::foo); + .def("foo", &PublicistB::foo) + .def("void_foo", &PublicistB::void_foo) + .def("get_self", &PublicistB::get_self); // test_brace_initialization struct BraceInitialization { diff --git a/tests/test_class.py b/tests/test_class.py index 47ba450fc..7c1ed2060 100644 --- a/tests/test_class.py +++ b/tests/test_class.py @@ -313,6 +313,8 @@ def test_bind_protected_functions(): b = m.ProtectedB() assert b.foo() == 42 + assert m.read_foo(b.void_foo()) == 42 + assert m.pointers_equal(b.get_self(), b) class C(m.ProtectedB): def __init__(self): From 4fe905d4f01ce47bdea5fdcbcb6037acc9e2596c Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Sun, 23 Oct 2022 00:32:17 -0400 Subject: [PATCH 11/53] fix: add flag for overriding classic Python search values (#4195) * fix: PyPy needs to overrite broken FindPythonInterp values Signed-off-by: Henry Schreiner * fix: add flag to opt-in to new (cross-compile) behavior Signed-off-by: Henry Schreiner * Apply suggestions from code review Signed-off-by: Henry Schreiner --- tools/FindPythonLibsNew.cmake | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tools/FindPythonLibsNew.cmake b/tools/FindPythonLibsNew.cmake index a5a628fd7..bbbd9f9ec 100644 --- a/tools/FindPythonLibsNew.cmake +++ b/tools/FindPythonLibsNew.cmake @@ -151,9 +151,13 @@ if(NOT _PYTHON_SUCCESS MATCHES 0) return() endif() +option( + PYBIND11_PYTHONLIBS_OVERWRITE + "Overwrite cached values read from Python library (classic search). Turn off if cross-compiling and manually setting these values." + ON) # Can manually set values when cross-compiling macro(_PYBIND11_GET_IF_UNDEF lst index name) - if(NOT DEFINED "${name}") + if(PYBIND11_PYTHONLIBS_OVERWRITE OR NOT DEFINED "${name}") list(GET "${lst}" "${index}" "${name}") endif() endmacro() From d1c31e9aa0b8f7594b4c816b8745493272791c27 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Sun, 23 Oct 2022 08:08:00 -0400 Subject: [PATCH 12/53] chore: improve issue template (#4276) --- .github/ISSUE_TEMPLATE/bug-report.yml | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index bd6a9a8e2..e6494ba6a 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -6,7 +6,8 @@ body: - type: markdown attributes: value: | - Maintainers will only make a best effort to triage PRs. Please do your best to make the issue as easy to act on as possible, and only open if clearly a problem with pybind11 (ask first if unsure). + Please do your best to make the issue as easy to act on as possible, and only submit here if there is clearly a problem with pybind11 (ask first if unsure). **Note that a reproducer in a PR is much more likely to get immediate attention.** + - type: checkboxes id: steps attributes: @@ -20,6 +21,12 @@ body: - label: Consider asking first in the [Gitter chat room](https://gitter.im/pybind/Lobby) or in a [Discussion](https:/pybind/pybind11/discussions/new). required: false + - type: Input + id: version + attributes: + label: What version (or hash if on master) of pybind11 are you using? + required: true + - type: textarea id: description attributes: @@ -40,6 +47,14 @@ body: The code should be minimal, have no external dependencies, isolate the function(s) that cause breakage. Submit matched and complete C++ and Python snippets that can be easily compiled and run to diagnose the - issue. If possible, make a PR with a new, failing test to give us a - starting point to work on! + issue. — Note that a reproducer in a PR is much more likely to get + immediate attention: failing tests in the pybind11 CI are the best + starting point for working out fixes. render: text + + - type: Input + id: regression + attributes: + label: Is this a regression? Put the last known working version here if it is. + description: Put the last known working version here if this is a regression. + value: Not a regression From 07a61aa1c0a331431165103d49e1848e69373d62 Mon Sep 17 00:00:00 2001 From: Vemund Handeland Date: Sun, 23 Oct 2022 20:57:45 +0200 Subject: [PATCH 13/53] Fix char8_t support (#4278) Standard library macro __cpp_lib_char8_t is only available after including standard header --- include/pybind11/detail/common.h | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 6da7ef859..b43100b95 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -205,10 +205,6 @@ # endif #endif -#if defined(__cpp_lib_char8_t) && __cpp_lib_char8_t >= 201811L -# define PYBIND11_HAS_U8STRING -#endif - #include #if PY_VERSION_HEX < 0x03060000 # error "PYTHON < 3.6 IS UNSUPPORTED. pybind11 v2.9 was the last to support Python 2 and 3.5." @@ -259,6 +255,11 @@ # endif #endif +// Must be after including or one of the other headers specified by the standard +#if defined(__cpp_lib_char8_t) && __cpp_lib_char8_t >= 201811L +# define PYBIND11_HAS_U8STRING +#endif + // #define PYBIND11_STR_LEGACY_PERMISSIVE // If DEFINED, pybind11::str can hold PyUnicodeObject or PyBytesObject // (probably surprising and never documented, but this was the From fcb5554d9fdd4ad3150bf12cb8be04099cb4976e Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Wed, 26 Oct 2022 10:41:51 -0400 Subject: [PATCH 14/53] ci: move to final release of 3.11 (#4286) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- .github/workflows/ci.yml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2951a0ad1..b56e8f447 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -30,7 +30,7 @@ jobs: - '3.6' - '3.9' - '3.10' - - '3.11-dev' + - '3.11' - 'pypy-3.7' - 'pypy-3.8' - 'pypy-3.9' @@ -119,7 +119,7 @@ jobs: - name: C++11 tests # TODO: Figure out how to load the DLL on Python 3.8+ - if: "!(runner.os == 'Windows' && (matrix.python == 3.8 || matrix.python == 3.9 || matrix.python == '3.10' || matrix.python == '3.11-dev' || matrix.python == 'pypy-3.8'))" + if: "!(runner.os == 'Windows' && (matrix.python == 3.8 || matrix.python == 3.9 || matrix.python == '3.10' || matrix.python == '3.11' || matrix.python == 'pypy-3.8'))" run: cmake --build . --target cpptest -j 2 - name: Interface test C++11 @@ -146,7 +146,7 @@ jobs: - name: C++ tests # TODO: Figure out how to load the DLL on Python 3.8+ - if: "!(runner.os == 'Windows' && (matrix.python == 3.8 || matrix.python == 3.9 || matrix.python == '3.10' || matrix.python == '3.11-dev' || matrix.python == 'pypy-3.8'))" + if: "!(runner.os == 'Windows' && (matrix.python == 3.8 || matrix.python == 3.9 || matrix.python == '3.10' || matrix.python == '3.11' || matrix.python == 'pypy-3.8'))" run: cmake --build build2 --target cpptest # Third build - C++17 mode with unstable ABI @@ -186,7 +186,7 @@ jobs: - python-version: "3.9" python-debug: true valgrind: true - - python-version: "3.11-dev" + - python-version: "3.11" python-debug: false name: "🐍 ${{ matrix.python-version }}${{ matrix.python-debug && '-dbg' || '' }} (deadsnakes)${{ matrix.valgrind && ' • Valgrind' || '' }} • x64" From b07223fa693f65da81e17c12858d636569947f98 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Sat, 29 Oct 2022 11:12:24 -0400 Subject: [PATCH 15/53] fix: improve bytes to str decoding error handling (#4294) * (bugfix): Improve bytes to str decoding error handling * regroup test * Further broaden tests * Add another decode error test * Fix bug in tests * Reviewer suggestions --- include/pybind11/pytypes.h | 9 +++++++++ tests/test_pytypes.cpp | 5 +++++ tests/test_pytypes.py | 14 ++++++++++++++ 3 files changed, 28 insertions(+) diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 37fc49a0e..80a2e2228 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -1432,6 +1432,9 @@ public: str(const char *c, const SzType &n) : object(PyUnicode_FromStringAndSize(c, ssize_t_cast(n)), stolen_t{}) { if (!m_ptr) { + if (PyErr_Occurred()) { + throw error_already_set(); + } pybind11_fail("Could not allocate string object!"); } } @@ -1441,6 +1444,9 @@ public: // NOLINTNEXTLINE(google-explicit-constructor) str(const char *c = "") : object(PyUnicode_FromString(c), stolen_t{}) { if (!m_ptr) { + if (PyErr_Occurred()) { + throw error_already_set(); + } pybind11_fail("Could not allocate string object!"); } } @@ -1598,6 +1604,9 @@ inline str::str(const bytes &b) { } auto obj = reinterpret_steal(PyUnicode_FromStringAndSize(buffer, length)); if (!obj) { + if (PyErr_Occurred()) { + throw error_already_set(); + } pybind11_fail("Could not allocate string object!"); } m_ptr = obj.release().ptr(); diff --git a/tests/test_pytypes.cpp b/tests/test_pytypes.cpp index 99237ccef..ea8d03958 100644 --- a/tests/test_pytypes.cpp +++ b/tests/test_pytypes.cpp @@ -206,7 +206,12 @@ TEST_SUBMODULE(pytypes, m) { m.def("str_from_char_ssize_t", []() { return py::str{"red", (py::ssize_t) 3}; }); m.def("str_from_char_size_t", []() { return py::str{"blue", (py::size_t) 4}; }); m.def("str_from_string", []() { return py::str(std::string("baz")); }); + m.def("str_from_std_string_input", [](const std::string &stri) { return py::str(stri); }); + m.def("str_from_cstr_input", [](const char *c_str) { return py::str(c_str); }); m.def("str_from_bytes", []() { return py::str(py::bytes("boo", 3)); }); + m.def("str_from_bytes_input", + [](const py::bytes &encoded_str) { return py::str(encoded_str); }); + m.def("str_from_object", [](const py::object &obj) { return py::str(obj); }); m.def("repr_from_object", [](const py::object &obj) { return py::repr(obj); }); m.def("str_from_handle", [](py::handle h) { return py::str(h); }); diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 3ed7b9c94..079ee7ca5 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -244,6 +244,20 @@ def test_str(doc): m.str_from_string_from_str(ucs_surrogates_str) +@pytest.mark.parametrize( + "func", + [ + m.str_from_bytes_input, + m.str_from_cstr_input, + m.str_from_std_string_input, + ], +) +def test_surrogate_pairs_unicode_error(func): + input_str = "\ud83d\ude4f".encode("utf-8", "surrogatepass") + with pytest.raises(UnicodeDecodeError): + func(input_str) + + def test_bytes(doc): assert m.bytes_from_char_ssize_t().decode() == "green" assert m.bytes_from_char_size_t().decode() == "purple" From b07d08f6009a52050bb588a3e6fb0489c98af85e Mon Sep 17 00:00:00 2001 From: Chekov2k Date: Sun, 30 Oct 2022 15:57:23 +0000 Subject: [PATCH 16/53] Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` option (cmake, C++ define) (#4216) * Add option to force the use of the PYPY GIL scoped acquire/release logic to support nested gil access, see https://github.com/pybind/pybind11/issues/1276 and https://github.com/pytorch/pytorch/issues/83101 * Apply suggestions from code review * Update CMakeLists.txt * docs: update upgrade guide * Update docs/upgrade.rst * All bells & whistles. * Add Reminder to common.h, so that we will not forget to purge `!WITH_THREAD` branches when dropping Python 3.6 * New sentence instead of semicolon. * Temporarily pull in snapshot of PR #4246 * Add `test_release_acquire` * Add more unit tests for nested gil locking * Add test_report_builtins_internals_keys * Very minor enhancement: sort list only after filtering. * Revert change in docs/upgrade.rst * Add test_multi_acquire_release_cross_module, while also forcing unique PYBIND11_INTERNALS_VERSION for cross_module_gil_utils.cpp * Hopefully fix apparently new ICC error. ``` 2022-10-28T07:57:54.5187728Z -- The CXX compiler identification is Intel 2021.7.0.20220726 ... 2022-10-28T07:58:53.6758994Z icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message. 2022-10-28T07:58:54.5801597Z In file included from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/type_caster_base.h(15), 2022-10-28T07:58:54.5803794Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../cast.h(15), 2022-10-28T07:58:54.5805740Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../attr.h(14), 2022-10-28T07:58:54.5809556Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/class.h(12), 2022-10-28T07:58:54.5812154Z from /home/runner/work/pybind11/pybind11/include/pybind11/pybind11.h(13), 2022-10-28T07:58:54.5948523Z from /home/runner/work/pybind11/pybind11/tests/cross_module_gil_utils.cpp(13): 2022-10-28T07:58:54.5949009Z /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/internals.h(177): error #2282: unrecognized GCC pragma 2022-10-28T07:58:54.5949374Z PYBIND11_TLS_KEY_INIT(tstate) 2022-10-28T07:58:54.5949579Z ^ 2022-10-28T07:58:54.5949695Z ``` * clang-tidy fixes * Workaround for PYPY WIN exitcode None * Revert "Temporarily pull in snapshot of PR #4246" This reverts commit 23ac16e859150f27fda25ca865cabcb4444e0770. * Another workaround for PYPY WIN exitcode None * Clean up how the tests are run "run in process" Part 1: uniformity * Clean up how the tests are run "run in process" Part 2: use `@pytest.mark.parametrize` and clean up the naming. * Skip some tests `#if defined(THREAD_SANITIZER)` (tested with TSAN using the Google-internal toolchain). * Run all tests again but ignore ThreadSanitizer exitcode 66 (this is less likely to mask unrelated ThreadSanitizer issues in the future). * bug fix: missing common.h include before using `PYBIND11_SIMPLE_GIL_MANAGEMENT` For the tests in the github CI this does not matter, because `PYBIND11_SIMPLE_GIL_MANAGEMENT` is always defined from the command line, but when monkey-patching common.h locally, it matters. * if process.exitcode is None: assert t_delta > 9.9 * More sophisiticated `_run_in_process()` implementation, clearly reporting `DEADLOCK`, additionally exercised via added `intentional_deadlock()` * Wrap m.intentional_deadlock in a Python function, for `ForkingPickler` compatibility. ``` > ForkingPickler(file, protocol).dump(obj) E TypeError: cannot pickle 'PyCapsule' object ``` Observed with all Windows builds including mingw but not PyPy, and macos-latest with Python 3.9, 3.10, 3.11 but not 3.6. * Add link to potential solution for WOULD-BE-NICE-TO-HAVE feature. * Add `SKIP_IF_DEADLOCK = True` option, to not pollute the CI results with expected `DEADLOCK` failures while we figure out what to do about them. * Add COPY-PASTE-THIS: gdb ... command (to be used for debugging the detected deadlock) * style: pre-commit fixes * Do better than automatic pre-commit fixes. * Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` to `pytest_report_header()` (so that we can easily know when harvesting deadlock information from the CI logs). Co-authored-by: Arnim Balzer Co-authored-by: Henry Schreiner Co-authored-by: Ralf W. Grosse-Kunstleve Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 4 + CMakeLists.txt | 6 + include/pybind11/detail/common.h | 5 + include/pybind11/detail/internals.h | 16 +- include/pybind11/gil.h | 53 ++++-- tests/conftest.py | 1 + tests/cross_module_gil_utils.cpp | 67 +++++++- tests/pybind11_tests.cpp | 6 + tests/test_embed/test_interpreter.cpp | 1 - tests/test_gil_scoped.cpp | 107 +++++++++++- tests/test_gil_scoped.py | 227 +++++++++++++++++++++----- 11 files changed, 433 insertions(+), 60 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index b56e8f447..a11cae1ab 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -102,10 +102,12 @@ jobs: run: python -m pip install pytest-github-actions-annotate-failures # First build - C++11 mode and inplace + # More-or-less randomly adding -DPYBIND11_SIMPLE_GIL_MANAGEMENT=ON here. - name: Configure C++11 ${{ matrix.args }} run: > cmake -S . -B . -DPYBIND11_WERROR=ON + -DPYBIND11_SIMPLE_GIL_MANAGEMENT=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON -DCMAKE_CXX_STANDARD=11 @@ -129,10 +131,12 @@ jobs: run: git clean -fdx # Second build - C++17 mode and in a build directory + # More-or-less randomly adding -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF here. - name: Configure C++17 run: > cmake -S . -B build2 -DPYBIND11_WERROR=ON + -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON -DCMAKE_CXX_STANDARD=17 diff --git a/CMakeLists.txt b/CMakeLists.txt index 3284e21eb..0d9320388 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -91,10 +91,16 @@ endif() option(PYBIND11_INSTALL "Install pybind11 header files?" ${PYBIND11_MASTER_PROJECT}) option(PYBIND11_TEST "Build pybind11 test suite?" ${PYBIND11_MASTER_PROJECT}) option(PYBIND11_NOPYTHON "Disable search for Python" OFF) +option(PYBIND11_SIMPLE_GIL_MANAGEMENT + "Use simpler GIL management logic that does not support disassociation" OFF) set(PYBIND11_INTERNALS_VERSION "" CACHE STRING "Override the ABI version, may be used to enable the unstable ABI.") +if(PYBIND11_SIMPLE_GIL_MANAGEMENT) + add_compile_definitions(PYBIND11_SIMPLE_GIL_MANAGEMENT) +endif() + cmake_dependent_option( USE_PYTHON_INCLUDE_DIR "Install pybind11 headers in Python include directory instead of default installation prefix" diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index b43100b95..a3e0bc9b3 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -206,6 +206,7 @@ #endif #include +// Reminder: WITH_THREAD is always defined if PY_VERSION_HEX >= 0x03070000 #if PY_VERSION_HEX < 0x03060000 # error "PYTHON < 3.6 IS UNSUPPORTED. pybind11 v2.9 was the last to support Python 2 and 3.5." #endif @@ -229,6 +230,10 @@ # undef copysign #endif +#if defined(PYPY_VERSION) && !defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) +# define PYBIND11_SIMPLE_GIL_MANAGEMENT +#endif + #if defined(_MSC_VER) # if defined(PYBIND11_DEBUG_MARKER) # define _DEBUG diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index d47084e26..7de779434 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -9,6 +9,12 @@ #pragma once +#include "common.h" + +#if defined(WITH_THREAD) && defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) +# include "../gil.h" +#endif + #include "../pytypes.h" #include @@ -49,7 +55,7 @@ inline PyObject *make_object_base_type(PyTypeObject *metaclass); // `Py_LIMITED_API` anyway. # if PYBIND11_INTERNALS_VERSION > 4 # define PYBIND11_TLS_KEY_REF Py_tss_t & -# ifdef __GNUC__ +# if defined(__GNUC__) && !defined(__INTEL_COMPILER) // Clang on macOS warns due to `Py_tss_NEEDS_INIT` not specifying an initializer // for every field. # define PYBIND11_TLS_KEY_INIT(var) \ @@ -169,10 +175,12 @@ struct internals { PyTypeObject *default_metaclass; PyObject *instance_base; #if defined(WITH_THREAD) + // Unused if PYBIND11_SIMPLE_GIL_MANAGEMENT is defined: PYBIND11_TLS_KEY_INIT(tstate) # if PYBIND11_INTERNALS_VERSION > 4 PYBIND11_TLS_KEY_INIT(loader_life_support_tls_key) # endif // PYBIND11_INTERNALS_VERSION > 4 + // Unused if PYBIND11_SIMPLE_GIL_MANAGEMENT is defined: PyInterpreterState *istate = nullptr; ~internals() { # if PYBIND11_INTERNALS_VERSION > 4 @@ -408,6 +416,10 @@ PYBIND11_NOINLINE internals &get_internals() { return **internals_pp; } +#if defined(WITH_THREAD) +# if defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) + gil_scoped_acquire gil; +# else // Ensure that the GIL is held since we will need to make Python calls. // Cannot use py::gil_scoped_acquire here since that constructor calls get_internals. struct gil_scoped_acquire_local { @@ -417,6 +429,8 @@ PYBIND11_NOINLINE internals &get_internals() { ~gil_scoped_acquire_local() { PyGILState_Release(state); } const PyGILState_STATE state; } gil; +# endif +#endif error_scope err_scope; PYBIND11_STR_TYPE id(PYBIND11_INTERNALS_ID); diff --git a/include/pybind11/gil.h b/include/pybind11/gil.h index 1ef5f0a8c..cb0028d50 100644 --- a/include/pybind11/gil.h +++ b/include/pybind11/gil.h @@ -10,7 +10,10 @@ #pragma once #include "detail/common.h" -#include "detail/internals.h" + +#if defined(WITH_THREAD) && !defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) +# include "detail/internals.h" +#endif PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) @@ -21,7 +24,9 @@ PyThreadState *get_thread_state_unchecked(); PYBIND11_NAMESPACE_END(detail) -#if defined(WITH_THREAD) && !defined(PYPY_VERSION) +#if defined(WITH_THREAD) + +# if !defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) /* The functions below essentially reproduce the PyGILState_* API using a RAII * pattern, but there are a few important differences: @@ -62,11 +67,11 @@ public: if (!tstate) { tstate = PyThreadState_New(internals.istate); -# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) +# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) if (!tstate) { pybind11_fail("scoped_acquire: could not create thread state!"); } -# endif +# endif tstate->gilstate_counter = 0; PYBIND11_TLS_REPLACE_VALUE(internals.tstate, tstate); } else { @@ -87,20 +92,20 @@ public: PYBIND11_NOINLINE void dec_ref() { --tstate->gilstate_counter; -# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) +# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) if (detail::get_thread_state_unchecked() != tstate) { pybind11_fail("scoped_acquire::dec_ref(): thread state must be current!"); } if (tstate->gilstate_counter < 0) { pybind11_fail("scoped_acquire::dec_ref(): reference count underflow!"); } -# endif +# endif if (tstate->gilstate_counter == 0) { -# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) +# if defined(PYBIND11_DETAILED_ERROR_MESSAGES) if (!release) { pybind11_fail("scoped_acquire::dec_ref(): internal error!"); } -# endif +# endif PyThreadState_Clear(tstate); if (active) { PyThreadState_DeleteCurrent(); @@ -178,12 +183,14 @@ private: bool disassoc; bool active = true; }; -#elif defined(PYPY_VERSION) + +# else // PYBIND11_SIMPLE_GIL_MANAGEMENT + class gil_scoped_acquire { PyGILState_STATE state; public: - gil_scoped_acquire() { state = PyGILState_Ensure(); } + gil_scoped_acquire() : state{PyGILState_Ensure()} {} gil_scoped_acquire(const gil_scoped_acquire &) = delete; gil_scoped_acquire &operator=(const gil_scoped_acquire &) = delete; ~gil_scoped_acquire() { PyGILState_Release(state); } @@ -194,19 +201,39 @@ class gil_scoped_release { PyThreadState *state; public: - gil_scoped_release() { state = PyEval_SaveThread(); } + gil_scoped_release() : state{PyEval_SaveThread()} {} gil_scoped_release(const gil_scoped_release &) = delete; gil_scoped_release &operator=(const gil_scoped_acquire &) = delete; ~gil_scoped_release() { PyEval_RestoreThread(state); } void disarm() {} }; -#else + +# endif // PYBIND11_SIMPLE_GIL_MANAGEMENT + +#else // WITH_THREAD + class gil_scoped_acquire { +public: + gil_scoped_acquire() { + // Trick to suppress `unused variable` error messages (at call sites). + (void) (this != (this + 1)); + } + gil_scoped_acquire(const gil_scoped_acquire &) = delete; + gil_scoped_acquire &operator=(const gil_scoped_acquire &) = delete; void disarm() {} }; + class gil_scoped_release { +public: + gil_scoped_release() { + // Trick to suppress `unused variable` error messages (at call sites). + (void) (this != (this + 1)); + } + gil_scoped_release(const gil_scoped_release &) = delete; + gil_scoped_release &operator=(const gil_scoped_acquire &) = delete; void disarm() {} }; -#endif + +#endif // WITH_THREAD PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/tests/conftest.py b/tests/conftest.py index 02ce263af..f5ddb9f12 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -210,4 +210,5 @@ def pytest_report_header(config): f" {pybind11_tests.compiler_info}" f" {pybind11_tests.cpp_std}" f" {pybind11_tests.PYBIND11_INTERNALS_ID}" + f" PYBIND11_SIMPLE_GIL_MANAGEMENT={pybind11_tests.PYBIND11_SIMPLE_GIL_MANAGEMENT}" ) diff --git a/tests/cross_module_gil_utils.cpp b/tests/cross_module_gil_utils.cpp index 1436c35d6..7c20849dd 100644 --- a/tests/cross_module_gil_utils.cpp +++ b/tests/cross_module_gil_utils.cpp @@ -6,9 +6,15 @@ All rights reserved. Use of this source code is governed by a BSD-style license that can be found in the LICENSE file. */ +#if defined(PYBIND11_INTERNALS_VERSION) +# undef PYBIND11_INTERNALS_VERSION +#endif +#define PYBIND11_INTERNALS_VERSION 21814642 // Ensure this module has its own `internals` instance. #include #include +#include +#include // This file mimics a DSO that makes pybind11 calls but does not define a // PYBIND11_MODULE. The purpose is to test that such a DSO can create a @@ -21,8 +27,54 @@ namespace { namespace py = pybind11; + void gil_acquire() { py::gil_scoped_acquire gil; } +std::string gil_multi_acquire_release(unsigned bits) { + if ((bits & 0x1u) != 0u) { + py::gil_scoped_acquire gil; + } + if ((bits & 0x2u) != 0u) { + py::gil_scoped_release gil; + } + if ((bits & 0x4u) != 0u) { + py::gil_scoped_acquire gil; + } + if ((bits & 0x8u) != 0u) { + py::gil_scoped_release gil; + } + return PYBIND11_INTERNALS_ID; +} + +struct CustomAutoGIL { + CustomAutoGIL() : gstate(PyGILState_Ensure()) {} + ~CustomAutoGIL() { PyGILState_Release(gstate); } + + PyGILState_STATE gstate; +}; +struct CustomAutoNoGIL { + CustomAutoNoGIL() : save(PyEval_SaveThread()) {} + ~CustomAutoNoGIL() { PyEval_RestoreThread(save); } + + PyThreadState *save; +}; + +template +void gil_acquire_inner() { + Acquire acquire_outer; + Acquire acquire_inner; + Release release; +} + +template +void gil_acquire_nested() { + Acquire acquire_outer; + Acquire acquire_inner; + Release release; + auto thread = std::thread(&gil_acquire_inner); + thread.join(); +} + constexpr char kModuleName[] = "cross_module_gil_utils"; struct PyModuleDef moduledef = { @@ -30,6 +82,9 @@ struct PyModuleDef moduledef = { } // namespace +#define ADD_FUNCTION(Name, ...) \ + PyModule_AddObject(m, Name, PyLong_FromVoidPtr(reinterpret_cast(&__VA_ARGS__))); + extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() { PyObject *m = PyModule_Create(&moduledef); @@ -37,8 +92,16 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() { if (m != nullptr) { static_assert(sizeof(&gil_acquire) == sizeof(void *), "Function pointer must have the same size as void*"); - PyModule_AddObject( - m, "gil_acquire_funcaddr", PyLong_FromVoidPtr(reinterpret_cast(&gil_acquire))); + ADD_FUNCTION("gil_acquire_funcaddr", gil_acquire) + ADD_FUNCTION("gil_multi_acquire_release_funcaddr", gil_multi_acquire_release) + ADD_FUNCTION("gil_acquire_inner_custom_funcaddr", + gil_acquire_inner) + ADD_FUNCTION("gil_acquire_nested_custom_funcaddr", + gil_acquire_nested) + ADD_FUNCTION("gil_acquire_inner_pybind11_funcaddr", + gil_acquire_inner) + ADD_FUNCTION("gil_acquire_nested_pybind11_funcaddr", + gil_acquire_nested) } return m; diff --git a/tests/pybind11_tests.cpp b/tests/pybind11_tests.cpp index aa3095594..624034648 100644 --- a/tests/pybind11_tests.cpp +++ b/tests/pybind11_tests.cpp @@ -89,6 +89,12 @@ PYBIND11_MODULE(pybind11_tests, m) { #endif m.attr("cpp_std") = cpp_std(); m.attr("PYBIND11_INTERNALS_ID") = PYBIND11_INTERNALS_ID; + m.attr("PYBIND11_SIMPLE_GIL_MANAGEMENT") = +#if defined(PYBIND11_SIMPLE_GIL_MANAGEMENT) + true; +#else + false; +#endif bind_ConstructorStats(m); diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 6299293b9..44dcd1fdb 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -293,7 +293,6 @@ TEST_CASE("Threads") { { py::gil_scoped_release gil_release{}; - REQUIRE(has_pybind11_internals_static()); auto threads = std::vector(); for (auto i = 0; i < num_threads; ++i) { diff --git a/tests/test_gil_scoped.cpp b/tests/test_gil_scoped.cpp index 97efdc161..f136086e8 100644 --- a/tests/test_gil_scoped.cpp +++ b/tests/test_gil_scoped.cpp @@ -11,6 +11,13 @@ #include "pybind11_tests.h" +#include +#include + +#define CROSS_MODULE(Function) \ + auto cm = py::module_::import("cross_module_gil_utils"); \ + auto target = reinterpret_cast(PyLong_AsVoidPtr(cm.attr(Function).ptr())); + class VirtClass { public: virtual ~VirtClass() = default; @@ -28,6 +35,16 @@ class PyVirtClass : public VirtClass { }; TEST_SUBMODULE(gil_scoped, m) { + m.attr("defined_THREAD_SANITIZER") = +#if defined(THREAD_SANITIZER) + true; +#else + false; +#endif + + m.def("intentional_deadlock", + []() { std::thread([]() { py::gil_scoped_acquire gil_acquired; }).join(); }); + py::class_(m, "VirtClass") .def(py::init<>()) .def("virtual_func", &VirtClass::virtual_func) @@ -37,11 +54,91 @@ TEST_SUBMODULE(gil_scoped, m) { m.def("test_callback_std_func", [](const std::function &func) { func(); }); m.def("test_callback_virtual_func", [](VirtClass &virt) { virt.virtual_func(); }); m.def("test_callback_pure_virtual_func", [](VirtClass &virt) { virt.pure_virtual_func(); }); - m.def("test_cross_module_gil", []() { - auto cm = py::module_::import("cross_module_gil_utils"); - auto gil_acquire = reinterpret_cast( - PyLong_AsVoidPtr(cm.attr("gil_acquire_funcaddr").ptr())); + m.def("test_cross_module_gil_released", []() { + CROSS_MODULE("gil_acquire_funcaddr") py::gil_scoped_release gil_release; - gil_acquire(); + target(); + }); + m.def("test_cross_module_gil_acquired", []() { + CROSS_MODULE("gil_acquire_funcaddr") + py::gil_scoped_acquire gil_acquire; + target(); + }); + m.def("test_cross_module_gil_inner_custom_released", []() { + CROSS_MODULE("gil_acquire_inner_custom_funcaddr") + py::gil_scoped_release gil_release; + target(); + }); + m.def("test_cross_module_gil_inner_custom_acquired", []() { + CROSS_MODULE("gil_acquire_inner_custom_funcaddr") + py::gil_scoped_acquire gil_acquire; + target(); + }); + m.def("test_cross_module_gil_inner_pybind11_released", []() { + CROSS_MODULE("gil_acquire_inner_pybind11_funcaddr") + py::gil_scoped_release gil_release; + target(); + }); + m.def("test_cross_module_gil_inner_pybind11_acquired", []() { + CROSS_MODULE("gil_acquire_inner_pybind11_funcaddr") + py::gil_scoped_acquire gil_acquire; + target(); + }); + m.def("test_cross_module_gil_nested_custom_released", []() { + CROSS_MODULE("gil_acquire_nested_custom_funcaddr") + py::gil_scoped_release gil_release; + target(); + }); + m.def("test_cross_module_gil_nested_custom_acquired", []() { + CROSS_MODULE("gil_acquire_nested_custom_funcaddr") + py::gil_scoped_acquire gil_acquire; + target(); + }); + m.def("test_cross_module_gil_nested_pybind11_released", []() { + CROSS_MODULE("gil_acquire_nested_pybind11_funcaddr") + py::gil_scoped_release gil_release; + target(); + }); + m.def("test_cross_module_gil_nested_pybind11_acquired", []() { + CROSS_MODULE("gil_acquire_nested_pybind11_funcaddr") + py::gil_scoped_acquire gil_acquire; + target(); + }); + m.def("test_release_acquire", [](const py::object &obj) { + py::gil_scoped_release gil_released; + py::gil_scoped_acquire gil_acquired; + return py::str(obj); + }); + m.def("test_nested_acquire", [](const py::object &obj) { + py::gil_scoped_release gil_released; + py::gil_scoped_acquire gil_acquired_outer; + py::gil_scoped_acquire gil_acquired_inner; + return py::str(obj); + }); + m.def("test_multi_acquire_release_cross_module", [](unsigned bits) { + py::set internals_ids; + internals_ids.add(PYBIND11_INTERNALS_ID); + { + py::gil_scoped_release gil_released; + auto thread_f = [bits, &internals_ids]() { + py::gil_scoped_acquire gil_acquired; + auto cm = py::module_::import("cross_module_gil_utils"); + auto target = reinterpret_cast( + PyLong_AsVoidPtr(cm.attr("gil_multi_acquire_release_funcaddr").ptr())); + std::string cm_internals_id = target(bits >> 3); + internals_ids.add(cm_internals_id); + }; + if ((bits & 0x1u) != 0u) { + thread_f(); + } + if ((bits & 0x2u) != 0u) { + std::thread non_python_thread(thread_f); + non_python_thread.join(); + } + if ((bits & 0x4u) != 0u) { + thread_f(); + } + } + return internals_ids; }); } diff --git a/tests/test_gil_scoped.py b/tests/test_gil_scoped.py index 52374b0cc..e890a7b0c 100644 --- a/tests/test_gil_scoped.py +++ b/tests/test_gil_scoped.py @@ -1,45 +1,199 @@ import multiprocessing +import sys import threading +import time + +import pytest from pybind11_tests import gil_scoped as m +class ExtendedVirtClass(m.VirtClass): + def virtual_func(self): + pass + + def pure_virtual_func(self): + pass + + +def test_callback_py_obj(): + m.test_callback_py_obj(lambda: None) + + +def test_callback_std_func(): + m.test_callback_std_func(lambda: None) + + +def test_callback_virtual_func(): + extended = ExtendedVirtClass() + m.test_callback_virtual_func(extended) + + +def test_callback_pure_virtual_func(): + extended = ExtendedVirtClass() + m.test_callback_pure_virtual_func(extended) + + +def test_cross_module_gil_released(): + """Makes sure that the GIL can be acquired by another module from a GIL-released state.""" + m.test_cross_module_gil_released() # Should not raise a SIGSEGV + + +def test_cross_module_gil_acquired(): + """Makes sure that the GIL can be acquired by another module from a GIL-acquired state.""" + m.test_cross_module_gil_acquired() # Should not raise a SIGSEGV + + +def test_cross_module_gil_inner_custom_released(): + """Makes sure that the GIL can be acquired/released by another module + from a GIL-released state using custom locking logic.""" + m.test_cross_module_gil_inner_custom_released() + + +def test_cross_module_gil_inner_custom_acquired(): + """Makes sure that the GIL can be acquired/acquired by another module + from a GIL-acquired state using custom locking logic.""" + m.test_cross_module_gil_inner_custom_acquired() + + +def test_cross_module_gil_inner_pybind11_released(): + """Makes sure that the GIL can be acquired/released by another module + from a GIL-released state using pybind11 locking logic.""" + m.test_cross_module_gil_inner_pybind11_released() + + +def test_cross_module_gil_inner_pybind11_acquired(): + """Makes sure that the GIL can be acquired/acquired by another module + from a GIL-acquired state using pybind11 locking logic.""" + m.test_cross_module_gil_inner_pybind11_acquired() + + +def test_cross_module_gil_nested_custom_released(): + """Makes sure that the GIL can be nested acquired/released by another module + from a GIL-released state using custom locking logic.""" + m.test_cross_module_gil_nested_custom_released() + + +def test_cross_module_gil_nested_custom_acquired(): + """Makes sure that the GIL can be nested acquired/acquired by another module + from a GIL-acquired state using custom locking logic.""" + m.test_cross_module_gil_nested_custom_acquired() + + +def test_cross_module_gil_nested_pybind11_released(): + """Makes sure that the GIL can be nested acquired/released by another module + from a GIL-released state using pybind11 locking logic.""" + m.test_cross_module_gil_nested_pybind11_released() + + +def test_cross_module_gil_nested_pybind11_acquired(): + """Makes sure that the GIL can be nested acquired/acquired by another module + from a GIL-acquired state using pybind11 locking logic.""" + m.test_cross_module_gil_nested_pybind11_acquired() + + +def test_release_acquire(): + assert m.test_release_acquire(0xAB) == "171" + + +def test_nested_acquire(): + assert m.test_nested_acquire(0xAB) == "171" + + +def test_multi_acquire_release_cross_module(): + for bits in range(16 * 8): + internals_ids = m.test_multi_acquire_release_cross_module(bits) + assert len(internals_ids) == 2 if bits % 8 else 1 + + +# Intentionally putting human review in the loop here, to guard against accidents. +VARS_BEFORE_ALL_BASIC_TESTS = dict(vars()) # Make a copy of the dict (critical). +ALL_BASIC_TESTS = ( + test_callback_py_obj, + test_callback_std_func, + test_callback_virtual_func, + test_callback_pure_virtual_func, + test_cross_module_gil_released, + test_cross_module_gil_acquired, + test_cross_module_gil_inner_custom_released, + test_cross_module_gil_inner_custom_acquired, + test_cross_module_gil_inner_pybind11_released, + test_cross_module_gil_inner_pybind11_acquired, + test_cross_module_gil_nested_custom_released, + test_cross_module_gil_nested_custom_acquired, + test_cross_module_gil_nested_pybind11_released, + test_cross_module_gil_nested_pybind11_acquired, + test_release_acquire, + test_nested_acquire, + test_multi_acquire_release_cross_module, +) + + +def test_all_basic_tests_completeness(): + num_found = 0 + for key, value in VARS_BEFORE_ALL_BASIC_TESTS.items(): + if not key.startswith("test_"): + continue + assert value in ALL_BASIC_TESTS + num_found += 1 + assert len(ALL_BASIC_TESTS) == num_found + + +def _intentional_deadlock(): + m.intentional_deadlock() + + +ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK = ALL_BASIC_TESTS + (_intentional_deadlock,) +SKIP_IF_DEADLOCK = True # See PR #4216 + + def _run_in_process(target, *args, **kwargs): - """Runs target in process and returns its exitcode after 10s (None if still alive).""" + if len(args) == 0: + test_fn = target + else: + test_fn = args[0] + # Do not need to wait much, 10s should be more than enough. + timeout = 0.1 if test_fn is _intentional_deadlock else 10 process = multiprocessing.Process(target=target, args=args, kwargs=kwargs) process.daemon = True try: + t_start = time.time() process.start() - # Do not need to wait much, 10s should be more than enough. - process.join(timeout=10) + if timeout >= 100: # For debugging. + print( + "\nprocess.pid STARTED", process.pid, (sys.argv, target, args, kwargs) + ) + print(f"COPY-PASTE-THIS: gdb {sys.argv[0]} -p {process.pid}", flush=True) + process.join(timeout=timeout) + if timeout >= 100: + print("\nprocess.pid JOINED", process.pid, flush=True) + t_delta = time.time() - t_start + if process.exitcode == 66 and m.defined_THREAD_SANITIZER: # Issue #2754 + # WOULD-BE-NICE-TO-HAVE: Check that the message below is actually in the output. + # Maybe this could work: + # https://gist.github.com/alexeygrigorev/01ce847f2e721b513b42ea4a6c96905e + pytest.skip( + "ThreadSanitizer: starting new threads after multi-threaded fork is not supported." + ) + elif test_fn is _intentional_deadlock: + assert process.exitcode is None + return 0 + elif process.exitcode is None: + assert t_delta > 0.9 * timeout + msg = "DEADLOCK, most likely, exactly what this test is meant to detect." + if SKIP_IF_DEADLOCK: + pytest.skip(msg) + raise RuntimeError(msg) return process.exitcode finally: if process.is_alive(): process.terminate() -def _python_to_cpp_to_python(): - """Calls different C++ functions that come back to Python.""" - - class ExtendedVirtClass(m.VirtClass): - def virtual_func(self): - pass - - def pure_virtual_func(self): - pass - - extended = ExtendedVirtClass() - m.test_callback_py_obj(lambda: None) - m.test_callback_std_func(lambda: None) - m.test_callback_virtual_func(extended) - m.test_callback_pure_virtual_func(extended) - - -def _python_to_cpp_to_python_from_threads(num_threads, parallel=False): - """Calls different C++ functions that come back to Python, from Python threads.""" +def _run_in_threads(test_fn, num_threads, parallel): threads = [] for _ in range(num_threads): - thread = threading.Thread(target=_python_to_cpp_to_python) + thread = threading.Thread(target=test_fn) thread.daemon = True thread.start() if parallel: @@ -51,43 +205,40 @@ def _python_to_cpp_to_python_from_threads(num_threads, parallel=False): # TODO: FIXME, sometimes returns -11 (segfault) instead of 0 on macOS Python 3.9 -def test_python_to_cpp_to_python_from_thread(): +@pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK) +def test_run_in_process_one_thread(test_fn): """Makes sure there is no GIL deadlock when running in a thread. It runs in a separate process to be able to stop and assert if it deadlocks. """ - assert _run_in_process(_python_to_cpp_to_python_from_threads, 1) == 0 + assert _run_in_process(_run_in_threads, test_fn, num_threads=1, parallel=False) == 0 # TODO: FIXME on macOS Python 3.9 -def test_python_to_cpp_to_python_from_thread_multiple_parallel(): +@pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK) +def test_run_in_process_multiple_threads_parallel(test_fn): """Makes sure there is no GIL deadlock when running in a thread multiple times in parallel. It runs in a separate process to be able to stop and assert if it deadlocks. """ - assert _run_in_process(_python_to_cpp_to_python_from_threads, 8, parallel=True) == 0 + assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=True) == 0 # TODO: FIXME on macOS Python 3.9 -def test_python_to_cpp_to_python_from_thread_multiple_sequential(): +@pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK) +def test_run_in_process_multiple_threads_sequential(test_fn): """Makes sure there is no GIL deadlock when running in a thread multiple times sequentially. It runs in a separate process to be able to stop and assert if it deadlocks. """ - assert ( - _run_in_process(_python_to_cpp_to_python_from_threads, 8, parallel=False) == 0 - ) + assert _run_in_process(_run_in_threads, test_fn, num_threads=8, parallel=False) == 0 # TODO: FIXME on macOS Python 3.9 -def test_python_to_cpp_to_python_from_process(): +@pytest.mark.parametrize("test_fn", ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK) +def test_run_in_process_direct(test_fn): """Makes sure there is no GIL deadlock when using processes. This test is for completion, but it was never an issue. """ - assert _run_in_process(_python_to_cpp_to_python) == 0 - - -def test_cross_module_gil(): - """Makes sure that the GIL can be acquired by another module from a GIL-released state.""" - m.test_cross_module_gil() # Should not raise a SIGSEGV + assert _run_in_process(test_fn) == 0 From 5bc0943ed96836f46489f53961f6c438d2935357 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sun, 30 Oct 2022 13:24:41 -0700 Subject: [PATCH 17/53] Ensure config, build, toolchain, spelling, etc. issues are not masked. (#4255) --- include/pybind11/eigen/matrix.h | 1 - tests/test_eigen_tensor.py | 12 ++++++++++-- 2 files changed, 10 insertions(+), 3 deletions(-) diff --git a/include/pybind11/eigen/matrix.h b/include/pybind11/eigen/matrix.h index 5f5ad3867..c30dac241 100644 --- a/include/pybind11/eigen/matrix.h +++ b/include/pybind11/eigen/matrix.h @@ -11,7 +11,6 @@ #include "../numpy.h" -// Similar to comments & pragma block in eigen_tensor.h. PLEASE KEEP IN SYNC. /* HINT: To suppress warnings originating from the Eigen headers, use -isystem. See also: https://stackoverflow.com/questions/2579576/i-dir-vs-isystem-dir diff --git a/tests/test_eigen_tensor.py b/tests/test_eigen_tensor.py index 5ee5fa01b..653c9f288 100644 --- a/tests/test_eigen_tensor.py +++ b/tests/test_eigen_tensor.py @@ -9,8 +9,16 @@ try: from pybind11_tests import eigen_tensor_avoid_stl_array as avoid submodules += [avoid.c_style, avoid.f_style] -except ImportError: - pass +except ImportError as e: + # Ensure config, build, toolchain, etc. issues are not masked here: + raise RuntimeError( + "import pybind11_tests.eigen_tensor_avoid_stl_array FAILED, while " + "import pybind11_tests.eigen_tensor succeeded. " + "Please ensure that " + "test_eigen_tensor.cpp & " + "test_eigen_tensor_avoid_stl_array.cpp " + "are built together (or both are not built if Eigen is not available)." + ) from e tensor_ref = np.empty((3, 5, 2), dtype=np.int64) From 3a2c96bd6f95f6b183c342380558238583415ab1 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 31 Oct 2022 09:18:05 -0700 Subject: [PATCH 18/53] fix: unicode surrogate character in Python exception message. (#4297) * Fix & test for issue #4288 (unicode surrogate character in Python exception message). * DRY `message_unavailable_exc` * fix: add a constexpr Co-authored-by: Aaron Gokaslan * style: pre-commit fixes Co-authored-by: Henry Schreiner Co-authored-by: Aaron Gokaslan Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- include/pybind11/pytypes.h | 22 ++++++++++++++++++++-- tests/test_exceptions.cpp | 5 ----- tests/test_exceptions.py | 14 ++++++++++++++ 3 files changed, 34 insertions(+), 7 deletions(-) diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 80a2e2228..91cbaaba2 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -501,11 +501,29 @@ struct error_fetch_and_normalize { std::string message_error_string; if (m_value) { auto value_str = reinterpret_steal(PyObject_Str(m_value.ptr())); + constexpr const char *message_unavailable_exc + = ""; if (!value_str) { message_error_string = detail::error_string(); - result = ""; + result = message_unavailable_exc; } else { - result = value_str.cast(); + // Not using `value_str.cast()`, to not potentially throw a secondary + // error_already_set that will then result in process termination (#4288). + auto value_bytes = reinterpret_steal( + PyUnicode_AsEncodedString(value_str.ptr(), "utf-8", "backslashreplace")); + if (!value_bytes) { + message_error_string = detail::error_string(); + result = message_unavailable_exc; + } else { + char *buffer = nullptr; + Py_ssize_t length = 0; + if (PyBytes_AsStringAndSize(value_bytes.ptr(), &buffer, &length) == -1) { + message_error_string = detail::error_string(); + result = message_unavailable_exc; + } else { + result = std::string(buffer, static_cast(length)); + } + } } } else { result = ""; diff --git a/tests/test_exceptions.cpp b/tests/test_exceptions.cpp index 3583f22a5..f57e09506 100644 --- a/tests/test_exceptions.cpp +++ b/tests/test_exceptions.cpp @@ -105,11 +105,6 @@ struct PythonAlreadySetInDestructor { py::str s; }; -std::string error_already_set_what(const py::object &exc_type, const py::object &exc_value) { - PyErr_SetObject(exc_type.ptr(), exc_value.ptr()); - return py::error_already_set().what(); -} - TEST_SUBMODULE(exceptions, m) { m.def("throw_std_exception", []() { throw std::runtime_error("This exception was intentionally thrown."); }); diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index 70b6ffea9..7eb1a9d62 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -275,6 +275,20 @@ def test_local_translator(msg): assert msg(excinfo.value) == "this mod" +def test_error_already_set_message_with_unicode_surrogate(): # Issue #4288 + assert m.error_already_set_what(RuntimeError, "\ud927") == ( + "RuntimeError: \\ud927", + False, + ) + + +def test_error_already_set_message_with_malformed_utf8(): + assert m.error_already_set_what(RuntimeError, b"\x80") == ( + "RuntimeError: b'\\x80'", + False, + ) + + class FlakyException(Exception): def __init__(self, failure_point): if failure_point == "failure_point_init": From b1bd7f2600d650bf59c88800c637703dd89317f3 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Mon, 31 Oct 2022 10:36:26 -0700 Subject: [PATCH 19/53] fix: define (non-empty) `PYBIND11_EXPORT_EXCEPTION` only under macOS. (#4298) Background: #2999, #4105, #4283, #4284 In a nutshell: * Only macOS actually needs `PYBIND11_EXPORT_EXCEPTION` (#4284). * Evidently (#4283), under macOS `PYBIND11_EXPORT_EXCEPTION` does not run the risk of introducing ODR violations, * but evidently (#4283) under Linux it does, in the presumably rare/unusual situation that `RTLD_GLOBAL` is used. * Windows does no have the equivalent of `RTLD_GLOBAL`, therefore `PYBIND11_EXPORT_EXCEPTION` has no practical benefit, on the contrary, noisy warning suppression pragmas are needed, therefore it is best left empty. --- docs/advanced/exceptions.rst | 9 ++++++--- include/pybind11/detail/common.h | 18 +++--------------- include/pybind11/pytypes.h | 9 --------- 3 files changed, 9 insertions(+), 27 deletions(-) diff --git a/docs/advanced/exceptions.rst b/docs/advanced/exceptions.rst index 2211caf5d..53981dc08 100644 --- a/docs/advanced/exceptions.rst +++ b/docs/advanced/exceptions.rst @@ -177,9 +177,12 @@ section. may be explicitly (re-)thrown to delegate it to the other, previously-declared existing exception translators. - Note that ``libc++`` and ``libstdc++`` `behave differently `_ - with ``-fvisibility=hidden``. Therefore exceptions that are used across ABI boundaries need to be explicitly exported, as exercised in ``tests/test_exceptions.h``. - See also: "Problems with C++ exceptions" under `GCC Wiki `_. + Note that ``libc++`` and ``libstdc++`` `behave differently under macOS + `_ + with ``-fvisibility=hidden``. Therefore exceptions that are used across ABI + boundaries need to be explicitly exported, as exercised in + ``tests/test_exceptions.h``. See also: + "Problems with C++ exceptions" under `GCC Wiki `_. Local vs Global Exception Translators diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index a3e0bc9b3..9b74323f6 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -96,13 +96,10 @@ #endif #if !defined(PYBIND11_EXPORT_EXCEPTION) -# ifdef __MINGW32__ -// workaround for: -// error: 'dllexport' implies default visibility, but xxx has already been declared with a -// different visibility -# define PYBIND11_EXPORT_EXCEPTION -# else +# if defined(__apple_build_version__) # define PYBIND11_EXPORT_EXCEPTION PYBIND11_EXPORT +# else +# define PYBIND11_EXPORT_EXCEPTION # endif #endif @@ -904,12 +901,6 @@ using expand_side_effects = bool[]; PYBIND11_NAMESPACE_END(detail) -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable : 4275) -// warning C4275: An exported class was derived from a class that wasn't exported. -// Can be ignored when derived from a STL class. -#endif /// C++ bindings of builtin Python exceptions class PYBIND11_EXPORT_EXCEPTION builtin_exception : public std::runtime_error { public: @@ -917,9 +908,6 @@ public: /// Set the error using the Python C API virtual void set_error() const = 0; }; -#if defined(_MSC_VER) -# pragma warning(pop) -#endif #define PYBIND11_RUNTIME_EXCEPTION(name, type) \ class PYBIND11_EXPORT_EXCEPTION name : public builtin_exception { \ diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 91cbaaba2..80b49ec39 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -623,12 +623,6 @@ inline std::string error_string() { PYBIND11_NAMESPACE_END(detail) -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable : 4275 4251) -// warning C4275: An exported class was derived from a class that wasn't exported. -// Can be ignored when derived from a STL class. -#endif /// Fetch and hold an error which was already set in Python. An instance of this is typically /// thrown to propagate python-side errors back through C++ which can either be caught manually or /// else falls back to the function dispatcher (which then raises the captured error back to @@ -688,9 +682,6 @@ private: /// crashes (undefined behavior) if the Python interpreter is finalizing. static void m_fetched_error_deleter(detail::error_fetch_and_normalize *raw_ptr); }; -#if defined(_MSC_VER) -# pragma warning(pop) -#endif /// Replaces the current Python error indicator with the chosen error, performing a /// 'raise from' to indicate that the chosen error was caused by the original error. From 252ed8fb52e46eb3ec3e3b8c621ca9f79b53b26a Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 31 Oct 2022 14:11:23 -0400 Subject: [PATCH 20/53] docs: prepare for 2.10.1 release (#4279) * docs: prepare for 2.10.1 release Signed-off-by: Henry Schreiner * Update changelog.rst * docs: update changelog with final list of PRs Signed-off-by: Henry Schreiner * Update docs/changelog.rst * chore: one more changelog bump Signed-off-by: Henry Schreiner --- docs/changelog.rst | 43 ++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 40 insertions(+), 3 deletions(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index df3181c52..7d6d0c0f5 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -10,19 +10,18 @@ Changes will be added here periodically from the "Suggested changelog entry" block in pull request descriptions. - IN DEVELOPMENT -------------- Changes will be summarized here periodically. -Version 2.10.1 (Oct 2?, 2022) +Version 2.10.1 (Oct 31, 2022) ----------------------------- +This is the first version to fully support embedding the newly released Python 3.11. Changes: - * Allow ``pybind11::capsule`` constructor to take null destructor pointers. `#4221 `_ @@ -30,8 +29,40 @@ Changes: (established behavior). `#4119 `_ +* A ``PYBIND11_SIMPLE_GIL_MANAGEMENT`` option was added (cmake, C++ define), + along with many additional tests in ``test_gil_scoped.py``. The option may be + useful to try when debugging GIL-related issues, to determine if the more + complex default implementation is or is not to blame. See #4216 for + background. WARNING: Please be careful to not create ODR violations when + using the option: everything that is linked together with mutual symbol + visibility needs to be rebuilt. + `#4216 `_ + +* ``PYBIND11_EXPORT_EXCEPTION`` was made non-empty only under macOS. This makes + Linux builds safer, and enables the removal of warning suppression pragmas for + Windows. + `#4298 `_ + Bug fixes: +* Fixed a bug where ``UnicodeDecodeError`` was not propagated from various + ``py::str`` ctors when decoding surrogate utf characters. + `#4294 `_ + +* Revert perfect forwarding for ``make_iterator``. This broke at least one + valid use case. May revisit later. + `#4234 `_ + +* Fix support for safe casts to ``void*`` (regression in 2.10.0). + `#4275 `_ + +* Fix ``char8_t`` support (regression in 2.9). + `#4278 `_ + +* Unicode surrogate character in Python exception message leads to process + termination in ``error_already_set::what()``. + `#4297 `_ + * Fix MSVC 2019 v.1924 & C++14 mode error for ``overload_cast``. `#4188 `_ @@ -100,9 +131,15 @@ Performance and style: * Optimize unpacking_collector when processing ``arg_v`` arguments. `#4219 `_ +* Optimize casting C++ object to ``None``. + `#4269 `_ + Build system improvements: +* CMake: revert overwrite behavior, now opt-in with ``PYBIND11_PYTHONLIBS_OVERRWRITE OFF``. + `#4195 `_ + * Include a pkg-config file when installing pybind11, such as in the Python package. `#4077 `_ From 2441d25b26af3e7b89d521a218694b604ec716e8 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 1 Nov 2022 00:10:13 -0400 Subject: [PATCH 21/53] chore(deps): update pre-commit hooks (#4302) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit updates: - [github.com/asottile/pyupgrade: v2.38.2 → v3.2.0](https://github.com/asottile/pyupgrade/compare/v2.38.2...v3.2.0) - [github.com/psf/black: 22.8.0 → 22.10.0](https://github.com/psf/black/compare/22.8.0...22.10.0) - [github.com/PyCQA/pylint: v2.15.3 → v2.15.5](https://github.com/PyCQA/pylint/compare/v2.15.3...v2.15.5) - [github.com/pre-commit/mirrors-mypy: v0.981 → v0.982](https://github.com/pre-commit/mirrors-mypy/compare/v0.981...v0.982) - [github.com/codespell-project/codespell: v2.2.1 → v2.2.2](https://github.com/codespell-project/codespell/compare/v2.2.1...v2.2.2) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 742e7a30a..a60d84f22 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -41,7 +41,7 @@ repos: # Upgrade old Python syntax - repo: https://github.com/asottile/pyupgrade - rev: "v2.38.2" + rev: "v3.2.0" hooks: - id: pyupgrade args: [--py36-plus] @@ -54,7 +54,7 @@ repos: # Black, the code formatter, natively supports pre-commit - repo: https://github.com/psf/black - rev: "22.8.0" # Keep in sync with blacken-docs + rev: "22.10.0" # Keep in sync with blacken-docs hooks: - id: black @@ -116,7 +116,7 @@ repos: # PyLint has native support - not always usable, but works for us - repo: https://github.com/PyCQA/pylint - rev: "v2.15.3" + rev: "v2.15.5" hooks: - id: pylint files: ^pybind11 @@ -132,7 +132,7 @@ repos: # Check static types with mypy - repo: https://github.com/pre-commit/mirrors-mypy - rev: "v0.981" + rev: "v0.982" hooks: - id: mypy args: [] @@ -152,7 +152,7 @@ repos: # Use tools/codespell_ignore_lines_from_errors.py # to rebuild .codespell-ignore-lines - repo: https://github.com/codespell-project/codespell - rev: "v2.2.1" + rev: "v2.2.2" hooks: - id: codespell exclude: ".supp$" From 0176632e8cef72a4223f2df54d6dfb9e552ec71f Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Tue, 1 Nov 2022 12:19:34 -0400 Subject: [PATCH 22/53] chore: sync blacken-docs hook with black (#4304) --- .pre-commit-config.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index a60d84f22..fd079dd03 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -64,7 +64,7 @@ repos: hooks: - id: blacken-docs additional_dependencies: - - black==22.8.0 # keep in sync with black hook + - black==22.10.0 # keep in sync with black hook # Changes tabs to spaces - repo: https://github.com/Lucas-C/pre-commit-hooks From ee2b5226295d67b690faddd446a329bb2840a1a8 Mon Sep 17 00:00:00 2001 From: Ethan Steinberg Date: Wed, 2 Nov 2022 11:32:53 -0700 Subject: [PATCH 23/53] Fix functional.h bug + introduce test to verify that it is fixed (#4254) * Illustrate bug in functional.h * style: pre-commit fixes * Make functional casting more robust / add workaround * Make function_record* casting even more robust * See if this fixes PyPy issue * It still fails on PyPy sadly * Do not make new CTOR just yet * Fix test * Add name to ensure correctness * style: pre-commit fixes * Clean up tests + remove ifdef guards * Add comments * Improve comments, error handling, and safety * Fix compile error * Fix magic logic * Extract helper function * Fix func signature * move to local internals * style: pre-commit fixes * Switch to simpler design * style: pre-commit fixes * Move to function_record * style: pre-commit fixes * Switch to internals, update tests and docs * Fix lint * Oops, forgot to resolve last comment * Fix typo * Update in response to comments * Implement suggestion to improve test * Update comment * Simple fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan --- include/pybind11/detail/internals.h | 31 ++++++++++++++++++++ include/pybind11/functional.h | 11 ++++++-- include/pybind11/pybind11.h | 44 ++++++++++++++++++++++------- tests/test_callbacks.cpp | 37 ++++++++++++++++++++++++ tests/test_callbacks.py | 13 +++++++++ 5 files changed, 124 insertions(+), 12 deletions(-) diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 7de779434..6fd61098c 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -43,6 +43,8 @@ using ExceptionTranslator = void (*)(std::exception_ptr); PYBIND11_NAMESPACE_BEGIN(detail) +constexpr const char *internals_function_record_capsule_name = "pybind11_function_record_capsule"; + // Forward declarations inline PyTypeObject *make_static_property_type(); inline PyTypeObject *make_default_metaclass(); @@ -182,6 +184,16 @@ struct internals { # endif // PYBIND11_INTERNALS_VERSION > 4 // Unused if PYBIND11_SIMPLE_GIL_MANAGEMENT is defined: PyInterpreterState *istate = nullptr; + +# if PYBIND11_INTERNALS_VERSION > 4 + // Note that we have to use a std::string to allocate memory to ensure a unique address + // We want unique addresses since we use pointer equality to compare function records + std::string function_record_capsule_name = internals_function_record_capsule_name; +# endif + + internals() = default; + internals(const internals &other) = delete; + internals &operator=(const internals &other) = delete; ~internals() { # if PYBIND11_INTERNALS_VERSION > 4 PYBIND11_TLS_FREE(loader_life_support_tls_key); @@ -548,6 +560,25 @@ const char *c_str(Args &&...args) { return strings.front().c_str(); } +inline const char *get_function_record_capsule_name() { +#if PYBIND11_INTERNALS_VERSION > 4 + return get_internals().function_record_capsule_name.c_str(); +#else + return nullptr; +#endif +} + +// Determine whether or not the following capsule contains a pybind11 function record. +// Note that we use `internals` to make sure that only ABI compatible records are touched. +// +// This check is currently used in two places: +// - An important optimization in functional.h to avoid overhead in C++ -> Python -> C++ +// - The sibling feature of cpp_function to allow overloads +inline bool is_function_record_capsule(const capsule &cap) { + // Pointer equality as we rely on internals() to ensure unique pointers + return cap.name() == get_function_record_capsule_name(); +} + PYBIND11_NAMESPACE_END(detail) /// Returns a named pointer that is shared among all extension modules (using the same diff --git a/include/pybind11/functional.h b/include/pybind11/functional.h index 102d1a938..87ec4d10c 100644 --- a/include/pybind11/functional.h +++ b/include/pybind11/functional.h @@ -48,9 +48,16 @@ public: */ if (auto cfunc = func.cpp_function()) { auto *cfunc_self = PyCFunction_GET_SELF(cfunc.ptr()); - if (isinstance(cfunc_self)) { + if (cfunc_self == nullptr) { + PyErr_Clear(); + } else if (isinstance(cfunc_self)) { auto c = reinterpret_borrow(cfunc_self); - auto *rec = (function_record *) c; + + function_record *rec = nullptr; + // Check that we can safely reinterpret the capsule into a function_record + if (detail::is_function_record_capsule(c)) { + rec = c.get_pointer(); + } while (rec != nullptr) { if (rec->is_stateless diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index e662236d0..76d6eadca 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -468,13 +468,20 @@ protected: if (rec->sibling) { if (PyCFunction_Check(rec->sibling.ptr())) { auto *self = PyCFunction_GET_SELF(rec->sibling.ptr()); - capsule rec_capsule = isinstance(self) ? reinterpret_borrow(self) - : capsule(self); - chain = (detail::function_record *) rec_capsule; - /* Never append a method to an overload chain of a parent class; - instead, hide the parent's overloads in this case */ - if (!chain->scope.is(rec->scope)) { + if (!isinstance(self)) { chain = nullptr; + } else { + auto rec_capsule = reinterpret_borrow(self); + if (detail::is_function_record_capsule(rec_capsule)) { + chain = rec_capsule.get_pointer(); + /* Never append a method to an overload chain of a parent class; + instead, hide the parent's overloads in this case */ + if (!chain->scope.is(rec->scope)) { + chain = nullptr; + } + } else { + chain = nullptr; + } } } // Don't trigger for things like the default __init__, which are wrapper_descriptors @@ -496,6 +503,7 @@ protected: capsule rec_capsule(unique_rec.release(), [](void *ptr) { destruct((detail::function_record *) ptr); }); + rec_capsule.set_name(detail::get_function_record_capsule_name()); guarded_strdup.release(); object scope_module; @@ -661,10 +669,13 @@ protected: /// Main dispatch logic for calls to functions bound using pybind11 static PyObject *dispatcher(PyObject *self, PyObject *args_in, PyObject *kwargs_in) { using namespace detail; + assert(isinstance(self)); /* Iterator over the list of potentially admissible overloads */ - const function_record *overloads = (function_record *) PyCapsule_GetPointer(self, nullptr), + const function_record *overloads = reinterpret_cast( + PyCapsule_GetPointer(self, get_function_record_capsule_name())), *it = overloads; + assert(overloads != nullptr); /* Need to know how many arguments + keyword arguments there are to pick the right overload */ @@ -1871,9 +1882,22 @@ private: static detail::function_record *get_function_record(handle h) { h = detail::get_function(h); - return h ? (detail::function_record *) reinterpret_borrow( - PyCFunction_GET_SELF(h.ptr())) - : nullptr; + if (!h) { + return nullptr; + } + + handle func_self = PyCFunction_GET_SELF(h.ptr()); + if (!func_self) { + throw error_already_set(); + } + if (!isinstance(func_self)) { + return nullptr; + } + auto cap = reinterpret_borrow(func_self); + if (!detail::is_function_record_capsule(cap)) { + return nullptr; + } + return cap.get_pointer(); } }; diff --git a/tests/test_callbacks.cpp b/tests/test_callbacks.cpp index 92b8053de..2fd05dec7 100644 --- a/tests/test_callbacks.cpp +++ b/tests/test_callbacks.cpp @@ -240,4 +240,41 @@ TEST_SUBMODULE(callbacks, m) { f(); } }); + + auto *custom_def = []() { + static PyMethodDef def; + def.ml_name = "example_name"; + def.ml_doc = "Example doc"; + def.ml_meth = [](PyObject *, PyObject *args) -> PyObject * { + if (PyTuple_Size(args) != 1) { + throw std::runtime_error("Invalid number of arguments for example_name"); + } + PyObject *first = PyTuple_GetItem(args, 0); + if (!PyLong_Check(first)) { + throw std::runtime_error("Invalid argument to example_name"); + } + auto result = py::cast(PyLong_AsLong(first) * 9); + return result.release().ptr(); + }; + def.ml_flags = METH_VARARGS; + return &def; + }(); + + // rec_capsule with name that has the same value (but not pointer) as our internal one + // This capsule should be detected by our code as foreign and not inspected as the pointers + // shouldn't match + constexpr const char *rec_capsule_name + = pybind11::detail::internals_function_record_capsule_name; + py::capsule rec_capsule(std::malloc(1), [](void *data) { std::free(data); }); + rec_capsule.set_name(rec_capsule_name); + m.add_object("custom_function", PyCFunction_New(custom_def, rec_capsule.ptr())); + + // This test requires a new ABI version to pass +#if PYBIND11_INTERNALS_VERSION > 4 + // rec_capsule with nullptr name + py::capsule rec_capsule2(std::malloc(1), [](void *data) { std::free(data); }); + m.add_object("custom_function2", PyCFunction_New(custom_def, rec_capsule2.ptr())); +#else + m.add_object("custom_function2", py::none()); +#endif } diff --git a/tests/test_callbacks.py b/tests/test_callbacks.py index 0b1047bbf..57b659988 100644 --- a/tests/test_callbacks.py +++ b/tests/test_callbacks.py @@ -193,3 +193,16 @@ def test_callback_num_times(): if len(rates) > 1: print("Min Mean Max") print(f"{min(rates):6.3f} {sum(rates) / len(rates):6.3f} {max(rates):6.3f}") + + +def test_custom_func(): + assert m.custom_function(4) == 36 + assert m.roundtrip(m.custom_function)(4) == 36 + + +@pytest.mark.skipif( + m.custom_function2 is None, reason="Current PYBIND11_INTERNALS_VERSION too low" +) +def test_custom_func2(): + assert m.custom_function2(3) == 27 + assert m.roundtrip(m.custom_function2)(3) == 27 From 1f04cc7062e33481c62c78231e9561b318bca67b Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 10 Nov 2022 08:33:26 -0800 Subject: [PATCH 24/53] Add windows_clang to ci.yml (#4323) * Add windows_clang to ci.yml (previously tested under PRs #4321, #4319) * Add `pip install --upgrade pip`, Show env, cosmetic changes Already tested under PR #4321 --- .github/workflows/ci.yml | 70 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a11cae1ab..adbbf626f 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -967,3 +967,73 @@ jobs: - name: Interface test C++17 run: PYTHONHOME=/${{matrix.sys}} PYTHONPATH=/${{matrix.sys}} cmake --build build3 --target test_cmake_build + + windows_clang: + + strategy: + matrix: + os: [windows-latest] + python: ['3.10'] + + runs-on: "${{ matrix.os }}" + + name: "🐍 ${{ matrix.python }} • ${{ matrix.os }} • clang-latest" + + steps: + - name: Show env + run: env + + - name: Checkout + uses: actions/checkout@v3 + + - name: Set up Clang + uses: egor-tensin/setup-clang@v1 + + - name: Setup Python ${{ matrix.python }} + uses: actions/setup-python@v4 + with: + python-version: ${{ matrix.python }} + + - name: Update CMake + uses: jwlawson/actions-setup-cmake@v1.13 + + - name: Install ninja-build tool + uses: seanmiddleditch/gha-setup-ninja@v3 + + - name: Run pip installs + run: | + python -m pip install --upgrade pip + python -m pip install -r tests/requirements.txt + + - name: Show Clang++ version + run: clang++ --version + + - name: Show CMake version + run: cmake --version + + # TODO: WERROR=ON + - name: Configure Clang + run: > + cmake -G Ninja -S . -B . + -DCMAKE_VERBOSE_MAKEFILE=ON + -DPYBIND11_WERROR=OFF + -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF + -DDOWNLOAD_CATCH=ON + -DDOWNLOAD_EIGEN=ON + -DCMAKE_CXX_COMPILER=clang++ + -DCMAKE_CXX_STANDARD=17 + + - name: Build + run: cmake --build . -j 2 + + - name: Python tests + run: cmake --build . --target pytest -j 2 + + - name: C++ tests + run: cmake --build . --target cpptest -j 2 + + - name: Interface test + run: cmake --build . --target test_cmake_build -j 2 + + - name: Clean directory + run: git clean -fdx From 88b019a8a5e116d7e4a4ffae6399a426364d4bcd Mon Sep 17 00:00:00 2001 From: gitartpiano <51239761+gitartpiano@users.noreply.github.com> Date: Fri, 11 Nov 2022 19:52:57 -0600 Subject: [PATCH 25/53] fix pybind11Tools.cmake typo causing Unknown arguments (#4327) * fix pybind11Tools.cmake typo causing Unknown arguments CMake Error at pybind11/tools/pybind11Tools.cmake:217 (if): if given arguments: "NOT" "MSVC" "AND" "NOT" "TEST" "MATCHES" "DEBUG|RELWITHDEBINFO" Unknown arguments specified https://github.com/pybind/pybind11/issues/4325 * Apply the same fix in tools/pybind11NewTools.cmake Co-authored-by: Ralf W. Grosse-Kunstleve --- tools/pybind11NewTools.cmake | 2 +- tools/pybind11Tools.cmake | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tools/pybind11NewTools.cmake b/tools/pybind11NewTools.cmake index 9e13daf1a..5a6a0cb8b 100644 --- a/tools/pybind11NewTools.cmake +++ b/tools/pybind11NewTools.cmake @@ -235,7 +235,7 @@ function(pybind11_add_module target_name) # Use case-insensitive comparison to match the result of $ string(TOUPPER "${CMAKE_BUILD_TYPE}" uppercase_CMAKE_BUILD_TYPE) - if(NOT MSVC AND NOT ${uppercase_CMAKE_BUILD_TYPE} MATCHES DEBUG|RELWITHDEBINFO) + if(NOT MSVC AND NOT "${uppercase_CMAKE_BUILD_TYPE}" MATCHES DEBUG|RELWITHDEBINFO) # Strip unnecessary sections of the binary on Linux/macOS pybind11_strip(${target_name}) endif() diff --git a/tools/pybind11Tools.cmake b/tools/pybind11Tools.cmake index 0dc61d399..66ad00a47 100644 --- a/tools/pybind11Tools.cmake +++ b/tools/pybind11Tools.cmake @@ -214,7 +214,7 @@ function(pybind11_add_module target_name) # Use case-insensitive comparison to match the result of $ string(TOUPPER "${CMAKE_BUILD_TYPE}" uppercase_CMAKE_BUILD_TYPE) - if(NOT MSVC AND NOT ${uppercase_CMAKE_BUILD_TYPE} MATCHES DEBUG|RELWITHDEBINFO) + if(NOT MSVC AND NOT "${uppercase_CMAKE_BUILD_TYPE}" MATCHES DEBUG|RELWITHDEBINFO) pybind11_strip(${target_name}) endif() From 296615ad34f9d8f680efbab22553881ad9843063 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sat, 12 Nov 2022 12:24:19 -0800 Subject: [PATCH 26/53] Add macos_brew_install_llvm to ci.yml (#4326) * Add macos_brew_install_llvm to ci.yml Added block transferred from PR #4324 * `test_cross_module_exception_translator` xfail 'Homebrew Clang' * Add `pip install numpy scipy` (tested already under PR #4324). --- .github/workflows/ci.yml | 67 ++++++++++++++++++++++++++++++++++++++++ tests/test_exceptions.py | 5 +-- 2 files changed, 70 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index adbbf626f..bd99ddd33 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1037,3 +1037,70 @@ jobs: - name: Clean directory run: git clean -fdx + + macos_brew_install_llvm: + name: "macos-latest • brew install llvm" + runs-on: macos-latest + + env: + # https://apple.stackexchange.com/questions/227026/how-to-install-recent-clang-with-homebrew + LDFLAGS: '-L/usr/local/opt/llvm/lib -Wl,-rpath,/usr/local/opt/llvm/lib' + + steps: + - name: Update PATH + run: echo "/usr/local/opt/llvm/bin" >> $GITHUB_PATH + + - name: Show env + run: env + + - name: Checkout + uses: actions/checkout@v3 + + - name: Show Clang++ version before brew install llvm + run: clang++ --version + + - name: brew install llvm + run: brew install llvm + + - name: Show Clang++ version after brew install llvm + run: clang++ --version + + - name: Update CMake + uses: jwlawson/actions-setup-cmake@v1.13 + + - name: Run pip installs + run: | + python3 -m pip install --upgrade pip + python3 -m pip install -r tests/requirements.txt + python3 -m pip install numpy + python3 -m pip install scipy + + - name: Show CMake version + run: cmake --version + + - name: CMake Configure + run: > + cmake -S . -B . + -DCMAKE_VERBOSE_MAKEFILE=ON + -DPYBIND11_WERROR=ON + -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF + -DDOWNLOAD_CATCH=ON + -DDOWNLOAD_EIGEN=ON + -DCMAKE_CXX_COMPILER=clang++ + -DCMAKE_CXX_STANDARD=17 + -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") + + - name: Build + run: cmake --build . -j 2 + + - name: Python tests + run: cmake --build . --target pytest -j 2 + + - name: C++ tests + run: cmake --build . --target cpptest -j 2 + + - name: Interface test + run: cmake --build . --target test_cmake_build -j 2 + + - name: Clean directory + run: git clean -fdx diff --git a/tests/test_exceptions.py b/tests/test_exceptions.py index 7eb1a9d62..0d2c80814 100644 --- a/tests/test_exceptions.py +++ b/tests/test_exceptions.py @@ -4,6 +4,7 @@ import pytest import env import pybind11_cross_module_tests as cm +import pybind11_tests # noqa: F401 from pybind11_tests import exceptions as m @@ -72,9 +73,9 @@ def test_cross_module_exceptions(msg): # TODO: FIXME @pytest.mark.xfail( - "env.PYPY and env.MACOS", + "env.MACOS and (env.PYPY or pybind11_tests.compiler_info.startswith('Homebrew Clang'))", raises=RuntimeError, - reason="Expected failure with PyPy and libc++ (Issue #2847 & PR #2999)", + reason="See Issue #2847, PR #2999, PR #4324", ) def test_cross_module_exception_translator(): with pytest.raises(KeyError): From 48949222c637da9fc72b0ed6526ae1b40bb55237 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 22 Nov 2022 15:14:49 -0800 Subject: [PATCH 27/53] Use `PyEval_InitThreads()` as intended (#4350) * Use `PyEval_InitThreads()` as intended (actually matters only for Python 3.6). * Add `if defined(WITH_THREAD)` condition. https://docs.python.org/3.6/c-api/init.html#c.PyEval_InitThreads > This function is not available when thread support is disabled at compile time. * Fix oversight pointed out by @EricCousineau-TRI: Remove condition that is always false. --- include/pybind11/detail/internals.h | 3 --- include/pybind11/embed.h | 3 +++ 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 6fd61098c..ef1849fbe 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -468,9 +468,6 @@ PYBIND11_NOINLINE internals &get_internals() { internals_ptr = new internals(); #if defined(WITH_THREAD) -# if PY_VERSION_HEX < 0x03090000 - PyEval_InitThreads(); -# endif PyThreadState *tstate = PyThreadState_Get(); if (!PYBIND11_TLS_KEY_CREATE(internals_ptr->tstate)) { pybind11_fail("get_internals: could not successfully initialize the tstate TSS key!"); diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index 0ac609e0f..5b77594b3 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -118,6 +118,9 @@ inline void initialize_interpreter(bool init_signal_handlers = true, #if PY_VERSION_HEX < 0x030B0000 Py_InitializeEx(init_signal_handlers ? 1 : 0); +# if defined(WITH_THREAD) && PY_VERSION_HEX < 0x03070000 + PyEval_InitThreads(); +# endif // Before it was special-cased in python 3.8, passing an empty or null argv // caused a segfault, so we have to reimplement the special case ourselves. From 9c18a74e377dece2f2acd22c2c6e63ecb2a59c77 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 22 Nov 2022 17:17:02 -0800 Subject: [PATCH 28/53] Use `multiprocessing` `start_method` `"forkserver"` (#4306) * Use `multiprocessing` `start_method` `"forkserver"` Alternative to PR #4305 * Add link to comment under PR #4105 * Unconditionally `pytest.skip("DEADLOCK")` for PyPy Windows * Remove `SKIP_IF_DEADLOCK` entirely, for simplicity. Hopefully this PR will resolve the deadlocks for good. * Add "In a nutshell" comment, in response to request by @EricCousineau-TRI --- tests/conftest.py | 12 ++++++++++++ tests/test_gil_scoped.py | 4 ++-- 2 files changed, 14 insertions(+), 2 deletions(-) diff --git a/tests/conftest.py b/tests/conftest.py index f5ddb9f12..96dffc81c 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -7,6 +7,8 @@ Adds docstring and exceptions message sanitizers. import contextlib import difflib import gc +import multiprocessing +import os import re import textwrap @@ -15,6 +17,16 @@ import pytest # Early diagnostic for failed imports import pybind11_tests +if os.name != "nt": + # Full background: https://github.com/pybind/pybind11/issues/4105#issuecomment-1301004592 + # In a nutshell: fork() after starting threads == flakiness in the form of deadlocks. + # It is actually a well-known pitfall, unfortunately without guard rails. + # "forkserver" is more performant than "spawn" (~9s vs ~13s for tests/test_gil_scoped.py, + # visit the issuecomment link above for details). + # Windows does not have fork() and the associated pitfall, therefore it is best left + # running with defaults. + multiprocessing.set_start_method("forkserver") + _long_marker = re.compile(r"([0-9])L") _hexadecimal = re.compile(r"0x[0-9a-fA-F]+") diff --git a/tests/test_gil_scoped.py b/tests/test_gil_scoped.py index e890a7b0c..6af6a472d 100644 --- a/tests/test_gil_scoped.py +++ b/tests/test_gil_scoped.py @@ -5,6 +5,7 @@ import time import pytest +import env from pybind11_tests import gil_scoped as m @@ -144,7 +145,6 @@ def _intentional_deadlock(): ALL_BASIC_TESTS_PLUS_INTENTIONAL_DEADLOCK = ALL_BASIC_TESTS + (_intentional_deadlock,) -SKIP_IF_DEADLOCK = True # See PR #4216 def _run_in_process(target, *args, **kwargs): @@ -181,7 +181,7 @@ def _run_in_process(target, *args, **kwargs): elif process.exitcode is None: assert t_delta > 0.9 * timeout msg = "DEADLOCK, most likely, exactly what this test is meant to detect." - if SKIP_IF_DEADLOCK: + if env.PYPY and env.WIN: pytest.skip(msg) raise RuntimeError(msg) return process.exitcode From 9907bedce517ea291c70c6a7b5dfec9e594f41df Mon Sep 17 00:00:00 2001 From: Xuehai Pan Date: Sat, 26 Nov 2022 07:15:54 +0800 Subject: [PATCH 29/53] fix(.github): fix bug-report issue template (#4363) --- .github/ISSUE_TEMPLATE/bug-report.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/.github/ISSUE_TEMPLATE/bug-report.yml b/.github/ISSUE_TEMPLATE/bug-report.yml index e6494ba6a..4f1e78f33 100644 --- a/.github/ISSUE_TEMPLATE/bug-report.yml +++ b/.github/ISSUE_TEMPLATE/bug-report.yml @@ -21,10 +21,11 @@ body: - label: Consider asking first in the [Gitter chat room](https://gitter.im/pybind/Lobby) or in a [Discussion](https:/pybind/pybind11/discussions/new). required: false - - type: Input + - type: input id: version attributes: label: What version (or hash if on master) of pybind11 are you using? + validations: required: true - type: textarea @@ -52,7 +53,7 @@ body: starting point for working out fixes. render: text - - type: Input + - type: input id: regression attributes: label: Is this a regression? Put the last known working version here if it is. From 06003e82b3ff48337b71b310b46c3d8b15ca6d5a Mon Sep 17 00:00:00 2001 From: Ethan Steinberg Date: Mon, 28 Nov 2022 07:39:38 -0800 Subject: [PATCH 30/53] Introduce a new style of warning suppression based on push/pop (#4285) * Introduce a new warning suppression system * Switch to better name * Nits --- include/pybind11/cast.h | 11 ++-- include/pybind11/detail/common.h | 82 +++++++++++++++++++++----- include/pybind11/detail/init.h | 9 ++- include/pybind11/eigen/matrix.h | 31 +++------- include/pybind11/eigen/tensor.h | 35 ++++------- include/pybind11/numpy.h | 15 ++--- include/pybind11/pybind11.h | 38 +++++------- include/pybind11/pytypes.h | 16 +++-- tests/test_builtin_casters.cpp | 9 ++- tests/test_class.cpp | 4 +- tests/test_constants_and_functions.cpp | 34 ++++------- tests/test_eigen_matrix.cpp | 4 +- tests/test_eigen_tensor.inl | 8 ++- tests/test_embed/catch.cpp | 4 +- tests/test_embed/test_interpreter.cpp | 4 +- tests/test_kwargs_and_defaults.cpp | 9 ++- tests/test_operator_overloading.cpp | 23 +++----- 17 files changed, 173 insertions(+), 163 deletions(-) diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 430c62f35..8f92df8ab 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -29,6 +29,9 @@ #include PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + +PYBIND11_WARNING_DISABLE_MSVC(4127) + PYBIND11_NAMESPACE_BEGIN(detail) template @@ -389,7 +392,7 @@ struct string_caster { // For UTF-8 we avoid the need for a temporary `bytes` object by using // `PyUnicode_AsUTF8AndSize`. - if (PYBIND11_SILENCE_MSVC_C4127(UTF_N == 8)) { + if (UTF_N == 8) { Py_ssize_t size = -1; const auto *buffer = reinterpret_cast(PyUnicode_AsUTF8AndSize(load_src.ptr(), &size)); @@ -416,7 +419,7 @@ struct string_caster { = reinterpret_cast(PYBIND11_BYTES_AS_STRING(utfNbytes.ptr())); size_t length = (size_t) PYBIND11_BYTES_SIZE(utfNbytes.ptr()) / sizeof(CharT); // Skip BOM for UTF-16/32 - if (PYBIND11_SILENCE_MSVC_C4127(UTF_N > 8)) { + if (UTF_N > 8) { buffer++; length--; } @@ -572,7 +575,7 @@ public: // figure out how long the first encoded character is in bytes to distinguish between these // two errors. We also allow want to allow unicode characters U+0080 through U+00FF, as // those can fit into a single char value. - if (PYBIND11_SILENCE_MSVC_C4127(StringCaster::UTF_N == 8) && str_len > 1 && str_len <= 4) { + if (StringCaster::UTF_N == 8 && str_len > 1 && str_len <= 4) { auto v0 = static_cast(value[0]); // low bits only: 0-127 // 0b110xxxxx - start of 2-byte sequence @@ -598,7 +601,7 @@ public: // UTF-16 is much easier: we can only have a surrogate pair for values above U+FFFF, thus a // surrogate pair with total length 2 instantly indicates a range error (but not a "your // string was too long" error). - else if (PYBIND11_SILENCE_MSVC_C4127(StringCaster::UTF_N == 16) && str_len == 2) { + else if (StringCaster::UTF_N == 16 && str_len == 2) { one_char = static_cast(value[0]); if (one_char >= 0xD800 && one_char < 0xE000) { throw value_error("Character code point not in range(0x10000)"); diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 9b74323f6..b58dc3afa 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -17,8 +17,69 @@ // Additional convention: 0xD = dev #define PYBIND11_VERSION_HEX 0x020B00D1 -#define PYBIND11_NAMESPACE_BEGIN(name) namespace name { -#define PYBIND11_NAMESPACE_END(name) } +// Define some generic pybind11 helper macros for warning management. +// +// Note that compiler-specific push/pop pairs are baked into the +// PYBIND11_NAMESPACE_BEGIN/PYBIND11_NAMESPACE_END pair of macros. Therefore manual +// PYBIND11_WARNING_PUSH/PYBIND11_WARNING_POP are usually only needed in `#include` sections. +// +// If you find you need to suppress a warning, please try to make the suppression as local as +// possible using these macros. Please also be sure to push/pop with the pybind11 macros. Please +// only use compiler specifics if you need to check specific versions, e.g. Apple Clang vs. vanilla +// Clang. +#if defined(_MSC_VER) +# define PYBIND11_COMPILER_MSVC +# define PYBIND11_PRAGMA(...) __pragma(__VA_ARGS__) +# define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(warning(push)) +# define PYBIND11_WARNING_POP PYBIND11_PRAGMA(warning(pop)) +#elif defined(__INTEL_COMPILER) +# define PYBIND11_COMPILER_INTEL +# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(warning push) +# define PYBIND11_WARNING_POP PYBIND11_PRAGMA(warning pop) +#elif defined(__clang__) +# define PYBIND11_COMPILER_CLANG +# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(clang diagnostic push) +# define PYBIND11_WARNING_POP PYBIND11_PRAGMA(clang diagnostic push) +#elif defined(__GNUC__) +# define PYBIND11_COMPILER_GCC +# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(GCC diagnostic push) +# define PYBIND11_WARNING_POP PYBIND11_PRAGMA(GCC diagnostic pop) +#endif + +#ifdef PYBIND11_COMPILER_MSVC +# define PYBIND11_WARNING_DISABLE_MSVC(name) PYBIND11_PRAGMA(warning(disable : name)) +#else +# define PYBIND11_WARNING_DISABLE_MSVC(name) +#endif + +#ifdef PYBIND11_COMPILER_CLANG +# define PYBIND11_WARNING_DISABLE_CLANG(name) PYBIND11_PRAGMA(clang diagnostic ignored name) +#else +# define PYBIND11_WARNING_DISABLE_CLANG(name) +#endif + +#ifdef PYBIND11_COMPILER_GCC +# define PYBIND11_WARNING_DISABLE_GCC(name) PYBIND11_PRAGMA(GCC diagnostic ignored name) +#else +# define PYBIND11_WARNING_DISABLE_GCC(name) +#endif + +#ifdef PYBIND11_COMPILER_INTEL +# define PYBIND11_WARNING_DISABLE_INTEL(name) PYBIND11_PRAGMA(warning disable name) +#else +# define PYBIND11_WARNING_DISABLE_INTEL(name) +#endif + +#define PYBIND11_NAMESPACE_BEGIN(name) \ + namespace name { \ + PYBIND11_WARNING_PUSH + +#define PYBIND11_NAMESPACE_END(name) \ + PYBIND11_WARNING_POP \ + } // Robust support for some features and loading modules compiled against different pybind versions // requires forcing hidden visibility on pybind code, so we enforce this by setting the attribute @@ -151,9 +212,9 @@ /// Include Python header, disable linking to pythonX_d.lib on Windows in debug mode #if defined(_MSC_VER) -# pragma warning(push) +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_MSVC(4505) // C4505: 'PySlice_GetIndicesEx': unreferenced local function has been removed (PyPy only) -# pragma warning(disable : 4505) # if defined(_DEBUG) && !defined(Py_DEBUG) // Workaround for a VS 2022 issue. // NOTE: This workaround knowingly violates the Python.h include order requirement: @@ -236,7 +297,7 @@ # define _DEBUG # undef PYBIND11_DEBUG_MARKER # endif -# pragma warning(pop) +PYBIND11_WARNING_POP #endif #include @@ -1136,17 +1197,6 @@ constexpr # define PYBIND11_WORKAROUND_INCORRECT_GCC_UNUSED_BUT_SET_PARAMETER(...) #endif -#if defined(_MSC_VER) // All versions (as of July 2021). - -// warning C4127: Conditional expression is constant -constexpr inline bool silence_msvc_c4127(bool cond) { return cond; } - -# define PYBIND11_SILENCE_MSVC_C4127(...) ::pybind11::detail::silence_msvc_c4127(__VA_ARGS__) - -#else -# define PYBIND11_SILENCE_MSVC_C4127(...) __VA_ARGS__ -#endif - #if defined(__clang__) \ && (defined(__apple_build_version__) /* AppleClang 13.0.0.13000029 was the only data point \ available. */ \ diff --git a/include/pybind11/detail/init.h b/include/pybind11/detail/init.h index 05f4fe54a..0938c9bde 100644 --- a/include/pybind11/detail/init.h +++ b/include/pybind11/detail/init.h @@ -12,6 +12,9 @@ #include "class.h" PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + +PYBIND11_WARNING_DISABLE_MSVC(4127) + PYBIND11_NAMESPACE_BEGIN(detail) template <> @@ -115,7 +118,7 @@ template void construct(value_and_holder &v_h, Cpp *ptr, bool need_alias) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); no_nullptr(ptr); - if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias && !is_alias(ptr)) { + if (Class::has_alias && need_alias && !is_alias(ptr)) { // We're going to try to construct an alias by moving the cpp type. Whether or not // that succeeds, we still need to destroy the original cpp pointer (either the // moved away leftover, if the alias construction works, or the value itself if we @@ -156,7 +159,7 @@ void construct(value_and_holder &v_h, Holder holder, bool need_alias) { auto *ptr = holder_helper>::get(holder); no_nullptr(ptr); // If we need an alias, check that the held pointer is actually an alias instance - if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias && !is_alias(ptr)) { + if (Class::has_alias && need_alias && !is_alias(ptr)) { throw type_error("pybind11::init(): construction failed: returned holder-wrapped instance " "is not an alias instance"); } @@ -174,7 +177,7 @@ void construct(value_and_holder &v_h, Cpp &&result, bool need_alias) { PYBIND11_WORKAROUND_INCORRECT_MSVC_C4100(need_alias); static_assert(std::is_move_constructible>::value, "pybind11::init() return-by-value factory function requires a movable class"); - if (PYBIND11_SILENCE_MSVC_C4127(Class::has_alias) && need_alias) { + if (Class::has_alias && need_alias) { construct_alias_from_cpp(is_alias_constructible{}, v_h, std::move(result)); } else { v_h.value_ptr() = new Cpp(std::move(result)); diff --git a/include/pybind11/eigen/matrix.h b/include/pybind11/eigen/matrix.h index c30dac241..34fe329a8 100644 --- a/include/pybind11/eigen/matrix.h +++ b/include/pybind11/eigen/matrix.h @@ -16,29 +16,15 @@ https://stackoverflow.com/questions/2579576/i-dir-vs-isystem-dir https://stackoverflow.com/questions/1741816/isystem-for-ms-visual-studio-c-compiler */ -// The C4127 suppression was introduced for Eigen 3.4.0. In theory we could -// make it version specific, or even remove it later, but considering that -// 1. C4127 is generally far more distracting than useful for modern template code, and -// 2. we definitely want to ignore any MSVC warnings originating from Eigen code, -// it is probably best to keep this around indefinitely. -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable : 4127) // C4127: conditional expression is constant -# pragma warning(disable : 5054) // https://github.com/pybind/pybind11/pull/3741 +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_MSVC(5054) // https://github.com/pybind/pybind11/pull/3741 // C5054: operator '&': deprecated between enumerations of different types -#elif defined(__MINGW32__) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wmaybe-uninitialized" -#endif +PYBIND11_WARNING_DISABLE_GCC("-Wmaybe-uninitialized") #include #include -#if defined(_MSC_VER) -# pragma warning(pop) -#elif defined(__MINGW32__) -# pragma GCC diagnostic pop -#endif +PYBIND11_WARNING_POP // Eigen prior to 3.2.7 doesn't have proper move constructors--but worse, some classes get implicit // move constructors that break things. We could detect this an explicitly copy, but an extra copy @@ -48,6 +34,8 @@ static_assert(EIGEN_VERSION_AT_LEAST(3, 2, 7), PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_WARNING_DISABLE_MSVC(4127) + // Provide a convenience alias for easier pass-by-ref usage with fully dynamic strides: using EigenDStride = Eigen::Stride; template @@ -189,8 +177,7 @@ struct EigenProps { EigenIndex np_rows = a.shape(0), np_cols = a.shape(1), np_rstride = a.strides(0) / static_cast(sizeof(Scalar)), np_cstride = a.strides(1) / static_cast(sizeof(Scalar)); - if ((PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && np_rows != rows) - || (PYBIND11_SILENCE_MSVC_C4127(fixed_cols) && np_cols != cols)) { + if ((fixed_rows && np_rows != rows) || (fixed_cols && np_cols != cols)) { return false; } @@ -203,7 +190,7 @@ struct EigenProps { stride = a.strides(0) / static_cast(sizeof(Scalar)); if (vector) { // Eigen type is a compile-time vector - if (PYBIND11_SILENCE_MSVC_C4127(fixed) && size != n) { + if (fixed && size != n) { return false; // Vector size mismatch } return {rows == 1 ? 1 : n, cols == 1 ? 1 : n, stride}; @@ -220,7 +207,7 @@ struct EigenProps { } return {1, n, stride}; } // Otherwise it's either fully dynamic, or column dynamic; both become a column vector - if (PYBIND11_SILENCE_MSVC_C4127(fixed_rows) && rows != n) { + if (fixed_rows && rows != n) { return false; } return {n, 1, stride}; diff --git a/include/pybind11/eigen/tensor.h b/include/pybind11/eigen/tensor.h index a823c0f39..a129f9929 100644 --- a/include/pybind11/eigen/tensor.h +++ b/include/pybind11/eigen/tensor.h @@ -13,28 +13,23 @@ static_assert(__GNUC__ > 5, "Eigen Tensor support in pybind11 requires GCC > 5.0"); #endif -#if defined(_MSC_VER) -# pragma warning(push) -# pragma warning(disable : 4554) // Tensor.h warning -# pragma warning(disable : 4127) // Tensor.h warning -#elif defined(__MINGW32__) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wmaybe-uninitialized" -#endif +// Disable warnings for Eigen +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_MSVC(4554) +PYBIND11_WARNING_DISABLE_MSVC(4127) +PYBIND11_WARNING_DISABLE_GCC("-Wmaybe-uninitialized") #include -#if defined(_MSC_VER) -# pragma warning(pop) -#elif defined(__MINGW32__) -# pragma GCC diagnostic pop -#endif +PYBIND11_WARNING_POP static_assert(EIGEN_VERSION_AT_LEAST(3, 3, 0), "Eigen Tensor support in pybind11 requires Eigen >= 3.3.0"); PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_WARNING_DISABLE_MSVC(4127) + PYBIND11_NAMESPACE_BEGIN(detail) inline bool is_tensor_aligned(const void *data) { @@ -138,10 +133,8 @@ struct get_tensor_descriptor { // // We need to disable the type-limits warning for the inner loop when size = 0. -#if defined(__GNUC__) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wtype-limits" -#endif +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_GCC("-Wtype-limits") template std::vector convert_dsizes_to_vector(const Eigen::DSizes &arr) { @@ -165,9 +158,7 @@ Eigen::DSizes get_shape_for_array(const array &arr) { return result; } -#if defined(__GNUC__) -# pragma GCC diagnostic pop -#endif +PYBIND11_WARNING_POP template struct type_caster::ValidType> { @@ -389,7 +380,7 @@ struct type_caster, constexpr bool is_aligned = (Options & Eigen::Aligned) != 0; - if (PYBIND11_SILENCE_MSVC_C4127(is_aligned) && !is_tensor_aligned(arr.data())) { + if (is_aligned && !is_tensor_aligned(arr.data())) { return false; } @@ -399,7 +390,7 @@ struct type_caster, return false; } - if (PYBIND11_SILENCE_MSVC_C4127(needs_writeable) && !arr.writeable()) { + if (needs_writeable && !arr.writeable()) { return false; } diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 369e18649..ea64aa7e7 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -36,6 +36,8 @@ static_assert(std::is_signed::value, "Py_intptr_t must be signed"); PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_WARNING_DISABLE_MSVC(4127) + class array; // Forward declaration PYBIND11_NAMESPACE_BEGIN(detail) @@ -875,7 +877,7 @@ public: */ template detail::unchecked_mutable_reference mutable_unchecked() & { - if (PYBIND11_SILENCE_MSVC_C4127(Dims >= 0) && ndim() != Dims) { + if (Dims >= 0 && ndim() != Dims) { throw std::domain_error("array has incorrect number of dimensions: " + std::to_string(ndim()) + "; expected " + std::to_string(Dims)); @@ -893,7 +895,7 @@ public: */ template detail::unchecked_reference unchecked() const & { - if (PYBIND11_SILENCE_MSVC_C4127(Dims >= 0) && ndim() != Dims) { + if (Dims >= 0 && ndim() != Dims) { throw std::domain_error("array has incorrect number of dimensions: " + std::to_string(ndim()) + "; expected " + std::to_string(Dims)); @@ -1865,9 +1867,10 @@ private: } auto result = returned_array::create(trivial, shape); + + PYBIND11_WARNING_PUSH #ifdef PYBIND11_DETECTED_CLANG_WITH_MISLEADING_CALL_STD_MOVE_EXPLICITLY_WARNING -# pragma clang diagnostic push -# pragma clang diagnostic ignored "-Wreturn-std-move" + PYBIND11_WARNING_DISABLE_CLANG("-Wreturn-std-move") #endif if (size == 0) { @@ -1883,9 +1886,7 @@ private: } return result; -#ifdef PYBIND11_DETECTED_CLANG_WITH_MISLEADING_CALL_STD_MOVE_EXPLICITLY_WARNING -# pragma clang diagnostic pop -#endif + PYBIND11_WARNING_POP } template diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 76d6eadca..10b27254e 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -35,6 +35,8 @@ # include #endif +PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) + /* https://stackoverflow.com/questions/46798456/handling-gccs-noexcept-type-warning This warning is about ABI compatibility, not code health. It is only actually needed in a couple places, but apparently GCC 7 "generates this warning if @@ -43,11 +45,10 @@ No other GCC version generates this warning. */ #if defined(__GNUC__) && __GNUC__ == 7 -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wnoexcept-type" +PYBIND11_WARNING_DISABLE_GCC("-Wnoexcept-type") #endif -PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_WARNING_DISABLE_MSVC(4127) PYBIND11_NAMESPACE_BEGIN(detail) @@ -177,22 +178,22 @@ protected: auto *rec = unique_rec.get(); /* Store the capture object directly in the function record if there is enough space */ - if (PYBIND11_SILENCE_MSVC_C4127(sizeof(capture) <= sizeof(rec->data))) { + if (sizeof(capture) <= sizeof(rec->data)) { /* Without these pragmas, GCC warns that there might not be enough space to use the placement new operator. However, the 'if' statement above ensures that this is the case. */ -#if defined(__GNUG__) && __GNUC__ >= 6 && !defined(__clang__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wplacement-new" + PYBIND11_WARNING_PUSH + +#if defined(__GNUG__) && __GNUC__ >= 6 + PYBIND11_WARNING_DISABLE_GCC("-Wplacement-new") #endif + new ((capture *) &rec->data) capture{std::forward(f)}; -#if defined(__GNUG__) && __GNUC__ >= 6 && !defined(__clang__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic pop -#endif -#if defined(__GNUG__) && !PYBIND11_HAS_STD_LAUNDER && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wstrict-aliasing" + +#if !PYBIND11_HAS_STD_LAUNDER + PYBIND11_WARNING_DISABLE_GCC("-Wstrict-aliasing") #endif + // UB without std::launder, but without breaking ABI and/or // a significant refactoring it's "impossible" to solve. if (!std::is_trivially_destructible::value) { @@ -202,9 +203,7 @@ protected: data->~capture(); }; } -#if defined(__GNUG__) && !PYBIND11_HAS_STD_LAUNDER && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic pop -#endif + PYBIND11_WARNING_POP } else { rec->data[0] = new capture{std::forward(f)}; rec->free_data = [](function_record *r) { delete ((capture *) r->data[0]); }; @@ -1841,8 +1840,7 @@ private: if (holder_ptr) { init_holder_from_existing(v_h, holder_ptr, std::is_copy_constructible()); v_h.set_holder_constructed(); - } else if (PYBIND11_SILENCE_MSVC_C4127(detail::always_construct_holder::value) - || inst->owned) { + } else if (detail::always_construct_holder::value || inst->owned) { new (std::addressof(v_h.holder())) holder_type(v_h.value_ptr()); v_h.set_holder_constructed(); } @@ -2876,7 +2874,3 @@ inline function get_overload(const T *this_ptr, const char *name) { PYBIND11_OVERRIDE_PURE(PYBIND11_TYPE(ret_type), PYBIND11_TYPE(cname), fn, __VA_ARGS__); PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) - -#if defined(__GNUC__) && __GNUC__ == 7 -# pragma GCC diagnostic pop // -Wnoexcept-type -#endif diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 80b49ec39..f913565d3 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -33,6 +33,8 @@ PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) +PYBIND11_WARNING_DISABLE_MSVC(4127) + /* A few forward declarations */ class handle; class object; @@ -884,10 +886,8 @@ object object_or_cast(T &&o); // Match a PyObject*, which we want to convert directly to handle via its converting constructor inline handle object_or_cast(PyObject *ptr) { return ptr; } -#if defined(_MSC_VER) && _MSC_VER < 1920 -# pragma warning(push) -# pragma warning(disable : 4522) // warning C4522: multiple assignment operators specified -#endif +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_MSVC(4522) // warning C4522: multiple assignment operators specified template class accessor : public object_api> { using key_type = typename Policy::key_type; @@ -951,9 +951,7 @@ private: key_type key; mutable object cache; }; -#if defined(_MSC_VER) && _MSC_VER < 1920 -# pragma warning(pop) -#endif +PYBIND11_WARNING_POP PYBIND11_NAMESPACE_BEGIN(accessor_policies) struct obj_attr { @@ -1693,7 +1691,7 @@ PYBIND11_NAMESPACE_BEGIN(detail) // unsigned type: (A)-1 != (B)-1 when A and B are unsigned types of different sizes). template Unsigned as_unsigned(PyObject *o) { - if (PYBIND11_SILENCE_MSVC_C4127(sizeof(Unsigned) <= sizeof(unsigned long))) { + if (sizeof(Unsigned) <= sizeof(unsigned long)) { unsigned long v = PyLong_AsUnsignedLong(o); return v == (unsigned long) -1 && PyErr_Occurred() ? (Unsigned) -1 : (Unsigned) v; } @@ -1710,7 +1708,7 @@ public: template ::value, int> = 0> // NOLINTNEXTLINE(google-explicit-constructor) int_(T value) { - if (PYBIND11_SILENCE_MSVC_C4127(sizeof(T) <= sizeof(long))) { + if (sizeof(T) <= sizeof(long)) { if (std::is_signed::value) { m_ptr = PyLong_FromLong((long) value); } else { diff --git a/tests/test_builtin_casters.cpp b/tests/test_builtin_casters.cpp index 6ff63edfc..0623b85dc 100644 --- a/tests/test_builtin_casters.cpp +++ b/tests/test_builtin_casters.cpp @@ -73,6 +73,9 @@ PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(pybind11) TEST_SUBMODULE(builtin_casters, m) { + PYBIND11_WARNING_PUSH + PYBIND11_WARNING_DISABLE_MSVC(4127) + // test_simple_string m.def("string_roundtrip", [](const char *s) { return s; }); @@ -86,7 +89,7 @@ TEST_SUBMODULE(builtin_casters, m) { std::wstring wstr; wstr.push_back(0x61); // a wstr.push_back(0x2e18); // ⸘ - if (PYBIND11_SILENCE_MSVC_C4127(sizeof(wchar_t) == 2)) { + if (sizeof(wchar_t) == 2) { wstr.push_back(mathbfA16_1); wstr.push_back(mathbfA16_2); } // 𝐀, utf16 @@ -113,7 +116,7 @@ TEST_SUBMODULE(builtin_casters, m) { // Under Python 2.7, invalid unicode UTF-32 characters didn't appear to trigger // UnicodeDecodeError m.def("bad_utf32_string", [=]() { return std::u32string({a32, char32_t(0xd800), z32}); }); - if (PYBIND11_SILENCE_MSVC_C4127(sizeof(wchar_t) == 2)) { + if (sizeof(wchar_t) == 2) { m.def("bad_wchar_string", [=]() { return std::wstring({wchar_t(0x61), wchar_t(0xd800)}); }); @@ -384,4 +387,6 @@ TEST_SUBMODULE(builtin_casters, m) { m.def("takes_const_ref", [](const ConstRefCasted &x) { return x.tag; }); m.def("takes_const_ref_wrap", [](std::reference_wrapper x) { return x.get().tag; }); + + PYBIND11_WARNING_POP } diff --git a/tests/test_class.cpp b/tests/test_class.cpp index fc1c17b17..9ea10fae1 100644 --- a/tests/test_class.cpp +++ b/tests/test_class.cpp @@ -22,10 +22,8 @@ #include -#if defined(_MSC_VER) -# pragma warning(disable : 4324) +PYBIND11_WARNING_DISABLE_MSVC(4324) // warning C4324: structure was padded due to alignment specifier -#endif // test_brace_initialization struct NoBraceInitialization { diff --git a/tests/test_constants_and_functions.cpp b/tests/test_constants_and_functions.cpp index 1918a429c..922375c5e 100644 --- a/tests/test_constants_and_functions.cpp +++ b/tests/test_constants_and_functions.cpp @@ -52,15 +52,12 @@ int f1(int x) noexcept { return x + 1; } #endif int f2(int x) noexcept(true) { return x + 2; } int f3(int x) noexcept(false) { return x + 3; } -#if defined(__GNUG__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wdeprecated" -#endif +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_GCC("-Wdeprecated") +PYBIND11_WARNING_DISABLE_CLANG("-Wdeprecated") // NOLINTNEXTLINE(modernize-use-noexcept) int f4(int x) throw() { return x + 4; } // Deprecated equivalent to noexcept(true) -#if defined(__GNUG__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic pop -#endif +PYBIND11_WARNING_POP struct C { int m1(int x) noexcept { return x - 1; } int m2(int x) const noexcept { return x - 2; } @@ -68,17 +65,14 @@ struct C { int m4(int x) const noexcept(true) { return x - 4; } int m5(int x) noexcept(false) { return x - 5; } int m6(int x) const noexcept(false) { return x - 6; } -#if defined(__GNUG__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic push -# pragma GCC diagnostic ignored "-Wdeprecated" -#endif + PYBIND11_WARNING_PUSH + PYBIND11_WARNING_DISABLE_GCC("-Wdeprecated") + PYBIND11_WARNING_DISABLE_CLANG("-Wdeprecated") // NOLINTNEXTLINE(modernize-use-noexcept) int m7(int x) throw() { return x - 7; } // NOLINTNEXTLINE(modernize-use-noexcept) int m8(int x) const throw() { return x - 8; } -#if defined(__GNUG__) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic pop -#endif + PYBIND11_WARNING_POP }; } // namespace test_exc_sp @@ -126,14 +120,12 @@ TEST_SUBMODULE(constants_and_functions, m) { .def("m8", &C::m8); m.def("f1", f1); m.def("f2", f2); -#if defined(__INTEL_COMPILER) -# pragma warning push -# pragma warning disable 878 // incompatible exception specifications -#endif + + PYBIND11_WARNING_PUSH + PYBIND11_WARNING_DISABLE_INTEL(878) // incompatible exception specifications m.def("f3", f3); -#if defined(__INTEL_COMPILER) -# pragma warning pop -#endif + PYBIND11_WARNING_POP + m.def("f4", f4); // test_function_record_leaks diff --git a/tests/test_eigen_matrix.cpp b/tests/test_eigen_matrix.cpp index 71e41d198..554cc4d7f 100644 --- a/tests/test_eigen_matrix.cpp +++ b/tests/test_eigen_matrix.cpp @@ -13,9 +13,7 @@ #include "constructor_stats.h" #include "pybind11_tests.h" -#if defined(_MSC_VER) -# pragma warning(disable : 4996) // C4996: std::unary_negation is deprecated -#endif +PYBIND11_WARNING_DISABLE_MSVC(4996) #include diff --git a/tests/test_eigen_tensor.inl b/tests/test_eigen_tensor.inl index 09b35fa13..f46eb1803 100644 --- a/tests/test_eigen_tensor.inl +++ b/tests/test_eigen_tensor.inl @@ -9,7 +9,9 @@ #include "pybind11_tests.h" -namespace PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE { +PYBIND11_NAMESPACE_BEGIN(PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE) + +PYBIND11_WARNING_DISABLE_MSVC(4127) template void reset_tensor(M &x) { @@ -90,7 +92,7 @@ struct CustomExample { template void init_tensor_module(pybind11::module &m) { const char *needed_options = ""; - if (PYBIND11_SILENCE_MSVC_C4127(Options == Eigen::ColMajor)) { + if (Options == Eigen::ColMajor) { needed_options = "F"; } else { needed_options = "C"; @@ -330,4 +332,4 @@ void test_module(py::module_ &m) { init_tensor_module(c_style); } -} // namespace PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE +PYBIND11_NAMESPACE_END(PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE) diff --git a/tests/test_embed/catch.cpp b/tests/test_embed/catch.cpp index a03a8b37c..558a7a35e 100644 --- a/tests/test_embed/catch.cpp +++ b/tests/test_embed/catch.cpp @@ -3,11 +3,9 @@ #include -#ifdef _MSC_VER // Silence MSVC C++17 deprecation warning from Catch regarding std::uncaught_exceptions (up to // catch 2.0.1; this should be fixed in the next catch release after 2.0.1). -# pragma warning(disable : 4996) -#endif +PYBIND11_WARNING_DISABLE_MSVC(4996) // Catch uses _ internally, which breaks gettext style defines #ifdef _ diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 44dcd1fdb..8cfd13fe5 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -1,10 +1,8 @@ #include -#ifdef _MSC_VER // Silence MSVC C++17 deprecation warning from Catch regarding std::uncaught_exceptions (up to // catch 2.0.1; this should be fixed in the next catch release after 2.0.1). -# pragma warning(disable : 4996) -#endif +PYBIND11_WARNING_DISABLE_MSVC(4996) #include #include diff --git a/tests/test_kwargs_and_defaults.cpp b/tests/test_kwargs_and_defaults.cpp index 2f3cabaf0..77e72c0c7 100644 --- a/tests/test_kwargs_and_defaults.cpp +++ b/tests/test_kwargs_and_defaults.cpp @@ -44,14 +44,13 @@ TEST_SUBMODULE(kwargs_and_defaults, m) { // test_args_and_kwargs m.def("args_function", [](py::args args) -> py::tuple { + PYBIND11_WARNING_PUSH + #ifdef PYBIND11_DETECTED_CLANG_WITH_MISLEADING_CALL_STD_MOVE_EXPLICITLY_WARNING -# pragma clang diagnostic push -# pragma clang diagnostic ignored "-Wreturn-std-move" + PYBIND11_WARNING_DISABLE_CLANG("-Wreturn-std-move") #endif return args; -#ifdef PYBIND11_DETECTED_CLANG_WITH_MISLEADING_CALL_STD_MOVE_EXPLICITLY_WARNING -# pragma clang diagnostic pop -#endif + PYBIND11_WARNING_POP }); m.def("args_kwargs_function", [](const py::args &args, const py::kwargs &kwargs) { return py::make_tuple(args, kwargs); diff --git a/tests/test_operator_overloading.cpp b/tests/test_operator_overloading.cpp index a4b895a89..112a363b4 100644 --- a/tests/test_operator_overloading.cpp +++ b/tests/test_operator_overloading.cpp @@ -132,22 +132,18 @@ struct hash { // Not a good abs function, but easy to test. std::string abs(const Vector2 &) { return "abs(Vector2)"; } -// MSVC & Intel warns about unknown pragmas, and warnings are errors. -#if !defined(_MSC_VER) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic push // clang 7.0.0 and Apple LLVM 10.0.1 introduce `-Wself-assign-overloaded` to // `-Wall`, which is used here for overloading (e.g. `py::self += py::self `). -// Here, we suppress the warning using `#pragma diagnostic`. +// Here, we suppress the warning // Taken from: https://github.com/RobotLocomotion/drake/commit/aaf84b46 // TODO(eric): This could be resolved using a function / functor (e.g. `py::self()`). -# if defined(__APPLE__) && defined(__clang__) -# if (__clang_major__ >= 10) -# pragma GCC diagnostic ignored "-Wself-assign-overloaded" -# endif -# elif defined(__clang__) -# if (__clang_major__ >= 7) -# pragma GCC diagnostic ignored "-Wself-assign-overloaded" -# endif +#if defined(__APPLE__) && defined(__clang__) +# if (__clang_major__ >= 10) +PYBIND11_WARNING_DISABLE_CLANG("-Wself-assign-overloaded") +# endif +#elif defined(__clang__) +# if (__clang_major__ >= 7) +PYBIND11_WARNING_DISABLE_CLANG("-Wself-assign-overloaded") # endif #endif @@ -283,6 +279,3 @@ TEST_SUBMODULE(operators, m) { m.def("get_unhashable_HashMe_set", []() { return std::unordered_set{{"one"}}; }); } -#if !defined(_MSC_VER) && !defined(__INTEL_COMPILER) -# pragma GCC diagnostic pop -#endif From 88699849264150f513dafb98d9ec1bb31a7fa904 Mon Sep 17 00:00:00 2001 From: Arman Date: Thu, 1 Dec 2022 07:17:59 +0200 Subject: [PATCH 31/53] scoped_interpreter. overloaded constructor: PyConfig param (#4330) * scoped_interpreter overloaded ctor: PyConfig param * style: pre-commit fixes * refact: some logics extracted into funcs (precheck_interpreter, _initialize_interpreter); config_guard * style: pre-commit fixes * refact: scoped_config, some funcs hidden in detail ns * refact: macro PYBIND11_PYCONFIG_SUPPORT_PY_VERSION + undef * feat: PYBIND11_PYCONFIG_SUPPORT_PY_VERSION set to 3.8 * tests: Custom PyConfig * ci: python 3.6 -> 3.8 * ci: reverted py 38 back to 36; refact: initialize_interpreter overloads * style: pre-commit fixes * fix: readability-implicit-bool-conversion * refact: each initialize_interpreter overloads in pybind11 ns * Move `initialize_interpreter_pre_pyconfig()` into the `detail` namespace. Move the `PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX` define down to where it is used for the first time, and check if it is defined already, so that it is possible to customize from the compilation command line, just in case there is some unforeseen issue for Python 3.8, 3.9, 3.10. * tests: Add program dir to path, Custom PyConfig with argv * refact: clang-formatted * tests: Add-program-dir-to-path covers both scoped_interpreter overloads * tests: Add-program-dir-to-path fixed * tests: Add-program-dir-to-path py_version dependant validation Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/embed.h | 119 +++++++++++++++++--------- tests/test_embed/test_interpreter.cpp | 66 ++++++++++++++ 2 files changed, 143 insertions(+), 42 deletions(-) diff --git a/include/pybind11/embed.h b/include/pybind11/embed.h index 5b77594b3..749c75beb 100644 --- a/include/pybind11/embed.h +++ b/include/pybind11/embed.h @@ -86,37 +86,22 @@ inline wchar_t *widen_chars(const char *safe_arg) { return widened_arg; } -PYBIND11_NAMESPACE_END(detail) - -/** \rst - Initialize the Python interpreter. No other pybind11 or CPython API functions can be - called before this is done; with the exception of `PYBIND11_EMBEDDED_MODULE`. The - optional `init_signal_handlers` parameter can be used to skip the registration of - signal handlers (see the `Python documentation`_ for details). Calling this function - again after the interpreter has already been initialized is a fatal error. - - If initializing the Python interpreter fails, then the program is terminated. (This - is controlled by the CPython runtime and is an exception to pybind11's normal behavior - of throwing exceptions on errors.) - - The remaining optional parameters, `argc`, `argv`, and `add_program_dir_to_path` are - used to populate ``sys.argv`` and ``sys.path``. - See the |PySys_SetArgvEx documentation|_ for details. - - .. _Python documentation: https://docs.python.org/3/c-api/init.html#c.Py_InitializeEx - .. |PySys_SetArgvEx documentation| replace:: ``PySys_SetArgvEx`` documentation - .. _PySys_SetArgvEx documentation: https://docs.python.org/3/c-api/init.html#c.PySys_SetArgvEx - \endrst */ -inline void initialize_interpreter(bool init_signal_handlers = true, - int argc = 0, - const char *const *argv = nullptr, - bool add_program_dir_to_path = true) { +inline void precheck_interpreter() { if (Py_IsInitialized() != 0) { pybind11_fail("The interpreter is already running"); } +} -#if PY_VERSION_HEX < 0x030B0000 +#if !defined(PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX) +# define PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX (0x03080000) +#endif +#if PY_VERSION_HEX < PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX +inline void initialize_interpreter_pre_pyconfig(bool init_signal_handlers, + int argc, + const char *const *argv, + bool add_program_dir_to_path) { + detail::precheck_interpreter(); Py_InitializeEx(init_signal_handlers ? 1 : 0); # if defined(WITH_THREAD) && PY_VERSION_HEX < 0x03070000 PyEval_InitThreads(); @@ -150,26 +135,30 @@ inline void initialize_interpreter(bool init_signal_handlers = true, auto *pysys_argv = widened_argv.get(); PySys_SetArgvEx(argc, pysys_argv, static_cast(add_program_dir_to_path)); -#else - PyConfig config; - PyConfig_InitIsolatedConfig(&config); - config.isolated = 0; - config.use_environment = 1; - config.install_signal_handlers = init_signal_handlers ? 1 : 0; +} +#endif - PyStatus status = PyConfig_SetBytesArgv(&config, argc, const_cast(argv)); - if (PyStatus_Exception(status)) { +PYBIND11_NAMESPACE_END(detail) + +#if PY_VERSION_HEX >= PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX +inline void initialize_interpreter(PyConfig *config, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { + detail::precheck_interpreter(); + PyStatus status = PyConfig_SetBytesArgv(config, argc, const_cast(argv)); + if (PyStatus_Exception(status) != 0) { // A failure here indicates a character-encoding failure or the python // interpreter out of memory. Give up. - PyConfig_Clear(&config); - throw std::runtime_error(PyStatus_IsError(status) ? status.err_msg - : "Failed to prepare CPython"); + PyConfig_Clear(config); + throw std::runtime_error(PyStatus_IsError(status) != 0 ? status.err_msg + : "Failed to prepare CPython"); } - status = Py_InitializeFromConfig(&config); - PyConfig_Clear(&config); - if (PyStatus_Exception(status)) { - throw std::runtime_error(PyStatus_IsError(status) ? status.err_msg - : "Failed to init CPython"); + status = Py_InitializeFromConfig(config); + if (PyStatus_Exception(status) != 0) { + PyConfig_Clear(config); + throw std::runtime_error(PyStatus_IsError(status) != 0 ? status.err_msg + : "Failed to init CPython"); } if (add_program_dir_to_path) { PyRun_SimpleString("import sys, os.path; " @@ -177,6 +166,43 @@ inline void initialize_interpreter(bool init_signal_handlers = true, "os.path.abspath(os.path.dirname(sys.argv[0])) " "if sys.argv and os.path.exists(sys.argv[0]) else '')"); } + PyConfig_Clear(config); +} +#endif + +/** \rst + Initialize the Python interpreter. No other pybind11 or CPython API functions can be + called before this is done; with the exception of `PYBIND11_EMBEDDED_MODULE`. The + optional `init_signal_handlers` parameter can be used to skip the registration of + signal handlers (see the `Python documentation`_ for details). Calling this function + again after the interpreter has already been initialized is a fatal error. + + If initializing the Python interpreter fails, then the program is terminated. (This + is controlled by the CPython runtime and is an exception to pybind11's normal behavior + of throwing exceptions on errors.) + + The remaining optional parameters, `argc`, `argv`, and `add_program_dir_to_path` are + used to populate ``sys.argv`` and ``sys.path``. + See the |PySys_SetArgvEx documentation|_ for details. + + .. _Python documentation: https://docs.python.org/3/c-api/init.html#c.Py_InitializeEx + .. |PySys_SetArgvEx documentation| replace:: ``PySys_SetArgvEx`` documentation + .. _PySys_SetArgvEx documentation: https://docs.python.org/3/c-api/init.html#c.PySys_SetArgvEx + \endrst */ +inline void initialize_interpreter(bool init_signal_handlers = true, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { +#if PY_VERSION_HEX < PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX + detail::initialize_interpreter_pre_pyconfig( + init_signal_handlers, argc, argv, add_program_dir_to_path); +#else + PyConfig config; + PyConfig_InitIsolatedConfig(&config); + config.isolated = 0; + config.use_environment = 1; + config.install_signal_handlers = init_signal_handlers ? 1 : 0; + initialize_interpreter(&config, argc, argv, add_program_dir_to_path); #endif } @@ -264,6 +290,15 @@ public: initialize_interpreter(init_signal_handlers, argc, argv, add_program_dir_to_path); } +#if PY_VERSION_HEX >= PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX + explicit scoped_interpreter(PyConfig *config, + int argc = 0, + const char *const *argv = nullptr, + bool add_program_dir_to_path = true) { + initialize_interpreter(config, argc, argv, add_program_dir_to_path); + } +#endif + scoped_interpreter(const scoped_interpreter &) = delete; scoped_interpreter(scoped_interpreter &&other) noexcept { other.is_valid = false; } scoped_interpreter &operator=(const scoped_interpreter &) = delete; diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 8cfd13fe5..5c306d363 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -166,6 +166,72 @@ TEST_CASE("There can be only one interpreter") { py::initialize_interpreter(); } +#if PY_VERSION_HEX >= PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX +TEST_CASE("Custom PyConfig") { + py::finalize_interpreter(); + PyConfig config; + PyConfig_InitPythonConfig(&config); + REQUIRE_NOTHROW(py::scoped_interpreter{&config}); + { + py::scoped_interpreter p{&config}; + REQUIRE(py::module_::import("widget_module").attr("add")(1, 41).cast() == 42); + } + py::initialize_interpreter(); +} + +TEST_CASE("Custom PyConfig with argv") { + py::finalize_interpreter(); + { + PyConfig config; + PyConfig_InitIsolatedConfig(&config); + char *argv[] = {strdup("a.out")}; + py::scoped_interpreter argv_scope{&config, 1, argv}; + std::free(argv[0]); + auto module = py::module::import("test_interpreter"); + auto py_widget = module.attr("DerivedWidget")("The question"); + const auto &cpp_widget = py_widget.cast(); + REQUIRE(cpp_widget.argv0() == "a.out"); + } + py::initialize_interpreter(); +} +#endif + +TEST_CASE("Add program dir to path") { + static auto get_sys_path_size = []() -> size_t { + auto sys_path = py::module::import("sys").attr("path"); + return py::len(sys_path); + }; + static auto validate_path_len = [](size_t default_len) { +#if PY_VERSION_HEX < 0x030A0000 + // It seems a value remains in sys.path + // left by the previous call of scoped_interpreter ctor. + REQUIRE(get_sys_path_size() > default_len); +#else + REQUIRE(get_sys_path_size() == default_len + 1); +#endif + }; + py::finalize_interpreter(); + + size_t sys_path_default_size = 0; + { + py::scoped_interpreter scoped_interp{true, 0, nullptr, false}; + sys_path_default_size = get_sys_path_size(); + } + { + py::scoped_interpreter scoped_interp{}; // expected to append some to sys.path + validate_path_len(sys_path_default_size); + } +#if PY_VERSION_HEX >= PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX + { + PyConfig config; + PyConfig_InitPythonConfig(&config); + py::scoped_interpreter scoped_interp{&config}; // expected to append some to sys.path + validate_path_len(sys_path_default_size); + } +#endif + py::initialize_interpreter(); +} + bool has_pybind11_internals_builtin() { auto builtins = py::handle(PyEval_GetBuiltins()); return builtins.contains(PYBIND11_INTERNALS_ID); From b14d58b6151b3cc7a9d9b2bf90d636940935db5c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Wed, 30 Nov 2022 23:30:36 -0800 Subject: [PATCH 32/53] chore(deps): bump pypa/gh-action-pypi-publish from 1.5.1 to 1.5.2 (#4370) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.5.1 to 1.5.2. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.5.1...v1.5.2) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pip.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pip.yml b/.github/workflows/pip.yml index f03a39701..b787187ee 100644 --- a/.github/workflows/pip.yml +++ b/.github/workflows/pip.yml @@ -98,13 +98,13 @@ jobs: - uses: actions/download-artifact@v3 - name: Publish standard package - uses: pypa/gh-action-pypi-publish@v1.5.1 + uses: pypa/gh-action-pypi-publish@v1.5.2 with: password: ${{ secrets.pypi_password }} packages_dir: standard/ - name: Publish global package - uses: pypa/gh-action-pypi-publish@v1.5.1 + uses: pypa/gh-action-pypi-publish@v1.5.2 with: password: ${{ secrets.pypi_password_global }} packages_dir: global/ From 358ba459d2f05321a15555c3b57c606fe2597ec7 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 1 Dec 2022 09:25:30 -0800 Subject: [PATCH 33/53] Fix test added with PR #4330 (#4372) --- tests/test_embed/test_interpreter.cpp | 47 ++++++++++++++------------- 1 file changed, 25 insertions(+), 22 deletions(-) diff --git a/tests/test_embed/test_interpreter.cpp b/tests/test_embed/test_interpreter.cpp index 5c306d363..10b20f371 100644 --- a/tests/test_embed/test_interpreter.cpp +++ b/tests/test_embed/test_interpreter.cpp @@ -14,6 +14,11 @@ PYBIND11_WARNING_DISABLE_MSVC(4996) namespace py = pybind11; using namespace py::literals; +size_t get_sys_path_size() { + auto sys_path = py::module::import("sys").attr("path"); + return py::len(sys_path); +} + class Widget { public: explicit Widget(std::string message) : message(std::move(message)) {} @@ -196,41 +201,39 @@ TEST_CASE("Custom PyConfig with argv") { } #endif -TEST_CASE("Add program dir to path") { - static auto get_sys_path_size = []() -> size_t { - auto sys_path = py::module::import("sys").attr("path"); - return py::len(sys_path); - }; - static auto validate_path_len = [](size_t default_len) { -#if PY_VERSION_HEX < 0x030A0000 - // It seems a value remains in sys.path - // left by the previous call of scoped_interpreter ctor. - REQUIRE(get_sys_path_size() > default_len); -#else - REQUIRE(get_sys_path_size() == default_len + 1); -#endif - }; +TEST_CASE("Add program dir to path pre-PyConfig") { py::finalize_interpreter(); - - size_t sys_path_default_size = 0; + size_t path_size_add_program_dir_to_path_false = 0; { py::scoped_interpreter scoped_interp{true, 0, nullptr, false}; - sys_path_default_size = get_sys_path_size(); + path_size_add_program_dir_to_path_false = get_sys_path_size(); } { - py::scoped_interpreter scoped_interp{}; // expected to append some to sys.path - validate_path_len(sys_path_default_size); + py::scoped_interpreter scoped_interp{}; + REQUIRE(get_sys_path_size() == path_size_add_program_dir_to_path_false + 1); } + py::initialize_interpreter(); +} + #if PY_VERSION_HEX >= PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX +TEST_CASE("Add program dir to path using PyConfig") { + py::finalize_interpreter(); + size_t path_size_add_program_dir_to_path_false = 0; { PyConfig config; PyConfig_InitPythonConfig(&config); - py::scoped_interpreter scoped_interp{&config}; // expected to append some to sys.path - validate_path_len(sys_path_default_size); + py::scoped_interpreter scoped_interp{&config, 0, nullptr, false}; + path_size_add_program_dir_to_path_false = get_sys_path_size(); + } + { + PyConfig config; + PyConfig_InitPythonConfig(&config); + py::scoped_interpreter scoped_interp{&config}; + REQUIRE(get_sys_path_size() == path_size_add_program_dir_to_path_false + 1); } -#endif py::initialize_interpreter(); } +#endif bool has_pybind11_internals_builtin() { auto builtins = py::handle(PyEval_GetBuiltins()); From e133c33d5c6b19acab55fb1d6c10331d918bd830 Mon Sep 17 00:00:00 2001 From: Aaron Gokaslan Date: Thu, 1 Dec 2022 15:15:47 -0500 Subject: [PATCH 34/53] chore: Convert direct multiprocessing.set_start_method("forkserver") call to a pytest fixture. (#4377) * chore: convert multiprocessing set_spawn to fixture in pytest * Switch to early return --- tests/conftest.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/tests/conftest.py b/tests/conftest.py index 96dffc81c..402fd4b25 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -17,7 +17,12 @@ import pytest # Early diagnostic for failed imports import pybind11_tests -if os.name != "nt": + +@pytest.fixture(scope="session", autouse=True) +def always_forkserver_on_unix(): + if os.name == "nt": + return + # Full background: https://github.com/pybind/pybind11/issues/4105#issuecomment-1301004592 # In a nutshell: fork() after starting threads == flakiness in the form of deadlocks. # It is actually a well-known pitfall, unfortunately without guard rails. @@ -27,6 +32,7 @@ if os.name != "nt": # running with defaults. multiprocessing.set_start_method("forkserver") + _long_marker = re.compile(r"([0-9])L") _hexadecimal = re.compile(r"0x[0-9a-fA-F]+") From 5b55f8fe8261dc50397e26b08435cd27a9e1f89c Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Sun, 4 Dec 2022 19:39:25 -0800 Subject: [PATCH 35/53] Replace `ubuntu-latest` with `ubuntu-22.04` (#4382) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * Replace 🐍 3.6 • ubuntu-latest • x64 -DPYBIND11_FINDPYTHON=ON -DCMAKE_CXX_FLAGS="-D_=1" with 3.9 * Revert "Replace 🐍 3.6 • ubuntu-latest • x64 -DPYBIND11_FINDPYTHON=ON -DCMAKE_CXX_FLAGS="-D_=1" with 3.9" This reverts commit 3ec984e1ed7265ac152e6760c68a0bad97d5a2bd. * Systematically change all active ubuntu-latest to ubuntu-20.04, except in upstream.yml * Revert "Systematically change all active ubuntu-latest to ubuntu-20.04, except in upstream.yml" This reverts commit cdfd99526a9d3c172691685ac17054bfe76590fc. * Change only some ubuntu-latest to ubuntu-20.04 --- .github/workflows/ci.yml | 8 ++++---- .github/workflows/configure.yml | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index bd99ddd33..a6f2624db 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,7 +25,7 @@ jobs: strategy: fail-fast: false matrix: - runs-on: [ubuntu-latest, windows-2022, macos-latest] + runs-on: [ubuntu-20.04, windows-2022, macos-latest] python: - '3.6' - '3.9' @@ -42,12 +42,12 @@ jobs: # We support an optional key: args, for cmake args include: # Just add a key - - runs-on: ubuntu-latest + - runs-on: ubuntu-20.04 python: '3.6' args: > -DPYBIND11_FINDPYTHON=ON -DCMAKE_CXX_FLAGS="-D_=1" - - runs-on: ubuntu-latest + - runs-on: ubuntu-20.04 python: 'pypy-3.8' args: > -DPYBIND11_FINDPYTHON=ON @@ -194,7 +194,7 @@ jobs: python-debug: false name: "🐍 ${{ matrix.python-version }}${{ matrix.python-debug && '-dbg' || '' }} (deadsnakes)${{ matrix.valgrind && ' • Valgrind' || '' }} • x64" - runs-on: ubuntu-latest + runs-on: ubuntu-20.04 steps: - uses: actions/checkout@v3 diff --git a/.github/workflows/configure.yml b/.github/workflows/configure.yml index 5ec0dd462..05e972c75 100644 --- a/.github/workflows/configure.yml +++ b/.github/workflows/configure.yml @@ -16,12 +16,12 @@ jobs: strategy: fail-fast: false matrix: - runs-on: [ubuntu-latest, macos-latest, windows-latest] + runs-on: [ubuntu-20.04, macos-latest, windows-latest] arch: [x64] cmake: ["3.23"] include: - - runs-on: ubuntu-latest + - runs-on: ubuntu-20.04 arch: x64 cmake: 3.4 From 6a1023e3e9ba32f7a9c4f59dfa8ea3f9e8dec341 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 5 Dec 2022 01:01:45 -0800 Subject: [PATCH 36/53] chore(deps): bump deadsnakes/action from 2.1.1 to 3.0.0 (#4383) Bumps [deadsnakes/action](https://github.com/deadsnakes/action) from 2.1.1 to 3.0.0. - [Release notes](https://github.com/deadsnakes/action/releases) - [Commits](https://github.com/deadsnakes/action/compare/v2.1.1...v3.0.0) --- updated-dependencies: - dependency-name: deadsnakes/action dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/ci.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a6f2624db..1c6a63f40 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -200,7 +200,7 @@ jobs: - uses: actions/checkout@v3 - name: Setup Python ${{ matrix.python-version }} (deadsnakes) - uses: deadsnakes/action@v2.1.1 + uses: deadsnakes/action@v3.0.0 with: python-version: ${{ matrix.python-version }} debug: ${{ matrix.python-debug }} From a6e75e4d8af51151587c13236fdc8570023fb57c Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Mon, 5 Dec 2022 01:03:05 -0800 Subject: [PATCH 37/53] chore(deps): bump pypa/gh-action-pypi-publish from 1.5.2 to 1.6.1 (#4384) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.5.2 to 1.6.1. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.5.2...v1.6.1) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pip.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pip.yml b/.github/workflows/pip.yml index b787187ee..1f4422d13 100644 --- a/.github/workflows/pip.yml +++ b/.github/workflows/pip.yml @@ -98,13 +98,13 @@ jobs: - uses: actions/download-artifact@v3 - name: Publish standard package - uses: pypa/gh-action-pypi-publish@v1.5.2 + uses: pypa/gh-action-pypi-publish@v1.6.1 with: password: ${{ secrets.pypi_password }} packages_dir: standard/ - name: Publish global package - uses: pypa/gh-action-pypi-publish@v1.5.2 + uses: pypa/gh-action-pypi-publish@v1.6.1 with: password: ${{ secrets.pypi_password_global }} packages_dir: global/ From a672de7cc88c053b77da0ae48f0f1a9b3afb86bb Mon Sep 17 00:00:00 2001 From: luzpaz Date: Tue, 6 Dec 2022 12:54:15 -0500 Subject: [PATCH 38/53] Fix source comment typo (#4388) --- include/pybind11/eigen/tensor.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/include/pybind11/eigen/tensor.h b/include/pybind11/eigen/tensor.h index a129f9929..568c641a2 100644 --- a/include/pybind11/eigen/tensor.h +++ b/include/pybind11/eigen/tensor.h @@ -279,7 +279,7 @@ struct type_caster::ValidType> { case return_value_policy::take_ownership: if (std::is_const::value) { // This cast is ugly, and might be UB in some cases, but we don't have an - // alterantive here as we must free that memory + // alternative here as we must free that memory Helper::free(const_cast(src)); pybind11_fail("Cannot take ownership of a const reference"); } From 4768a6f8f5e1abe106b4e3c9899d5866d88d77f6 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Tue, 6 Dec 2022 10:10:48 -0800 Subject: [PATCH 39/53] chore(deps): update pre-commit hooks (#4386) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * chore(deps): update pre-commit hooks updates: - [github.com/pre-commit/pre-commit-hooks: v4.3.0 → v4.4.0](https://github.com/pre-commit/pre-commit-hooks/compare/v4.3.0...v4.4.0) - [github.com/asottile/pyupgrade: v3.2.0 → v3.3.0](https://github.com/asottile/pyupgrade/compare/v3.2.0...v3.3.0) - [github.com/hadialqattan/pycln: v2.1.1 → v2.1.2](https://github.com/hadialqattan/pycln/compare/v2.1.1...v2.1.2) - [github.com/PyCQA/flake8: 5.0.4 → 6.0.0](https://github.com/PyCQA/flake8/compare/5.0.4...6.0.0) - [github.com/PyCQA/pylint: v2.15.5 → v2.15.8](https://github.com/PyCQA/pylint/compare/v2.15.5...v2.15.8) - [github.com/pre-commit/mirrors-mypy: v0.982 → v0.991](https://github.com/pre-commit/mirrors-mypy/compare/v0.982...v0.991) - [github.com/mgedmin/check-manifest: 0.48 → 0.49](https://github.com/mgedmin/check-manifest/compare/0.48...0.49) - [github.com/pre-commit/mirrors-clang-format: v14.0.6 → v15.0.4](https://github.com/pre-commit/mirrors-clang-format/compare/v14.0.6...v15.0.4) * style: pre-commit fixes Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> --- .pre-commit-config.yaml | 16 ++++++++-------- include/pybind11/cast.h | 3 ++- include/pybind11/detail/common.h | 6 +++--- include/pybind11/detail/init.h | 27 +++++++++++++++------------ include/pybind11/numpy.h | 2 +- include/pybind11/pybind11.h | 6 +++--- include/pybind11/pytypes.h | 3 ++- tests/test_virtual_functions.cpp | 3 ++- 8 files changed, 36 insertions(+), 30 deletions(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index fd079dd03..4e80fbad1 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -24,7 +24,7 @@ exclude: ^tools/JoinPaths.cmake$ repos: # Standard hooks - repo: https://github.com/pre-commit/pre-commit-hooks - rev: "v4.3.0" + rev: "v4.4.0" hooks: - id: check-added-large-files - id: check-case-conflict @@ -41,7 +41,7 @@ repos: # Upgrade old Python syntax - repo: https://github.com/asottile/pyupgrade - rev: "v3.2.0" + rev: "v3.3.0" hooks: - id: pyupgrade args: [--py36-plus] @@ -80,7 +80,7 @@ repos: # Autoremoves unused imports - repo: https://github.com/hadialqattan/pycln - rev: "v2.1.1" + rev: "v2.1.2" hooks: - id: pycln stages: [manual] @@ -108,7 +108,7 @@ repos: # Flake8 also supports pre-commit natively (same author) - repo: https://github.com/PyCQA/flake8 - rev: "5.0.4" + rev: "6.0.0" hooks: - id: flake8 exclude: ^(docs/.*|tools/.*)$ @@ -116,7 +116,7 @@ repos: # PyLint has native support - not always usable, but works for us - repo: https://github.com/PyCQA/pylint - rev: "v2.15.5" + rev: "v2.15.8" hooks: - id: pylint files: ^pybind11 @@ -132,7 +132,7 @@ repos: # Check static types with mypy - repo: https://github.com/pre-commit/mirrors-mypy - rev: "v0.982" + rev: "v0.991" hooks: - id: mypy args: [] @@ -141,7 +141,7 @@ repos: # Checks the manifest for missing files (native support) - repo: https://github.com/mgedmin/check-manifest - rev: "0.48" + rev: "0.49" hooks: - id: check-manifest # This is a slow hook, so only run this if --hook-stage manual is passed @@ -175,7 +175,7 @@ repos: # Clang format the codebase automatically - repo: https://github.com/pre-commit/mirrors-clang-format - rev: "v14.0.6" + rev: "v15.0.4" hooks: - id: clang-format types_or: [c++, c, cuda] diff --git a/include/pybind11/cast.h b/include/pybind11/cast.h index 8f92df8ab..3a4046027 100644 --- a/include/pybind11/cast.h +++ b/include/pybind11/cast.h @@ -91,7 +91,8 @@ public: template >::value, \ - int> = 0> \ + int> \ + = 0> \ static ::pybind11::handle cast( \ T_ *src, ::pybind11::return_value_policy policy, ::pybind11::handle parent) { \ if (!src) \ diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index b58dc3afa..6e295fd2f 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -34,17 +34,17 @@ # define PYBIND11_WARNING_POP PYBIND11_PRAGMA(warning(pop)) #elif defined(__INTEL_COMPILER) # define PYBIND11_COMPILER_INTEL -# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_PRAGMA(...) _Pragma(#__VA_ARGS__) # define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(warning push) # define PYBIND11_WARNING_POP PYBIND11_PRAGMA(warning pop) #elif defined(__clang__) # define PYBIND11_COMPILER_CLANG -# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_PRAGMA(...) _Pragma(#__VA_ARGS__) # define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(clang diagnostic push) # define PYBIND11_WARNING_POP PYBIND11_PRAGMA(clang diagnostic push) #elif defined(__GNUC__) # define PYBIND11_COMPILER_GCC -# define PYBIND11_PRAGMA(...) _Pragma(# __VA_ARGS__) +# define PYBIND11_PRAGMA(...) _Pragma(#__VA_ARGS__) # define PYBIND11_WARNING_PUSH PYBIND11_PRAGMA(GCC diagnostic push) # define PYBIND11_WARNING_POP PYBIND11_PRAGMA(GCC diagnostic pop) #endif diff --git a/include/pybind11/detail/init.h b/include/pybind11/detail/init.h index 0938c9bde..9f71278c2 100644 --- a/include/pybind11/detail/init.h +++ b/include/pybind11/detail/init.h @@ -209,10 +209,11 @@ struct constructor { extra...); } - template , Args...>::value, - int> = 0> + template < + typename Class, + typename... Extra, + enable_if_t, Args...>::value, int> + = 0> static void execute(Class &cl, const Extra &...extra) { cl.def( "__init__", @@ -229,10 +230,11 @@ struct constructor { extra...); } - template , Args...>::value, - int> = 0> + template < + typename Class, + typename... Extra, + enable_if_t, Args...>::value, int> + = 0> static void execute(Class &cl, const Extra &...extra) { cl.def( "__init__", @@ -248,10 +250,11 @@ struct constructor { // Implementing class for py::init_alias<...>() template struct alias_constructor { - template , Args...>::value, - int> = 0> + template < + typename Class, + typename... Extra, + enable_if_t, Args...>::value, int> + = 0> static void execute(Class &cl, const Extra &...extra) { cl.def( "__init__", diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index ea64aa7e7..8f072af26 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -1471,7 +1471,7 @@ private: } // Extract name, offset and format descriptor for a struct field -# define PYBIND11_FIELD_DESCRIPTOR(T, Field) PYBIND11_FIELD_DESCRIPTOR_EX(T, Field, # Field) +# define PYBIND11_FIELD_DESCRIPTOR(T, Field) PYBIND11_FIELD_DESCRIPTOR_EX(T, Field, #Field) // The main idea of this macro is borrowed from https://github.com/swansontec/map-macro // (C) William Swanson, Paul Fultz diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 10b27254e..a7310dfd4 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1426,9 +1426,9 @@ template ::value, int> = 0> void call_operator_delete(T *p, size_t, size_t) { T::operator delete(p); } -template < - typename T, - enable_if_t::value && has_operator_delete_size::value, int> = 0> +template ::value && has_operator_delete_size::value, int> + = 0> void call_operator_delete(T *p, size_t s, size_t) { T::operator delete(p, s); } diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index f913565d3..328287931 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -232,7 +232,8 @@ public: detail::enable_if_t, detail::is_pyobj_ptr_or_nullptr_t>, std::is_convertible>::value, - int> = 0> + int> + = 0> // NOLINTNEXTLINE(google-explicit-constructor) handle(T &obj) : m_ptr(obj) {} diff --git a/tests/test_virtual_functions.cpp b/tests/test_virtual_functions.cpp index 323aa0d22..93b136ad3 100644 --- a/tests/test_virtual_functions.cpp +++ b/tests/test_virtual_functions.cpp @@ -173,7 +173,8 @@ struct AdderBase { using DataVisitor = std::function; virtual void - operator()(const Data &first, const Data &second, const DataVisitor &visitor) const = 0; + operator()(const Data &first, const Data &second, const DataVisitor &visitor) const + = 0; virtual ~AdderBase() = default; AdderBase() = default; AdderBase(const AdderBase &) = delete; From 65cc9d2a291c983c289b35afe970bdda5710d706 Mon Sep 17 00:00:00 2001 From: "dependabot[bot]" <49699333+dependabot[bot]@users.noreply.github.com> Date: Tue, 6 Dec 2022 23:36:11 -0500 Subject: [PATCH 40/53] chore(deps): bump pypa/gh-action-pypi-publish from 1.6.1 to 1.6.4 (#4389) Bumps [pypa/gh-action-pypi-publish](https://github.com/pypa/gh-action-pypi-publish) from 1.6.1 to 1.6.4. - [Release notes](https://github.com/pypa/gh-action-pypi-publish/releases) - [Commits](https://github.com/pypa/gh-action-pypi-publish/compare/v1.6.1...v1.6.4) --- updated-dependencies: - dependency-name: pypa/gh-action-pypi-publish dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] Signed-off-by: dependabot[bot] Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> --- .github/workflows/pip.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.github/workflows/pip.yml b/.github/workflows/pip.yml index 1f4422d13..7c6fc67a3 100644 --- a/.github/workflows/pip.yml +++ b/.github/workflows/pip.yml @@ -98,13 +98,13 @@ jobs: - uses: actions/download-artifact@v3 - name: Publish standard package - uses: pypa/gh-action-pypi-publish@v1.6.1 + uses: pypa/gh-action-pypi-publish@v1.6.4 with: password: ${{ secrets.pypi_password }} packages_dir: standard/ - name: Publish global package - uses: pypa/gh-action-pypi-publish@v1.6.1 + uses: pypa/gh-action-pypi-publish@v1.6.4 with: password: ${{ secrets.pypi_password_global }} packages_dir: global/ From 65374c8e62488c557dccf70c9276e3f9fe04b007 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 8 Dec 2022 22:06:51 -0800 Subject: [PATCH 41/53] `pybind11::handle` `inc_ref()` & `dec_ref()` `PyGILState_Check()` **excluding** `nullptr` (#4246) * pybind11/pytypes.h `inc_ref()`, `dec_ref()` `PyGILState_Check()` **excluding** `nullptr` Guarded by `PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF` * Disable `PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF` for PyPy under Windows. * Add reference to PR #4268 (PyPy Windows) --- include/pybind11/detail/common.h | 9 +++++++++ include/pybind11/pytypes.h | 10 ++++++++++ 2 files changed, 19 insertions(+) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 6e295fd2f..9acf10466 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -323,6 +323,15 @@ PYBIND11_WARNING_POP # define PYBIND11_HAS_U8STRING #endif +// See description of PR #4246: +#if !defined(NDEBUG) && !defined(PY_ASSERT_GIL_HELD_INCREF_DECREF) \ + && !(defined(PYPY_VERSION) \ + && defined(_MSC_VER)) /* PyPy Windows: pytest hangs indefinitely at the end of the \ + process (see PR #4268) */ \ + && !defined(PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF) +# define PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF +#endif + // #define PYBIND11_STR_LEGACY_PERMISSIVE // If DEFINED, pybind11::str can hold PyUnicodeObject or PyBytesObject // (probably surprising and never documented, but this was the diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 328287931..56e0423ac 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -249,6 +249,11 @@ public: const handle &inc_ref() const & { #ifdef PYBIND11_HANDLE_REF_DEBUG inc_ref_counter(1); +#endif +#if defined(PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF) + if (m_ptr != nullptr && !PyGILState_Check()) { + throw std::runtime_error("pybind11::handle::inc_ref() PyGILState_Check() failure."); + } #endif Py_XINCREF(m_ptr); return *this; @@ -260,6 +265,11 @@ public: this function automatically. Returns a reference to itself. \endrst */ const handle &dec_ref() const & { +#if defined(PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF) + if (m_ptr != nullptr && !PyGILState_Check()) { + throw std::runtime_error("pybind11::handle::dec_ref() PyGILState_Check() failure."); + } +#endif Py_XDECREF(m_ptr); return *this; } From 00126859a550d10fb9f5bc0d4fdf86988c2012f4 Mon Sep 17 00:00:00 2001 From: Frank Date: Fri, 9 Dec 2022 08:10:10 +0100 Subject: [PATCH 42/53] Add option for enable/disable enum members in docstring. (#2768) * Add option for enable/disable enum members in docstring * Add tests for disable enum members docstring option * Add docstring options to documentation * style: pre-commit fixes * Fix typos in documentation * Improve documentation wording * Apply suggestions by @Skylion007 Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf W. Grosse-Kunstleve --- docs/advanced/misc.rst | 9 ++++++ include/pybind11/options.h | 16 ++++++++++ include/pybind11/pybind11.h | 50 +++++++++++++++++------------- tests/test_docstring_options.cpp | 53 ++++++++++++++++++++++++++++++++ tests/test_docstring_options.py | 23 ++++++++++++++ 5 files changed, 129 insertions(+), 22 deletions(-) diff --git a/docs/advanced/misc.rst b/docs/advanced/misc.rst index 71960b803..35a6ebcd6 100644 --- a/docs/advanced/misc.rst +++ b/docs/advanced/misc.rst @@ -324,6 +324,15 @@ The class ``options`` allows you to selectively suppress auto-generated signatur m.def("add", [](int a, int b) { return a + b; }, "A function which adds two numbers"); } +pybind11 also appends all members of an enum to the resulting enum docstring. +This default behavior can be disabled by using the ``disable_enum_members_docstring()`` +function of the ``options`` class. + +With ``disable_user_defined_docstrings()`` all user defined docstrings of +``module_::def()``, ``class_::def()`` and ``enum_()`` are disabled, but the +function signatures and enum members are included in the docstring, unless they +are disabled separately. + Note that changes to the settings affect only function bindings created during the lifetime of the ``options`` instance. When it goes out of scope at the end of the module's init function, the default settings are restored to prevent unwanted side effects. diff --git a/include/pybind11/options.h b/include/pybind11/options.h index 1e493bdcc..1b2122522 100644 --- a/include/pybind11/options.h +++ b/include/pybind11/options.h @@ -47,6 +47,16 @@ public: return *this; } + options &disable_enum_members_docstring() & { + global_state().show_enum_members_docstring = false; + return *this; + } + + options &enable_enum_members_docstring() & { + global_state().show_enum_members_docstring = true; + return *this; + } + // Getter methods (return the global state): static bool show_user_defined_docstrings() { @@ -55,6 +65,10 @@ public: static bool show_function_signatures() { return global_state().show_function_signatures; } + static bool show_enum_members_docstring() { + return global_state().show_enum_members_docstring; + } + // This type is not meant to be allocated on the heap. void *operator new(size_t) = delete; @@ -63,6 +77,8 @@ private: bool show_user_defined_docstrings = true; //< Include user-supplied texts in docstrings. bool show_function_signatures = true; //< Include auto-generated function signatures // in docstrings. + bool show_enum_members_docstring = true; //< Include auto-generated member list in enum + // docstrings. }; static state &global_state() { diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index a7310dfd4..6205effd6 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1972,29 +1972,35 @@ struct enum_base { name("name"), is_method(m_base)); - m_base.attr("__doc__") = static_property( - cpp_function( - [](handle arg) -> std::string { - std::string docstring; - dict entries = arg.attr("__entries"); - if (((PyTypeObject *) arg.ptr())->tp_doc) { - docstring += std::string(((PyTypeObject *) arg.ptr())->tp_doc) + "\n\n"; - } - docstring += "Members:"; - for (auto kv : entries) { - auto key = std::string(pybind11::str(kv.first)); - auto comment = kv.second[int_(1)]; - docstring += "\n\n " + key; - if (!comment.is_none()) { - docstring += " : " + (std::string) pybind11::str(comment); + if (options::show_enum_members_docstring()) { + m_base.attr("__doc__") = static_property( + cpp_function( + [](handle arg) -> std::string { + std::string docstring; + dict entries = arg.attr("__entries"); + if (((PyTypeObject *) arg.ptr())->tp_doc) { + docstring += std::string( + reinterpret_cast(arg.ptr())->tp_doc); + docstring += "\n\n"; } - } - return docstring; - }, - name("__doc__")), - none(), - none(), - ""); + docstring += "Members:"; + for (auto kv : entries) { + auto key = std::string(pybind11::str(kv.first)); + auto comment = kv.second[int_(1)]; + docstring += "\n\n "; + docstring += key; + if (!comment.is_none()) { + docstring += " : "; + docstring += pybind11::str(comment).cast(); + } + } + return docstring; + }, + name("__doc__")), + none(), + none(), + ""); + } m_base.attr("__members__") = static_property(cpp_function( [](handle arg) -> dict { diff --git a/tests/test_docstring_options.cpp b/tests/test_docstring_options.cpp index 4d44f4e20..dda1cf6e4 100644 --- a/tests/test_docstring_options.cpp +++ b/tests/test_docstring_options.cpp @@ -85,4 +85,57 @@ TEST_SUBMODULE(docstring_options, m) { &DocstringTestFoo::setValue, "This is a property docstring"); } + + { + enum class DocstringTestEnum1 { Member1, Member2 }; + + py::enum_(m, "DocstringTestEnum1", "Enum docstring") + .value("Member1", DocstringTestEnum1::Member1) + .value("Member2", DocstringTestEnum1::Member2); + } + + { + py::options options; + options.enable_enum_members_docstring(); + + enum class DocstringTestEnum2 { Member1, Member2 }; + + py::enum_(m, "DocstringTestEnum2", "Enum docstring") + .value("Member1", DocstringTestEnum2::Member1) + .value("Member2", DocstringTestEnum2::Member2); + } + + { + py::options options; + options.disable_enum_members_docstring(); + + enum class DocstringTestEnum3 { Member1, Member2 }; + + py::enum_(m, "DocstringTestEnum3", "Enum docstring") + .value("Member1", DocstringTestEnum3::Member1) + .value("Member2", DocstringTestEnum3::Member2); + } + + { + py::options options; + options.disable_user_defined_docstrings(); + + enum class DocstringTestEnum4 { Member1, Member2 }; + + py::enum_(m, "DocstringTestEnum4", "Enum docstring") + .value("Member1", DocstringTestEnum4::Member1) + .value("Member2", DocstringTestEnum4::Member2); + } + + { + py::options options; + options.disable_user_defined_docstrings(); + options.disable_enum_members_docstring(); + + enum class DocstringTestEnum5 { Member1, Member2 }; + + py::enum_(m, "DocstringTestEnum5", "Enum docstring") + .value("Member1", DocstringTestEnum5::Member1) + .value("Member2", DocstringTestEnum5::Member2); + } } diff --git a/tests/test_docstring_options.py b/tests/test_docstring_options.py index fcd16b89f..e6f5a9d98 100644 --- a/tests/test_docstring_options.py +++ b/tests/test_docstring_options.py @@ -39,3 +39,26 @@ def test_docstring_options(): # Suppression of user-defined docstrings for non-function objects assert not m.DocstringTestFoo.__doc__ assert not m.DocstringTestFoo.value_prop.__doc__ + + # Check existig behaviour of enum docstings + assert ( + m.DocstringTestEnum1.__doc__ + == "Enum docstring\n\nMembers:\n\n Member1\n\n Member2" + ) + + # options.enable_enum_members_docstring() + assert ( + m.DocstringTestEnum2.__doc__ + == "Enum docstring\n\nMembers:\n\n Member1\n\n Member2" + ) + + # options.disable_enum_members_docstring() + assert m.DocstringTestEnum3.__doc__ == "Enum docstring" + + # options.disable_user_defined_docstrings() + assert m.DocstringTestEnum4.__doc__ == "Members:\n\n Member1\n\n Member2" + + # options.disable_user_defined_docstrings() + # options.disable_enum_members_docstring() + # When all options are disabled, no docstring (instead of an empty one) should be generated + assert m.DocstringTestEnum5.__doc__ is None From 9db988013ceb54ab15fb775b229ac9180fd08fbe Mon Sep 17 00:00:00 2001 From: aimir <48388610+aimir@users.noreply.github.com> Date: Fri, 9 Dec 2022 09:15:11 +0200 Subject: [PATCH 43/53] Correct class names for KeysView, ValuesView and ItemsView in bind_map (#4353) * Create templated abstract classes KeysView, ValuesView and ItemsView, and implement them on-the-fly when wrapping any specific map type * We don't want to wrap different ValuesView objects for double values and const double, for example, as both wrappers will be named ValuesView[float] * Fallback to C++ names if key or values types are not wrapped * Added a test for .keys(), .values() and .items() returning the same types for similarly-typed maps * Fixed wrong use of auto in a declarator list: the two descriptions might have different types * Fixes for clang-tidy issues: explicit single-argument constructor, using the 'override' keyword when overriding functions * Bugfix for old versions of clang++, which seem to have trouble with the struct being defined inside a module, which was also needlessly ugly anyway * Bugfix for clang++, which doesn't have some of the names in runtime uness they are specified to be static * A fix for clang-tidy performance-inefficient-string-concatenation issues - I personally think this looks uglier, but it's probably worth it for clang-tidy to be happy * Possible fix for clang++ linking issues - make the descriptions static constexpr to make sure they are known before linking * Correct names for previously-wrapped types as keys/values of maps * Bugfix - typo in type info names which caused things to segfault * Apply suggestions from code review Co-authored-by: Aaron Gokaslan * Use detail::remove_cvref_t instead of doing remove_cv and remove_reference separately * Avoid names with double underscore, as they are reserved * Improved testing for KeysView, ValuesView and ItemsView: check type names + stricter asserts * Moved description logic to helper function in type_caster_base.h * style: pre-commit fixes * Fix a clang-tidy issue: do not use 'else' after 'return' * Apply suggestion by @Skylion007, with additional trivial simplification. Co-authored-by: Amir Co-authored-by: aimir Co-authored-by: Aaron Gokaslan Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf W. Grosse-Kunstleve --- include/pybind11/detail/type_caster_base.h | 9 ++ include/pybind11/stl_bind.h | 154 ++++++++++++++------- tests/test_stl_binders.py | 26 ++++ 3 files changed, 142 insertions(+), 47 deletions(-) diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 21f69c289..0b710d7e4 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -1006,5 +1006,14 @@ protected: static Constructor make_move_constructor(...) { return nullptr; } }; +PYBIND11_NOINLINE std::string type_info_description(const std::type_info &ti) { + if (auto *type_data = get_type_info(ti)) { + handle th((PyObject *) type_data->type); + return th.attr("__module__").cast() + '.' + + th.attr("__qualname__").cast(); + } + return clean_type_id(ti.name()); +} + PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/include/pybind11/stl_bind.h b/include/pybind11/stl_bind.h index 22a29b476..0c634597e 100644 --- a/include/pybind11/stl_bind.h +++ b/include/pybind11/stl_bind.h @@ -10,10 +10,13 @@ #pragma once #include "detail/common.h" +#include "detail/type_caster_base.h" +#include "cast.h" #include "operators.h" #include #include +#include PYBIND11_NAMESPACE_BEGIN(PYBIND11_NAMESPACE) PYBIND11_NAMESPACE_BEGIN(detail) @@ -636,18 +639,52 @@ auto map_if_insertion_operator(Class_ &cl, std::string const &name) "Return the canonical string representation of this map."); } -template +template struct keys_view { - Map ↦ + virtual size_t len() = 0; + virtual iterator iter() = 0; + virtual bool contains(const KeyType &k) = 0; + virtual bool contains(const object &k) = 0; + virtual ~keys_view() = default; }; -template +template struct values_view { + virtual size_t len() = 0; + virtual iterator iter() = 0; + virtual ~values_view() = default; +}; + +template +struct items_view { + virtual size_t len() = 0; + virtual iterator iter() = 0; + virtual ~items_view() = default; +}; + +template +struct KeysViewImpl : public KeysView { + explicit KeysViewImpl(Map &map) : map(map) {} + size_t len() override { return map.size(); } + iterator iter() override { return make_key_iterator(map.begin(), map.end()); } + bool contains(const typename Map::key_type &k) override { return map.find(k) != map.end(); } + bool contains(const object &) override { return false; } Map ↦ }; -template -struct items_view { +template +struct ValuesViewImpl : public ValuesView { + explicit ValuesViewImpl(Map &map) : map(map) {} + size_t len() override { return map.size(); } + iterator iter() override { return make_value_iterator(map.begin(), map.end()); } + Map ↦ +}; + +template +struct ItemsViewImpl : public ItemsView { + explicit ItemsViewImpl(Map &map) : map(map) {} + size_t len() override { return map.size(); } + iterator iter() override { return make_iterator(map.begin(), map.end()); } Map ↦ }; @@ -657,9 +694,11 @@ template , typename... class_ bind_map(handle scope, const std::string &name, Args &&...args) { using KeyType = typename Map::key_type; using MappedType = typename Map::mapped_type; - using KeysView = detail::keys_view; - using ValuesView = detail::values_view; - using ItemsView = detail::items_view; + using StrippedKeyType = detail::remove_cvref_t; + using StrippedMappedType = detail::remove_cvref_t; + using KeysView = detail::keys_view; + using ValuesView = detail::values_view; + using ItemsView = detail::items_view; using Class_ = class_; // If either type is a non-module-local bound type then make the map binding non-local as well; @@ -673,12 +712,57 @@ class_ bind_map(handle scope, const std::string &name, Args && } Class_ cl(scope, name.c_str(), pybind11::module_local(local), std::forward(args)...); - class_ keys_view( - scope, ("KeysView[" + name + "]").c_str(), pybind11::module_local(local)); - class_ values_view( - scope, ("ValuesView[" + name + "]").c_str(), pybind11::module_local(local)); - class_ items_view( - scope, ("ItemsView[" + name + "]").c_str(), pybind11::module_local(local)); + static constexpr auto key_type_descr = detail::make_caster::name; + static constexpr auto mapped_type_descr = detail::make_caster::name; + std::string key_type_name(key_type_descr.text), mapped_type_name(mapped_type_descr.text); + + // If key type isn't properly wrapped, fall back to C++ names + if (key_type_name == "%") { + key_type_name = detail::type_info_description(typeid(KeyType)); + } + // Similarly for value type: + if (mapped_type_name == "%") { + mapped_type_name = detail::type_info_description(typeid(MappedType)); + } + + // Wrap KeysView[KeyType] if it wasn't already wrapped + if (!detail::get_type_info(typeid(KeysView))) { + class_ keys_view( + scope, ("KeysView[" + key_type_name + "]").c_str(), pybind11::module_local(local)); + keys_view.def("__len__", &KeysView::len); + keys_view.def("__iter__", + &KeysView::iter, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + keys_view.def("__contains__", + static_cast(&KeysView::contains)); + // Fallback for when the object is not of the key type + keys_view.def("__contains__", + static_cast(&KeysView::contains)); + } + // Similarly for ValuesView: + if (!detail::get_type_info(typeid(ValuesView))) { + class_ values_view(scope, + ("ValuesView[" + mapped_type_name + "]").c_str(), + pybind11::module_local(local)); + values_view.def("__len__", &ValuesView::len); + values_view.def("__iter__", + &ValuesView::iter, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + } + // Similarly for ItemsView: + if (!detail::get_type_info(typeid(ItemsView))) { + class_ items_view( + scope, + ("ItemsView[" + key_type_name + ", ").append(mapped_type_name + "]").c_str(), + pybind11::module_local(local)); + items_view.def("__len__", &ItemsView::len); + items_view.def("__iter__", + &ItemsView::iter, + keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ + ); + } cl.def(init<>()); @@ -698,19 +782,25 @@ class_ bind_map(handle scope, const std::string &name, Args && cl.def( "keys", - [](Map &m) { return KeysView{m}; }, + [](Map &m) { + return std::unique_ptr(new detail::KeysViewImpl(m)); + }, keep_alive<0, 1>() /* Essential: keep map alive while view exists */ ); cl.def( "values", - [](Map &m) { return ValuesView{m}; }, + [](Map &m) { + return std::unique_ptr(new detail::ValuesViewImpl(m)); + }, keep_alive<0, 1>() /* Essential: keep map alive while view exists */ ); cl.def( "items", - [](Map &m) { return ItemsView{m}; }, + [](Map &m) { + return std::unique_ptr(new detail::ItemsViewImpl(m)); + }, keep_alive<0, 1>() /* Essential: keep map alive while view exists */ ); @@ -749,36 +839,6 @@ class_ bind_map(handle scope, const std::string &name, Args && cl.def("__len__", &Map::size); - keys_view.def("__len__", [](KeysView &view) { return view.map.size(); }); - keys_view.def( - "__iter__", - [](KeysView &view) { return make_key_iterator(view.map.begin(), view.map.end()); }, - keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ - ); - keys_view.def("__contains__", [](KeysView &view, const KeyType &k) -> bool { - auto it = view.map.find(k); - if (it == view.map.end()) { - return false; - } - return true; - }); - // Fallback for when the object is not of the key type - keys_view.def("__contains__", [](KeysView &, const object &) -> bool { return false; }); - - values_view.def("__len__", [](ValuesView &view) { return view.map.size(); }); - values_view.def( - "__iter__", - [](ValuesView &view) { return make_value_iterator(view.map.begin(), view.map.end()); }, - keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ - ); - - items_view.def("__len__", [](ItemsView &view) { return view.map.size(); }); - items_view.def( - "__iter__", - [](ItemsView &view) { return make_iterator(view.map.begin(), view.map.end()); }, - keep_alive<0, 1>() /* Essential: keep view alive while iterator exists */ - ); - return cl; } diff --git a/tests/test_stl_binders.py b/tests/test_stl_binders.py index d5e9ccced..9eb906f06 100644 --- a/tests/test_stl_binders.py +++ b/tests/test_stl_binders.py @@ -309,3 +309,29 @@ def test_map_delitem(): del um["ua"] assert sorted(list(um)) == ["ub"] assert sorted(list(um.items())) == [("ub", 2.6)] + + +def test_map_view_types(): + map_string_double = m.MapStringDouble() + unordered_map_string_double = m.UnorderedMapStringDouble() + map_string_double_const = m.MapStringDoubleConst() + unordered_map_string_double_const = m.UnorderedMapStringDoubleConst() + + assert map_string_double.keys().__class__.__name__ == "KeysView[str]" + assert map_string_double.values().__class__.__name__ == "ValuesView[float]" + assert map_string_double.items().__class__.__name__ == "ItemsView[str, float]" + + keys_type = type(map_string_double.keys()) + assert type(unordered_map_string_double.keys()) is keys_type + assert type(map_string_double_const.keys()) is keys_type + assert type(unordered_map_string_double_const.keys()) is keys_type + + values_type = type(map_string_double.values()) + assert type(unordered_map_string_double.values()) is values_type + assert type(map_string_double_const.values()) is values_type + assert type(unordered_map_string_double_const.values()) is values_type + + items_type = type(map_string_double.items()) + assert type(unordered_map_string_double.items()) is items_type + assert type(map_string_double_const.items()) is items_type + assert type(unordered_map_string_double_const.items()) is items_type From 663b86c26ca07c51f2981681ba79e2595e17fbb9 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Fri, 9 Dec 2022 10:53:03 -0800 Subject: [PATCH 44/53] Add flake8 `B905` to `extend-ignore` in setup.cfg (#4391) * Add flake8 `--ignore=B905,N818,W503` * Add B905 to `extend-ignore` in setup.cfg (thanks @Skylion007), leave .pre-commit-config.yaml as-is on master. --- setup.cfg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup.cfg b/setup.cfg index 9af50ea48..16479ab00 100644 --- a/setup.cfg +++ b/setup.cfg @@ -46,5 +46,5 @@ zip_safe = False max-line-length = 120 show_source = True exclude = .git, __pycache__, build, dist, docs, tools, venv -extend-ignore = E203, E722, B903, B950 +extend-ignore = E203, E722, B903, B905, B950 extend-select = B9 From 9727dcdae5b723a01ec04bec5eb1a5a681cbb0fd Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Fri, 9 Dec 2022 23:18:36 -0500 Subject: [PATCH 45/53] chore: future safe bugbear opinionated warnings (#4393) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- setup.cfg | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/setup.cfg b/setup.cfg index 16479ab00..8b3361981 100644 --- a/setup.cfg +++ b/setup.cfg @@ -46,5 +46,5 @@ zip_safe = False max-line-length = 120 show_source = True exclude = .git, __pycache__, build, dist, docs, tools, venv -extend-ignore = E203, E722, B903, B905, B950 -extend-select = B9 +extend-ignore = E203, E722 +extend-select = B902, B904 From ff42f5254a2f1315d3b56e2692590f1ce10b8193 Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Tue, 13 Dec 2022 11:04:05 -0800 Subject: [PATCH 46/53] Systematically add `-DCMAKE_VERBOSE_MAKEFILE=ON` to obtain full command lines related to `-Wodr` (#4398) --- .github/workflows/ci.yml | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 1c6a63f40..cd0f33d27 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -106,6 +106,7 @@ jobs: - name: Configure C++11 ${{ matrix.args }} run: > cmake -S . -B . + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DPYBIND11_SIMPLE_GIL_MANAGEMENT=ON -DDOWNLOAD_CATCH=ON @@ -135,6 +136,7 @@ jobs: - name: Configure C++17 run: > cmake -S . -B build2 + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF -DDOWNLOAD_CATCH=ON @@ -157,6 +159,7 @@ jobs: - name: Configure (unstable ABI) run: > cmake -S . -B build3 + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON @@ -243,6 +246,7 @@ jobs: SETUPTOOLS_USE_DISTUTILS: stdlib run: > cmake -S . -B build + -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -307,6 +311,7 @@ jobs: shell: bash run: > cmake -S . -B build + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=${{ matrix.std }} @@ -339,7 +344,7 @@ jobs: run: apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y cmake git python3-dev python3-pytest python3-numpy - name: Configure - run: cmake -S . -B build -DPYBIND11_CUDA_TESTS=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON + run: cmake -S . -B build -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_CUDA_TESTS=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON - name: Build run: cmake --build build -j2 --verbose @@ -377,7 +382,7 @@ jobs: # run: | # source /etc/profile.d/modules.sh # module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/20.11 -# cmake -S . -B build -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=14 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") +# cmake -S . -B build -DCMAKE_VERBOSE_MAKEFILE=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=14 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") # # - name: Build # run: cmake --build build -j 2 --verbose @@ -475,6 +480,7 @@ jobs: shell: bash run: > cmake -S . -B build + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=${{ matrix.std }} @@ -529,6 +535,7 @@ jobs: run: | set +e; source /opt/intel/oneapi/setvars.sh; set -e cmake -S . -B build-11 \ + -DCMAKE_VERBOSE_MAKEFILE=ON \ -DPYBIND11_WERROR=ON \ -DDOWNLOAD_CATCH=ON \ -DDOWNLOAD_EIGEN=OFF \ @@ -561,6 +568,7 @@ jobs: run: | set +e; source /opt/intel/oneapi/setvars.sh; set -e cmake -S . -B build-17 \ + -DCMAKE_VERBOSE_MAKEFILE=ON \ -DPYBIND11_WERROR=ON \ -DDOWNLOAD_CATCH=ON \ -DDOWNLOAD_EIGEN=OFF \ @@ -627,6 +635,7 @@ jobs: shell: bash run: > cmake -S . -B build + -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=MinSizeRel -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -774,6 +783,7 @@ jobs: run: > cmake -S . -B build -G "Visual Studio 16 2019" -A Win32 + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON @@ -827,6 +837,7 @@ jobs: run: > cmake -S . -B build -G "Visual Studio 16 2019" -A Win32 + -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -867,6 +878,7 @@ jobs: - name: Configure C++20 run: > cmake -S . -B build + -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON From 89c3561d9a217f494900e82cd35723402c7d3e7c Mon Sep 17 00:00:00 2001 From: Martin Blanchard <49144287+mablanchard@users.noreply.github.com> Date: Wed, 14 Dec 2022 20:03:37 +0100 Subject: [PATCH 47/53] Fix multi-context new Python linking mode (#4401) Allow CMake find_package() from multiple directories. https://github.com/pybind/pybind11/issues/4400 Co-authored-by: Martin Blanchard --- tools/pybind11NewTools.cmake | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/pybind11NewTools.cmake b/tools/pybind11NewTools.cmake index 5a6a0cb8b..7d7424a79 100644 --- a/tools/pybind11NewTools.cmake +++ b/tools/pybind11NewTools.cmake @@ -9,7 +9,7 @@ if(CMAKE_VERSION VERSION_LESS 3.12) message(FATAL_ERROR "You cannot use the new FindPython module with CMake < 3.12") endif() -include_guard(GLOBAL) +include_guard(DIRECTORY) get_property( is_config From ece1206b8ae4d1fc350d6f8129f7a098d223519d Mon Sep 17 00:00:00 2001 From: "Ralf W. Grosse-Kunstleve" Date: Thu, 15 Dec 2022 13:19:13 -0800 Subject: [PATCH 48/53] ci: set `env: VERBOSE: 1` (#4405) * Revert "Systematically add `-DCMAKE_VERBOSE_MAKEFILE=ON` to obtain full command lines related to `-Wodr` (#4398)" This reverts commit ff42f5254a2f1315d3b56e2692590f1ce10b8193. * Set `env: VERBOSE: 1` as suggested by @henryiii * Set `env: VERBOSE: 1` also in all other .yml files using cmake --- .github/workflows/ci.yml | 26 +++++++------------------- .github/workflows/configure.yml | 4 ++++ .github/workflows/format.yml | 2 ++ .github/workflows/upstream.yml | 2 ++ 4 files changed, 15 insertions(+), 19 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index cd0f33d27..af5a29d62 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -17,6 +17,8 @@ env: PIP_ONLY_BINARY: numpy FORCE_COLOR: 3 PYTEST_TIMEOUT: 300 + # For cmake: + VERBOSE: 1 jobs: # This is the "main" test suite, which tests a large number of different @@ -106,7 +108,6 @@ jobs: - name: Configure C++11 ${{ matrix.args }} run: > cmake -S . -B . - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DPYBIND11_SIMPLE_GIL_MANAGEMENT=ON -DDOWNLOAD_CATCH=ON @@ -136,7 +137,6 @@ jobs: - name: Configure C++17 run: > cmake -S . -B build2 - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF -DDOWNLOAD_CATCH=ON @@ -159,7 +159,6 @@ jobs: - name: Configure (unstable ABI) run: > cmake -S . -B build3 - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON @@ -246,7 +245,6 @@ jobs: SETUPTOOLS_USE_DISTUTILS: stdlib run: > cmake -S . -B build - -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -311,7 +309,6 @@ jobs: shell: bash run: > cmake -S . -B build - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=${{ matrix.std }} @@ -344,7 +341,7 @@ jobs: run: apt-get update && DEBIAN_FRONTEND="noninteractive" apt-get install -y cmake git python3-dev python3-pytest python3-numpy - name: Configure - run: cmake -S . -B build -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_CUDA_TESTS=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON + run: cmake -S . -B build -DPYBIND11_CUDA_TESTS=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON - name: Build run: cmake --build build -j2 --verbose @@ -382,7 +379,7 @@ jobs: # run: | # source /etc/profile.d/modules.sh # module load /opt/nvidia/hpc_sdk/modulefiles/nvhpc/20.11 -# cmake -S . -B build -DCMAKE_VERBOSE_MAKEFILE=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=14 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") +# cmake -S . -B build -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=14 -DPYTHON_EXECUTABLE=$(python3 -c "import sys; print(sys.executable)") # # - name: Build # run: cmake --build build -j 2 --verbose @@ -480,7 +477,6 @@ jobs: shell: bash run: > cmake -S . -B build - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DCMAKE_CXX_STANDARD=${{ matrix.std }} @@ -535,7 +531,6 @@ jobs: run: | set +e; source /opt/intel/oneapi/setvars.sh; set -e cmake -S . -B build-11 \ - -DCMAKE_VERBOSE_MAKEFILE=ON \ -DPYBIND11_WERROR=ON \ -DDOWNLOAD_CATCH=ON \ -DDOWNLOAD_EIGEN=OFF \ @@ -568,7 +563,6 @@ jobs: run: | set +e; source /opt/intel/oneapi/setvars.sh; set -e cmake -S . -B build-17 \ - -DCMAKE_VERBOSE_MAKEFILE=ON \ -DPYBIND11_WERROR=ON \ -DDOWNLOAD_CATCH=ON \ -DDOWNLOAD_EIGEN=OFF \ @@ -635,7 +629,6 @@ jobs: shell: bash run: > cmake -S . -B build - -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=MinSizeRel -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -783,7 +776,6 @@ jobs: run: > cmake -S . -B build -G "Visual Studio 16 2019" -A Win32 - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON @@ -837,7 +829,6 @@ jobs: run: > cmake -S . -B build -G "Visual Studio 16 2019" -A Win32 - -DCMAKE_VERBOSE_MAKEFILE=ON -DCMAKE_BUILD_TYPE=Debug -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON @@ -878,7 +869,6 @@ jobs: - name: Configure C++20 run: > cmake -S . -B build - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -DDOWNLOAD_EIGEN=ON @@ -930,7 +920,7 @@ jobs: - name: Configure C++11 # LTO leads to many undefined reference like # `pybind11::detail::function_call::function_call(pybind11::detail::function_call&&) - run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=11 -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=11 -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build - name: Build C++11 run: cmake --build build -j 2 @@ -948,7 +938,7 @@ jobs: run: git clean -fdx - name: Configure C++14 - run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=14 -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build2 + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=14 -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build2 - name: Build C++14 run: cmake --build build2 -j 2 @@ -966,7 +956,7 @@ jobs: run: git clean -fdx - name: Configure C++17 - run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=17 -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build3 + run: cmake -G "MinGW Makefiles" -DCMAKE_CXX_STANDARD=17 -DPYBIND11_WERROR=ON -DDOWNLOAD_CATCH=ON -S . -B build3 - name: Build C++17 run: cmake --build build3 -j 2 @@ -1027,7 +1017,6 @@ jobs: - name: Configure Clang run: > cmake -G Ninja -S . -B . - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=OFF -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF -DDOWNLOAD_CATCH=ON @@ -1093,7 +1082,6 @@ jobs: - name: CMake Configure run: > cmake -S . -B . - -DCMAKE_VERBOSE_MAKEFILE=ON -DPYBIND11_WERROR=ON -DPYBIND11_SIMPLE_GIL_MANAGEMENT=OFF -DDOWNLOAD_CATCH=ON diff --git a/.github/workflows/configure.yml b/.github/workflows/configure.yml index 05e972c75..29b041168 100644 --- a/.github/workflows/configure.yml +++ b/.github/workflows/configure.yml @@ -9,6 +9,10 @@ on: - stable - v* +env: + # For cmake: + VERBOSE: 1 + jobs: # This tests various versions of CMake in various combinations, to make sure # the configure step passes. diff --git a/.github/workflows/format.yml b/.github/workflows/format.yml index 31d893c47..b18474bc3 100644 --- a/.github/workflows/format.yml +++ b/.github/workflows/format.yml @@ -14,6 +14,8 @@ on: env: FORCE_COLOR: 3 + # For cmake: + VERBOSE: 1 jobs: pre-commit: diff --git a/.github/workflows/upstream.yml b/.github/workflows/upstream.yml index 366284acf..a15861ee4 100644 --- a/.github/workflows/upstream.yml +++ b/.github/workflows/upstream.yml @@ -11,6 +11,8 @@ concurrency: env: PIP_ONLY_BINARY: numpy + # For cmake: + VERBOSE: 1 jobs: standard: From 09db6445d8da6e918c2d2be3aa4e7b0ddd8077c7 Mon Sep 17 00:00:00 2001 From: Martin Blanchard <49144287+mablanchard@users.noreply.github.com> Date: Fri, 16 Dec 2022 06:10:46 +0100 Subject: [PATCH 49/53] IPO/LTO support for ICX (IntelLLVM) compiler (#4402) * IPO/LTO support for ICX (IntelLLVM) compiler https://github.com/pybind/pybind11/issues/4080 * style: pre-commit fixes * Add WARNING/HELP WANTED comment. Co-authored-by: Martin Blanchard Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Ralf W. Grosse-Kunstleve --- tools/pybind11Common.cmake | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/tools/pybind11Common.cmake b/tools/pybind11Common.cmake index e1fb601ac..0c985bc8e 100644 --- a/tools/pybind11Common.cmake +++ b/tools/pybind11Common.cmake @@ -311,6 +311,16 @@ function(_pybind11_generate_lto target prefer_thin_lto) HAS_FLTO "-flto${cxx_append}" "-flto${linker_append}" PYBIND11_LTO_CXX_FLAGS PYBIND11_LTO_LINKER_FLAGS) endif() + elseif(CMAKE_CXX_COMPILER_ID MATCHES "IntelLLVM") + # IntelLLVM equivalent to LTO is called IPO; also IntelLLVM is WIN32/UNIX + # WARNING/HELP WANTED: This block of code is currently not covered by pybind11 GitHub Actions! + if(WIN32) + _pybind11_return_if_cxx_and_linker_flags_work( + HAS_INTEL_IPO "-Qipo" "-Qipo" PYBIND11_LTO_CXX_FLAGS PYBIND11_LTO_LINKER_FLAGS) + else() + _pybind11_return_if_cxx_and_linker_flags_work( + HAS_INTEL_IPO "-ipo" "-ipo" PYBIND11_LTO_CXX_FLAGS PYBIND11_LTO_LINKER_FLAGS) + endif() elseif(CMAKE_CXX_COMPILER_ID MATCHES "Intel") # Intel equivalent to LTO is called IPO _pybind11_return_if_cxx_and_linker_flags_work(HAS_INTEL_IPO "-ipo" "-ipo" From a97c4d220ed8f10b6f2a67b0382887699a5ecf07 Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Mon, 19 Dec 2022 14:36:39 -0500 Subject: [PATCH 50/53] fix(cmake): support Windows ARM cross-compilation (#4406) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- tools/FindPythonLibsNew.cmake | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/tools/FindPythonLibsNew.cmake b/tools/FindPythonLibsNew.cmake index bbbd9f9ec..ce558d4ec 100644 --- a/tools/FindPythonLibsNew.cmake +++ b/tools/FindPythonLibsNew.cmake @@ -208,7 +208,9 @@ string(REGEX REPLACE "\\\\" "/" PYTHON_PREFIX "${PYTHON_PREFIX}") string(REGEX REPLACE "\\\\" "/" PYTHON_INCLUDE_DIR "${PYTHON_INCLUDE_DIR}") string(REGEX REPLACE "\\\\" "/" PYTHON_SITE_PACKAGES "${PYTHON_SITE_PACKAGES}") -if(CMAKE_HOST_WIN32) +if(DEFINED PYTHON_LIBRARY) + # Don't write to PYTHON_LIBRARY if it's already set +elseif(CMAKE_HOST_WIN32) set(PYTHON_LIBRARY "${PYTHON_PREFIX}/libs/python${PYTHON_LIBRARY_SUFFIX}.lib") # when run in a venv, PYTHON_PREFIX points to it. But the libraries remain in the @@ -274,7 +276,7 @@ if(NOT PYTHON_DEBUG_LIBRARY) endif() set(PYTHON_DEBUG_LIBRARIES "${PYTHON_DEBUG_LIBRARY}") -find_package_message(PYTHON "Found PythonLibs: ${PYTHON_LIBRARY}" +find_package_message(PYTHON "Found PythonLibs: ${PYTHON_LIBRARIES}" "${PYTHON_EXECUTABLE}${PYTHON_VERSION_STRING}") set(PYTHONLIBS_FOUND TRUE) From ee4b9f5d7b43383c1d4344dd11caf3dd5b066695 Mon Sep 17 00:00:00 2001 From: Ethan Steinberg Date: Tue, 20 Dec 2022 05:34:00 -0800 Subject: [PATCH 51/53] Fix ODR violations in our Eigen Tensor tests (#4412) * First * Fix centos 7 again :( * Fix minor nits --- tests/CMakeLists.txt | 15 +++++---------- ...array.cpp => eigen_tensor_avoid_stl_array.cpp} | 6 ++---- tests/test_eigen_tensor.cpp | 6 ++++-- tests/test_eigen_tensor.inl | 10 ++++------ tests/test_eigen_tensor.py | 8 ++++---- 5 files changed, 19 insertions(+), 26 deletions(-) rename tests/{test_eigen_tensor_avoid_stl_array.cpp => eigen_tensor_avoid_stl_array.cpp} (65%) diff --git a/tests/CMakeLists.txt b/tests/CMakeLists.txt index 491f215ce..9beb268ed 100644 --- a/tests/CMakeLists.txt +++ b/tests/CMakeLists.txt @@ -130,7 +130,6 @@ set(PYBIND11_TEST_FILES test_docstring_options test_eigen_matrix test_eigen_tensor - test_eigen_tensor_avoid_stl_array.cpp test_enum test_eval test_exceptions @@ -293,6 +292,11 @@ if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) set(EIGEN3_VERSION ${EIGEN3_VERSION_STRING}) endif() message(STATUS "Building tests with Eigen v${EIGEN3_VERSION}") + + if(NOT (CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_LESS 5.0)) + tests_extra_targets("test_eigen_tensor.py" "eigen_tensor_avoid_stl_array") + endif() + else() list(FIND PYBIND11_TEST_FILES test_eigen_matrix.cpp PYBIND11_TEST_FILES_EIGEN_I) if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) @@ -303,11 +307,6 @@ if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) endif() - list(FIND PYBIND11_TEST_FILES test_eigen_tensor_avoid_stl_array.cpp - PYBIND11_TEST_FILES_EIGEN_I) - if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) - list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) - endif() message( STATUS "Building tests WITHOUT Eigen, use -DDOWNLOAD_EIGEN=ON on CMake 3.11+ to download") endif() @@ -319,10 +318,6 @@ if(CMAKE_CXX_COMPILER_ID STREQUAL "GNU" AND CMAKE_CXX_COMPILER_VERSION VERSION_L if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) endif() - list(FIND PYBIND11_TEST_FILES test_eigen_tensor_avoid_stl_array.cpp PYBIND11_TEST_FILES_EIGEN_I) - if(PYBIND11_TEST_FILES_EIGEN_I GREATER -1) - list(REMOVE_AT PYBIND11_TEST_FILES ${PYBIND11_TEST_FILES_EIGEN_I}) - endif() endif() # Optional dependency for some tests (boost::variant is only supported with version >= 1.56) diff --git a/tests/test_eigen_tensor_avoid_stl_array.cpp b/tests/eigen_tensor_avoid_stl_array.cpp similarity index 65% rename from tests/test_eigen_tensor_avoid_stl_array.cpp rename to tests/eigen_tensor_avoid_stl_array.cpp index 58bedf62d..eacc9e9bd 100644 --- a/tests/test_eigen_tensor_avoid_stl_array.cpp +++ b/tests/eigen_tensor_avoid_stl_array.cpp @@ -5,12 +5,10 @@ BSD-style license that can be found in the LICENSE file. */ -constexpr const char *test_eigen_tensor_module_name = "eigen_tensor_avoid_stl_array"; - #ifndef EIGEN_AVOID_STL_ARRAY # define EIGEN_AVOID_STL_ARRAY #endif -#define PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE eigen_tensor_avoid_stl_array - #include "test_eigen_tensor.inl" + +PYBIND11_MODULE(eigen_tensor_avoid_stl_array, m) { eigen_tensor_test::test_module(m); } diff --git a/tests/test_eigen_tensor.cpp b/tests/test_eigen_tensor.cpp index 40b494005..503c69c7d 100644 --- a/tests/test_eigen_tensor.cpp +++ b/tests/test_eigen_tensor.cpp @@ -5,8 +5,6 @@ BSD-style license that can be found in the LICENSE file. */ -constexpr const char *test_eigen_tensor_module_name = "eigen_tensor"; - #define PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE eigen_tensor #ifdef EIGEN_AVOID_STL_ARRAY @@ -14,3 +12,7 @@ constexpr const char *test_eigen_tensor_module_name = "eigen_tensor"; #endif #include "test_eigen_tensor.inl" + +#include "pybind11_tests.h" + +test_initializer egien_tensor("eigen_tensor", eigen_tensor_test::test_module); diff --git a/tests/test_eigen_tensor.inl b/tests/test_eigen_tensor.inl index f46eb1803..d864ce737 100644 --- a/tests/test_eigen_tensor.inl +++ b/tests/test_eigen_tensor.inl @@ -7,9 +7,9 @@ #include -#include "pybind11_tests.h" +PYBIND11_NAMESPACE_BEGIN(eigen_tensor_test) -PYBIND11_NAMESPACE_BEGIN(PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE) +namespace py = pybind11; PYBIND11_WARNING_DISABLE_MSVC(4127) @@ -108,7 +108,7 @@ void init_tensor_module(pybind11::module &m) { return check_tensor(get_tensor()) && check_tensor(get_fixed_tensor()); }); - py::class_>(m, "CustomExample") + py::class_>(m, "CustomExample", py::module_local()) .def(py::init<>()) .def_readonly( "member", &CustomExample::member, py::return_value_policy::reference_internal) @@ -322,8 +322,6 @@ void init_tensor_module(pybind11::module &m) { py::return_value_policy::reference); } -void test_module(py::module_ &); -test_initializer name(test_eigen_tensor_module_name, test_module); void test_module(py::module_ &m) { auto f_style = m.def_submodule("f_style"); auto c_style = m.def_submodule("c_style"); @@ -332,4 +330,4 @@ void test_module(py::module_ &m) { init_tensor_module(c_style); } -PYBIND11_NAMESPACE_END(PYBIND11_TEST_EIGEN_TENSOR_NAMESPACE) +PYBIND11_NAMESPACE_END(eigen_tensor_test) diff --git a/tests/test_eigen_tensor.py b/tests/test_eigen_tensor.py index 653c9f288..dc8aa4643 100644 --- a/tests/test_eigen_tensor.py +++ b/tests/test_eigen_tensor.py @@ -6,17 +6,17 @@ np = pytest.importorskip("numpy") eigen_tensor = pytest.importorskip("pybind11_tests.eigen_tensor") submodules = [eigen_tensor.c_style, eigen_tensor.f_style] try: - from pybind11_tests import eigen_tensor_avoid_stl_array as avoid + import eigen_tensor_avoid_stl_array as avoid submodules += [avoid.c_style, avoid.f_style] except ImportError as e: # Ensure config, build, toolchain, etc. issues are not masked here: raise RuntimeError( - "import pybind11_tests.eigen_tensor_avoid_stl_array FAILED, while " + "import eigen_tensor_avoid_stl_array FAILED, while " "import pybind11_tests.eigen_tensor succeeded. " "Please ensure that " "test_eigen_tensor.cpp & " - "test_eigen_tensor_avoid_stl_array.cpp " + "eigen_tensor_avoid_stl_array.cpp " "are built together (or both are not built if Eigen is not available)." ) from e @@ -42,7 +42,7 @@ def cleanup(): def test_import_avoid_stl_array(): - pytest.importorskip("pybind11_tests.eigen_tensor_avoid_stl_array") + pytest.importorskip("eigen_tensor_avoid_stl_array") assert len(submodules) == 4 From 3fd1520de294035325469a7fcb3d60a2d818a85b Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 20 Dec 2022 14:47:41 -0500 Subject: [PATCH 52/53] docs: changelog for next version (#4413) * docs: changelog for next version Signed-off-by: Henry Schreiner * docs: address feedback Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- docs/changelog.rst | 55 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 54 insertions(+), 1 deletion(-) diff --git a/docs/changelog.rst b/docs/changelog.rst index 7d6d0c0f5..9ff7281bc 100644 --- a/docs/changelog.rst +++ b/docs/changelog.rst @@ -15,6 +15,60 @@ IN DEVELOPMENT Changes will be summarized here periodically. +Version 2.10.2 (Dec 20, 2022) +----------------------------- + +Changes: + +* ``scoped_interpreter`` constructor taking ``PyConfig``. + `#4330 `_ + +* ``pybind11/eigen/tensor.h`` adds converters to and from ``Eigen::Tensor`` and + ``Eigen::TensorMap``. + `#4201 `_ + +* ``PyGILState_Check()``'s were integrated to ``pybind11::handle`` + ``inc_ref()`` & ``dec_ref()``. The added GIL checks are guarded by + ``PYBIND11_ASSERT_GIL_HELD_INCREF_DECREF``, which is the default only if + ``NDEBUG`` is not defined. + `#4246 `_ + +* Add option for enable/disable enum members in docstring. + `#2768 `_ + +* Fixed typing of ``KeysView``, ``ValuesView`` and ``ItemsView`` in ``bind_map``. + `#4353 `_ + +Bug fixes: + +* Bug fix affecting only Python 3.6 under very specific, uncommon conditions: + move ``PyEval_InitThreads()`` call to the correct location. + `#4350 `_ + +* Fix segfault bug when passing foreign native functions to functional.h. + `#4254 `_ + +Build system improvements: + +* Support setting PYTHON_LIBRARIES manually for Windows ARM cross-compilation + (classic mode). + `#4406 `_ + +* Extend IPO/LTO detection for ICX (a.k.a IntelLLVM) compiler. + `#4402 `_ + +* Allow calling ``find_package(pybind11 CONFIG)`` multiple times from separate + directories in the same CMake project and properly link Python (new mode). + `#4401 `_ + +* ``multiprocessing_set_spawn`` in pytest fixture for added safety. + `#4377 `_ + +* Fixed a bug in two pybind11/tools cmake scripts causing "Unknown arguments specified" errors. + `#4327 `_ + + + Version 2.10.1 (Oct 31, 2022) ----------------------------- @@ -95,7 +149,6 @@ Bug fixes: finalization. `#4192 `_ - Performance and style: * Reserve space in set and STL map casters if possible. This will prevent From 0694ec6a15863bff2e0ea5efe07c78de39b9a33c Mon Sep 17 00:00:00 2001 From: Henry Schreiner Date: Tue, 20 Dec 2022 17:57:47 -0500 Subject: [PATCH 53/53] chore: preapre for 2.10.2 release (#4414) Signed-off-by: Henry Schreiner Signed-off-by: Henry Schreiner --- include/pybind11/detail/common.h | 6 +++--- pybind11/_version.py | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 9acf10466..068ae0137 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -10,12 +10,12 @@ #pragma once #define PYBIND11_VERSION_MAJOR 2 -#define PYBIND11_VERSION_MINOR 11 -#define PYBIND11_VERSION_PATCH 0.dev1 +#define PYBIND11_VERSION_MINOR 10 +#define PYBIND11_VERSION_PATCH 2 // Similar to Python's convention: https://docs.python.org/3/c-api/apiabiversion.html // Additional convention: 0xD = dev -#define PYBIND11_VERSION_HEX 0x020B00D1 +#define PYBIND11_VERSION_HEX 0x020A0200 // Define some generic pybind11 helper macros for warning management. // diff --git a/pybind11/_version.py b/pybind11/_version.py index 1cb51fc5c..4c814edfa 100644 --- a/pybind11/_version.py +++ b/pybind11/_version.py @@ -8,5 +8,5 @@ def _to_int(s: str) -> Union[int, str]: return s -__version__ = "2.11.0.dev1" +__version__ = "2.10.2" version_info = tuple(_to_int(s) for s in __version__.split("."))