The SMART_HOLDER_WIP comments are mostly from 2021. In the meantime, the
smart_holder code was extremely thoroughly tested in the Google codebase
(production code). Additionally, testing via PyCLIF-pybind11 provided
thousands of diverse use cases that needed to satisfy many million unit tests
(the success rate was about 99.999%).
* Replace `unique_ptr_cref_roundtrip()` with `pass_unique_ptr_cref()`, `rtrn_unique_ptr_cref()` to make the current behavior obvious.
* add in unique_ptr_storage, unique_ptr_storage_deleter
* Add shared_ptr_storage (with that disowning fails as expected).
* Add load_as_const_unique_ptr()
* Restore original struct_smart_holder.h
* factor out `smart_holder::extract_deleter()`
* Better error message.
* Misc cleanup/tidying.
* Use `re.match("ctor_arg(_MvCtor)*_MvCtor", ...)` for compatibility with MSVC, NVHPC, ICC
* Add small comments.
* Fix small, inconsequential oversight in test code.
* Apply suggestion by @iwanders under PR #5334
* Remove `std::move()` in `smart_holder::extract_deleter()`
* Add `static_assert()` following a suggestion by @iwanders under PR #5334
PREPARATION for:
PR #5332 — Fix handling of const unique_ptr<T, D> & (do not disown).
Splitting out so that the functional changes under PR #5332 will be more obvious.
The only functional change under this PR is that
```
assert(custom_deleter_ptr != nullptr);
```
is replaced with:
```
if (custom_deleter_ptr == nullptr) {
throw std::runtime_error(
std::string("smart_holder::extract_deleter() precondition failure (") + context
+ ").");
}
```
PREPARATION for:
PR #5332 — Fix handling of const unique_ptr<T, D> & (do not disown).
Splitting out so that the functional changes under PR #5332 will be more obvious.
* DOC: Clarify requirements for including pybind11
Inherited from requirements for including Python.h
Closes#4999
* style: pre-commit fixes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: allow -Wpedantic again
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: ignore pedantic warning for PYBIND11_DECLARE_HOLDER_TYPE
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: try just turning off pedantic for one file
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: only run pedantic in C++20 mode
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Update tests/local_bindings.h
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* feat: add `<ranges>` support for `py::tuple` and `py::list`
* fix: format the code
* fix: disable `ranges` in clang < 16
* refactor: move `<ranges>` test macro to `test_pytypes.h`
* refactor: seperate `ranges` test into 3 funcs
* style: compress the if statement
* style: pre-commit fixes
* style: better formatting
---------
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix: never use `..` in a header include
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix: one more parent include
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix: Escape paths with spaces in include list from --includes
* fix: --includes should not use shlex on Windows platforms
* Apply suggestions from code review
* fix: use custom impl
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Support trailing backslashes
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Markus Bauer <markus.bauer@cispa.saarland>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
* Configure only one job per OS kind to build and run tests with `-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT`
Remove ci_sh_def.yml
Piggy-backed: Add `smart_holder` to `branches` in emscripten.yaml
* Fix misunderstanding.
* Add `/GR /EHsc` for Windows.
* #error "Intentional Breakage" (to verify: PYBIND11_ACTUALLY_USING_SMART_HOLDER_AS_DEFAULT)
* Revert "#error "Intentional Breakage" (to verify: PYBIND11_ACTUALLY_USING_SMART_HOLDER_AS_DEFAULT)"
This reverts commit 4f6f5cb6e9.
* Add `test_pass_std_vector_int()`, `test_pass_std_set_int()` in test_stl
* Change `list_caster` to also accept generator objects (`PyGen_Check(src.ptr()`).
Note for completeness: This is a more conservative change than https://github.com/google/pywrapcc/pull/30042
* Drop in (currently unpublished) PyCLIF code, use in `list_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdSet()` in `set_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdMap()` in `map_caster`, add tests.
* Simplify `list_caster` `load()` implementation, push str/bytes check into `PyObjectTypeIsConvertibleToStdVector()`.
* clang-tidy cleanup with a few extra `(... != 0)` to be more consistent.
* Also use `PyObjectTypeIsConvertibleToStdVector()` in `array_caster`.
* Update comment pointing to clif/python/runtime.cc (code is unchanged).
* Comprehensive test coverage, enhanced set_caster load implementation.
* Resolve clang-tidy eror.
* Add a long C++ comment explaining what led to the `PyObjectTypeIsConvertibleTo*()` implementations.
* Minor function name change in test.
* strcmp -> std::strcmp (thanks @Skylion007 for catching this)
* Add `PyCallable_Check(items)` in `PyObjectTypeIsConvertibleToStdMap()`
* Resolve clang-tidy error
* Use `PyMapping_Items()` instead of `src.attr("items")()`, to be internally consistent with `PyMapping_Check()`
* Update link to PyCLIF sources.
* Fix typo (thanks @wangxf123456 for catching this)
* Add `test_pass_std_vector_int()`, `test_pass_std_set_int()` in test_stl
* Change `list_caster` to also accept generator objects (`PyGen_Check(src.ptr()`).
Note for completeness: This is a more conservative change than https://github.com/google/pywrapcc/pull/30042
* Drop in (currently unpublished) PyCLIF code, use in `list_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdSet()` in `set_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdMap()` in `map_caster`, add tests.
* Simplify `list_caster` `load()` implementation, push str/bytes check into `PyObjectTypeIsConvertibleToStdVector()`.
* clang-tidy cleanup with a few extra `(... != 0)` to be more consistent.
* Also use `PyObjectTypeIsConvertibleToStdVector()` in `array_caster`.
* Update comment pointing to clif/python/runtime.cc (code is unchanged).
* Comprehensive test coverage, enhanced set_caster load implementation.
* Resolve clang-tidy eror.
* Add a long C++ comment explaining what led to the `PyObjectTypeIsConvertibleTo*()` implementations.
* Minor function name change in test.
* strcmp -> std::strcmp (thanks @Skylion007 for catching this)
* Add `PyCallable_Check(items)` in `PyObjectTypeIsConvertibleToStdMap()`
* Resolve clang-tidy error
* Use `PyMapping_Items()` instead of `src.attr("items")()`, to be internally consistent with `PyMapping_Check()`
* Update link to PyCLIF sources.
* Fix typo (thanks @wangxf123456 for catching this)
* Fix typo discovered by new version of codespell.
* fix(cmake): add required emscripten flags
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Update emscripten.yaml
* fix(cmake): add required emscripten flags to headers target
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): incorrect detection of Emscripten
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): allow pybind11::headers to be modified
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): hide a warning when building the tests standalone
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): use explicit variable for is config
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): go back to ALIAS target
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: reduce overall diff
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: reduce overall diff
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: shorten code a bit
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Allow specializations based on callback function return values.
* clang-tidy auto fix
* Add a test case for function specialization.
* Add test for callback function that raises Python exception.
* Fix test failures.
* style: pre-commit fixes
* Add `#define PYBIND11_HAS_TYPE_CASTER_STD_FUNCTION_SPECIALIZATIONS`
---------
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Put bakein branch @ 18b72c0ffa on top of smart_holder branch:
Commands used:
```
git checkout bakein
git diff smart_holder > ~/zd
git checkout smart_holder
git checkout -b bakein_sh
patch -p 1 < ~/zd
git checkout smart_holder \
MANIFEST.in \
README.rst \
README_smart_holder.rst \
docs/advanced/smart_ptrs.rst \
ubench/holder_comparison.cpp \
ubench/holder_comparison.py \
ubench/holder_comparison_extract_sheet_data.py \
ubench/number_bucket.h \
ubench/python/number_bucket.clif
git add -A
```
* Add back README_smart_holder.rst in tests/extra_python_package/test_files.py
* Restore smart_holder_poc.h as-is on smart_holder branch (i.e. undo `PYBIND11_SMART_HOLDER_PADDING`, which was meant for stress-testing only).
* Insert `std::move()` as suggested by @laramiel
* `property_cpp_function_sh_*` named specializations, as suggested by @laramiel (https://github.com/pybind/pybind11/pull/5257#discussion_r1688346807)
* Call `property_cpp_function_classic` member functions, rather than inlining the implementations.
* Use `PYBIND11_HAVE_INTERNALS_WITH_SMART_HOLDER_SUPPORT` in holder_comparison.cpp (holder_comparison.py is NOT changed accordingly in this commit, i.e. can still only be run if the smart_holder functionality is available).
* Systematically rename `loaded_as` to `load_as` (`shared_ptr`, `unique_ptr`) as suggested by @laramiel
* Make change as suggested by @laramiel. This makes it much more obvious that the latest implementation of `smart_holder_from_unique_ptr()` accepts all existing `return_value_policy` enum values except `copy`.
* Resolve `BAKEIN_WIP: Rewrite comment.` for `property_cpp_function_*` specializations.
* Resolve `BAKEIN_WIP: Add comment to explain: This is meant for stress-testing only.`
* Resolve all remaining BAKEIN_WIP (in pybind11/cast.h).
Leave only two pairs of SMART_HOLDER_BAKEIN_FOLLOW_ON comments: refactoring of copyable_holder_caster, move_only_holder_caster. This is best left until after the smart_holder branch is merged into the master branch.
* Remove obsolete `using holder_type = smart_holder;` in `load_helper`
* Add SMART_HOLDER_BAKEIN_FOLLOW_ON comment for `internals::default_holder`
* README_smart_holder.rst update (line count reduced from 356 to 123).
Follow-on to https://github.com/pybind/pybind11/pull/5272 — clang-tidy upgrade (to version 18)
Fixes this error:
```
/__w/pybind11/pybind11/tests/pure_cpp/smart_holder_poc_test.cpp:167:12: error: the value returned by this function should not be disregarded; neglecting it may lead to errors [bugprone-unused-return-value,-warnings-as-errors]
167 | (void) new_owner.release(); // Manually verified: without this, clang++ -fsanitize=address
| ^~~~~~~~~~~~~~~~~~~
```
See also: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/unused-return-value.html
Specifically there:
> std::unique_ptr::release(). Not using the return value can lead to resource leaks if the same pointer isn’t stored anywhere else. Often, ignoring the release() return value indicates that the programmer confused the function with reset().
I.e. the only way to tell clang-tidy that the smart_holder_poc_test.cpp is correct is to add the `NOLINT`.
* `container: silkeh/clang:18-bookworm` in .github/workflows/format.yml
* clang-tidy auto-fix (trivial, in test only)
* Disable `performance-enum-size` (noisy, low value)
* Temporarily turn off 3 diagnostics (to be tackled one-by-one).
* Add explicit `switch` `default` to resolve clang-tidy `bugprone-switch-missing-default-case`
Debian clang version 18.1.8 (++20240718080534+3b5b5c1ec4a3-1~exp1~20240718200641.143)
Target: x86_64-pc-linux-gnu
tests/test_numpy_dtypes.cpp:212:5: warning: switching on non-enum value without default case may not cover all cases [bugprone-switch-missing-default-case]
* Add clang-17 and clang-18 testing.
* Add `NOLINTNEXTLINE(clang-analyzer-optin.core.EnumCastOutOfRange)` in test_tagbased_polymorphic.cpp
Debian clang version 18.1.8 (++20240718080534+3b5b5c1ec4a3-1~exp1~20240718200641.143)
Target: x86_64-pc-linux-gnu
tests/test_tagbased_polymorphic.cpp:77:40: warning: The value '150' provided to the cast expression is not in the valid range of values for 'Kind' [clang-analyzer-optin.core.EnumCastOutOfRange]
* Fix inconsistent pybind11/eigen/tensor.h behavior:
This existing comment in pybind11/eigen/tensor.h
```
// move, take_ownership don't make any sense for a ref/map:
```
is at odds with the `delete src;` three lines up.
In real-world client code `take_ownership` will not exist (unless the client code is untested and unused). I.e. the `delete` is essentially only useful to avoid leaks in the pybind11 unit tests.
While upgrading to clang-tidy 18, the warning below appeared. Apparently it is produced during LTO, and it appears difficult to suppress. Regardless, the best way to resolve this is to remove the `delete` and to simply make the test objects `static` in the unit test code.
________
Debian clang version 18.1.8 (++20240718080534+3b5b5c1ec4a3-1~exp1~20240718200641.143)
Target: x86_64-pc-linux-gnu
________
```
lto-wrapper: warning: using serial compilation of 3 LTRANS jobs
lto-wrapper: note: see the ‘-flto’ option documentation for more information
In function ‘cast_impl’,
inlined from ‘cast’ at /mounted_pybind11/include/pybind11/eigen/tensor.h:414:25,
inlined from ‘operator()’ at /mounted_pybind11/include/pybind11/eigen/../pybind11.h:296:40,
inlined from ‘_FUN’ at /mounted_pybind11/include/pybind11/eigen/../pybind11.h:267:21:
/mounted_pybind11/include/pybind11/eigen/tensor.h:475:17: warning: ‘operator delete’ called on unallocated object ‘<anonymous>’ [-Wfree-nonheap-object]
475 | delete src;
| ^
/mounted_pybind11/include/pybind11/eigen/../pybind11.h: In function ‘_FUN’:
/mounted_pybind11/include/pybind11/eigen/../pybind11.h:297:75: note: declared here
297 | std::move(args_converter).template call<Return, Guard>(cap->f),
| ^
```
* Disable `bugprone-chained-comparison`: this clang-tidy check is incompatible with the Catch2 `REQUIRE` macro (26 warnings like the one below).
________
Debian clang version 18.1.8 (++20240718080534+3b5b5c1ec4a3-1~exp1~20240718200641.143)
Target: x86_64-pc-linux-gnu
________
```
/mounted_pybind11/tests/test_embed/test_interpreter.cpp:127:9: warning: chained comparison 'v0 <= v1 == v2' may generate unintended results, use parentheses to specify order of evaluation or a logical operator to separate comparison expressions [bugprone-chained-comparison]
127 | REQUIRE(ret == 42);
| ^
/build/tests/catch/catch.hpp:17670:24: note: expanded from macro 'REQUIRE'
17670 | #define REQUIRE( ... ) INTERNAL_CATCH_TEST( "REQUIRE", Catch::ResultDisposition::Normal, __VA_ARGS__ )
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/build/tests/catch/catch.hpp:2710:47: note: expanded from macro 'INTERNAL_CATCH_TEST'
2710 | catchAssertionHandler.handleExpr( Catch::Decomposer() <= __VA_ARGS__ ); \
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/mounted_pybind11/tests/test_embed/test_interpreter.cpp:127:9: note: operand 'v0' is here
127 | REQUIRE(ret == 42);
| ^
/build/tests/catch/catch.hpp:17670:24: note: expanded from macro 'REQUIRE'
17670 | #define REQUIRE( ... ) INTERNAL_CATCH_TEST( "REQUIRE", Catch::ResultDisposition::Normal, __VA_ARGS__ )
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/build/tests/catch/catch.hpp:2710:47: note: expanded from macro 'INTERNAL_CATCH_TEST'
2710 | catchAssertionHandler.handleExpr( Catch::Decomposer() <= __VA_ARGS__ ); \
| ^~~~~~~~~~~~~~~~~~~
/mounted_pybind11/tests/test_embed/test_interpreter.cpp:127:17: note: operand 'v1' is here
127 | REQUIRE(ret == 42);
| ^
/build/tests/catch/catch.hpp:17670:90: note: expanded from macro 'REQUIRE'
17670 | #define REQUIRE( ... ) INTERNAL_CATCH_TEST( "REQUIRE", Catch::ResultDisposition::Normal, __VA_ARGS__ )
| ^~~~~~~~~~~
/build/tests/catch/catch.hpp:2710:70: note: expanded from macro 'INTERNAL_CATCH_TEST'
2710 | catchAssertionHandler.handleExpr( Catch::Decomposer() <= __VA_ARGS__ ); \
| ^~~~~~~~~~~
/mounted_pybind11/tests/test_embed/test_interpreter.cpp:127:24: note: operand 'v2' is here
127 | REQUIRE(ret == 42);
| ^
/build/tests/catch/catch.hpp:17670:90: note: expanded from macro 'REQUIRE'
17670 | #define REQUIRE( ... ) INTERNAL_CATCH_TEST( "REQUIRE", Catch::ResultDisposition::Normal, __VA_ARGS__ )
| ^~~~~~~~~~~
/build/tests/catch/catch.hpp:2710:70: note: expanded from macro 'INTERNAL_CATCH_TEST'
2710 | catchAssertionHandler.handleExpr( Catch::Decomposer() <= __VA_ARGS__ ); \
| ^~~~~~~~~~~
```
* Add 8 `// NOLINT(bugprone-empty-catch)`
* Resolve clang-tidy `bugprone-multi-level-implicit-pointer-conversion` warnings.
________
Debian clang version 18.1.8 (++20240718080534+3b5b5c1ec4a3-1~exp1~20240718200641.143)
Target: x86_64-pc-linux-gnu
________
```
pybind11/detail/internals.h:556:53: warning: multilevel pointer conversion from 'internals **' to 'const void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/detail/type_caster_base.h:431:20: warning: multilevel pointer conversion from 'void **' to 'void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/numpy.h:904:81: warning: multilevel pointer conversion from '_object *const *' to 'const void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/numpy.h:1989:39: warning: multilevel pointer conversion from 'typename vectorize_arg<const double *>::type *' (aka 'const double **') to 'void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/numpy.h:1989:39: warning: multilevel pointer conversion from 'typename vectorize_arg<const VectorizeTestClass *>::type *' (aka 'const VectorizeTestClass **') to 'void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/stl/filesystem.h:75:44: warning: multilevel pointer conversion from 'PyObject **' (aka '_object **') to 'void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
pybind11/stl/filesystem.h:83:42: warning: multilevel pointer conversion from 'PyObject **' (aka '_object **') to 'void *', please use explicit cast [bugprone-multi-level-implicit-pointer-conversion]
```
* Introduce `PYBIND11_REINTERPRET_CAST_VOID_PTR_IF_NOT_PYPY` to resolve PyPy build errors:
```
In file included from /Users/runner/work/pybind11/pybind11/tests/test_stl.cpp:18:
/Users/runner/work/pybind11/pybind11/include/pybind11/stl/filesystem.h:75:17: error: no matching function for call to 'PyPyUnicode_FSConverter'
if (PyUnicode_FSConverter(buf, reinterpret_cast<void *>(&native)) != 0) {
^~~~~~~~~~~~~~~~~~~~~
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:969:31: note: expanded from macro 'PyUnicode_FSConverter'
^~~~~~~~~~~~~~~~~~~~~~~
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:970:17: note: candidate function not viable: cannot convert argument of incomplete type 'void *' to 'struct _object **' for 2nd argument
PyAPI_FUNC(int) PyUnicode_FSConverter(struct _object *arg0, struct _object **arg1);
^
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:969:31: note: expanded from macro 'PyUnicode_FSConverter'
^
In file included from /Users/runner/work/pybind11/pybind11/tests/test_stl.cpp:18:
/Users/runner/work/pybind11/pybind11/include/pybind11/stl/filesystem.h:83:17: error: no matching function for call to 'PyPyUnicode_FSDecoder'
if (PyUnicode_FSDecoder(buf, reinterpret_cast<void *>(&native)) != 0) {
^~~~~~~~~~~~~~~~~~~
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:971:29: note: expanded from macro 'PyUnicode_FSDecoder'
^~~~~~~~~~~~~~~~~~~~~
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:972:17: note: candidate function not viable: cannot convert argument of incomplete type 'void *' to 'struct _object **' for 2nd argument
PyAPI_FUNC(int) PyUnicode_FSDecoder(struct _object *arg0, struct _object **arg1);
^
/Users/runner/hostedtoolcache/PyPy/3.10.14/x64/include/pypy3.10/pypy_decl.h:971:29: note: expanded from macro 'PyUnicode_FSDecoder'
^
```
* clang-tidy auto-fix
* Fix silly oversight.
