* Fix data race all_type_info_populate in free-threading mode
Description:
- fixed data race all_type_info_populate in free-threading mode
- added test
For example, we have 2 threads entering `all_type_info`.
Both enter `all_type_info_get_cache`` function and
there is a first one which inserts a tuple (type, empty_vector) to the map
and second is waiting. Inserting thread gets the (iter_to_key, True) and non-inserting thread
after waiting gets (iter_to_key, False).
Inserting thread than will add a weakref and will then call into `all_type_info_populate`.
However, non-inserting thread is not entering `if (ins.second) {` clause and
returns `ins.first->second;`` which is just empty_vector.
Finally, non-inserting thread is failing the check in `allocate_layout`:
```c++
if (n_types == 0) {
pybind11_fail(
"instance allocation failed: new instance has no pybind11-registered base types");
}
```
* style: pre-commit fixes
* Addressed PR comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix buffer protocol implementation
According to the buffer protocol, `ndim` is a _required_ field [1], and
should always be set correctly. Additionally, `shape` should be set if
flags includes `PyBUF_ND` or higher [2]. The current implementation only
set those fields if flags was `PyBUF_STRIDES`.
[1] https://docs.python.org/3/c-api/buffer.html#request-independent-fields
[2] https://docs.python.org/3/c-api/buffer.html#shape-strides-suboffsets
* Apply suggestions from review
* Obey contiguity requests for buffer protocol
If a contiguous buffer is requested, and the underlying buffer isn't,
then that should raise. This matches NumPy behaviour if you do something
like:
```
struct.unpack_from('5d', np.arange(20.0)[::4]) # Raises for contiguity
```
Also, if a buffer is contiguous, then it can masquerade as a
less-complex buffer, either by dropping strides, or even pretending to
be 1D. This matches NumPy behaviour if you do something like:
```
a = np.full((3, 5), 30.0)
struct.unpack_from('15d', a) # --> Produces 1D tuple from 2D buffer.
```
* Handle review comments
* Test buffer protocol against NumPy
* Also check PyBUF_FORMAT results
* Incomplete attempt to address regression introduced in #5381
* style: pre-commit fixes
* Revert "style: pre-commit fixes"
This reverts commit 9d107d2f75.
* Revert "Incomplete attempt to address regression introduced in #5381"
This reverts commit 8cf1cdbc96.
* Simpler fix for the regression introduced in #5381
* style: pre-commit fixes
* Added if constexpr workaround
This can probably be done better but at least this is a start.
* style: pre-commit fixes
* Replace if constexpr with template struct
if constexpr was not added until C++ 17.
I think this should do the same thing as before.
* style: pre-commit fixes
* Made comment clearer
* Added test cases
* style: pre-commit fixes
* Fixed is_same_or_base_of reference
* style: pre-commit fixes
* Added static assert messages
* style: pre-commit fixes
* Replaced typedef with using
* style: pre-commit fixes
* Back out `ForwardClassPtr` (to be discussed separately). Tested locally with clang-tidy.
* Shuffle new `static_assert()` and leave error messages blank (they are more distracting than helpful here).
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: gentlegiantJGC <gentlegiantJGC@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
* Print key in map.getitem/delitem KeyError
* Add tests
* Fix tests
* Make robust
* Make clang-tidy happy
* Return a Python str
* Show beginning and end of the message
* Avoid implicit conversion
* Split out `format_message_key_error_key_object()` to reduce amount of templated code.
* Use `"✄✄✄"` instead of `"..."`
Also rename variable to `cut_length`, to not get into even/odd issues with the meaning of "half".
---------
Co-authored-by: Ralf W. Grosse-Kunstleve <rgrossekunst@nvidia.com>
* Initial support for GraalPy
* Mark tests that currently fail on GraalPy with xfail
* Add graalpy to CI
* Limit test deps on graalpy to available binary wheels
* Skip cmake test installed_function on GraalPy
CMake won't find libpython on GraalPy, it either fails or silently picks
CPython's libpython.
* Factor out setting function docstrings into a macro
* Try to narrow down skipped tests
* Allow subclasses of py::args and py::kwargs
The current implementation does not allow subclasses of args or kwargs.
This change allows subclasses to be used.
* Added test case
* style: pre-commit fixes
* Added missing semi-colons
* style: pre-commit fixes
* Added handle_type_name
* Moved classes outside of function
* Added namespaces
* style: pre-commit fixes
* Refactored tests
Added more tests and moved tests to more appropriate locations.
* style: pre-commit fixes
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* `self.__cpp_transporter__()` proof of concept: Enable passing C++ pointers across extensions even if the `PYBIND11_INTERNALS_VERSION`s do not match.
* Include cleanup (mainly to resolve PyPy build failures).
* Fix clang-tidy errors.
* Resolve `error: extra
* factor out platform_abi_id.h from internals.h (no functional changes)
* factor out internals_version.h from internals.h (no functional changes)
* Update CMakeLists.txt, tests/extra_python_package/test_files.py
* Revert "factor out internals_version.h from internals.h (no functional changes)"
This reverts commit 3ccea8cd43.
* Remove internals_version.h from CMakeLists.txt, tests/extra_python_package/test_files.py
* `.__cpp_transporter__()` implementation: compare `pybind11_platform_abi_id`, `cpp_typeid_name`
* Add PremiumTraveler
* Rename test_cpp_transporter_traveler_type.h -> test_cpp_transporter_traveler_types.h
* Expand tests: `PremiumTraveler`, `get_points()`
* Shuffle order of tests (no real changes).
* Move `__cpp_transporter__` lambda to `py::cpp_transporter()` regular function.
* Use `type_caster_generic::load(self)` instead of `cast<T *>(self)`
* Pass `const std::type_info *` via `py::capsule` (instead of `cpp_typeid_name`).
* Make platform_abi_id.h completely stand-alone.
* rename exo_planet.cpp -> exo_planet_pybind11.cpp
* Add exo_planet_c_api.cpp (incomplete).
* Fix silly oversight (wrong filename in `#include`).
* Resolve clang-tidy errors:
```
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:10:18: error: 'wrapGetLuggage' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
10 | static PyObject *wrapGetLuggage(PyObject *, PyObject *) { return PyUnicode_FromString("TODO"); }
| ~~~~~~ ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:14:20: error: 'ThisMethodDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
14 | static PyMethodDef ThisMethodDef[]
| ~~~~~~ ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:17:27: error: 'ThisModuleDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
17 | static struct PyModuleDef ThisModuleDef = {
| ~~~~~~ ^
```
* Implement exo_planet_c_api GetLuggage(), GetPoints()
* Move new code from test_cpp_transporter_traveler_bindings.h to pybind11/detail/type_caster_base.h, under the name `class_dunder_cpp_transporter()`
* Fix oversight.
* Unconditionally add `__cpp_transporter__` method to all `py::class_` objects, but do not include that magic method in docstring signatures.
* Back out pybind11/detail/platform_abi_id.h for now. Maximizing reusability can be handled separately, later.
* Small cleanup.
* Restore and add to `test_call_cpp_transporter_*()`
* Ensure https://github.com/pybind/pybind11/issues/3788 does not bite again.
* `class_dunder_cpp_transporter()`: replace `obj.cast<std::string>()` with `std::string(obj)`
* Add (simple) copyright notices in all newly added files.
* Globally replace cpp_transporter with cpp_conduit
* style: pre-commit fixes
* IWYU fixes
* Rename `class_dunder_cpp_conduit()` -> `cpp_conduit_method()`
* Change `pybind11_platform_abi_id`, `pointer_kind` argument types from `str` to `bytes`.
This avoids the unicode decode/encode roundtrips:
* More robust (no decode/encode errors).
* Minor runtime optimization.
* Systematically rename `cap_cpp_type_info` -> `cpp_type_info_capsule` (no functional changes).
* Systematically replace `cpp_type_info_capsule` `name`: `"const std::type_info *"` -> `typeid(std::type_info).name()` (this IS a functional change).
This provides an extra layer of protection against C++ ABI mismatches:
* The first and most important layer is that the `PYBIND11_PLATFORM_ABI_ID`s must match between extensions.
* The second layer is that the `typeid(std::type_info).name()`s must match between extensions.
* Fix sort order accident in tests/CMakeLists.txt
* Apply suggestions from code review
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
* style: pre-commit fixes
* refactor: rename to _pybind_conduit_v1_
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Add test_home_planet_wrap_very_lonely_traveler(), test_exo_planet_pybind11_wrap_very_lonely_traveler()
* Resolve clang-tidy errors:
```
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:39:32: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
10 | py::class_<LonelyTraveler>(m, "LonelyTraveler");
| ^
| std::move( )
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:43:52: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
43 | py::class_<VeryLonelyTraveler, LonelyTraveler>(m, "VeryLonelyTraveler");
| ^
| std::move( )
```
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
* Add test for throwing def_buffer case
* Translate C++ -> Python exceptions in buffer creation
This required a little refactoring to extract exception translation to a separate method
* Fix code formatting
* Fix "unused parameter" warning
* Refactor per review
* style: pre-commit fixes
* Address review comments
* Address review comments
---------
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Step 1: Establish new `PYBIND11_SMART_HOLDER_ENABLED` macro, but only under include/pybind11/
At the stage `PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT` and `PYBIND11_SMART_HOLDER_ENABLED` are still equivalent.
* Systematically replace all `PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT` with `PYBIND11_SMART_HOLDER_ENABLED` under tests/ and ubench/
As at the previous stage, `PYBIND11_HAS_INTERNALS_WITH_SMART_HOLDER_SUPPORT` and `PYBIND11_SMART_HOLDER_ENABLED` are still equivalent.
* Introduce `PYBIND11_SMART_HOLDER_DISABLE` option.
* `#ifdef` out entire `wrap()` function to avoid `unused-parameter` warning-as-error under macos-13
```
/Users/runner/work/pybind11/pybind11/tests/test_class_sh_trampoline_basic.cpp:67:23: error: unused parameter 'm' [-Werror,-Wunused-parameter]
void wrap(py::module_ m, const char *py_class_name) {
^
/Users/runner/work/pybind11/pybind11/tests/test_class_sh_trampoline_basic.cpp:67:38: error: unused parameter 'py_class_name' [-Werror,-Wunused-parameter]
void wrap(py::module_ m, const char *py_class_name) {
^
2 errors generated.
```
The SMART_HOLDER_WIP comments are mostly from 2021. In the meantime, the
smart_holder code was extremely thoroughly tested in the Google codebase
(production code). Additionally, testing via PyCLIF-pybind11 provided
thousands of diverse use cases that needed to satisfy many million unit tests
(the success rate was about 99.999%).
* fix: use `__cpp_nontype_template_args` instead of gnu extensions
* fix: add feature test value
* fix: change `PYBIND11_TYPING_H_HAS_STRING_LITERAL` skip reason
* Replace `unique_ptr_cref_roundtrip()` with `pass_unique_ptr_cref()`, `rtrn_unique_ptr_cref()` to make the current behavior obvious.
* add in unique_ptr_storage, unique_ptr_storage_deleter
* Add shared_ptr_storage (with that disowning fails as expected).
* Add load_as_const_unique_ptr()
* Restore original struct_smart_holder.h
* factor out `smart_holder::extract_deleter()`
* Better error message.
* Misc cleanup/tidying.
* Use `re.match("ctor_arg(_MvCtor)*_MvCtor", ...)` for compatibility with MSVC, NVHPC, ICC
* Add small comments.
* Fix small, inconsequential oversight in test code.
* Apply suggestion by @iwanders under PR #5334
* Remove `std::move()` in `smart_holder::extract_deleter()`
* Add `static_assert()` following a suggestion by @iwanders under PR #5334
PREPARATION for:
PR #5332 — Fix handling of const unique_ptr<T, D> & (do not disown).
Splitting out so that the functional changes under PR #5332 will be more obvious.
* fix: allow -Wpedantic again
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: ignore pedantic warning for PYBIND11_DECLARE_HOLDER_TYPE
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: try just turning off pedantic for one file
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: only run pedantic in C++20 mode
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Update tests/local_bindings.h
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* feat: add `<ranges>` support for `py::tuple` and `py::list`
* fix: format the code
* fix: disable `ranges` in clang < 16
* refactor: move `<ranges>` test macro to `test_pytypes.h`
* refactor: seperate `ranges` test into 3 funcs
* style: compress the if statement
* style: pre-commit fixes
* style: better formatting
---------
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add `test_pass_std_vector_int()`, `test_pass_std_set_int()` in test_stl
* Change `list_caster` to also accept generator objects (`PyGen_Check(src.ptr()`).
Note for completeness: This is a more conservative change than https://github.com/google/pywrapcc/pull/30042
* Drop in (currently unpublished) PyCLIF code, use in `list_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdSet()` in `set_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdMap()` in `map_caster`, add tests.
* Simplify `list_caster` `load()` implementation, push str/bytes check into `PyObjectTypeIsConvertibleToStdVector()`.
* clang-tidy cleanup with a few extra `(... != 0)` to be more consistent.
* Also use `PyObjectTypeIsConvertibleToStdVector()` in `array_caster`.
* Update comment pointing to clif/python/runtime.cc (code is unchanged).
* Comprehensive test coverage, enhanced set_caster load implementation.
* Resolve clang-tidy eror.
* Add a long C++ comment explaining what led to the `PyObjectTypeIsConvertibleTo*()` implementations.
* Minor function name change in test.
* strcmp -> std::strcmp (thanks @Skylion007 for catching this)
* Add `PyCallable_Check(items)` in `PyObjectTypeIsConvertibleToStdMap()`
* Resolve clang-tidy error
* Use `PyMapping_Items()` instead of `src.attr("items")()`, to be internally consistent with `PyMapping_Check()`
* Update link to PyCLIF sources.
* Fix typo (thanks @wangxf123456 for catching this)
* Add `test_pass_std_vector_int()`, `test_pass_std_set_int()` in test_stl
* Change `list_caster` to also accept generator objects (`PyGen_Check(src.ptr()`).
Note for completeness: This is a more conservative change than https://github.com/google/pywrapcc/pull/30042
* Drop in (currently unpublished) PyCLIF code, use in `list_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdSet()` in `set_caster`, adjust tests.
* Use `PyObjectTypeIsConvertibleToStdMap()` in `map_caster`, add tests.
* Simplify `list_caster` `load()` implementation, push str/bytes check into `PyObjectTypeIsConvertibleToStdVector()`.
* clang-tidy cleanup with a few extra `(... != 0)` to be more consistent.
* Also use `PyObjectTypeIsConvertibleToStdVector()` in `array_caster`.
* Update comment pointing to clif/python/runtime.cc (code is unchanged).
* Comprehensive test coverage, enhanced set_caster load implementation.
* Resolve clang-tidy eror.
* Add a long C++ comment explaining what led to the `PyObjectTypeIsConvertibleTo*()` implementations.
* Minor function name change in test.
* strcmp -> std::strcmp (thanks @Skylion007 for catching this)
* Add `PyCallable_Check(items)` in `PyObjectTypeIsConvertibleToStdMap()`
* Resolve clang-tidy error
* Use `PyMapping_Items()` instead of `src.attr("items")()`, to be internally consistent with `PyMapping_Check()`
* Update link to PyCLIF sources.
* Fix typo (thanks @wangxf123456 for catching this)
* Fix typo discovered by new version of codespell.
* fix(cmake): add required emscripten flags
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Update emscripten.yaml
* fix(cmake): add required emscripten flags to headers target
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): incorrect detection of Emscripten
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): allow pybind11::headers to be modified
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): hide a warning when building the tests standalone
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): use explicit variable for is config
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix(cmake): go back to ALIAS target
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: reduce overall diff
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: reduce overall diff
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* chore: shorten code a bit
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Allow specializations based on callback function return values.
* clang-tidy auto fix
* Add a test case for function specialization.
* Add test for callback function that raises Python exception.
* Fix test failures.
* style: pre-commit fixes
* Add `#define PYBIND11_HAS_TYPE_CASTER_STD_FUNCTION_SPECIALIZATIONS`
---------
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Put bakein branch @ 18b72c0ffa on top of smart_holder branch:
Commands used:
```
git checkout bakein
git diff smart_holder > ~/zd
git checkout smart_holder
git checkout -b bakein_sh
patch -p 1 < ~/zd
git checkout smart_holder \
MANIFEST.in \
README.rst \
README_smart_holder.rst \
docs/advanced/smart_ptrs.rst \
ubench/holder_comparison.cpp \
ubench/holder_comparison.py \
ubench/holder_comparison_extract_sheet_data.py \
ubench/number_bucket.h \
ubench/python/number_bucket.clif
git add -A
```
* Add back README_smart_holder.rst in tests/extra_python_package/test_files.py
* Restore smart_holder_poc.h as-is on smart_holder branch (i.e. undo `PYBIND11_SMART_HOLDER_PADDING`, which was meant for stress-testing only).
* Insert `std::move()` as suggested by @laramiel
* `property_cpp_function_sh_*` named specializations, as suggested by @laramiel (https://github.com/pybind/pybind11/pull/5257#discussion_r1688346807)
* Call `property_cpp_function_classic` member functions, rather than inlining the implementations.
* Use `PYBIND11_HAVE_INTERNALS_WITH_SMART_HOLDER_SUPPORT` in holder_comparison.cpp (holder_comparison.py is NOT changed accordingly in this commit, i.e. can still only be run if the smart_holder functionality is available).
* Systematically rename `loaded_as` to `load_as` (`shared_ptr`, `unique_ptr`) as suggested by @laramiel
* Make change as suggested by @laramiel. This makes it much more obvious that the latest implementation of `smart_holder_from_unique_ptr()` accepts all existing `return_value_policy` enum values except `copy`.
* Resolve `BAKEIN_WIP: Rewrite comment.` for `property_cpp_function_*` specializations.
* Resolve `BAKEIN_WIP: Add comment to explain: This is meant for stress-testing only.`
* Resolve all remaining BAKEIN_WIP (in pybind11/cast.h).
Leave only two pairs of SMART_HOLDER_BAKEIN_FOLLOW_ON comments: refactoring of copyable_holder_caster, move_only_holder_caster. This is best left until after the smart_holder branch is merged into the master branch.
* Remove obsolete `using holder_type = smart_holder;` in `load_helper`
* Add SMART_HOLDER_BAKEIN_FOLLOW_ON comment for `internals::default_holder`
* README_smart_holder.rst update (line count reduced from 356 to 123).
Follow-on to https://github.com/pybind/pybind11/pull/5272 — clang-tidy upgrade (to version 18)
Fixes this error:
```
/__w/pybind11/pybind11/tests/pure_cpp/smart_holder_poc_test.cpp:167:12: error: the value returned by this function should not be disregarded; neglecting it may lead to errors [bugprone-unused-return-value,-warnings-as-errors]
167 | (void) new_owner.release(); // Manually verified: without this, clang++ -fsanitize=address
| ^~~~~~~~~~~~~~~~~~~
```
See also: https://clang.llvm.org/extra/clang-tidy/checks/bugprone/unused-return-value.html
Specifically there:
> std::unique_ptr::release(). Not using the return value can lead to resource leaks if the same pointer isn’t stored anywhere else. Often, ignoring the release() return value indicates that the programmer confused the function with reset().
I.e. the only way to tell clang-tidy that the smart_holder_poc_test.cpp is correct is to add the `NOLINT`.
