Now also passes the open_spiel iterated_prisoners_dilemma_test ASAN clean, in addition to all pybind11 and PyCLIF unit tests.
The problem was that calling `std::shared_ptr<void>::reset()` with a `void` pointer cannot possibly update the `shared_from_this` `weak_ptr`.
The solution is to store a `shd_ptr_reset` function pointer in `guarded_deleter` (similar in idea to the stored function pointer for calling `delete`).
This commit still includes all debugging code, i.e. is "dirty". The code will be cleaned up after the GitHub CI is fully successful.
* Using new smart_holder::reclaim_disowned in smart_holder_type_caster for unique_ptr.
* Systematically renaming was_disowned to is_disowned (because disowning is now reversible: reclaim_disowned).
* Systematically renaming virtual_overrider_self_life_support to trampoline_self_life_support (to reuse existing terminology instead of introducing new one).
* Systematically renaming test_class_sh_with_alias to test_class_sh_trampoline_basic.
* Adding a Trampolines and std::unique_ptr section to README_smart_holder.rst.
* MSVC compatibility.
* Porting subset of absltest code from reproducer provided by @elkhrt. Baseline for debugging ASAN heap-use-after-free.
* Moving Py_DECREF to resolve ASAN heap-use-after-free failure.
* Fixing trivial formatting issue.
* Workaround for clang 3.6 and 3.7.
* Adding test_class_sh_disowning.
* Fixing minor namespace naming inconsistency between test_class_sh_*.cpp files.
* Replacing py::overload_cast with plain cast for C++11 compatibility.
* Accommodate that the C++ order of evaluation of function arguments is unspecified.
* Adaption of PyCLIF virtual_py_cpp_mix test.
* Removing ValueError: Ownership of instance with virtual overrides in Python cannot be transferred to C++. TODO: static_assert alias class needs to inherit from virtual_overrider_self_life_support.
* Bringing back ValueError: "... instance cannot safely be transferred to C++.", but based on dynamic_cast<AliasType>.
* Fixing oversight: adding test_class_sh_virtual_py_cpp_mix.cpp to cmake file.
* clang <= 3.6 compatibility.
* Fixing oversight: dynamic_raw_ptr_cast_if_possible needs special handling for To = void. Adding corresponding missing test in test_class_sh_virtual_py_cpp_mix. Moving dynamic_raw_ptr_cast_if_possible to separate header.
* Changing py::detail::virtual_overrider_self_life_support to py::virtual_overrider_self_life_support.
* Adding test_unique_ptr_member (for desired PyCLIF behavior).
See also: https://github.com/pybind/pybind11/issues/2583
Does not build with upstream master or
https://github.com/pybind/pybind11/pull/2047, but builds with
https://github.com/RobotLocomotion/pybind11 and almost runs:
```
Running tests in directory "/usr/local/google/home/rwgk/forked/EricCousineau-TRI/pybind11/tests":
================================================================================= test session starts =================================================================================
platform linux -- Python 3.8.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /usr/local/google/home/rwgk/forked/EricCousineau-TRI/pybind11/tests, inifile: pytest.ini
collected 2 items
test_unique_ptr_member.py .F [100%]
====================================================================================== FAILURES =======================================================================================
_____________________________________________________________________________ test_pointee_and_ptr_owner ______________________________________________________________________________
def test_pointee_and_ptr_owner():
obj = m.pointee()
assert obj.get_int() == 213
m.ptr_owner(obj)
with pytest.raises(ValueError) as exc_info:
> obj.get_int()
E Failed: DID NOT RAISE <class 'ValueError'>
test_unique_ptr_member.py:17: Failed
============================================================================= 1 failed, 1 passed in 0.06s =============================================================================
```
* unique_ptr or shared_ptr return
* new test_variant_unique_shared with vptr_holder prototype
* moving prototype code to pybind11/vptr_holder.h, adding type_caster specialization to make the bindings involving unique_ptr passing compile, but load and cast implementations are missing
* disabling GitHub Actions on pull_request (for this PR)
* disabling AppVeyor (for this PR)
* TRIGGER_SEGSEV macro, annotations for GET_STACK (vptr::get), GET_INT_STACK (pointee)
* adding test_promotion_of_disowned_to_shared
* Copying tests as-is from xxx_value_ptr_xxx_holder branch.
https://github.com/rwgk/pybind11/tree/xxx_value_ptr_xxx_holder
Systematically exercising returning and passing unique_ptr<T>, shared_ptr<T>
with unique_ptr, shared_ptr holder.
Observations:
test_holder_unique_ptr:
make_unique_pointee OK
pass_unique_pointee BUILD_FAIL (as documented)
make_shared_pointee Abort free(): double free detected
pass_shared_pointee RuntimeError: Unable to load a custom holder type from a default-holder instance
test_holder_shared_ptr:
make_unique_pointee Segmentation fault (#1138)
pass_unique_pointee BUILD_FAIL (as documented)
make_shared_pointee OK
pass_shared_pointee OK
* Copying tests as-is from xxx_value_ptr_xxx_holder branch.
https://github.com/rwgk/pybind11/tree/xxx_value_ptr_xxx_holder
Systematically exercising casting between shared_ptr<base>, shared_ptr<derived>.
* Demonstration of Undefined Behavior in handling of shared_ptr holder.
Based on https://godbolt.org/z/4fdjaW by jorgbrown@ (thanks Jorg!).
* Additional demonstration of Undefined Behavior in handling of shared_ptr holder.
* fixing up-down mixup in comment
* Demonstration of Undefined Behavior in handling of polymorphic pointers.
(This demo does NOT involve smart pointers at all, unlike the otherwise similar test_smart_ptr_private_first_base.)
* minor test_private_first_base.cpp simplification (after discovering that this can be wrapped with Boost.Python, using boost::noncopyable)
* pybind11 equivalent of Boost.Python test similar to reproducer under #1333
* Snapshot of WIP, TODO: shared_ptr deleter with on/off switch
* Adding vptr_deleter.
* Adding from/as unique_ptr<T> and unique_ptr<T, D>.
* Adding from_shared_ptr. Some polishing.
* New tests/core/smart_holder_poc_test.cpp, using Catch2.
* Adding in vptr_deleter_guard_flag.
* Improved labeling of TEST_CASEs.
* Shuffling existing TEST_CASEs into systematic matrix.
* Implementing all [S]uccess tests.
* Implementing all [E]xception tests.
* Testing of exceptions not covered by the from-as matrix.
* Adding top-level comment.
* Converting from methods to factory functions (no functional change).
* Removing obsolete and very incomplete test (replaced by Catch2-based test).
* Removing stray file.
* Adding type_caster_bare_interface_demo.
* Adding shared_ptr<mpty>, shared_ptr<mpty const> casters.
* Adding unique_ptr<mpty>, unique_ptr<mpty const> casters.
* Pure copy of `class class_` implementation in pybind11.h (master commit 98f1bbb800).
* classh.h: renaming of class_ to classh + namespace; forking test_classh_wip from test_type_caster_bare_interface_demo.
* Hard-coding smart_holder into classh.
* Adding mpty::mtxt string member.
* Adding isinstance<mpty> in type_caster::load functions.
* Adding rvalue_ref, renaming const_value_ref to lvalue_ref & removing const.
* Retrieving smart_holder pointer in type_caster<mpty>::load, and using it cast_op operators.
* Factoring out smart_holder_type_caster_load.
* Retrieving smart_holder pointer in type_caster<std::shared_ptr<mpty[ const]>>::load, and using it cast_op operators.
* Improved error messaging: Cannot disown nullptr (as_unique_ptr).
* Retrieving smart_holder pointer in type_caster<std::unique_ptr<mpty[ const]>>::load, and using it cast_op operators.
* Pure `clang-format --style=file -i` change.
* Pure `clang-format --style=file -i` change, with two `clang-format off` directives.
* Fixing oversight (discovered by flake8).
* flake8 cleanup
* Systematically setting mtxt for all rtrn_mpty_* functions (preparation, the values are not actually used yet).
* static cast handle for rtrn_cptr works by simply dropping in code from type_caster_base (marked with comments).
* static cast handle for rtrn_cref works by simply dropping in code from type_caster_base (marked with comments). rtrn_mref and rtrn_mptr work via const_cast (to add const).
* static cast handle for rtrn_valu works by simply dropping in code from type_caster_base (marked with comments). rtrn_rref raises a RuntimeError, to be investigated.
* Copying type_caster_generic::cast into type_caster<mpty> as-is (preparation for handling smart pointers).
* Pure clang-format change (applied to original type_caster_generic::cast).
* Adding comment re potential use_count data race.
* static handle cast implementations for rtrn_shmp, rtrn_shcp.
* Adding MISSING comments in operator std::unique_ptr<mpty[ const]>.
* static handle cast implementations for rtrn_uqmp, rtrn_uqcp.
* Bug fix: vptr_deleter_armed_flag_ptr has to live on the heap.
See new bullet point in comment section near the top.
The variable was also renamed to reflect its function more accurately.
* Fixing bugs discovered by ASAN. The code is now ASAN, MSAN, UBSAN clean.
* Making test_type_caster_bare_interface_demo.cpp slightly more realistic, ASAN, MSAN, UBSAN clean.
* Calling deregister_instance after disowning via unique_ptr.
* Removing enable_shared_from_this stub, simplifying existing code, clang-format.
Open question, with respect to the original code:
76a160070b/include/pybind11/pybind11.h (L1510)
To me it looks like the exact situation marked as `std::shared_ptr<Good> gp1 = not_so_good.getptr();` here: https://en.cppreference.com/w/cpp/memory/enable_shared_from_this
The comment there is: `// undefined behavior (until C++17) and std::bad_weak_ptr thrown (since C++17)`
Does the existing code have UB pre C++17?
I'll leave handling of enable_shared_from_this for later, as the need arises.
* Cosmetical change around helper functions.
* Using type_caster_base<mpty>::src_and_type directly, removing copy. Also renaming one cast to cast_const_raw_ptr, for clarity.
* Fixing clang-format oversight.
* Using factored-out make_constructor (PR #2798), removing duplicate code.
* Inserting additional assert to ensure a returned unique_ptr is always a new Python instance.
* Adding minor comment (change to internals needed to distinguish uninitialized/disowned in error message).
* Factoring out find_existing_python_instance().
* Moving factored-out make_constructor to test_classh_wip.cpp, restoring previous version of cast.h. This is currently the most practical approach. See PR #2798 for background.
* Copying classh type_casters from test_classh_wip.cpp UNMODIFIED, as a baseline for generalizing the code.
* Using pybind11/detail/classh_type_casters.h from test_classh_wip.cpp.
* Adding & using PYBIND11_CLASSH_TYPE_CASTERS define.
* Adding test_classh_inheritance, currently failing (passes with class_).
* Removing .clang-format before git rebase master (where the file was added).
* Bringing back .clang-format, the previous rm was a bad idea.
* Folding in modified_type_caster_generic_load_impl, just enough to pass test_class_wip. test_classh_inheritance is still failing, but with a different error: [RuntimeError: Incompatible type (as_raw_ptr_unowned).]
* Minimal changes needed to pass test_classh_inheritance.
* First pass adjusting try_implicit_casts and try_load_foreign_module_local to capture loaded_v_h, but untested and guarded with pybind11_failure("Untested"). This was done mainly to determine general feasibility. Note the TODO in pybind11.h, where type_caster_generic::local_load is currently hard-coded. test_classh_wip and test_classh_inheritance still pass, as before.
* Decoupling generic_type from type_caster_generic.
* Changes and tests covering classh_type_casters try_implicit_casts.
* Minimal test covering classh_type_casters load_impl Case 2b.
* Removing stray isinstance<T>(src): it interferes with the py::module_local feature. Adding missing #includes.
* Tests for classh py::module_local() feature.
* Pure renaming of function names in test_classh_inheritance, similar to the systematic approach used in test_class_wip. NO functional changes.
* Pure renaming of function and variable names, for better generalization when convoluting with inheritance. NO functional changes.
* Adopting systematic naming scheme from test_classh_wip. NO functional changes.
* Moving const after type name, for functions that cover a systematic scheme. NO functional changes.
* Adding smart_holder_type_caster_load::loaded_as_shared_ptr, currently bypassing smart_holder shared_ptr tracking completely, but the tests pass and are sanitizer clean.
* Removing rtti_held from smart_holder. See updated comment.
* Cleaning up loaded_as_raw_ptr_unowned, loaded_as_shared_ptr.
* Factoring out convert_type and folding into loaded_as_unique_ptr.
* Folding convert_type into lvalue_ref and rvalue_ref paths. Some smart_holder_type_caster_load cleanup.
* Using unique_ptr in local_load to replace static variable. Also adding local_load_safety_guard.
* Converting test_unique_ptr_member to using classh: fully working, ASAN, MSAN, UBSAN clean.
* Removing debugging comments (GET_STACK, GET_INT_STACK). cast.h is identical to current master again, pybind11.h only has the generic_type::initialize(..., &type_caster_generic::local_load) change.
* Purging obsolete pybind11/vptr_holder.h and associated test.
* Moving several tests to github.com/rwgk/rwgk_tbx/tree/main/pybind11_tests
a2c2f88174
These tests are from experimenting, and for demonstrating UB in pybind11 multiple inheritance handling ("first_base"), to be fixed later.
* Adding py::smart_holder support to py::class_, purging py::classh completely.
* Renaming files in include directory, creating pybind11/smart_holder.h.
* Renaming all "classh" to "smart_holder" in pybind11/detail/smart_holder_type_casters.h.
The user-facing macro is now PYBIND11_SMART_HOLDER_TYPE_CASTERS.
* Systematically renaming tests to use "class_sh" in the name.
* Renaming test_type_caster_bare_interface_demo to test_type_caster_bare_interface.
* Renaming new tests/core subdirectory to tests/pure_cpp.
* Adding new tests to CMake config, resetting CI config.
* Changing CMake file so that test_class_sh_module_local.py actually runs.
* clang-tidy fixes.
* 32-bit compatibility.
* Reusing type_caster_base make_copy_constructor, make_move_constructor with a trick.
* CMake COMPARE NATURAL is not available with older versions.
* Adding copyright notices to new header files.
* Explicitly define copy/move constructors/assignments.
* Adding new header files to tests/extra_python_package/test_files.py.
* Adding tests/pure_cpp/CMakeLists.txt.
* Making use of the new find_existing_python_instance() function factored out with PR #2822.
* Moving define PYBIND11_SMART_HOLDER_TYPE_CASTERS(T) down in the file. NO functional changes. Preparation for follow-up work (to keep that diff smaller).
* Reintroducing py::classh, this time as a simple alias for py::class_<U, py::smart_holder>.
* Replacing detail::is_smart_holder<H> in cast.h with detail::is_smart_holder_type_caster<T>.
Moving get_local_load_function_ptr, init_instance_for_type to smart_holder_type_caster_class_hooks.
Expanding static_assert in py::type::handle_of<> to accommodate smart_holder_type_casters.
* Fixing oversight.
* Adding classu alias for class_<U, std::unique_ptr<U>>.
* Giving up on idea to use legacy init_instance only if is_base_of<type_caster_generic, type_caster<T>. There are use cases in the wild that define both a custom type_caster and class_.
* Removing test_type_caster_bare_interface, which was moved to the separate PR #2834.
* Moving up is_smart_holder_type_caster, to also use in cast_is_temporary_value_reference.
* Adding smart_holder_type_casters for unique_ptr with custom deleter. SEVERE CODE DUPLICATION. This commit is to establish a baseline for consolidating the unique_ptr code.
* Unification of unique_ptr, unique_ptr_with_deleter code in smart_holder_poc.h. Leads to more fitting error messages. Enables use of unique_ptr<T, D> smart_holder_type_casters also for unique_ptr<T>.
* Copying files as-is from branch test_unique_ptr_member (PR #2672).
* Adding comment, simplifying naming, cmake addition.
* Introducing PYBIND11_USE_SMART_HOLDER_AS_DEFAULT macro (tested only undefined; there are many errors with the macro defined).
* Removing test_type_caster_bare_interface, which was moved to the separate PR #2834.
* Fixing oversight introduced with commit 95425f13d6.
* Setting record.default_holder correctly for PYBIND11_USE_SMART_HOLDER_AS_DEFAULT.
With this test_class.cpp builds and even mostly runs, except
`test_multiple_instances_with_same_pointer`, which segfaults because it is
using a `unique_ptr` holder but `smart_holder` `type_caster`.
Also adding `static_assert`s to generate build errors for such situations,
but guarding with `#if 0` to first pivot to test_factory_constructors.cpp.
* Fixing up cast.h and smart_holder.h after rebase.
* Removing detail/smart_holder_type_casters.h in separate commit.
* Commenting out const in def_buffer(... const). With this, test_buffers builds and runs with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT. Explanation why the const needs to be removed, or fix elsewhere, is still needed, but left for later.
* Adding test_class_sh_factory_constructors, reproducing test_factory_constructors failure. Using py::class_ in this commit, to be changed to py::classh for debugging.
* Removing include/pybind11/detail/smart_holder_type_casters.h from CMakeLists.txt, test_files.py (since it does not exist in this branch).
* Adding // DANGER ZONE reminders.
* Converting as many py::class_ to py::classh as possible, not breaking tests.
* Adding initimpl::construct() overloads, resulting in test_class_sh_factory_constructors feature parity for py::class_ and py::classh.
* Adding enable_if !is_smart_holder_type_caster to existing initimpl::construct(). With this test_factory_constructors.cpp builds with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT.
* Disabling shared_ptr&, shared_ptr* tests when building with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT for now, pending work on smart_holder_type_caster<shared_ptr>.
* Factoring out struct and class definitions into anonymous namespace. Preparation for building with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT.
* Simplifying from_unique_ptr(): typename D = std::default_delete<T> is not needed. Factoring out is_std_default_delete<T>() for consistentcy between ensure_compatible_rtti_uqp_del() and from_unique_ptr().
* Introducing PYBIND11_SMART_POINTER_HOLDER_TYPE_CASTERS. Using it in test_smart_ptr.cpp. With this test_smart_ptr builds with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT and all but one test run successfully.
* Introducing 1. type_caster_for_class_, used in PYBIND11_MAKE_OPAQUE, and 2. default_holder_type, used in stl_bind.h.
* Using __VA_ARGS__ in PYBIND11_SMART_POINTER_HOLDER_TYPE_CASTERS.
* Replacing condense_for_macro with much simpler approach.
* Softening static_assert, to only check specifically that smart_holder is not mixed with type_caster_base, and unique_ptr/shared_ptr holders are not mixed with smart_holder_type_casters.
* Adding PYBIND11_SMART_POINTER_HOLDER_TYPE_CASTERS in test_class.cpp (with this all but one test succeed with PYBIND11_USE_SMART_HOLDER_AS_DEFAULT).
* Adding remaining PYBIND11_SMART_POINTER_HOLDER_TYPE_CASTERS. static_assert for "necessary conditions" for both types of default holder, static_assert for "strict conditions" guarded by new PYBIND11_STRICT_ASSERTS_CLASS_HOLDER_VS_TYPE_CASTER_MIX. All tests build & run as before with unique_ptr as the default holder, all tests build for smart_holder as the default holder, even with the strict static_assert.
* Introducing check_is_smart_holder_type_caster() function for runtime check, and reinterpreting record.default_holder as "uses_unique_ptr_holder". With this test_smart_ptr succeeds. (All 42 tests build, 35 tests succeed, 5 run but have some failures, 2 segfault.)
* Bug fix: Adding have_value() to smart_holder_type_caster_load. With this test_builtin_casters succeeds. (All 42 tests build, 36 tests succeed, 5 run but have some failures, 1 segfault.)
* Adding unowned_void_ptr_from_direct_conversion to modified_type_caster_generic_load_impl. This fixes the last remaining segfault (test_numpy_dtypes). New stats for all tests combined: 12 failed, 458 passed.
* Adding "Lazy allocation for unallocated values" (for old-style __init__) into load_value_and_holder. Deferring destruction of disowned holder until clear_instance, to remain inspectable for "uninitialized" or "disowned" detection. New stats for all tests combined: 5 failed, 465 passed.
* Changing std::shared_ptr pointer/reference to const pointer/reference. New stats for all tests combined: 4 failed, 466 passed.
* Adding return_value_policy::move to permissible policies for unique_ptr returns. New stats for all tests combined: 3 failed, 467 passed.
* Overlooked flake8 fixes.
* Manipulating failing ConstructorStats test to pass, to be able to run all tests with ASAN.
This version of the code is ASAN clean with unique_ptr or smart_holder as the default.
This change needs to be reverted after adopting the existing move-only-if-refcount-is-1
logic used by type_caster_base.
* Adding copy constructor and move constructor tracking to atyp. Preparation for a follow-up change in smart_holder_type_caster, to make this test sensitive to the changing behavior.
[skip ci]
* Removing `operator T&&() &&` from smart_holder_type_caster, for compatibility with the behavior of type_caster_base. Enables reverting 2 of 3 test manipulations applied under commit 249df7cbdb. The manipulation in test_factory_constructors.py is NOT reverted in this commit.
[skip ci]
* Fixing unfortunate editing mishap. This reverts the last remaining test manipulation in commit 249df7cbdb and makes all existing unit tests pass with smart_holder as default holder.
* GitHub CI clang-tidy fixes.
* Adding messages to terse `static_assert`s, for pre-C++17 compatibility.
* Using @pytest.mark.parametrize to run each assert separately (to see all errors, not just the first).
* Systematically removing _atyp from function names, to make the test code simpler.
* Using re.match to accommodate variable number of intermediate MvCtor.
* Also removing `operator T()` from smart_holder_type_caster, to fix gcc compilation errors. The only loss is pass_rref in test_class_sh_basic.
* Systematically replacing `detail::enable_if_t<...smart_holder...>` with `typename std::enable_if<...smart_holder...>::type`. Attempt to work around MSVC 2015 issues, to be tested via GitHub CI. The idea for this change originates from this comment: https://github.com/pybind/pybind11/issues/1616#issuecomment-444536813
* Importing re before pytest after observing a PyPy CI flake when importing pytest first.
* Copying MSVC 2015 compatibility change from branch pr2672_use_smart_holder_as_default.
* Introducing is_smart_holder_type_caster_base_tag, to keep smart_holder code more disconnected.
* Working around MSVC 2015 bug.
* Expanding comment for MSVC 2015 workaround.
* Systematically changing std::enable_if back to detail::enable_if_t, effectively reverting commit 5d4b6890a3.
* Removing unused smart_holder_type_caster_load::loaded_as_rvalue_ref (it was an oversight that it was not removed with commit 23036a45eb).
* Removing py::classu, because it does not seem useful enough.
* Reverting commit 6349531306 by un-commenting `const` in `def_buffer(...)`. To make this possible, `operator T const&` and `operator T const*` in `smart_holder_type_caster` need to be marked as `const` member functions.
* Adding construct() overloads for constructing smart_holder from alias unique_ptr, shared_ptr returns.
* Adding test_class_sh_factory_constructors.cpp to tests/CMakeLists.txt (fixes oversight, this should have been added long before).
* Compatibility with old clang versions (clang 3.6, 3.7 C++11).
* Cleaning up changes to existing unit tests.
* Systematically adding SMART_HOLDER_WIP tag. Removing minor UNTESTED tags (only the throw are not actually exercised, investing time there has a high cost but very little benefit).
* Splitting out smart_holder_type_casters again, into new detail/smart_holder_type_casters_inline_include.h.
* Splitting out smart_holder_init_inline_include.h.
* Adding additional new include files to CMakeLists.txt, tests/extra_python_package/test_files.py.
* clang-format cleanup of most smart_holder code.
* Adding source code comments in response to review.
* Simple micro-benchmark ("ubench") comparing runtime performance for several holders.
Tested using github.com/rwgk/pybind11_scons and Google-internal build system.
Sorry, no cmake support at the moment.
First results: https://docs.google.com/spreadsheets/d/1InapCYws2Gt-stmFf_Bwl33eOMo3aLE_gc9adveY7RU/edit#gid=0
* Breaking out number_bucket.h, adding hook for also collecting performance data for PyCLIF.
* Accounting for ubench in MANIFEST.in (simply prune, for now).
* Smarter determination of call_repetitions.
[skip ci]
* Also scaling performance data to PyCLIF.
[skip ci]
* Adding ubench/python/number_bucket.clif here for general visibility.
* Fix after rebase
* Merging detail/smart_holder_init_inline_include.h into detail/init.h.
* Renaming detail/is_smart_holder_type_caster.h -> detail/smart_holder_sfinae_hooks_only.h.
* Renaming is_smart_holder_type_caster -> type_uses_smart_holder_type_caster for clarity.
* Renaming type_caster_type_is_smart_holder_type_caster -> wrapped_type_uses_smart_holder_type_caster for clarity.
* Renaming is_smart_holder_type_caster_base_tag -> smart_holder_type_caster_base_tag for simplicity.
* Adding copyright notices and minor colateral cleanup.
* iwyu cleanup (comprehensive only for cast.h and smart_holder*.h files).
* Fixing `git rebase master` accident.
* Moving large `pragma warning` block from pybind11.h to detail/common.h.
* Fixing another `git rebase master` accident.
* Adding a valgrind build on debug Python 3.9
Co-authored-by: Boris Staletic <boris.staletic@gmail.com>
* Add Valgrind suppression files
- Introduce suppression file, populate it with a first suppression taken from CPython, and fix one leak in the tests
- Suppress leak in NumPy
- More clean tests!
- Tests with names a-e passing (except for test_buffer)
- Suppress multiprocessing errors
- Merge multiprocessing suppressions into other suppression files
- Numpy seems to be spelled with a big P
- Append single entry from valgrind-misc.supp to valgrind-python.supp, and make clear valgrind-python.supp is only CPython
Co-authored-by: Boris Staletic <boris.staletic@gmail.com>
* Enable test_virtual_functions with a workaround
* Add a memcheck cmake target
- Add a memcheck cmake target
- Reformat cmake
- Appease the formatting overlords - they are angry
- Format CMake valgrind target decently
* Update CI config to new action versions
* fix: separate memcheck from pytest
* ci: cleanup
* Merge Valgrind and other deadsnakes builds
Co-authored-by: Boris Staletic <boris.staletic@gmail.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
* Adding missing virtual destructors, to silence clang -Wnon-virtual-dtor warnings.
Tested with clang version 9.0.1-12 under an Ubuntu-like OS.
Originally discovered in the Google-internal environment.
* adding -Wnon-virtual-dtor for GNU|Intel|Clang
* Added guards to the includes
Added new CI config
Added new trigger
Changed CI workflow name
Debug CI
Debug CI
Debug CI
Debug CI
Added flags fro PGI
Disable Eigen
Removed tests that fail
Uncomment lines
* fix: missing include
fix: minor style cleanup
tests: support skipping
ci: remove and tighten a bit
fix: try msvc workaround for pgic
* tests: split up prealoc tests
* fix: PGI compiler fix
* fix: PGI void_t only
* fix: try to appease nvcc
* ci: better ordering for slow tests
* ci: minor improvements to testing
* ci: Add NumPy to testing
* ci: Eigen generates CUDA warnings / PGI errors
* Added CentOS7 back for a moment
* Fix YAML
* ci: runs-on missing
* centos7 is missing pytest
* ci: use C++11 on CentOS 7
* ci: test something else
* Try just adding flags on CentOS 7
* fix: CentOS 7
* refactor: move include to shared location
* Added verbose flag
* Try to use system cmake3 on CI
* Try to use system cmake3 on CI, attempt2
* Try to use system cmake3 on CI, attempt3
* tests: not finding pytest should be a warning, not a fatal error
* tests: cleanup
* Weird issue?
* fix: final polish
Co-authored-by: Andrii Verbytskyi <andrii.verbytskyi@mpp.mpg.de>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Andrii Verbytskyi <averbyts@cern.ch>
The variables PYBIND11_HAS_OPTIONAL, PYBIND11_HAS_EXP_OPTIONAL, PYBIND11_HAS_VARIANT,
__clang__, __APPLE__ were not checked for defined in a minortity of instances.
If the project using pybind11 sets -Wundef, the warnings will show.
The test build is also modified to catch the problem.
* fix: support nvcc and test
* fixup! fix: support nvcc and test
* docs: mention what compilers fail
* fix: much simpler logic
* refactor: slightly faster / clearer
* tests: keep source dir clean
* ci: make first build inplace
* ci: drop dev setting (wasn't doing anything)
* tests: warn if source directory is dirty
* docs: move helpers to .github where allowed
* docs: more guidelines in CONTRIBUTING
* chore: update issue templates
* fix: review from @bstaletic
* refactor: a few points from @rwgk
* docs: more touchup, review changes
* tests: refactor and cleanup
* refactor: more consistent
* tests: vendor six
* tests: more xfails, nicer system
* tests: simplify to info
* tests: suggestions from @YannickJadoul and @bstaletic
* tests: restore some pypy tests that now pass
* tests: rename info to env
* tests: strict False/True
* tests: drop explicit strict=True again
* tests: reduce minimum PyTest to 3.1
This is only necessary if `get_internals` is called for the first time in a given module when the running thread is in a GIL-released state.
Fixes#1364
In def_readonly and def_readwrite, there is an assertion that the member comes
from the class or a base class:
static_assert(std::is_base_of<C, type>::value, "...");
However, if C and type are the same type, is_base_of will still only be true
if they are the same _non-union_ type. This means we can't define accessors
for the members of a union type because of this assertion.
Update the assertion to test
std::is_same<C, type>::value || std::is_base_of<C, type>::value
which will allow union types, or members of base classes.
Also add a basic unit test for accessing unions.
This avoids GIL deadlocking when pybind11 tries to acquire the GIL in a thread that already acquired it using standard Python API (e.g. when running from a Python thread).
* Add basic support for tag-based static polymorphism
Sometimes it is possible to look at a C++ object and know what its dynamic type is,
even if it doesn't use C++ polymorphism, because instances of the object and its
subclasses conform to some other mechanism for being self-describing; for example,
perhaps there's an enumerated "tag" or "kind" member in the base class that's always
set to an indication of the correct type. This might be done for performance reasons,
or to permit most-derived types to be trivially copyable. One of the most widely-known
examples is in LLVM: https://llvm.org/docs/HowToSetUpLLVMStyleRTTI.html
This PR permits pybind11 to be informed of such conventions via a new specializable
detail::polymorphic_type_hook<> template, which generalizes the previous logic for
determining the runtime type of an object based on C++ RTTI. Implementors provide
a way to map from a base class object to a const std::type_info* for the dynamic
type; pybind11 then uses this to ensure that casting a Base* to Python creates a
Python object that knows it's wrapping the appropriate sort of Derived.
There are a number of restrictions with this tag-based static polymorphism support
compared to pybind11's existing support for built-in C++ polymorphism:
- there is no support for this-pointer adjustment, so only single inheritance is permitted
- there is no way to make C++ code call new Python-provided subclasses
- when binding C++ classes that redefine a method in a subclass, the .def() must be
repeated in the binding for Python to know about the update
But these are not much of an issue in practice in many cases, the impact on the
complexity of pybind11's innards is minimal and localized, and the support for
automatic downcasting improves usability a great deal.
This commit turns on `-Wdeprecated` in the test suite and fixes several
associated deprecation warnings that show up as a result:
- in C++17 `static constexpr` members are implicitly inline; our
redeclaration (needed for C++11/14) is deprecated in C++17.
- various test suite classes have destructors and rely on implicit copy
constructors, but implicit copy constructor definitions when a
user-declared destructor is present was deprecated in C++11.
- Eigen also has various implicit copy constructors, so just disable
`-Wdeprecated` in `eigen.h`.
E.g. trying to convert a `list` to a `std::vector<int>` without
including <pybind11/stl.h> will now raise an error with a note that
suggests checking the headers.
The note is only appended if `std::` is found in the function
signature. This should only be the case when a header is missing.
E.g. when stl.h is included, the signature would contain `List[int]`
instead of `std::vector<int>` while using stl_bind.h would produce
something like `MyVector`. Similarly for `std::map`/`Dict`, `complex`,
`std::function`/`Callable`, etc.
There's a possibility for false positives, but it's pretty low.
To avoid an ODR violation in the test suite while testing
both `stl.h` and `std_bind.h` with `std::vector<bool>`,
the `py::bind_vector<std::vector<bool>>` test is moved to
the secondary module (which does not include `stl.h`).
This allows you to use:
cls.def(py::init(&factory_function));
where `factory_function` returns a pointer, holder, or value of the
class type (or a derived type). Various compile-time checks
(static_asserts) are performed to ensure the function is valid, and
various run-time type checks where necessary.
Some other details of this feature:
- The `py::init` name doesn't conflict with the templated no-argument
`py::init<...>()`, but keeps the naming consistent: the existing
templated, no-argument one wraps constructors, the no-template,
function-argument one wraps factory functions.
- If returning a CppClass (whether by value or pointer) when an CppAlias
is required (i.e. python-side inheritance and a declared alias), a
dynamic_cast to the alias is attempted (for the pointer version); if
it fails, or if returned by value, an Alias(Class &&) constructor
is invoked. If this constructor doesn't exist, a runtime error occurs.
- for holder returns when an alias is required, we try a dynamic_cast of
the wrapped pointer to the alias to see if it is already an alias
instance; if it isn't, we raise an error.
- `py::init(class_factory, alias_factory)` is also available that takes
two factories: the first is called when an alias is not needed, the
second when it is.
- Reimplement factory instance clearing. The previous implementation
failed under python-side multiple inheritance: *each* inherited
type's factory init would clear the instance instead of only setting
its own type value. The new implementation here clears just the
relevant value pointer.
- dealloc is updated to explicitly set the leftover value pointer to
nullptr and the `holder_constructed` flag to false so that it can be
used to clear preallocated value without needing to rebuild the
instance internals data.
- Added various tests to test out new allocation/deallocation code.
- With preallocation now done lazily, init factory holders can
completely avoid the extra overhead of needing an extra
allocation/deallocation.
- Updated documentation to make factory constructors the default
advanced constructor style.
- If an `__init__` is called a second time, we have two choices: we can
throw away the first instance, replacing it with the second; or we can
ignore the second call. The latter is slightly easier, so do that.
In C++11 mode, `boost::apply_visitor` requires an explicit `result_type`.
This also adds optional tests for `boost::variant` in C++11/14, if boost
is available. In C++17 mode, `std::variant` is tested instead.
This commit adds a `py::module_local` attribute that lets you confine a
registered type to the module (more technically, the shared object) in
which it is defined, by registering it with:
py::class_<C>(m, "C", py::module_local())
This will allow the same C++ class `C` to be registered in different
modules with independent sets of class definitions. On the Python side,
two such types will be completely distinct; on the C++ side, the C++
type resolves to a different Python type in each module.
This applies `py::module_local` automatically to `stl_bind.h` bindings
when the container value type looks like something global: i.e. when it
is a converting type (for example, when binding a `std::vector<int>`),
or when it is a registered type itself bound with `py::module_local`.
This should help resolve potential future conflicts (e.g. if two
completely unrelated modules both try to bind a `std::vector<int>`.
Users can override the automatic selection by adding a
`py::module_local()` or `py::module_local(false)`.
Note that this does mildly break backwards compatibility: bound stl
containers of basic types like `std::vector<int>` cannot be bound in one
module and returned in a different module. (This can be re-enabled with
`py::module_local(false)` as described above, but with the potential for
eventual load conflicts).
The builtin exception handler currently doesn't work across modules
under clang/libc++ for builtin pybind exceptions like
`pybind11::error_already_set` or `pybind11::stop_iteration`: under
RTLD_LOCAL module loading clang considers each module's exception
classes distinct types. This then means that the base exception
translator fails to catch the exceptions and the fall through to the
generic `std::exception` handler, which completely breaks things like
`stop_iteration`: only the `stop_iteration` of the first module loaded
actually works properly; later modules raise a RuntimeError with no
message when trying to invoke their iterators.
For example, two modules defined like this exhibit the behaviour under
clang++/libc++:
z1.cpp:
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z1, m) {
py::bind_vector<std::vector<long>>(m, "IntVector");
}
z2.cpp:
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z2, m) {
py::bind_vector<std::vector<double>>(m, "FloatVector");
}
Python:
import z1, z2
for i in z2.FloatVector():
pass
results in:
Traceback (most recent call last):
File "zs.py", line 2, in <module>
for i in z2.FloatVector():
RuntimeError
This commit fixes the issue by adding a new exception translator each
time the internals pointer is initialized from python builtins: this
generally means the internals data was initialized by some other
module. (The extra translator(s) are skipped under libstdc++).
This adds the infrastructure for a separate test plugin for cross-module
tests. (This commit contains no tests that actually use it, but the
following commits do; this is separated simply to provide a cleaner
commit history).
At this point, there is only a single test for interpreter basics.
Apart from embedding itself, having a C++ test framework will also
benefit the C++-side features by allowing them to be tested directly.
All targets provided by pybind11:
* pybind11::module - the existing target for creating extension modules
* pybind11::embed - new target for embedding the interpreter
* pybind11::pybind11 - common "base" target (headers only)
MSVC by default uses the local codepage, which fails when it sees the
utf-8 in test_python_types.cpp. This adds the /utf-8 flag to the test
suite compilation to force it to interpret source code as utf-8.
Fixes#869
This commit allows type_casters to allow their local values to be moved
away, rather than copied, when the type caster instance itself is an rvalue.
This only applies (automatically) to type casters using
PYBIND11_TYPE_CASTER; the generic type type casters don't own their own
pointer, and various value casters (e.g. std::string, std::pair,
arithmetic types) already cast to an rvalue (i.e. they return by value).
This updates various calling code to attempt to get a movable value
whenever the value is itself coming from a type caster about to be
destroyed: for example, when constructing an std::pair or various stl.h
containers. For types that don't support value moving, the cast_op
falls back to an lvalue cast.
There wasn't an obvious place to add the tests, so I added them to
test_copy_move_policies, but also renamed it to drop the _policies as it
now tests more than just policies.
* Add `pytest.ini` config file and set default options there instead of
in `CMakeLists.txt` (command line arguments).
* Change all output capture from `capfd` (filedescriptors) to `capsys`
(Python's `sys.stdout` and `sys.stderr`). This avoids capturing
low-level C errors, e.g. from the debug build of Python.
* Set pytest minimum version to 3.0 to make it easier to use new
features. Removed conditional use of `excinfo.match()`.
* Clean up some leftover function-level `@pytest.requires_numpy`.
* Make tests buildable independently
This makes "tests" buildable as a separate project that uses
find_package(pybind11 CONFIG) when invoked independently.
This also moves the WERROR option into tests/CMakeLists.txt, as that's
the only place it is used.
* Use Eigen 3.3.1's cmake target, if available
This changes the eigen finding code to attempt to use Eigen's
system-installed Eigen3Config first. In Eigen 3.3.1, it exports a cmake
Eigen3::Eigen target to get dependencies from (rather than setting the
include path directly).
If it fails, we fall back to the trying to load allowing modules (i.e.
allowing our tools/FindEigen3.cmake). If we either fallback, or the
eigen version is older than 3.3.1 (or , we still set the include
directory manually; otherwise, for CONFIG + new Eigen, we get it via
the target.
This is also needed to allow 'tests' to be built independently, when
the find_package(Eigen3) is going to find via the system-installed
Eigen3Config.cmake.
* Add a install-then-build test, using clang on linux
This tests that `make install` to the actual system, followed by a build
of the tests (without the main pybind11 repository available) works as
expected.
To also expand the testing variety a bit, it also builds using
clang-3.9 instead of gcc.
* Don't try loading Eigen3Config in cmake < 3.0
It could FATAL_ERROR as the newer cmake includes a cmake 3.0 required
line.
If doing an independent, out-of-tree "tests" build, the regular
find_package(Eigen3) is likely to fail with the same error, but I think
we can just let that be: if you want a recent Eigen with proper cmake
loading support *and* want to do an independent tests build, you'll
need at least cmake 3.0.
Clang on linux currently fails to run cmake:
$ CC=clang CXX=clang++ cmake ..
...
-- Configuring done
CMake Error at tools/pybind11Tools.cmake:135 (target_compile_options):
Error evaluating generator expression:
$<:-flto>
Expression did not evaluate to a known generator expression
Call Stack (most recent call first):
tests/CMakeLists.txt:68 (pybind11_add_module)
But investigating this led to various other -flto detection problems;
this commit thus overhauls LTO flag detection:
- -flto needs to be passed to the linker as well
- Also compile with -fno-fat-lto-objects under GCC
- Pass the equivalent flags to MSVC
- Enable LTO flags for via generator expressions (for non-debug builds
only), so that multi-config builds (like on Windows) still work
properly. This seems reasonable, however, even on single-config
builds (and simplifies the cmake code a bit).
- clang's lto linker plugins don't accept '-Os', so replace it with
'-O3' when doing a MINSIZEREL build
- Enable trying ThinLTO by default for test suite (only affects clang)
- Match Clang$ rather than ^Clang$ because, for cmake with 3.0+
policies in effect, the compiler ID will be AppleClang on macOS.
Use PROJECT_SOURCE_DIR instead of CMAKE_SOURCE_DIR as the base of the
path to libsize.py. This fixes an error if pybind11 is being built
directly within another project.
On a debian jessie machine, running 'python --version --noconftest' caused
pytest to try and run the test suite with the not-yet-compiled extension
module, thus failing the test. This commit chages the pytest detection
so that it only attempts to run an import statement.
Add a BUILD_INTERFACE and a pybind11::pybind11 alias for the interface
library to match the installed target.
Add new cmake tests for add_subdirectory and consolidates the
.cpp and .py files needed for the cmake build tests:
Before:
tests
|-- test_installed_module
| |-- CMakeLists.txt
| |-- main.cpp
| \-- test.py
\-- test_installed_target
|-- CMakeLists.txt
|-- main.cpp
\-- test.py
After:
tests
\-- test_cmake_build
|-- installed_module/CMakeLists.txt
|-- installed_target/CMakeLists.txt
|-- subdirectory_module/CMakeLists.txt
|-- subdirectory_target/CMakeLists.txt
|-- main.cpp
\-- test.py
When working on some particular feature, it's nice to be able to disable
all the tests except for the one I'm working on; this is currently
possible by editing tests/CMakeLists.txt, and commenting out the tests
you don't want.
This commit goes a step further by letting you give a list of tests you
do want when invoking cmake, e.g.:
cmake -DPYBIND11_TEST_OVERRIDE="test_issues.cpp;test_pickling.cpp" ..
changes the build to build just those two tests (and changes the `pytest`
target to invoke just the two associated tests).
This persists in the build directory until you disable it again by
running cmake with `-DPYBIND11_TEST_OVERRIDE=`. It also adds a message
after the pytest output to remind you that it is in effect:
Note: not all tests run: -DPYBIND11_TEST_OVERRIDE is in effect
* Add debugging info about so size to build output
This adds a small python script to tools that captures before-and-after
.so sizes between builds and outputs this in the build output via a
string such as:
------ pybind11_tests.cpython-35m-x86_64-linux-gnu.so file size: 924696 (decrease of 73680 bytes = 7.38%)
------ pybind11_tests.cpython-35m-x86_64-linux-gnu.so file size: 998376 (increase of 73680 bytes = 7.97%)
------ pybind11_tests.cpython-35m-x86_64-linux-gnu.so file size: 998376 (no change)
Or, if there was no .so during the build, just the .so size by itself:
------ pybind11_tests.cpython-35m-x86_64-linux-gnu.so file size: 998376
This allows you to, for example, build, checkout a different branch,
rebuild, and easily see exactly the change in the pybind11_tests.so
size.
It also allows looking at the travis and appveyor build logs to get an
idea of .so/.dll sizes across different build systems.
* Minor libsize.py script changes
- Use RAII open
- Remove unused libsize=-1
- Report change as [+-]xyz bytes = [+-]a.bc%
