============================== pybind11 — smart_holder branch ============================== Overview ======== - The smart_holder git branch is a strict superset of the master branch. Everything that works on master is expected to work exactly the same with the smart_holder branch. - **Smart-pointer interoperability** (``std::unique_ptr``, ``std::shared_ptr``) is implemented as an **add-on**. - The add-on also supports * passing a Python object back to C++ via ``std::unique_ptr``, safely **disowning** the Python object. * safely passing `"trampoline" `_ objects (objects with C++ virtual function overrides implemented in Python) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++: associated Python objects are automatically kept alive for the lifetime of the smart-pointer. - The smart_holder branch can be used in two modes: * **Conservative mode**: ``py::class_`` works exactly as on master. ``py::classh`` uses ``py::smart_holder``. * **Progressive mode**: ``py::class_`` uses ``py::smart_holder`` (i.e. ``py::smart_holder`` is the default holder). What is fundamentally different? -------------------------------- - Traditional pybind11 has the concept of "smart-pointer is holder". Interoperability between smart-pointers is completely missing. For example, when using ``std::shared_ptr`` as holder, ``return``-ing a ``std::unique_ptr`` leads to undefined runtime behavior (`#1138 `_). A `systematic analysis is here `_. - ``py::smart_holder`` has a richer concept in comparison, with well-defined runtime behavior. The holder "knows" about both ``std::unique_ptr`` and ``std::shared_ptr`` and how they interoperate. - Caveat (#HelpAppreciated): currently the ``smart_holder`` branch does not have a well-lit path for including interoperability with custom smart-pointers. It is expected to be a fairly obvious extension of the ``smart_holder`` implementation, but will depend on the exact specifications of each custom smart-pointer type (generalizations are very likely possible). What motivated the development of the smart_holder code? -------------------------------------------------------- - Necessity is the mother. The bigger context is the ongoing retooling of `PyCLIF `_, to use pybind11 underneath instead of directly targeting the Python C API. Essentially, the smart_holder branch is porting established PyCLIF functionality into pybind11. Installation ============ Currently ``git clone`` is the only option. We do not have released packages. .. code-block:: bash git clone --branch smart_holder https://github.com/pybind/pybind11.git Everything else is exactly identical to using the default (master) branch. Conservative or Progressive mode? ================================= It depends. To a first approximation, for a stand-alone, new project, the progressive mode will be easiest to use. For larger projects or projects that integrate with third-party pybind11-based projects, the conservative mode may be more practical, at least initially, although it comes with the disadvantage of having to use the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro. Conservative mode ----------------- Here is a minimal example for wrapping a C++ type with ``py::smart_holder`` as holder: .. code-block:: cpp #include struct Foo {}; PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo) PYBIND11_MODULE(example_bindings, m) { namespace py = pybind11; py::classh(m, "Foo"); } There are three small differences compared to traditional pybind11: - ``#include `` is used instead of ``#include ``. - ``py::classh`` is used instead of ``py::class_``. - The ``PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)`` macro is needed. To the 2nd bullet point, ``py::classh`` is simply a shortcut for ``py::class_``. The shortcut makes it possible to switch to using ``py::smart_holder`` without messing up the indentation of existing code. However, when migrating code that uses ``py::class_>``, currently ``std::shared_ptr`` needs to be removed manually when switching to ``py::classh`` (#HelpAppreciated this could probably be avoided with a little bit of template metaprogramming). To the 3rd bullet point, the macro also needs to appear in other translation units with pybind11 bindings that involve Python⇄C++ conversions for `Foo`. This is the biggest inconvenience of the conservative mode. Practially, at a larger scale it is best to work with a pair of `.h` and `.cpp` files for the bindings code, with the macros in the `.h` files. Progressive mode ---------------- To work in progressive mode: - Add ``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` to the compilation commands. - Remove any ``std::shared_ptr<...>`` holders from existing ``py::class_`` instantiations (#HelpAppreciated this could probably be avoided with a little bit of template metaprogramming). - Only if custom smart-pointers are used: the `PYBIND11_TYPE_CASTER_BASE_HOLDER` macro is needed [`example `_]. Overall this is probably easier to work with than the conservative mode, but - the macro inconvenience is shifted from ``py::smart_holder`` to custom smart-pointers (but probably much more rarely needed). - it will not interoperate with other extensions built against master or stable, or extensions built in conservative mode. Transition from conservative to progressive mode ------------------------------------------------ This still has to be tried out more in practice, but in small-scale situations it may be feasible to switch directly to progressive mode in a break-fix fashion. In large-scale situations it seems more likely that an incremental approach is needed, which could mean incrementally converting ``py::class_`` to ``py::classh`` including addition of the macros, then flip the switch, and convert ``py::classh`` back to ``py:class_`` combined with removal of the macros if desired (at that point it will work equivalently either way). It may be smart to delay the final cleanup step until all third-party projects of interest have made the switch, because then the code will continue to work in either mode. Using py::classh but with fallback to classic pybind11 ------------------------------------------------------ This could be viewed as super-conservative mode, for situations in which compatibility with classic pybind11 (without smart_holder) is needed for some period of time. The main idea is to enable use of ``py::classh`` and the associated ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro while still being able to build the same code with classic pybind11. Please see tests/test_classh_mock.cpp for an example. Trampolines and std::unique_ptr ------------------------------- A pybind11 `"trampoline" `_ is a C++ helper class with virtual function overrides that transparently call back from C++ into Python. To enable safely passing a ``std::unique_ptr`` to a trampoline object between Python and C++, the trampoline class must inherit from ``py::trampoline_self_life_support``, for example: .. code-block:: cpp class PyAnimal : public Animal, public py::trampoline_self_life_support { ... }; This is the only difference compared to traditional pybind11. A fairly minimal but complete example is tests/test_class_sh_trampoline_unique_ptr.cpp. Ideas for the long-term ----------------------- The macros are clearly an inconvenience in many situations. Highly speculative: to avoid the need for the macros, a potential approach would be to combine the traditional implementation (``type_caster_base``) with the ``smart_holder_type_caster``, but this will probably be very messy and not great as a long-term solution. The ``type_caster_base`` code is very complex already. A more maintainable approach long-term could be to work out and document a smart_holder-based solution for custom smart-pointers in pybind11 version ``N``, then purge ``type_caster_base`` in version ``N+1``. #HelpAppreciated. GitHub testing of PRs against the smart_holder branch ----------------------------------------------------- PRs against the smart_holder branch need to be tested in both modes (conservative, progressive), with the only difference that ``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` is defined for progressive mode testing. Currently this is handled simply by creating a secondary PR with a one-line change in ``include/pybind11/detail/smart_holder_sfinae_hooks_only.h`` (as in e.g. `PR #2879 `_). It will be best to mark the secondary PR as Draft. Often it is convenient to reuse the same secondary PR for a series of primary PRs, simply by rebasing on a primary PR as needed: .. code-block:: bash git checkout -b sh_primary_pr # Code development ... git push # Create a PR as usual, selecting smart_holder from the branch pulldown. git checkout sh_secondary_pr git rebase -X theirs sh_primary_pr git diff # To verify that the one-line change in smart_holder_sfinae_hooks_only.h is the only diff. git push --force-with-lease # This will trigger the GitHub Actions for the progressive mode. The second time through this will only take a minute or two. Related links ============= * The smart_holder branch addresses issue `#1138 `_ and the ten issues enumerated in the `description of PR 2839 `_. * `Description of PR #2672 `_, from which the smart_holder branch was created. * Small `slide deck `_ presented in meeting with pybind11 maintainers on Feb 22, 2021. Slides 5 and 6 show performance comparisons.