pybind11/README_smart_holder.rst

254 lines
10 KiB
ReStructuredText
Raw Normal View History

==============================
pybind11 — smart_holder branch
==============================
Overview
========
- The smart_holder git branch is a strict superset of the master
branch. Everything that works on master is expected to work exactly the same
with the smart_holder branch.
- **Smart-pointer interoperability** (``std::unique_ptr``, ``std::shared_ptr``)
is implemented as an **add-on**.
- The add-on also supports
* passing a Python object back to C++ via ``std::unique_ptr``, safely
**disowning** the Python object.
* safely passing `"trampoline"
<https://pybind11.readthedocs.io/en/stable/advanced/classes.html#overriding-virtual-functions-in-python>`_
objects (objects with C++ virtual function overrides implemented in
Python) via ``std::unique_ptr`` or ``std::shared_ptr`` back to C++:
associated Python objects are automatically kept alive for the lifetime
of the smart-pointer.
- The smart_holder branch can be used in two modes:
* **Conservative mode**: ``py::class_`` works exactly as on master.
``py::classh`` uses ``py::smart_holder``.
* **Progressive mode**: ``py::class_`` uses ``py::smart_holder``
(i.e. ``py::smart_holder`` is the default holder).
What is fundamentally different?
--------------------------------
- Traditional pybind11 has the concept of "smart-pointer is holder".
Interoperability between smart-pointers is completely missing. For
example, when using ``std::shared_ptr`` as holder, ``return``-ing
a ``std::unique_ptr`` leads to undefined runtime behavior
(`#1138 <https://github.com/pybind/pybind11/issues/1138>`_). A
`systematic analysis is here <https://github.com/pybind/pybind11/pull/2672#issuecomment-748392993>`_.
- ``py::smart_holder`` has a richer concept in comparison, with well-defined
runtime behavior. The holder "knows" about both ``std::unique_ptr`` and
``std::shared_ptr`` and how they interoperate.
- Caveat (#HelpAppreciated): currently the ``smart_holder`` branch does
not have a well-lit path for including interoperability with custom
smart-pointers. It is expected to be a fairly obvious extension of the
``smart_holder`` implementation, but will depend on the exact specifications
of each custom smart-pointer type (generalizations are very likely possible).
What motivated the development of the smart_holder code?
--------------------------------------------------------
- Necessity is the mother. The bigger context is the ongoing retooling of
`PyCLIF <https://github.com/google/clif/>`_, to use pybind11 underneath
instead of directly targeting the Python C API. Essentially, the smart_holder
branch is porting established PyCLIF functionality into pybind11.
Installation
============
Currently ``git clone`` is the only option. We do not have released packages.
.. code-block:: bash
git clone --branch smart_holder https://github.com/pybind/pybind11.git
Everything else is exactly identical to using the default (master) branch.
Conservative or Progressive mode?
=================================
It depends. To a first approximation, for a stand-alone, new project, the
progressive mode will be easiest to use. For larger projects or projects
that integrate with third-party pybind11-based projects, the conservative
mode may be more practical, at least initially, although it comes with the
disadvantage of having to use the ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro.
Conservative mode
-----------------
Here is a minimal example for wrapping a C++ type with ``py::smart_holder`` as
holder:
.. code-block:: cpp
#include <pybind11/smart_holder.h>
struct Foo {};
PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)
PYBIND11_MODULE(example_bindings, m) {
namespace py = pybind11;
py::classh<Foo>(m, "Foo");
}
There are three small differences compared to traditional pybind11:
- ``#include <pybind11/smart_holder.h>`` is used instead of
``#include <pybind11/pybind11.h>``.
- ``py::classh`` is used instead of ``py::class_``.
- The ``PYBIND11_SMART_HOLDER_TYPE_CASTERS(Foo)`` macro is needed.
To the 2nd bullet point, ``py::classh<Foo>`` is simply a shortcut for
``py::class_<Foo, py::smart_holder>``. The shortcut makes it possible to
switch to using ``py::smart_holder`` without messing up the indentation of
existing code. However, when migrating code that uses ``py::class_<Foo,
std::shared_ptr<Foo>>``, currently ``std::shared_ptr<Foo>`` needs to be
removed manually when switching to ``py::classh`` (#HelpAppreciated this
could probably be avoided with a little bit of template metaprogramming).
To the 3rd bullet point, the macro also needs to appear in other translation
units with pybind11 bindings that involve Python⇄C++ conversions for
`Foo`. This is the biggest inconvenience of the conservative mode. Practially,
at a larger scale it is best to work with a pair of `.h` and `.cpp` files
for the bindings code, with the macros in the `.h` files.
Progressive mode
----------------
To work in progressive mode:
- Add ``-DPYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` to the compilation commands.
- Remove any ``std::shared_ptr<...>`` holders from existing ``py::class_``
instantiations (#HelpAppreciated this could probably be avoided with a little
bit of template metaprogramming).
- Only if custom smart-pointers are used: the
`PYBIND11_TYPE_CASTER_BASE_HOLDER` macro is needed [`example
<https://github.com/pybind/pybind11/blob/2f624af1ac8571d603df2d70cb54fc7e2e3a356a/tests/test_multiple_inheritance.cpp#L72>`_].
Overall this is probably easier to work with than the conservative mode, but
- the macro inconvenience is shifted from ``py::smart_holder`` to custom
smart-pointers (but probably much more rarely needed).
- it will not interoperate with other extensions built against master or
stable, or extensions built in conservative mode.
Transition from conservative to progressive mode
------------------------------------------------
This still has to be tried out more in practice, but in small-scale situations
it may be feasible to switch directly to progressive mode in a break-fix
fashion. In large-scale situations it seems more likely that an incremental
approach is needed, which could mean incrementally converting ``py::class_``
to ``py::classh`` including addition of the macros, then flip the switch,
and convert ``py::classh`` back to ``py:class_`` combined with removal of the
macros if desired (at that point it will work equivalently either way). It
may be smart to delay the final cleanup step until all third-party projects
of interest have made the switch, because then the code will continue to
work in either mode.
Using py::classh but with fallback to classic pybind11
------------------------------------------------------
This could be viewed as super-conservative mode, for situations in which
compatibility with classic pybind11 (without smart_holder) is needed for
some period of time. The main idea is to enable use of ``py::classh``
and the associated ``PYBIND11_SMART_HOLDER_TYPE_CASTERS`` macro while
still being able to build the same code with classic pybind11. Please see
tests/test_classh_mock.cpp for an example.
Trampolines and std::unique_ptr
-------------------------------
A pybind11 `"trampoline"
<https://pybind11.readthedocs.io/en/stable/advanced/classes.html#overriding-virtual-functions-in-python>`_
is a C++ helper class with virtual function overrides that transparently
call back from C++ into Python. To enable safely passing a ``std::unique_ptr``
to a trampoline object between Python and C++, the trampoline class must
inherit from ``py::trampoline_self_life_support``, for example:
.. code-block:: cpp
class PyAnimal : public Animal, public py::trampoline_self_life_support {
...
};
This is the only difference compared to traditional pybind11. A fairly
minimal but complete example is tests/test_class_sh_trampoline_unique_ptr.cpp.
Ideas for the long-term
-----------------------
The macros are clearly an inconvenience in many situations. Highly
speculative: to avoid the need for the macros, a potential approach would
be to combine the traditional implementation (``type_caster_base``) with
the ``smart_holder_type_caster``, but this will probably be very messy and
not great as a long-term solution. The ``type_caster_base`` code is very
complex already. A more maintainable approach long-term could be to work
out and document a smart_holder-based solution for custom smart-pointers
in pybind11 version ``N``, then purge ``type_caster_base`` in version
``N+1``. #HelpAppreciated.
GitHub testing of PRs against the smart_holder branch
-----------------------------------------------------
PRs against the smart_holder branch need to be tested in both
modes (conservative, progressive), with the only difference that
``PYBIND11_USE_SMART_HOLDER_AS_DEFAULT`` is defined for progressive mode
testing. Currently this is handled simply by creating a secondary PR with a
one-line change in ``include/pybind11/detail/smart_holder_sfinae_hooks_only.h``
(as in e.g. `PR #2879 <https://github.com/pybind/pybind11/pull/2879>`_). It
will be best to mark the secondary PR as Draft. Often it is convenient to reuse
the same secondary PR for a series of primary PRs, simply by rebasing on a
primary PR as needed:
.. code-block:: bash
git checkout -b sh_primary_pr
# Code development ...
git push # Create a PR as usual, selecting smart_holder from the branch pulldown.
git checkout sh_secondary_pr
git rebase -X theirs sh_primary_pr
git diff # To verify that the one-line change in smart_holder_sfinae_hooks_only.h is the only diff.
git push --force-with-lease # This will trigger the GitHub Actions for the progressive mode.
The second time through this will only take a minute or two.
Related links
=============
* The smart_holder branch addresses issue
`#1138 <https://github.com/pybind/pybind11/issues/1138>`_ and
the ten issues enumerated in the `description of PR 2839
<https://github.com/pybind/pybind11/pull/2839#issue-564808678>`_.
* `Description of PR #2672
<https://github.com/pybind/pybind11/pull/2672#issue-522688184>`_, from which
the smart_holder branch was created.
* Small `slide deck
<https://docs.google.com/presentation/d/1r7auDN0x-b6uf-XCvUnZz6z09raasRcCHBMVDh7PsnQ/>`_
presented in meeting with pybind11 maintainers on Feb 22, 2021. Slides 5
and 6 show performance comparisons.