From 6ba98650e22f068ceac1915e3c6995446cf304e0 Mon Sep 17 00:00:00 2001 From: Wenzel Jakob Date: Mon, 24 Oct 2016 23:48:20 +0200 Subject: [PATCH] a bit of work on the new documentation structure --- docs/advanced/cast/chrono.rst | 2 +- docs/advanced/cast/index.rst | 151 +++++--------------------------- docs/advanced/cast/overview.rst | 140 +++++++++++++++++++++++++++++ docs/advanced/functions.rst | 106 +++++++++++++++------- 4 files changed, 235 insertions(+), 164 deletions(-) create mode 100644 docs/advanced/cast/overview.rst diff --git a/docs/advanced/cast/chrono.rst b/docs/advanced/cast/chrono.rst index 1213e2cf0..6d4a5ee55 100644 --- a/docs/advanced/cast/chrono.rst +++ b/docs/advanced/cast/chrono.rst @@ -39,7 +39,7 @@ depending on the system. If it is a typedef of the system clock, python will get datetime objects, but if it is a different clock they will be timedelta objects. -Conversions Provided +Provided conversions -------------------- .. rubric:: C++ to Python diff --git a/docs/advanced/cast/index.rst b/docs/advanced/cast/index.rst index a0e4d5666..36586af5c 100644 --- a/docs/advanced/cast/index.rst +++ b/docs/advanced/cast/index.rst @@ -1,148 +1,41 @@ Type conversions ################ -There are 3 mechanisms that pybind11 uses to move data between C++ and Python. -We'll take a quick look at each one to get an overview of what's happening. +Apart from enabling cross-language function calls, a fundamental problem +that a binding tool like pybind11 must address is to provide access to +native Python types in C++ and vice versa. There are three fundamentally +different ways to do this—which approach is preferable for a particular type +depends on the situation at hand. -.. rubric:: 1. Native type in C++, wrapper in Python +1. Use a native C++ type everywhere. In this case, the type must be wrapped + using pybind11-generated bindings so that Python can interact with it. -Exposing a custom C++ type using :class:`py::class_` was covered in detail in -the :doc:`/classes` section. There, the underlying data structure is always the -original C++ class while the :class:`py::class_` wrapper provides a Python -interface. Internally, when an object like this is sent from C++ to Python, -pybind11 will just add the outer wrapper layer over the native C++ object. -Getting it back from Python is just a matter of peeling off the wrapper. +2. Use a native Python type everywhere. It will need to be wrapped so that + C++ functions can interact with it. -.. rubric:: 2. Wrapper in C++, native type in Python +3. Use a native C++ type on the C++ side and a native Python type on the + Python side. pybind11 refers to this as a *type conversion*. -This is the exact opposite situation. Now, we have a type which is native to -Python, like a ``tuple`` or a ``list``. One way to get this data into C++ is -with the :class:`py::object` family of wrappers. These are explained in more -detail in the :doc:`/advanced/pycpp/object` section. We'll just give a quick -example here: + Type conversions are the most "natural" option in the sense that native + (non-wrapped) types are used everywhere. The main downside is that a copy + of the data must be made on every Python ↔ C++ transition: this is + needed since the C++ and Python versions of the same type generally won't + have the same memory layout. -.. code-block:: cpp + pybind11 can perform many kinds of conversions automatically. An overview + is provided in the table ":ref:`conversion_table`". - void print_list(py::list my_list) { - for (auto item : my_list) - std::cout << item << " "; - } - -.. code-block:: pycon - - >>> print_list([1, 2, 3]) - 1 2 3 - -The Python ``list`` is not converted in any way -- it's just wrapped in a C++ -:class:`py::list` class. At its core it's still a Python object. Copying a -:class:`py::list` will do the usual reference-counting like in Python. -Returning the object to Python will just remove the thin wrapper. - -.. rubric:: 3. Converting between native C++ and Python types - -In the previous two cases we had a native type in one language and a wrapper in -the other. Now, we have native types on both sides and we convert between them. - -.. code-block:: cpp - - void print_vector(const std::vector &v) { - for (auto item : v) - std::cout << item << "\n"; - } - -.. code-block:: pycon - - >>> print_vector([1, 2, 3]) - 1 2 3 - -In this case, pybind11 will construct a new ``std::vector`` and copy each -element from the Python ``list``. The newly constructed object will be passed -to ``print_vector``. The same thing happens in the other direction: a new -``list`` is made to match the value returned from C++. - -Lots of these conversions are supported out of the box, as shown in the table -below. They are very convenient, but keep in mind that these conversions are -fundamentally based on copying data. This is perfectly fine for small immutable -types but it may become quite expensive for large data structures. This can be -avoided by overriding the automatic conversion with a custom wrapper (i.e. the -above-mentioned approach 1). This requires some manual effort and more details -are available in the :ref:`opaque` section. - -.. rubric:: Supported automatic conversions +The following subsections discuss the differences between these options in more +detail. The main focus in this section is on type conversions, which represent +the last case of the above list. .. toctree:: :maxdepth: 1 + overview stl functional chrono eigen custom -The following basic data types are supported out of the box (some may require -an additional extension header to be included). To pass other data structures -as arguments and return values, refer to the section on binding :ref:`classes`. - -+---------------------------------+--------------------------+-------------------------------+ -| Data type | Description | Header file | -+=================================+==========================+===============================+ -| ``int8_t``, ``uint8_t`` | 8-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int16_t``, ``uint16_t`` | 16-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int32_t``, ``uint32_t`` | 32-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int64_t``, ``uint64_t`` | 64-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``ssize_t``, ``size_t`` | Platform-dependent size | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``float``, ``double`` | Floating point types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``bool`` | Two-state Boolean type | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``char`` | Character literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``wchar_t`` | Wide character literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``const char *`` | UTF-8 string literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``const wchar_t *`` | Wide string literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::string`` | STL dynamic UTF-8 string | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::wstring`` | STL dynamic wide string | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::pair`` | Pair of two custom types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::tuple<...>`` | Arbitrary tuple of types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::reference_wrapper<...>`` | Reference type wrapper | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::complex`` | Complex numbers | :file:`pybind11/complex.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::array`` | STL static array | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::vector`` | STL dynamic array | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::list`` | STL linked list | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::map`` | STL ordered map | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::unordered_map`` | STL unordered map | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::set`` | STL ordered set | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::unordered_set`` | STL unordered set | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::function<...>`` | STL polymorphic function | :file:`pybind11/functional.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::chrono::duration<...>`` | STL time duration | :file:`pybind11/chrono.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::chrono::time_point<...>``| STL date/time | :file:`pybind11/chrono.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::Matrix<...>`` | Eigen: dense matrix | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::Map<...>`` | Eigen: mapped memory | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::SparseMatrix<...>`` | Eigen: sparse matrix | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ diff --git a/docs/advanced/cast/overview.rst b/docs/advanced/cast/overview.rst new file mode 100644 index 000000000..4df1bd103 --- /dev/null +++ b/docs/advanced/cast/overview.rst @@ -0,0 +1,140 @@ +Overview +######## + +.. rubric:: 1. Native type in C++, wrapper in Python + +Exposing a custom C++ type using :class:`py::class_` was covered in detail +in the :doc:`/classes` section. There, the underlying data structure is +always the original C++ class while the :class:`py::class_` wrapper provides +a Python interface. Internally, when an object like this is sent from C++ to +Python, pybind11 will just add the outer wrapper layer over the native C++ +object. Getting it back from Python is just a matter of peeling off the +wrapper. + +.. rubric:: 2. Wrapper in C++, native type in Python + +This is the exact opposite situation. Now, we have a type which is native to +Python, like a ``tuple`` or a ``list``. One way to get this data into C++ is +with the :class:`py::object` family of wrappers. These are explained in more +detail in the :doc:`/advanced/pycpp/object` section. We'll just give a quick +example here: + +.. code-block:: cpp + + void print_list(py::list my_list) { + for (auto item : my_list) + std::cout << item << " "; + } + +.. code-block:: pycon + + >>> print_list([1, 2, 3]) + 1 2 3 + +The Python ``list`` is not converted in any way -- it's just wrapped in a C++ +:class:`py::list` class. At its core it's still a Python object. Copying a +:class:`py::list` will do the usual reference-counting like in Python. +Returning the object to Python will just remove the thin wrapper. + +.. rubric:: 3. Converting between native C++ and Python types + +In the previous two cases we had a native type in one language and a wrapper in +the other. Now, we have native types on both sides and we convert between them. + +.. code-block:: cpp + + void print_vector(const std::vector &v) { + for (auto item : v) + std::cout << item << "\n"; + } + +.. code-block:: pycon + + >>> print_vector([1, 2, 3]) + 1 2 3 + +In this case, pybind11 will construct a new ``std::vector`` and copy each +element from the Python ``list``. The newly constructed object will be passed +to ``print_vector``. The same thing happens in the other direction: a new +``list`` is made to match the value returned from C++. + +Lots of these conversions are supported out of the box, as shown in the table +below. They are very convenient, but keep in mind that these conversions are +fundamentally based on copying data. This is perfectly fine for small immutable +types but it may become quite expensive for large data structures. This can be +avoided by overriding the automatic conversion with a custom wrapper (i.e. the +above-mentioned approach 1). This requires some manual effort and more details +are available in the :ref:`opaque` section. + +.. _conversion_table: + +List of all builtin conversions +------------------------------- + +The following basic data types are supported out of the box (some may require +an additional extension header to be included). To pass other data structures +as arguments and return values, refer to the section on binding :ref:`classes`. + ++---------------------------------+--------------------------+-------------------------------+ +| Data type | Description | Header file | ++=================================+==========================+===============================+ +| ``int8_t``, ``uint8_t`` | 8-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int16_t``, ``uint16_t`` | 16-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int32_t``, ``uint32_t`` | 32-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int64_t``, ``uint64_t`` | 64-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``ssize_t``, ``size_t`` | Platform-dependent size | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``float``, ``double`` | Floating point types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``bool`` | Two-state Boolean type | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``char`` | Character literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``wchar_t`` | Wide character literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``const char *`` | UTF-8 string literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``const wchar_t *`` | Wide string literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::string`` | STL dynamic UTF-8 string | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::wstring`` | STL dynamic wide string | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::pair`` | Pair of two custom types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::tuple<...>`` | Arbitrary tuple of types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::reference_wrapper<...>`` | Reference type wrapper | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::complex`` | Complex numbers | :file:`pybind11/complex.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::array`` | STL static array | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::vector`` | STL dynamic array | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::list`` | STL linked list | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::map`` | STL ordered map | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::unordered_map`` | STL unordered map | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::set`` | STL ordered set | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::unordered_set`` | STL unordered set | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::function<...>`` | STL polymorphic function | :file:`pybind11/functional.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::chrono::duration<...>`` | STL time duration | :file:`pybind11/chrono.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::chrono::time_point<...>``| STL date/time | :file:`pybind11/chrono.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::Matrix<...>`` | Eigen: dense matrix | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::Map<...>`` | Eigen: mapped memory | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::SparseMatrix<...>`` | Eigen: sparse matrix | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ diff --git a/docs/advanced/functions.rst b/docs/advanced/functions.rst index 5c697b12c..d9886e2a2 100644 --- a/docs/advanced/functions.rst +++ b/docs/advanced/functions.rst @@ -9,48 +9,62 @@ functions, i.e. *methods* in Python. Return value policies ===================== -Python and C++ use wildly different ways of managing the memory and lifetime of -objects managed by them. This can lead to issues when creating bindings for -functions that return a non-trivial type. Just by looking at the type -information, it is not clear whether Python should take charge of the returned -value and eventually free its resources, or if this is handled on the C++ side. -For this reason, pybind11 provides a several `return value policy` annotations -that can be passed to the :func:`module::def` and :func:`class_::def` -functions. The default policy is :enum:`return_value_policy::automatic`. +Python and C++ use fundamentally different ways of managing the memory and +lifetime of objects managed by them. This can lead to issues when creating +bindings for functions that return a non-trivial type. Just by looking at the +type information, it is not clear whether Python should take charge of the +returned value and eventually free its resources, or if this is handled on the +C++ side. For this reason, pybind11 provides a several `return value policy` +annotations that can be passed to the :func:`module::def` and +:func:`class_::def` functions. The default policy is +:enum:`return_value_policy::automatic`. -Return value policies can also be applied to properties, in which case the -arguments must be passed through the :class:`cpp_function` constructor: +Return value policies are tricky, and it's very important to get them right. +Just to illustrate what can go wrong, consider the following simple example: .. code-block:: cpp - class_(m, "MyClass") - def_property("data" - py::cpp_function(&MyClass::getData, py::return_value_policy::copy), - py::cpp_function(&MyClass::setData) - ); + /* Function declaration */ + Data *get_data() { return _data; /* (pointer to a static data structure) */ } + ... -The following table provides an overview of the available return value policies: + /* Binding code */ + m.def("get_data", &get_data); // <-- KABOOM, will cause crash when called from Python + +What's going on here? When ``get_data()`` is called from Python, the return +value (a native C++ type) must be wrapped to turn it into a usable Python type. +In this case, the default return value policy (:enum:`return_value_policy::automatic`) +causes pybind11 to assume ownership of the static ``_data`` instance. + +When Python's garbage collector eventually deletes the Python +wrapper, pybind11 will also attempt to delete the C++ instance (via ``operator +delete()``) due to the implied ownership. At this point, the entire application +will come crashing down, though errors could also be more subtle and involve +silent data corruption. + +In the above example, the policy :enum:`return_value_policy::reference` should have +been specified so that the global data instance is only *referenced* without any +implied transfer of ownership, i.e.: + +.. code-block:: cpp + + m.def("get_data", &get_data, return_value_policy::reference); + +On the other hand, this is not the right policy for many other situations, +where ignoring ownership could lead to resource leaks. +As a developer using pybind11, it's important to be familiar with the different +return value policies, including which situation calls for which one of them. +The following table provides an overview of available policies: .. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| +--------------------------------------------------+----------------------------------------------------------------------------+ | Return value policy | Description | +==================================================+============================================================================+ -| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | -| | :enum:`return_value_policy::take_ownership` when the return value is a | -| | pointer. Otherwise, it uses :enum:`return_value::move` or | -| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | -| | See below for a description of what all of these different policies do. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | -| | return value is a pointer. This is the default conversion policy for | -| | function arguments when calling Python functions manually from C++ code | -| | (i.e. via handle::operator()). You probably won't need to use this. | -+--------------------------------------------------+----------------------------------------------------------------------------+ | :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | | | ownership. Python will call the destructor and delete operator when the | | | object's reference count reaches zero. Undefined behavior ensues when the | -| | C++ side does the same. | +| | C++ side does the same, or when the data was not dynamically allocated. | +--------------------------------------------------+----------------------------------------------------------------------------+ | :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | | | This policy is comparably safe because the lifetimes of the two instances | @@ -74,6 +88,28 @@ The following table provides an overview of the available return value policies: | | return value is referenced by Python. This is the default policy for | | | property getters created via ``def_property``, ``def_readwrite``, etc. | +--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | +| | :enum:`return_value_policy::take_ownership` when the return value is a | +| | pointer. Otherwise, it uses :enum:`return_value::move` or | +| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | +| | See above for a description of what all of these different policies do. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | +| | return value is a pointer. This is the default conversion policy for | +| | function arguments when calling Python functions manually from C++ code | +| | (i.e. via handle::operator()). You probably won't need to use this. | ++--------------------------------------------------+----------------------------------------------------------------------------+ + +Return value policies can also be applied to properties, in which case the +arguments must be passed through the :class:`cpp_function` constructor: + +.. code-block:: cpp + + class_(m, "MyClass") + def_property("data" + py::cpp_function(&MyClass::getData, py::return_value_policy::copy), + py::cpp_function(&MyClass::setData) + ); .. warning:: @@ -82,12 +118,14 @@ The following table provides an overview of the available return value policies: non-determinism and segmentation faults, hence it is worth spending the time to understand all the different options in the table above. -One important aspect of the above policies is that they only apply to instances -which pybind11 has *not* seen before, in which case the policy clarifies -essential questions about the return value's lifetime and ownership. When -pybind11 knows the instance already (as identified by its type and address in -memory), it will return the existing Python object wrapper rather than creating -a new copy. +.. note:: + + One important aspect of the above policies is that they only apply to + instances which pybind11 has *not* seen before, in which case the policy + clarifies essential questions about the return value's lifetime and + ownership. When pybind11 knows the instance already (as identified by its + type and address in memory), it will return the existing Python object + wrapper rather than creating a new copy. .. note::