From 932966968362533e94c12a6156875b8273a58219 Mon Sep 17 00:00:00 2001 From: Wenzel Jakob Date: Tue, 13 Oct 2015 23:21:54 +0200 Subject: [PATCH] remainder of documentation --- docs/advanced.rst | 541 +++++++++++++++++++++++++++++++++++++++++++-- docs/basics.rst | 11 +- docs/classes.rst | 28 ++- docs/cmake.rst | 2 +- docs/conf.py | 5 + docs/intro.rst | 6 +- docs/reference.rst | 16 +- 7 files changed, 568 insertions(+), 41 deletions(-) diff --git a/docs/advanced.rst b/docs/advanced.rst index 6b7cc1ea3..962869514 100644 --- a/docs/advanced.rst +++ b/docs/advanced.rst @@ -3,23 +3,506 @@ Advanced topics ############### +For brevity, the rest of this chapter assumes that the following two lines are +present: + +.. code-block:: cpp + + #include + + namespace py = pybind; + Operator overloading ==================== +Suppose that we're given the following ``Vector2`` class with a vector addition +and scalar multiplication operation, all implemented using overloaded operators +in C++. + +.. code-block:: cpp + + class Vector2 { + public: + Vector2(float x, float y) : x(x), y(y) { } + + std::string toString() const { return "[" + std::to_string(x) + ", " + std::to_string(y) + "]"; } + + Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); } + Vector2 operator*(float value) const { return Vector2(x * value, y * value); } + Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; } + Vector2& operator*=(float v) { x *= v; y *= v; return *this; } + + friend Vector2 operator*(float f, const Vector2 &v) { return Vector2(f * v.x, f * v.y); } + + private: + float x, y; + }; + +The following snippet shows how the above operators can be conveniently exposed +to Python. + +.. code-block:: cpp + + #include + + PYBIND_PLUGIN(example) { + py::module m("example", "pybind example plugin"); + + py::class_(m, "Vector2") + .def(py::init()) + .def(py::self + py::self) + .def(py::self += py::self) + .def(py::self *= float()) + .def(float() * py::self) + .def("__repr__", &Vector2::toString); + + return m.ptr(); + } + +Note that a line like + +.. code-block:: cpp + + .def(py::self * float()) + +is really just short hand notation for + +.. code-block:: cpp + + .def("__mul__", [](const Vector2 &a, float b) { + return a * b; + }) + +This can be useful for exposing additional operators that don't exist on the +C++ side, or to perform other types of customization. + +.. note:: + + To use the more convenient ``py::self`` notation, the additional + header file :file:`pybind/operators.h` must be included. + +.. seealso:: + + The file :file:`example/example3.cpp` contains a complete example that + demonstrates how to work with overloaded operators in more detail. + +Callbacks and passing anonymous functions +========================================= + +The C++11 standard brought lambda functions and the generic polymorphic +function wrapper ``std::function<>`` to the C++ programming language, which +enable powerful new ways of working with functions. Lambda functions come in +two flavors: stateless lambda function resemble classic function pointers that +link to an anonymous piece of code, while stateful lambda functions +additionally depend on captured variables that are stored in an anonymous +*lambda closure object*. + +Here is a simple example of a C++ function that takes an arbitrary function +(stateful or stateless) with signature ``int -> int`` as an argument and runs +it with the value 10. + +.. code-block:: cpp + + int func_arg(const std::function &f) { + return f(10); + } + +The example below is more involved: it takes a function of signature ``int -> int`` +and returns another function of the same kind. The return value is a stateful +lambda function, which stores the value ``f`` in the capture object and adds 1 to +its return value upon execution. + +.. code-block:: cpp + + std::function func_ret(const std::function &f) { + return [f](int i) { + return f(i) + 1; + }; + } + +After including the extra header file :file:`pybind/functional.h`, it is almost +trivial to generate binding code for both of these functions. + +.. code-block:: cpp + + #include + + PYBIND_PLUGIN(example) { + py::module m("example", "pybind example plugin"); + + m.def("func_arg", &func_arg); + m.def("func_ret", &func_ret); + + return m.ptr(); + } + +The following interactive session shows how to call them from Python. + +.. code-block:: python + + $ python + >>> import example + >>> def square(i): + ... return i * i + ... + >>> example.func_arg(square) + 100L + >>> square_plus_1 = example.func_ret(square) + >>> square_plus_1(4) + 17L + >>> + +.. note:: + + This functionality is very useful when generating bindings for callbacks in + C++ libraries (e.g. a graphical user interface library). + + The file :file:`example/example5.cpp` contains a complete example that + demonstrates how to work with callbacks and anonymous functions in more detail. + Overriding virtual functions in Python ====================================== -Passing anonymous functions +Suppose that a C++ class or interface has a virtual function that we'd like to +to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is +given as a specific example of how one would do this with traditional C++ +code). + +.. code-block:: cpp + + class Animal { + public: + virtual ~Animal() { } + virtual std::string go(int n_times) = 0; + }; + + class Dog : public Animal { + public: + std::string go(int n_times) { + std::string result; + for (int i=0; igo(3); + } + +Normally, the binding code for these classes would look as follows: + +.. code-block:: cpp + + PYBIND_PLUGIN(example) { + py::module m("example", "pybind example plugin"); + + py::class_ animal(m, "Animal"); + animal + .def("go", &Animal::go); + + py::class_(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +However, these bindings are impossible to extend: ``Animal`` is not +constructible, and we clearly require some kind of "trampoline" that +redirects virtual calls back to Python. + +Defining a new type of ``Animal`` from within Python is possible but requires a +helper class that is defined as follows: + +.. code-block:: cpp + + class PyAnimal : public Animal { + public: + /* Inherit the constructors */ + using Animal::Animal; + + /* Trampoline (need one for each virtual function) */ + std::string go(int n_times) { + PYBIND_OVERLOAD_PURE( + std::string, /* Return type */ + Animal, /* Parent class */ + go, /* Name of function */ + n_times /* Argument(s) */ + ); + } + }; + +The macro :func:`PYBIND_OVERLOAD_PURE` should be used for pure virtual +functions, and :func:`PYBIND_OVERLOAD` should be used for functions which have +a default implementation. The binding code also needs a few minor adaptations +(highlighted): + +.. code-block:: cpp + :emphasize-lines: 4,6,7 + + PYBIND_PLUGIN(example) { + py::module m("example", "pybind example plugin"); + + py::class_ animal(m, "Animal"); + animal + .alias() + .def(py::init<>()) + .def("go", &Animal::go); + + py::class_(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +Importantly, the trampoline helper class is used as the template argument to +:class:`class_`, and a call to :func:`class_::alias` informs the binding +generator that this is merely an alias for the underlying type ``Animal``. +Following this, we are able to define a constructor as usual. + +The Python session below shows how to override ``Animal::go`` and invoke it via +a virtual method call. + +.. code-block:: cpp + + >>> from example import * + >>> d = Dog() + >>> call_go(d) + u'woof! woof! woof! ' + >>> class Cat(Animal): + ... def go(self, n_times): + ... return "meow! " * n_times + ... + >>> c = Cat() + >>> call_go(c) + u'meow! meow! meow! ' + +.. seealso:: + + The file :file:`example/example12.cpp` contains a complete example that + demonstrates how to override virtual functions using pybind11 in more + detail. + +Passing STL data structures =========================== +When including the additional header file :file:`pybind/stl.h`, conversions +between ``std::vector<>`` and ``std::map<>`` and the Python ``list`` and +``dict`` data structures are automatically enabled. The types ``std::pair<>`` +and ``std::tuple<>`` are already supported out of the box with just the core +:file:`pybind/pybind.h` header. + +.. note:: + + Arbitrary nesting of any of these types is explicitly permitted. + +.. seealso:: + + The file :file:`example/example2.cpp` contains a complete example that + demonstrates how to pass STL data types in more detail. + +Binding sequence data types, the slicing protocol, etc. +======================================================= + +Please refer to the supplemental example for details. + +.. seealso:: + + The file :file:`example/example6.cpp` contains a complete example that + shows how to bind a sequence data type, including length queries + (``__len__``), iterators (``__iter__``), the slicing protocol and other + kinds of useful operations. + Return value policies ===================== -Functions taking Python objects as arguments -============================================ +Python and C++ use wildly different ways of managing the memory and lifetime of +objects managed by them. This can lead to issues when creating bindings for +functions that return a non-trivial type. Just by looking at the type +information, it is not clear whether Python should take charge of the returned +value and eventually free its resources, or if this is handled on the C++ side. +For this reason, pybind11 provides a several `return value policy` annotations +that can be passed to the :func:`module::def` and :func:`class_::def` +functions. The default policy is :enum:`return_value_policy::automatic``. -Callbacks -========= + ++--------------------------------------------------+---------------------------------------------------------------------------+ +| Return value policy | Description | ++==================================================+===========================================================================+ +| :enum:`return_value_policy::automatic` | Automatic: copy objects returned as values and take ownership of | +| | objects returned as pointers | ++--------------------------------------------------+---------------------------------------------------------------------------+ +| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python | ++--------------------------------------------------+---------------------------------------------------------------------------+ +| :enum:`return_value_policy::take_ownership` | Reference the existing object and take ownership. Python will call | +| | the destructor and delete operator when the reference count reaches zero | ++--------------------------------------------------+---------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference` | Reference the object, but do not take ownership and defer responsibility | +| | for deleting it to C++ (dangerous when C++ code at some point decides to | +| | delete it while Python still has a nonzero reference count) | ++--------------------------------------------------+---------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference_internal` | Reference the object, but do not take ownership. The object is considered | +| | be owned by the C++ instance whose method or property returned it. The | +| | Python object will increase the reference count of this 'parent' by 1 | +| | to ensure that it won't be deallocated while Python is using the 'child' | ++--------------------------------------------------+---------------------------------------------------------------------------+ + +.. warning:: + + Code with invalid call policies might access unitialized memory and free + data structures multiple times, which can lead to hard-to-debug + non-determinism and segmentation faults, hence it is worth spending the + time to understand all the different options above. + +See below for an example that uses the +:enum:`return_value_policy::reference_internal` policy. + +.. code-block:: cpp + + class Example { + public: + Internal &get_internal() { return internal; } + private: + Internal internal; + }; + + PYBIND_PLUGIN(example) { + py::module m("example", "pybind example plugin"); + + py::class_(m, "Example") + .def(py::init<>()) + .def("get_internal", &Example::get_internal, "Return the internal data", py::return_value_policy::reference_internal) + + return m.ptr(); + } + +Implicit type conversions +========================= + +Suppose that instances of two types ``A`` and ``B`` are used in a project, and +that an ``A`` can easily be converted into a an instance of type ``B`` (examples of this +could be a fixed and an arbitrary precision number type). + +.. code-block:: cpp + + py::class_(m, "A") + /// ... members ... + + py::class_(m, "B") + .def(py::init()) + /// ... members ... + + m.def("func", + [](const B &) { /* .... */ } + ); + +To invoke the function ``func`` using a variable ``a`` containing an ``A`` +instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++ +will automatically apply an implicit type conversion, which makes it possible +to directly write ``func(a)``. + +In this situation (i.e. where ``B`` has a constructor that converts from +``A``), the following statement enables similar implicit conversions on the +Python side: + +.. code-block:: cpp + + py::implicitly_convertible(); + +Smart pointers +============== + +The binding generator for classes (:class:`class_`) takes an optional second +template type, which denotes a special *holder* type that is used to manage +references to the object. When wrapping a type named ``Type``, the default +value of this template parameter is ``std::unique_ptr``, which means that +the object is deallocated when Python's reference count goes to zero. + +It is possible to switch to other types of smart pointers, which is useful in +codebases that rely on them. For instance, the following snippet causes +``std::shared_ptr`` to be used instead. + +.. code-block:: cpp + + py::class_> obj(m, "Example"); + +.. seealso:: + + The file :file:`example/example8.cpp` contains a complete example that + demonstrates how to work with custom smart pointer types in more detail. + +.. _custom_constructors: + +Custom constructors +=================== + +The syntax for binding constructors was previously introduced, but it only +works when a constructor with the given parameters actually exists on the C++ +side. To extend this to more general cases, let's take a look at what actually +happens under the hood: the following statement + +.. code-block:: cpp + + py::class_(m, "Example") + .def(py::init()); + +is short hand notation for + +.. code-block:: cpp + + py::class_(m, "Example") + .def("__init__", + [](Example &instance, int arg) { + new (&instance) Example(arg); + } + ); + +In other words, :func:`init` creates an anonymous function that invokes an +in-place constructor. Memory allocation etc. is already take care of beforehand +within pybind11. + +Catching and throwing exceptions +================================ + +When C++ code invoked from Python throws an ``std::exception``, it is +automatically converted into a Python ``Exception``. pybind11 defines multiple +special exception classes that will map to different types of Python +exceptions: + ++----------------------------+------------------------------+ +| C++ exception type | Python exception type | ++============================+==============================+ +| :class:`std::exception` | ``Exception`` | ++----------------------------+------------------------------+ +| :class:`stop_iteration` | ``StopIteration`` (used to | +| | implement custom iterators) | ++----------------------------+------------------------------+ +| :class:`index_error` | ``IndexError`` (used to | +| | indicate out of bounds | +| | accesses in ``__getitem__``, | +| | ``__setitem__``, etc.) | ++----------------------------+------------------------------+ +| :class:`error_already_set` | Indicates that the Python | +| | exception flag has already | +| | been initialized. | ++----------------------------+------------------------------+ + +When a Python function invoked from C++ throws an exception, it is converted +into a C++ exception of type :class:`error_already_set` whose string payload +contains a textual summary. + +There is also a special exception :class:`cast_error` that is thrown by +:func:`handle::call` when the input arguments cannot be converted to Python +objects. Buffer protocol =============== @@ -114,6 +597,11 @@ objects (e.g. a NumPy matrix). } }); +.. seealso:: + + The file :file:`example/example7.cpp` contains a complete example that + demonstrates using the buffer protocol with pybind11 in more detail. + NumPy support ============= @@ -122,13 +610,13 @@ restrict the function so that it only accepts NumPy arrays (rather than any type of Python object satisfying the buffer object protocol). In many situations, we want to define a function which only accepts a NumPy -array of a certain data type. This is possible via the ``py::array_dtype`` +array of a certain data type. This is possible via the ``py::array_t`` template. For instance, the following function requires the argument to be a dense array of doubles in C-style ordering. .. code-block:: cpp - void f(py::array_dtype array); + void f(py::array_t array); When it is invoked with a different type (e.g. an integer), the binding code will attempt to cast the input into a NumPy array of the requested type. @@ -181,22 +669,41 @@ This can be done with a stateful Lambda closure: // Vectorize a lambda function with a capture object (e.g. to exclude some arguments from the vectorization) m.def("vectorized_func", - [](py::array_dtype x, py::array_dtype y, my_custom_type *z) { + [](py::array_t x, py::array_t y, my_custom_type *z) { auto stateful_closure = [z](int x, float y) { return my_func(x, y, z); }; return py::vectorize(stateful_closure)(x, y); } ); -Throwing exceptions -=================== +.. seealso:: -STL data structures -=================== + The file :file:`example/example10.cpp` contains a complete example that + demonstrates using :func:`vectorize` in more detail. -Smart pointers -============== +Functions taking Python objects as arguments +============================================ -.. _custom_constructors: +pybind11 exposes all major Python types using thin C++ wrapper classes. These +wrapper classes can also be used as parameters of functions in bindings, which +makes it possible to directly work with native Python types on the C++ side. +For instance, the following statement iterates over a Python ``dict``: + +.. code-block:: cpp + + void print_dict(py::dict dict) { + /* Easily interact with Python types */ + for (auto item : dict) + std::cout << "key=" << item.first << ", " + << "value=" << item.second << std::endl; + } + +Available types include :class:`handle`, :class:`object`, :class:`bool_`, +:class:`int_`, :class:`float_`, :class:`str`, :class:`tuple`, :class:`list`, +:class:`dict`, :class:`slice`, :class:`capsule`, :class:`function`, +:class:`buffer`, :class:`array`, and :class:`array_t`. + +.. seealso:: + + The file :file:`example/example2.cpp` contains a complete example that + demonstrates passing native Python types in more detail. -Custom constructors -=================== diff --git a/docs/basics.rst b/docs/basics.rst index eb37da54f..a4e727190 100644 --- a/docs/basics.rst +++ b/docs/basics.rst @@ -80,14 +80,14 @@ a file named :file:`example.cpp` with the following contents: .. code-block:: cpp #include - + int add(int i, int j) { return i + j; } namespace py = pybind; - PYTHON_PLUGIN(example) { + PYBIND_PLUGIN(example) { py::module m("example", "pybind example plugin"); m.def("add", &add, "A function which adds two numbers"); @@ -95,7 +95,7 @@ a file named :file:`example.cpp` with the following contents: return m.ptr(); } -The :func:`PYTHON_PLUGIN` macro creates a function that will be called when an +The :func:`PYBIND_PLUGIN` macro creates a function that will be called when an ``import`` statement is issued from within Python. The next line creates a module named ``example`` (with the supplied docstring). The method :func:`module::def` generates binding code that exposes the @@ -130,13 +130,14 @@ shows how to load and execute the example. .. code-block:: python - % python + $ python Python 2.7.10 (default, Aug 22 2015, 20:33:39) [GCC 4.2.1 Compatible Apple LLVM 7.0.0 (clang-700.0.59.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import example >>> example.add(1, 2) 3L + >>> .. _keyword_args: @@ -219,7 +220,7 @@ Supported data types The following basic data types are supported out of the box (some may require an additional extension header to be included). To pass other data structures -as arguments and return values, refer to the section on :ref:`classes`. +as arguments and return values, refer to the section on binding :ref:`classes`. +------------------------+--------------------------+---------------------+ | Data type | Description | Header file | diff --git a/docs/classes.rst b/docs/classes.rst index c141c1610..9f3b56cb6 100644 --- a/docs/classes.rst +++ b/docs/classes.rst @@ -24,10 +24,10 @@ The binding code for ``Pet`` looks as follows: .. code-block:: cpp #include - + namespace py = pybind; - PYTHON_PLUGIN(example) { + PYBIND_PLUGIN(example) { py::module m("example", "pybind example plugin"); py::class_(m, "Pet") @@ -140,7 +140,8 @@ that can only be accessed via setters and getters. In this case, the method :func:`class_::def_property` (:func:`class_::def_property_readonly` for read-only data) can be used to -provide an interface that is indistinguishable from within Python: +provide a field-like interface within Python that will transparently call +the setter and getter functions: .. code-block:: cpp @@ -190,7 +191,7 @@ class. Instances then expose fields and methods of both types: -.. code-block:: python +.. code-block:: python >>> p = example.Dog('Molly') >>> p.name @@ -249,13 +250,18 @@ The overload signatures are also visible in the method's docstring: | 2. Signature : (Pet, str) -> None | | Set the pet's name - | + +.. note:: + + To define multiple overloaded constructors, simply declare one after the + other using the ``.def(py::init<...>())`` syntax. The existing machinery + for specifying keyword and default arguments also works. Enumerations and internal types =============================== -Let's now suppose that the example class also contains an internal enumeration -type. +Let's now suppose that the example class contains an internal enumeration type, +e.g.: .. code-block:: cpp @@ -288,9 +294,9 @@ The binding code for this example looks as follows: To ensure that the ``Kind`` type is created within the scope of ``Pet``, the ``pet`` :class:`class_` instance must be supplied to the :class:`enum_`. -constructor. The :func:`enum_::export_values` function ensures that the enum -entries are exported into the parent scope; skip this call for new C++11-style -strongly typed enums. +constructor. The :func:`enum_::export_values` function exports the enum entries +into the parent scope, which should be skipped for newer C++11-style strongly +typed enums. .. code-block:: python @@ -301,4 +307,4 @@ strongly typed enums. 1L -.. [#f1] (those with an empty pair of brackets ``[]`` as the capture object) +.. [#f1] Stateless closures are those with an empty pair of brackets ``[]`` as the capture object. diff --git a/docs/cmake.rst b/docs/cmake.rst index a5d074011..a3277c085 100644 --- a/docs/cmake.rst +++ b/docs/cmake.rst @@ -78,7 +78,7 @@ and that the pybind11 repository is located in a subdirectory named :file:`pybin # into Blender or Maya later on, this will cause segfaults when multiple # conflicting Python instances are active at the same time. - # Windows is not affected by this issue since it handles DLL imports + # Windows is not affected by this issue since it handles DLL imports # differently. The solution for Linux and Mac OS is simple: we just don't # link against the Python library. The resulting shared library will have # missing symbols, but that's perfectly fine -- they will be resolved at diff --git a/docs/conf.py b/docs/conf.py index f883fef6e..74009a1e2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -116,6 +116,11 @@ if not on_rtd: # only import and set the theme if we're building docs locally import sphinx_rtd_theme html_theme = 'sphinx_rtd_theme' html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] + html_context = { + 'css_files': [ + '_static/theme_overrides.css', + ] + } #import alabaster diff --git a/docs/intro.rst b/docs/intro.rst index 3fa362676..738d937ad 100644 --- a/docs/intro.rst +++ b/docs/intro.rst @@ -4,7 +4,7 @@ About this project and vice versa, mainly to create Python bindings of existing C++ code. Its goals and syntax are similar to the excellent `Boost.Python`_ library by David Abrahams: to minimize boilerplate code in traditional extension modules by -inferring type information using compile-time introspection. +inferring type information using compile-time introspection. .. _Boost.Python: http://www.boost.org/doc/libs/release/libs/python/doc/index.html @@ -21,7 +21,9 @@ everything stripped away that isn't relevant for binding generation. The whole codebase requires less than 3000 lines of code and only depends on Python (2.7 or 3.x) and the C++ standard library. This compact implementation was possible thanks to some of the new C++11 language features (tuples, lambda functions and -variadic templates). +variadic templates). Since its creation, this library has grown beyond +Boost.Python in many ways, leading to dramatically simpler binding code in many +common situations. Core features ************* diff --git a/docs/reference.rst b/docs/reference.rst index 8fbb39624..5fade60a1 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -1,12 +1,18 @@ .. _reference: +.. warning:: + + Please be advised that the reference documentation discussing pybind11 + internals is currently incomplete. Please refer to the previous sections + and the pybind header files for the nitty gritty details. + Reference ######### Macros ====== -.. function:: PYTHON_PLUGIN(const char *name) +.. function:: PYBIND_PLUGIN(const char *name) This macro creates the entry point that will be invoked when the Python interpreter imports a plugin library. Please create a @@ -15,7 +21,7 @@ Macros .. code-block:: cpp - PYTHON_PLUGIN(example) { + PYBIND_PLUGIN(example) { pybind::module m("example", "pybind example plugin"); /// Set up bindings here return m.ptr(); @@ -37,7 +43,7 @@ Without reference counting various Python API functions. .. seealso:: - + The :class:`object` class inherits from :class:`handle` and adds automatic reference counting features. @@ -122,7 +128,7 @@ Without reference counting Assuming the Python object is a function or implements the ``__call__`` protocol, ``call()`` invokes the underlying function, passing an arbitrary set of parameters. The result is returned as a :class:`object` and may need - to be converted back into a Python object using :func:`template handle::cast`. + to be converted back into a Python object using :func:`handle::cast`. When some of the arguments cannot be converted to Python objects, the function will throw a :class:`cast_error` exception. When the Python @@ -206,7 +212,7 @@ Passing extra arguments to the def function .. class:: template arg_t : public arg Represents a named argument with a default value - + .. class:: sibling Used to specify a handle to an existing sibling function; used internally