From 67b52d808e973bbdaed2d371df7ee74b173eb7ae Mon Sep 17 00:00:00 2001 From: Dean Moldovan Date: Sun, 16 Oct 2016 19:12:43 +0200 Subject: [PATCH] Reorganize documentation --- docs/advanced.rst | 2276 ----------------------------- docs/advanced/cast/chrono.rst | 81 + docs/advanced/cast/custom.rst | 79 + docs/advanced/cast/eigen.rst | 50 + docs/advanced/cast/functional.rst | 113 ++ docs/advanced/cast/index.rst | 79 + docs/advanced/cast/stl.rst | 154 ++ docs/advanced/classes.rst | 634 ++++++++ docs/advanced/exceptions.rst | 142 ++ docs/advanced/functions.rst | 263 ++++ docs/advanced/misc.rst | 187 +++ docs/advanced/pycpp/index.rst | 13 + docs/advanced/pycpp/numpy.rst | 299 ++++ docs/advanced/pycpp/object.rst | 96 ++ docs/advanced/pycpp/utilities.rst | 57 + docs/advanced/smart_ptrs.rst | 149 ++ docs/basics.rst | 116 +- docs/index.rst | 29 +- docs/reference.rst | 4 +- 19 files changed, 2468 insertions(+), 2353 deletions(-) delete mode 100644 docs/advanced.rst create mode 100644 docs/advanced/cast/chrono.rst create mode 100644 docs/advanced/cast/custom.rst create mode 100644 docs/advanced/cast/eigen.rst create mode 100644 docs/advanced/cast/functional.rst create mode 100644 docs/advanced/cast/index.rst create mode 100644 docs/advanced/cast/stl.rst create mode 100644 docs/advanced/classes.rst create mode 100644 docs/advanced/exceptions.rst create mode 100644 docs/advanced/functions.rst create mode 100644 docs/advanced/misc.rst create mode 100644 docs/advanced/pycpp/index.rst create mode 100644 docs/advanced/pycpp/numpy.rst create mode 100644 docs/advanced/pycpp/object.rst create mode 100644 docs/advanced/pycpp/utilities.rst create mode 100644 docs/advanced/smart_ptrs.rst diff --git a/docs/advanced.rst b/docs/advanced.rst deleted file mode 100644 index 47595f63f..000000000 --- a/docs/advanced.rst +++ /dev/null @@ -1,2276 +0,0 @@ -.. _advanced: - -Advanced topics -############### - -For brevity, the rest of this chapter assumes that the following two lines are -present: - -.. code-block:: cpp - - #include - - namespace py = pybind11; - -Exporting constants and mutable objects -======================================= - -To expose a C++ constant, use the ``attr`` function to register it in a module -as shown below. The ``int_`` class is one of many small wrapper objects defined -in ``pybind11/pytypes.h``. General objects (including integers) can also be -converted using the function ``cast``. - -.. code-block:: cpp - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - m.attr("MY_CONSTANT") = py::int_(123); - m.attr("MY_CONSTANT_2") = py::cast(new MyObject()); - } - -Operator overloading -==================== - -Suppose that we're given the following ``Vector2`` class with a vector addition -and scalar multiplication operation, all implemented using overloaded operators -in C++. - -.. code-block:: cpp - - class Vector2 { - public: - Vector2(float x, float y) : x(x), y(y) { } - - Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); } - Vector2 operator*(float value) const { return Vector2(x * value, y * value); } - Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; } - Vector2& operator*=(float v) { x *= v; y *= v; return *this; } - - friend Vector2 operator*(float f, const Vector2 &v) { - return Vector2(f * v.x, f * v.y); - } - - std::string toString() const { - return "[" + std::to_string(x) + ", " + std::to_string(y) + "]"; - } - private: - float x, y; - }; - -The following snippet shows how the above operators can be conveniently exposed -to Python. - -.. code-block:: cpp - - #include - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - - py::class_(m, "Vector2") - .def(py::init()) - .def(py::self + py::self) - .def(py::self += py::self) - .def(py::self *= float()) - .def(float() * py::self) - .def("__repr__", &Vector2::toString); - - return m.ptr(); - } - -Note that a line like - -.. code-block:: cpp - - .def(py::self * float()) - -is really just short hand notation for - -.. code-block:: cpp - - .def("__mul__", [](const Vector2 &a, float b) { - return a * b; - }, py::is_operator()) - -This can be useful for exposing additional operators that don't exist on the -C++ side, or to perform other types of customization. The ``py::is_operator`` -flag marker is needed to inform pybind11 that this is an operator, which -returns ``NotImplemented`` when invoked with incompatible arguments rather than -throwing a type error. - -.. note:: - - To use the more convenient ``py::self`` notation, the additional - header file :file:`pybind11/operators.h` must be included. - -.. seealso:: - - The file :file:`tests/test_operator_overloading.cpp` contains a - complete example that demonstrates how to work with overloaded operators in - more detail. - -Callbacks and passing anonymous functions -========================================= - -The C++11 standard brought lambda functions and the generic polymorphic -function wrapper ``std::function<>`` to the C++ programming language, which -enable powerful new ways of working with functions. Lambda functions come in -two flavors: stateless lambda function resemble classic function pointers that -link to an anonymous piece of code, while stateful lambda functions -additionally depend on captured variables that are stored in an anonymous -*lambda closure object*. - -Here is a simple example of a C++ function that takes an arbitrary function -(stateful or stateless) with signature ``int -> int`` as an argument and runs -it with the value 10. - -.. code-block:: cpp - - int func_arg(const std::function &f) { - return f(10); - } - -The example below is more involved: it takes a function of signature ``int -> int`` -and returns another function of the same kind. The return value is a stateful -lambda function, which stores the value ``f`` in the capture object and adds 1 to -its return value upon execution. - -.. code-block:: cpp - - std::function func_ret(const std::function &f) { - return [f](int i) { - return f(i) + 1; - }; - } - -This example demonstrates using python named parameters in C++ callbacks which -requires using ``py::cpp_function`` as a wrapper. Usage is similar to defining -methods of classes: - -.. code-block:: cpp - - py::cpp_function func_cpp() { - return py::cpp_function([](int i) { return i+1; }, - py::arg("number")); - } - -After including the extra header file :file:`pybind11/functional.h`, it is almost -trivial to generate binding code for all of these functions. - -.. code-block:: cpp - - #include - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - - m.def("func_arg", &func_arg); - m.def("func_ret", &func_ret); - m.def("func_cpp", &func_cpp); - - return m.ptr(); - } - -The following interactive session shows how to call them from Python. - -.. code-block:: pycon - - $ python - >>> import example - >>> def square(i): - ... return i * i - ... - >>> example.func_arg(square) - 100L - >>> square_plus_1 = example.func_ret(square) - >>> square_plus_1(4) - 17L - >>> plus_1 = func_cpp() - >>> plus_1(number=43) - 44L - -.. warning:: - - Keep in mind that passing a function from C++ to Python (or vice versa) - will instantiate a piece of wrapper code that translates function - invocations between the two languages. Naturally, this translation - increases the computational cost of each function call somewhat. A - problematic situation can arise when a function is copied back and forth - between Python and C++ many times in a row, in which case the underlying - wrappers will accumulate correspondingly. The resulting long sequence of - C++ -> Python -> C++ -> ... roundtrips can significantly decrease - performance. - - There is one exception: pybind11 detects case where a stateless function - (i.e. a function pointer or a lambda function without captured variables) - is passed as an argument to another C++ function exposed in Python. In this - case, there is no overhead. Pybind11 will extract the underlying C++ - function pointer from the wrapped function to sidestep a potential C++ -> - Python -> C++ roundtrip. This is demonstrated in :file:`tests/test_callbacks.cpp`. - -.. note:: - - This functionality is very useful when generating bindings for callbacks in - C++ libraries (e.g. GUI libraries, asynchronous networking libraries, etc.). - - The file :file:`tests/test_callbacks.cpp` contains a complete example - that demonstrates how to work with callbacks and anonymous functions in - more detail. - -.. _overriding_virtuals: - -Overriding virtual functions in Python -====================================== - -Suppose that a C++ class or interface has a virtual function that we'd like to -to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is -given as a specific example of how one would do this with traditional C++ -code). - -.. code-block:: cpp - - class Animal { - public: - virtual ~Animal() { } - virtual std::string go(int n_times) = 0; - }; - - class Dog : public Animal { - public: - std::string go(int n_times) override { - std::string result; - for (int i=0; igo(3); - } - -Normally, the binding code for these classes would look as follows: - -.. code-block:: cpp - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - - py::class_ animal(m, "Animal"); - animal - .def("go", &Animal::go); - - py::class_(m, "Dog", animal) - .def(py::init<>()); - - m.def("call_go", &call_go); - - return m.ptr(); - } - -However, these bindings are impossible to extend: ``Animal`` is not -constructible, and we clearly require some kind of "trampoline" that -redirects virtual calls back to Python. - -Defining a new type of ``Animal`` from within Python is possible but requires a -helper class that is defined as follows: - -.. code-block:: cpp - - class PyAnimal : public Animal { - public: - /* Inherit the constructors */ - using Animal::Animal; - - /* Trampoline (need one for each virtual function) */ - std::string go(int n_times) override { - PYBIND11_OVERLOAD_PURE( - std::string, /* Return type */ - Animal, /* Parent class */ - go, /* Name of function */ - n_times /* Argument(s) */ - ); - } - }; - -The macro :func:`PYBIND11_OVERLOAD_PURE` should be used for pure virtual -functions, and :func:`PYBIND11_OVERLOAD` should be used for functions which have -a default implementation. There are also two alternate macros -:func:`PYBIND11_OVERLOAD_PURE_NAME` and :func:`PYBIND11_OVERLOAD_NAME` which -take a string-valued name argument between the *Parent class* and *Name of the -function* slots. This is useful when the C++ and Python versions of the -function have different names, e.g. ``operator()`` vs ``__call__``. - -The binding code also needs a few minor adaptations (highlighted): - -.. code-block:: cpp - :emphasize-lines: 4,6,7 - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - - py::class_ animal(m, "Animal"); - animal - .def(py::init<>()) - .def("go", &Animal::go); - - py::class_(m, "Dog", animal) - .def(py::init<>()); - - m.def("call_go", &call_go); - - return m.ptr(); - } - -Importantly, pybind11 is made aware of the trampoline helper class by -specifying it as an extra template argument to :class:`class_`. (This can also -be combined with other template arguments such as a custom holder type; the -order of template types does not matter). Following this, we are able to -define a constructor as usual. - -Note, however, that the above is sufficient for allowing python classes to -extend ``Animal``, but not ``Dog``: see ref:`virtual_and_inheritance` for the -necessary steps required to providing proper overload support for inherited -classes. - -The Python session below shows how to override ``Animal::go`` and invoke it via -a virtual method call. - -.. code-block:: pycon - - >>> from example import * - >>> d = Dog() - >>> call_go(d) - u'woof! woof! woof! ' - >>> class Cat(Animal): - ... def go(self, n_times): - ... return "meow! " * n_times - ... - >>> c = Cat() - >>> call_go(c) - u'meow! meow! meow! ' - -Please take a look at the :ref:`macro_notes` before using this feature. - -.. note:: - - When the overridden type returns a reference or pointer to a type that - pybind11 converts from Python (for example, numeric values, std::string, - and other built-in value-converting types), there are some limitations to - be aware of: - - - because in these cases there is no C++ variable to reference (the value - is stored in the referenced Python variable), pybind11 provides one in - the PYBIND11_OVERLOAD macros (when needed) with static storage duration. - Note that this means that invoking the overloaded method on *any* - instance will change the referenced value stored in *all* instances of - that type. - - - Attempts to modify a non-const reference will not have the desired - effect: it will change only the static cache variable, but this change - will not propagate to underlying Python instance, and the change will be - replaced the next time the overload is invoked. - -.. seealso:: - - The file :file:`tests/test_virtual_functions.cpp` contains a complete - example that demonstrates how to override virtual functions using pybind11 - in more detail. - -.. _virtual_and_inheritance: - -Combining virtual functions and inheritance -=========================================== - -When combining virtual methods with inheritance, you need to be sure to provide -an override for each method for which you want to allow overrides from derived -python classes. For example, suppose we extend the above ``Animal``/``Dog`` -example as follows: - -.. code-block:: cpp - - class Animal { - public: - virtual std::string go(int n_times) = 0; - virtual std::string name() { return "unknown"; } - }; - class Dog : public class Animal { - public: - std::string go(int n_times) override { - std::string result; - for (int i=0; i class PyAnimal : public AnimalBase { - using AnimalBase::AnimalBase; // Inherit constructors - std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, AnimalBase, go, n_times); } - std::string name() override { PYBIND11_OVERLOAD(std::string, AnimalBase, name, ); } - }; - template class PyDog : public PyAnimal { - using PyAnimal::PyAnimal; // Inherit constructors - // Override PyAnimal's pure virtual go() with a non-pure one: - std::string go(int n_times) override { PYBIND11_OVERLOAD(std::string, DogBase, go, n_times); } - std::string bark() override { PYBIND11_OVERLOAD(std::string, DogBase, bark, ); } - }; - -This technique has the advantage of requiring just one trampoline method to be -declared per virtual method and pure virtual method override. It does, -however, require the compiler to generate at least as many methods (and -possibly more, if both pure virtual and overridden pure virtual methods are -exposed, as above). - -The classes are then registered with pybind11 using: - -.. code-block:: cpp - - py::class_> animal(m, "Animal"); - py::class_> dog(m, "Dog"); - py::class_> husky(m, "Husky"); - // ... add animal, dog, husky definitions - -Note that ``Husky`` did not require a dedicated trampoline template class at -all, since it neither declares any new virtual methods nor provides any pure -virtual method implementations. - -With either the repeated-virtuals or templated trampoline methods in place, you -can now create a python class that inherits from ``Dog``: - -.. code-block:: python - - class ShihTzu(Dog): - def bark(self): - return "yip!" - -.. seealso:: - - See the file :file:`tests/test_virtual_functions.cpp` for complete examples - using both the duplication and templated trampoline approaches. - -Extended trampoline class functionality -======================================= - -The trampoline classes described in the previous sections are, by default, only -initialized when needed. More specifically, they are initialized when a python -class actually inherits from a registered type (instead of merely creating an -instance of the registered type), or when a registered constructor is only -valid for the trampoline class but not the registered class. This is primarily -for performance reasons: when the trampoline class is not needed for anything -except virtual method dispatching, not initializing the trampoline class -improves performance by avoiding needing to do a run-time check to see if the -inheriting python instance has an overloaded method. - -Sometimes, however, it is useful to always initialize a trampoline class as an -intermediate class that does more than just handle virtual method dispatching. -For example, such a class might perform extra class initialization, extra -destruction operations, and might define new members and methods to enable a -more python-like interface to a class. - -In order to tell pybind11 that it should *always* initialize the trampoline -class when creating new instances of a type, the class constructors should be -declared using ``py::init_alias()`` instead of the usual -``py::init()``. This forces construction via the trampoline class, -ensuring member initialization and (eventual) destruction. - -.. seealso:: - - See the file :file:`tests/test_alias_initialization.cpp` for complete examples - showing both normal and forced trampoline instantiation. - -.. _macro_notes: - -General notes regarding convenience macros -========================================== - -pybind11 provides a few convenience macros such as -:func:`PYBIND11_MAKE_OPAQUE` and :func:`PYBIND11_DECLARE_HOLDER_TYPE`, and -``PYBIND11_OVERLOAD_*``. Since these are "just" macros that are evaluated -in the preprocessor (which has no concept of types), they *will* get confused -by commas in a template argument such as ``PYBIND11_OVERLOAD(MyReturnValue, myFunc)``. In this case, the preprocessor assumes that the comma indicates -the beginning of the next parameter. Use a ``typedef`` to bind the template to -another name and use it in the macro to avoid this problem. - - -Global Interpreter Lock (GIL) -============================= - -The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be -used to acquire and release the global interpreter lock in the body of a C++ -function call. In this way, long-running C++ code can be parallelized using -multiple Python threads. Taking the previous section as an example, this could -be realized as follows (important changes highlighted): - -.. code-block:: cpp - :emphasize-lines: 8,9,33,34 - - class PyAnimal : public Animal { - public: - /* Inherit the constructors */ - using Animal::Animal; - - /* Trampoline (need one for each virtual function) */ - std::string go(int n_times) { - /* Acquire GIL before calling Python code */ - py::gil_scoped_acquire acquire; - - PYBIND11_OVERLOAD_PURE( - std::string, /* Return type */ - Animal, /* Parent class */ - go, /* Name of function */ - n_times /* Argument(s) */ - ); - } - }; - - PYBIND11_PLUGIN(example) { - py::module m("example", "pybind11 example plugin"); - - py::class_ animal(m, "Animal"); - animal - .def(py::init<>()) - .def("go", &Animal::go); - - py::class_(m, "Dog", animal) - .def(py::init<>()); - - m.def("call_go", [](Animal *animal) -> std::string { - /* Release GIL before calling into (potentially long-running) C++ code */ - py::gil_scoped_release release; - return call_go(animal); - }); - - return m.ptr(); - } - -Passing STL data structures -=========================== - -When including the additional header file :file:`pybind11/stl.h`, conversions -between ``std::vector<>``, ``std::list<>``, ``std::set<>``, and ``std::map<>`` -and the Python ``list``, ``set`` and ``dict`` data structures are automatically -enabled. The types ``std::pair<>`` and ``std::tuple<>`` are already supported -out of the box with just the core :file:`pybind11/pybind11.h` header. - -The major downside of these implicit conversions is that containers must be -converted (i.e. copied) on every Python->C++ and C++->Python transition, which -can have implications on the program semantics and performance. Please read the -next sections for more details and alternative approaches that avoid this. - -.. note:: - - Arbitrary nesting of any of these types is possible. - -.. seealso:: - - The file :file:`tests/test_python_types.cpp` contains a complete - example that demonstrates how to pass STL data types in more detail. - -.. _opaque: - -Treating STL data structures as opaque objects -============================================== - -pybind11 heavily relies on a template matching mechanism to convert parameters -and return values that are constructed from STL data types such as vectors, -linked lists, hash tables, etc. This even works in a recursive manner, for -instance to deal with lists of hash maps of pairs of elementary and custom -types, etc. - -However, a fundamental limitation of this approach is that internal conversions -between Python and C++ types involve a copy operation that prevents -pass-by-reference semantics. What does this mean? - -Suppose we bind the following function - -.. code-block:: cpp - - void append_1(std::vector &v) { - v.push_back(1); - } - -and call it from Python, the following happens: - -.. code-block:: pycon - - >>> v = [5, 6] - >>> append_1(v) - >>> print(v) - [5, 6] - -As you can see, when passing STL data structures by reference, modifications -are not propagated back the Python side. A similar situation arises when -exposing STL data structures using the ``def_readwrite`` or ``def_readonly`` -functions: - -.. code-block:: cpp - - /* ... definition ... */ - - class MyClass { - std::vector contents; - }; - - /* ... binding code ... */ - - py::class_(m, "MyClass") - .def(py::init<>) - .def_readwrite("contents", &MyClass::contents); - -In this case, properties can be read and written in their entirety. However, an -``append`` operation involving such a list type has no effect: - -.. code-block:: pycon - - >>> m = MyClass() - >>> m.contents = [5, 6] - >>> print(m.contents) - [5, 6] - >>> m.contents.append(7) - >>> print(m.contents) - [5, 6] - -Finally, the involved copy operations can be costly when dealing with very -large lists. To deal with all of the above situations, pybind11 provides a -macro named ``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based -conversion machinery of types, thus rendering them *opaque*. The contents of -opaque objects are never inspected or extracted, hence they *can* be passed by -reference. For instance, to turn ``std::vector`` into an opaque type, add -the declaration - -.. code-block:: cpp - - PYBIND11_MAKE_OPAQUE(std::vector); - -before any binding code (e.g. invocations to ``class_::def()``, etc.). This -macro must be specified at the top level (and outside of any namespaces), since -it instantiates a partial template overload. If your binding code consists of -multiple compilation units, it must be present in every file preceding any -usage of ``std::vector``. Opaque types must also have a corresponding -``class_`` declaration to associate them with a name in Python, and to define a -set of available operations, e.g.: - -.. code-block:: cpp - - py::class_>(m, "IntVector") - .def(py::init<>()) - .def("clear", &std::vector::clear) - .def("pop_back", &std::vector::pop_back) - .def("__len__", [](const std::vector &v) { return v.size(); }) - .def("__iter__", [](std::vector &v) { - return py::make_iterator(v.begin(), v.end()); - }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */ - // .... - -The ability to expose STL containers as native Python objects is a fairly -common request, hence pybind11 also provides an optional header file named -:file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try -to match the behavior of their native Python counterparts as much as possible. - -The following example showcases usage of :file:`pybind11/stl_bind.h`: - -.. code-block:: cpp - - // Don't forget this - #include - - PYBIND11_MAKE_OPAQUE(std::vector); - PYBIND11_MAKE_OPAQUE(std::map); - - // ... - - // later in binding code: - py::bind_vector>(m, "VectorInt"); - py::bind_map>(m, "MapStringDouble"); - -Please take a look at the :ref:`macro_notes` before using the -``PYBIND11_MAKE_OPAQUE`` macro. - -.. seealso:: - - The file :file:`tests/test_opaque_types.cpp` contains a complete - example that demonstrates how to create and expose opaque types using - pybind11 in more detail. - - The file :file:`tests/test_stl_binders.cpp` shows how to use the - convenience STL container wrappers. - - -Binding sequence data types, iterators, the slicing protocol, etc. -================================================================== - -Please refer to the supplemental example for details. - -.. seealso:: - - The file :file:`tests/test_sequences_and_iterators.cpp` contains a - complete example that shows how to bind a sequence data type, including - length queries (``__len__``), iterators (``__iter__``), the slicing - protocol and other kinds of useful operations. - -C++11 chrono datatypes -====================== - -When including the additional header file :file:`pybind11/chrono.h` conversions -from C++11 chrono datatypes to python datetime objects are automatically enabled. -This header also enables conversions of python floats (often from sources such -as `time.monotonic()`, `time.perf_counter()` and `time.process_time()`) into -durations. - -An overview of clocks in C++11 ------------------------------- - -A point of confusion when using these conversions is the differences between -clocks provided in C++11. There are three clock types defined by the C++11 -standard and users can define their own if needed. Each of these clocks have -different properties and when converting to and from python will give different -results. - -The first clock defined by the standard is ``std::chrono::system_clock``. This -clock measures the current date and time. However, this clock changes with to -updates to the operating system time. For example, if your time is synchronised -with a time server this clock will change. This makes this clock a poor choice -for timing purposes but good for measuring the wall time. - -The second clock defined in the standard is ``std::chrono::steady_clock``. -This clock ticks at a steady rate and is never adjusted. This makes it excellent -for timing purposes, however the value in this clock does not correspond to the -current date and time. Often this clock will be the amount of time your system -has been on, although it does not have to be. This clock will never be the same -clock as the system clock as the system clock can change but steady clocks -cannot. - -The third clock defined in the standard is ``std::chrono::high_resolution_clock``. -This clock is the clock that has the highest resolution out of the clocks in the -system. It is normally a typedef to either the system clock or the steady clock -but can be its own independent clock. This is important as when using these -conversions as the types you get in python for this clock might be different -depending on the system. -If it is a typedef of the system clock, python will get datetime objects, but if -it is a different clock they will be timedelta objects. - -Conversions Provided --------------------- - -C++ to Python - - ``std::chrono::system_clock::time_point`` → ``datetime.datetime`` - System clock times are converted to python datetime instances. They are - in the local timezone, but do not have any timezone information attached - to them (they are naive datetime objects). - - - ``std::chrono::duration`` → ``datetime.timedelta`` - Durations are converted to timedeltas, any precision in the duration - greater than microseconds is lost by rounding towards zero. - - - ``std::chrono::[other_clocks]::time_point`` → ``datetime.timedelta`` - Any clock time that is not the system clock is converted to a time delta. This timedelta measures the time from the clocks epoch to now. - -Python to C++ - - ``datetime.datetime`` → ``std::chrono::system_clock::time_point`` - Date/time objects are converted into system clock timepoints. Any - timezone information is ignored and the type is treated as a naive - object. - - - ``datetime.timedelta`` → ``std::chrono::duration`` - Time delta are converted into durations with microsecond precision. - - - ``datetime.timedelta`` → ``std::chrono::[other_clocks]::time_point`` - Time deltas that are converted into clock timepoints are treated as - the amount of time from the start of the clocks epoch. - - - ``float`` → ``std::chrono::duration`` - Floats that are passed to C++ as durations be interpreted as a number of - seconds. These will be converted to the duration using ``duration_cast`` - from the float. - - - ``float`` → ``std::chrono::[other_clocks]::time_point`` - Floats that are passed to C++ as time points will be interpreted as the - number of seconds from the start of the clocks epoch. - -Return value policies -===================== - -Python and C++ use wildly different ways of managing the memory and lifetime of -objects managed by them. This can lead to issues when creating bindings for -functions that return a non-trivial type. Just by looking at the type -information, it is not clear whether Python should take charge of the returned -value and eventually free its resources, or if this is handled on the C++ side. -For this reason, pybind11 provides a several `return value policy` annotations -that can be passed to the :func:`module::def` and :func:`class_::def` -functions. The default policy is :enum:`return_value_policy::automatic`. - -Return value policies can also be applied to properties, in which case the -arguments must be passed through the :class:`cpp_function` constructor: - -.. code-block:: cpp - - class_(m, "MyClass") - def_property("data" - py::cpp_function(&MyClass::getData, py::return_value_policy::copy), - py::cpp_function(&MyClass::setData) - ); - -The following table provides an overview of the available return value policies: - -.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| - -+--------------------------------------------------+----------------------------------------------------------------------------+ -| Return value policy | Description | -+==================================================+============================================================================+ -| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | -| | :enum:`return_value_policy::take_ownership` when the return value is a | -| | pointer. Otherwise, it uses :enum:`return_value::move` or | -| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | -| | See below for a description of what all of these different policies do. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | -| | return value is a pointer. This is the default conversion policy for | -| | function arguments when calling Python functions manually from C++ code | -| | (i.e. via handle::operator()). You probably won't need to use this. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | -| | ownership. Python will call the destructor and delete operator when the | -| | object's reference count reaches zero. Undefined behavior ensues when the | -| | C++ side does the same. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | -| | This policy is comparably safe because the lifetimes of the two instances | -| | are decoupled. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | -| | that will be owned by Python. This policy is comparably safe because the | -| | lifetimes of the two instances (move source and destination) are decoupled.| -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | -| | responsible for managing the object's lifetime and deallocating it when | -| | it is no longer used. Warning: undefined behavior will ensue when the C++ | -| | side deletes an object that is still referenced and used by Python. | -+--------------------------------------------------+----------------------------------------------------------------------------+ -| :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | -| | of a parent object, namely the implicit ``this``, or ``self`` argument of | -| | the called method or property. Internally, this policy works just like | -| | :enum:`return_value_policy::reference` but additionally applies a | -| | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | -| | prevents the parent object from being garbage collected as long as the | -| | return value is referenced by Python. This is the default policy for | -| | property getters created via ``def_property``, ``def_readwrite``, etc. | -+--------------------------------------------------+----------------------------------------------------------------------------+ - -.. warning:: - - Code with invalid return value policies might access unitialized memory or - free data structures multiple times, which can lead to hard-to-debug - non-determinism and segmentation faults, hence it is worth spending the - time to understand all the different options in the table above. - -One important aspect of the above policies is that they only apply to instances -which pybind11 has *not* seen before, in which case the policy clarifies -essential questions about the return value's lifetime and ownership. When -pybind11 knows the instance already (as identified by its type and address in -memory), it will return the existing Python object wrapper rather than creating -a new copy. - -.. note:: - - The next section on :ref:`call_policies` discusses *call policies* that can be - specified *in addition* to a return value policy from the list above. Call - policies indicate reference relationships that can involve both return values - and parameters of functions. - -.. note:: - - As an alternative to elaborate call policies and lifetime management logic, - consider using smart pointers (see the section on :ref:`smart_pointers` for - details). Smart pointers can tell whether an object is still referenced from - C++ or Python, which generally eliminates the kinds of inconsistencies that - can lead to crashes or undefined behavior. For functions returning smart - pointers, it is not necessary to specify a return value policy. - -.. _call_policies: - -Additional call policies -======================== - -In addition to the above return value policies, further `call policies` can be -specified to indicate dependencies between parameters. There is currently just -one policy named ``keep_alive``, which indicates that the -argument with index ``Patient`` should be kept alive at least until the -argument with index ``Nurse`` is freed by the garbage collector. Argument -indices start at one, while zero refers to the return value. For methods, index -``1`` refers to the implicit ``this`` pointer, while regular arguments begin at -index ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` -with value ``None`` is detected at runtime, the call policy does nothing. - -This feature internally relies on the ability to create a *weak reference* to -the nurse object, which is permitted by all classes exposed via pybind11. When -the nurse object does not support weak references, an exception will be thrown. - -Consider the following example: here, the binding code for a list append -operation ties the lifetime of the newly added element to the underlying -container: - -.. code-block:: cpp - - py::class_(m, "List") - .def("append", &List::append, py::keep_alive<1, 2>()); - -.. note:: - - ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, - Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == - 0) policies from Boost.Python. - -.. seealso:: - - The file :file:`tests/test_keep_alive.cpp` contains a complete example - that demonstrates using :class:`keep_alive` in more detail. - -Implicit type conversions -========================= - -Suppose that instances of two types ``A`` and ``B`` are used in a project, and -that an ``A`` can easily be converted into an instance of type ``B`` (examples of this -could be a fixed and an arbitrary precision number type). - -.. code-block:: cpp - - py::class_(m, "A") - /// ... members ... - - py::class_(m, "B") - .def(py::init()) - /// ... members ... - - m.def("func", - [](const B &) { /* .... */ } - ); - -To invoke the function ``func`` using a variable ``a`` containing an ``A`` -instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++ -will automatically apply an implicit type conversion, which makes it possible -to directly write ``func(a)``. - -In this situation (i.e. where ``B`` has a constructor that converts from -``A``), the following statement enables similar implicit conversions on the -Python side: - -.. code-block:: cpp - - py::implicitly_convertible(); - -.. note:: - - Implicit conversions from ``A`` to ``B`` only work when ``B`` is a custom - data type that is exposed to Python via pybind11. - -.. _static_properties: - -Static properties -================= - -The section on :ref:`properties` discussed the creation of instance properties -that are implemented in terms of C++ getters and setters. - -Static properties can also be created in a similar way to expose getters and -setters of static class attributes. It is important to note that the implicit -``self`` argument also exists in this case and is used to pass the Python -``type`` subclass instance. This parameter will often not be needed by the C++ -side, and the following example illustrates how to instantiate a lambda getter -function that ignores it: - -.. code-block:: cpp - - py::class_(m, "Foo") - .def_property_readonly_static("foo", [](py::object /* self */) { return Foo(); }); - -Unique pointers -=============== - -Given a class ``Example`` with Python bindings, it's possible to return -instances wrapped in C++11 unique pointers, like so - -.. code-block:: cpp - - std::unique_ptr create_example() { return std::unique_ptr(new Example()); } - -.. code-block:: cpp - - m.def("create_example", &create_example); - -In other words, there is nothing special that needs to be done. While returning -unique pointers in this way is allowed, it is *illegal* to use them as function -arguments. For instance, the following function signature cannot be processed -by pybind11. - -.. code-block:: cpp - - void do_something_with_example(std::unique_ptr ex) { ... } - -The above signature would imply that Python needs to give up ownership of an -object that is passed to this function, which is generally not possible (for -instance, the object might be referenced elsewhere). - -.. _smart_pointers: - -Smart pointers -============== - -This section explains how to pass values that are wrapped in "smart" pointer -types with internal reference counting. For the simpler C++11 unique pointers, -refer to the previous section. - -The binding generator for classes, :class:`class_`, can be passed a template -type that denotes a special *holder* type that is used to manage references to -the object. If no such holder type template argument is given, the default for -a type named ``Type`` is ``std::unique_ptr``, which means that the object -is deallocated when Python's reference count goes to zero. - -It is possible to switch to other types of reference counting wrappers or smart -pointers, which is useful in codebases that rely on them. For instance, the -following snippet causes ``std::shared_ptr`` to be used instead. - -.. code-block:: cpp - - py::class_ /* <- holder type */> obj(m, "Example"); - -Note that any particular class can only be associated with a single holder type. - -To enable transparent conversions for functions that take shared pointers as an -argument or that return them, a macro invocation similar to the following must -be declared at the top level before any binding code: - -.. code-block:: cpp - - PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr); - -.. note:: - - The first argument of :func:`PYBIND11_DECLARE_HOLDER_TYPE` should be a - placeholder name that is used as a template parameter of the second - argument. Thus, feel free to use any identifier, but use it consistently on - both sides; also, don't use the name of a type that already exists in your - codebase. - -One potential stumbling block when using holder types is that they need to be -applied consistently. Can you guess what's broken about the following binding -code? - -.. code-block:: cpp - - PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr); - - class Child { }; - - class Parent { - public: - Parent() : child(std::make_shared()) { } - Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */ - private: - std::shared_ptr child; - }; - - PYBIND11_PLUGIN(example) { - py::module m("example"); - - py::class_>(m, "Child"); - - py::class_>(m, "Parent") - .def(py::init<>()) - .def("get_child", &Parent::get_child); - - return m.ptr(); - } - -The following Python code will cause undefined behavior (and likely a -segmentation fault). - -.. code-block:: python - - from example import Parent - print(Parent().get_child()) - -The problem is that ``Parent::get_child()`` returns a pointer to an instance of -``Child``, but the fact that this instance is already managed by -``std::shared_ptr<...>`` is lost when passing raw pointers. In this case, -pybind11 will create a second independent ``std::shared_ptr<...>`` that also -claims ownership of the pointer. In the end, the object will be freed **twice** -since these shared pointers have no way of knowing about each other. - -There are two ways to resolve this issue: - -1. For types that are managed by a smart pointer class, never use raw pointers - in function arguments or return values. In other words: always consistently - wrap pointers into their designated holder types (such as - ``std::shared_ptr<...>``). In this case, the signature of ``get_child()`` - should be modified as follows: - -.. code-block:: cpp - - std::shared_ptr get_child() { return child; } - -2. Adjust the definition of ``Child`` by specifying - ``std::enable_shared_from_this`` (see cppreference_ for details) as a - base class. This adds a small bit of information to ``Child`` that allows - pybind11 to realize that there is already an existing - ``std::shared_ptr<...>`` and communicate with it. In this case, the - declaration of ``Child`` should look as follows: - -.. _cppreference: http://en.cppreference.com/w/cpp/memory/enable_shared_from_this - -.. code-block:: cpp - - class Child : public std::enable_shared_from_this { }; - - -Please take a look at the :ref:`macro_notes` before using this feature. - -.. seealso:: - - The file :file:`tests/test_smart_ptr.cpp` contains a complete example - that demonstrates how to work with custom reference-counting holder types - in more detail. - -.. _custom_constructors: - -Custom constructors -=================== - -The syntax for binding constructors was previously introduced, but it only -works when a constructor with the given parameters actually exists on the C++ -side. To extend this to more general cases, let's take a look at what actually -happens under the hood: the following statement - -.. code-block:: cpp - - py::class_(m, "Example") - .def(py::init()); - -is short hand notation for - -.. code-block:: cpp - - py::class_(m, "Example") - .def("__init__", - [](Example &instance, int arg) { - new (&instance) Example(arg); - } - ); - -In other words, :func:`init` creates an anonymous function that invokes an -in-place constructor. Memory allocation etc. is already take care of beforehand -within pybind11. - -.. _classes_with_non_public_destructors: - -Classes with non-public destructors -=================================== - -If a class has a private or protected destructor (as might e.g. be the case in -a singleton pattern), a compile error will occur when creating bindings via -pybind11. The underlying issue is that the ``std::unique_ptr`` holder type that -is responsible for managing the lifetime of instances will reference the -destructor even if no deallocations ever take place. In order to expose classes -with private or protected destructors, it is possible to override the holder -type via a holder type argument to ``class_``. Pybind11 provides a helper class -``py::nodelete`` that disables any destructor invocations. In this case, it is -crucial that instances are deallocated on the C++ side to avoid memory leaks. - -.. code-block:: cpp - - /* ... definition ... */ - - class MyClass { - private: - ~MyClass() { } - }; - - /* ... binding code ... */ - - py::class_>(m, "MyClass") - .def(py::init<>) - -.. _catching_and_throwing_exceptions: - -Catching and throwing exceptions -================================ - -When C++ code invoked from Python throws an ``std::exception``, it is -automatically converted into a Python ``Exception``. pybind11 defines multiple -special exception classes that will map to different types of Python -exceptions: - -.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| - -+--------------------------------------+------------------------------+ -| C++ exception type | Python exception type | -+======================================+==============================+ -| :class:`std::exception` | ``RuntimeError`` | -+--------------------------------------+------------------------------+ -| :class:`std::bad_alloc` | ``MemoryError`` | -+--------------------------------------+------------------------------+ -| :class:`std::domain_error` | ``ValueError`` | -+--------------------------------------+------------------------------+ -| :class:`std::invalid_argument` | ``ValueError`` | -+--------------------------------------+------------------------------+ -| :class:`std::length_error` | ``ValueError`` | -+--------------------------------------+------------------------------+ -| :class:`std::out_of_range` | ``ValueError`` | -+--------------------------------------+------------------------------+ -| :class:`std::range_error` | ``ValueError`` | -+--------------------------------------+------------------------------+ -| :class:`pybind11::stop_iteration` | ``StopIteration`` (used to | -| | implement custom iterators) | -+--------------------------------------+------------------------------+ -| :class:`pybind11::index_error` | ``IndexError`` (used to | -| | indicate out of bounds | -| | accesses in ``__getitem__``, | -| | ``__setitem__``, etc.) | -+--------------------------------------+------------------------------+ -| :class:`pybind11::value_error` | ``ValueError`` (used to | -| | indicate wrong value passed | -| | in ``container.remove(...)`` | -+--------------------------------------+------------------------------+ -| :class:`pybind11::key_error` | ``KeyError`` (used to | -| | indicate out of bounds | -| | accesses in ``__getitem__``, | -| | ``__setitem__`` in dict-like | -| | objects, etc.) | -+--------------------------------------+------------------------------+ -| :class:`pybind11::error_already_set` | Indicates that the Python | -| | exception flag has already | -| | been initialized | -+--------------------------------------+------------------------------+ - -When a Python function invoked from C++ throws an exception, it is converted -into a C++ exception of type :class:`error_already_set` whose string payload -contains a textual summary. - -There is also a special exception :class:`cast_error` that is thrown by -:func:`handle::call` when the input arguments cannot be converted to Python -objects. - -Registering custom exception translators -======================================== - -If the default exception conversion policy described -:ref:`above ` -is insufficient, pybind11 also provides support for registering custom -exception translators. - -To register a simple exception conversion that translates a C++ exception into -a new Python exception using the C++ exception's ``what()`` method, a helper -function is available: - -.. code-block:: cpp - - py::register_exception(module, "PyExp"); - -This call creates a Python exception class with the name ``PyExp`` in the given -module and automatically converts any encountered exceptions of type ``CppExp`` -into Python exceptions of type ``PyExp``. - -When more advanced exception translation is needed, the function -``py::register_exception_translator(translator)`` can be used to register -functions that can translate arbitrary exception types (and which may include -additional logic to do so). The function takes a stateless callable (e.g. a -function pointer or a lambda function without captured variables) with the call -signature ``void(std::exception_ptr)``. - -When a C++ exception is thrown, the registered exception translators are tried -in reverse order of registration (i.e. the last registered translator gets the -first shot at handling the exception). - -Inside the translator, ``std::rethrow_exception`` should be used within -a try block to re-throw the exception. One or more catch clauses to catch -the appropriate exceptions should then be used with each clause using -``PyErr_SetString`` to set a Python exception or ``ex(string)`` to set -the python exception to a custom exception type (see below). - -To declare a custom Python exception type, declare a ``py::exception`` variable -and use this in the associated exception translator (note: it is often useful -to make this a static declaration when using it inside a lambda expression -without requiring capturing). - - -The following example demonstrates this for a hypothetical exception classes -``MyCustomException`` and ``OtherException``: the first is translated to a -custom python exception ``MyCustomError``, while the second is translated to a -standard python RuntimeError: - -.. code-block:: cpp - - static py::exception exc(m, "MyCustomError"); - py::register_exception_translator([](std::exception_ptr p) { - try { - if (p) std::rethrow_exception(p); - } catch (const MyCustomException &e) { - exc(e.what()); - } catch (const OtherException &e) { - PyErr_SetString(PyExc_RuntimeError, e.what()); - } - }); - -Multiple exceptions can be handled by a single translator, as shown in the -example above. If the exception is not caught by the current translator, the -previously registered one gets a chance. - -If none of the registered exception translators is able to handle the -exception, it is handled by the default converter as described in the previous -section. - -.. seealso:: - - The file :file:`tests/test_exceptions.cpp` contains examples - of various custom exception translators and custom exception types. - -.. note:: - - You must call either ``PyErr_SetString`` or a custom exception's call - operator (``exc(string)``) for every exception caught in a custom exception - translator. Failure to do so will cause Python to crash with ``SystemError: - error return without exception set``. - - Exceptions that you do not plan to handle should simply not be caught, or - may be explicity (re-)thrown to delegate it to the other, - previously-declared existing exception translators. - -.. _eigen: - -Transparent conversion of dense and sparse Eigen data types -=========================================================== - -Eigen [#f1]_ is C++ header-based library for dense and sparse linear algebra. Due to -its popularity and widespread adoption, pybind11 provides transparent -conversion support between Eigen and Scientific Python linear algebra data types. - -Specifically, when including the optional header file :file:`pybind11/eigen.h`, -pybind11 will automatically and transparently convert - -1. Static and dynamic Eigen dense vectors and matrices to instances of - ``numpy.ndarray`` (and vice versa). - -2. Returned matrix expressions such as blocks (including columns or rows) and - diagonals will be converted to ``numpy.ndarray`` of the expression - values. - -3. Returned matrix-like objects such as Eigen::DiagonalMatrix or - Eigen::SelfAdjointView will be converted to ``numpy.ndarray`` containing the - expressed value. - -4. Eigen sparse vectors and matrices to instances of - ``scipy.sparse.csr_matrix``/``scipy.sparse.csc_matrix`` (and vice versa). - -This makes it possible to bind most kinds of functions that rely on these types. -One major caveat are functions that take Eigen matrices *by reference* and modify -them somehow, in which case the information won't be propagated to the caller. - -.. code-block:: cpp - - /* The Python bindings of these functions won't replicate - the intended effect of modifying the function arguments */ - void scale_by_2(Eigen::Vector3f &v) { - v *= 2; - } - void scale_by_2(Eigen::Ref &v) { - v *= 2; - } - -To see why this is, refer to the section on :ref:`opaque` (although that -section specifically covers STL data types, the underlying issue is the same). -The next two sections discuss an efficient alternative for exposing the -underlying native Eigen types as opaque objects in a way that still integrates -with NumPy and SciPy. - -.. [#f1] http://eigen.tuxfamily.org - -.. seealso:: - - The file :file:`tests/test_eigen.cpp` contains a complete example that - shows how to pass Eigen sparse and dense data types in more detail. - -Buffer protocol -=============== - -Python supports an extremely general and convenient approach for exchanging -data between plugin libraries. Types can expose a buffer view [#f2]_, which -provides fast direct access to the raw internal data representation. Suppose we -want to bind the following simplistic Matrix class: - -.. code-block:: cpp - - class Matrix { - public: - Matrix(size_t rows, size_t cols) : m_rows(rows), m_cols(cols) { - m_data = new float[rows*cols]; - } - float *data() { return m_data; } - size_t rows() const { return m_rows; } - size_t cols() const { return m_cols; } - private: - size_t m_rows, m_cols; - float *m_data; - }; - -The following binding code exposes the ``Matrix`` contents as a buffer object, -making it possible to cast Matrices into NumPy arrays. It is even possible to -completely avoid copy operations with Python expressions like -``np.array(matrix_instance, copy = False)``. - -.. code-block:: cpp - - py::class_(m, "Matrix") - .def_buffer([](Matrix &m) -> py::buffer_info { - return py::buffer_info( - m.data(), /* Pointer to buffer */ - sizeof(float), /* Size of one scalar */ - py::format_descriptor::format(), /* Python struct-style format descriptor */ - 2, /* Number of dimensions */ - { m.rows(), m.cols() }, /* Buffer dimensions */ - { sizeof(float) * m.rows(), /* Strides (in bytes) for each index */ - sizeof(float) } - ); - }); - -The snippet above binds a lambda function, which can create ``py::buffer_info`` -description records on demand describing a given matrix. The contents of -``py::buffer_info`` mirror the Python buffer protocol specification. - -.. code-block:: cpp - - struct buffer_info { - void *ptr; - size_t itemsize; - std::string format; - int ndim; - std::vector shape; - std::vector strides; - }; - -To create a C++ function that can take a Python buffer object as an argument, -simply use the type ``py::buffer`` as one of its arguments. Buffers can exist -in a great variety of configurations, hence some safety checks are usually -necessary in the function body. Below, you can see an basic example on how to -define a custom constructor for the Eigen double precision matrix -(``Eigen::MatrixXd``) type, which supports initialization from compatible -buffer objects (e.g. a NumPy matrix). - -.. code-block:: cpp - - /* Bind MatrixXd (or some other Eigen type) to Python */ - typedef Eigen::MatrixXd Matrix; - - typedef Matrix::Scalar Scalar; - constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit; - - py::class_(m, "Matrix") - .def("__init__", [](Matrix &m, py::buffer b) { - typedef Eigen::Stride Strides; - - /* Request a buffer descriptor from Python */ - py::buffer_info info = b.request(); - - /* Some sanity checks ... */ - if (info.format != py::format_descriptor::format()) - throw std::runtime_error("Incompatible format: expected a double array!"); - - if (info.ndim != 2) - throw std::runtime_error("Incompatible buffer dimension!"); - - auto strides = Strides( - info.strides[rowMajor ? 0 : 1] / sizeof(Scalar), - info.strides[rowMajor ? 1 : 0] / sizeof(Scalar)); - - auto map = Eigen::Map( - static_cat(info.ptr), info.shape[0], info.shape[1], strides); - - new (&m) Matrix(map); - }); - -For reference, the ``def_buffer()`` call for this Eigen data type should look -as follows: - -.. code-block:: cpp - - .def_buffer([](Matrix &m) -> py::buffer_info { - return py::buffer_info( - m.data(), /* Pointer to buffer */ - sizeof(Scalar), /* Size of one scalar */ - /* Python struct-style format descriptor */ - py::format_descriptor::format(), - /* Number of dimensions */ - 2, - /* Buffer dimensions */ - { (size_t) m.rows(), - (size_t) m.cols() }, - /* Strides (in bytes) for each index */ - { sizeof(Scalar) * (rowMajor ? m.cols() : 1), - sizeof(Scalar) * (rowMajor ? 1 : m.rows()) } - ); - }) - -For a much easier approach of binding Eigen types (although with some -limitations), refer to the section on :ref:`eigen`. - -.. seealso:: - - The file :file:`tests/test_buffers.cpp` contains a complete example - that demonstrates using the buffer protocol with pybind11 in more detail. - -.. [#f2] http://docs.python.org/3/c-api/buffer.html - -NumPy support -============= - -By exchanging ``py::buffer`` with ``py::array`` in the above snippet, we can -restrict the function so that it only accepts NumPy arrays (rather than any -type of Python object satisfying the buffer protocol). - -In many situations, we want to define a function which only accepts a NumPy -array of a certain data type. This is possible via the ``py::array_t`` -template. For instance, the following function requires the argument to be a -NumPy array containing double precision values. - -.. code-block:: cpp - - void f(py::array_t array); - -When it is invoked with a different type (e.g. an integer or a list of -integers), the binding code will attempt to cast the input into a NumPy array -of the requested type. Note that this feature requires the -:file:``pybind11/numpy.h`` header to be included. - -Data in NumPy arrays is not guaranteed to packed in a dense manner; -furthermore, entries can be separated by arbitrary column and row strides. -Sometimes, it can be useful to require a function to only accept dense arrays -using either the C (row-major) or Fortran (column-major) ordering. This can be -accomplished via a second template argument with values ``py::array::c_style`` -or ``py::array::f_style``. - -.. code-block:: cpp - - void f(py::array_t array); - -The ``py::array::forcecast`` argument is the default value of the second -template parameter, and it ensures that non-conforming arguments are converted -into an array satisfying the specified requirements instead of trying the next -function overload. - -NumPy structured types -====================== - -In order for ``py::array_t`` to work with structured (record) types, we first need -to register the memory layout of the type. This can be done via ``PYBIND11_NUMPY_DTYPE`` -macro which expects the type followed by field names: - -.. code-block:: cpp - - struct A { - int x; - double y; - }; - - struct B { - int z; - A a; - }; - - PYBIND11_NUMPY_DTYPE(A, x, y); - PYBIND11_NUMPY_DTYPE(B, z, a); - - /* now both A and B can be used as template arguments to py::array_t */ - -Vectorizing functions -===================== - -Suppose we want to bind a function with the following signature to Python so -that it can process arbitrary NumPy array arguments (vectors, matrices, general -N-D arrays) in addition to its normal arguments: - -.. code-block:: cpp - - double my_func(int x, float y, double z); - -After including the ``pybind11/numpy.h`` header, this is extremely simple: - -.. code-block:: cpp - - m.def("vectorized_func", py::vectorize(my_func)); - -Invoking the function like below causes 4 calls to be made to ``my_func`` with -each of the array elements. The significant advantage of this compared to -solutions like ``numpy.vectorize()`` is that the loop over the elements runs -entirely on the C++ side and can be crunched down into a tight, optimized loop -by the compiler. The result is returned as a NumPy array of type -``numpy.dtype.float64``. - -.. code-block:: pycon - - >>> x = np.array([[1, 3],[5, 7]]) - >>> y = np.array([[2, 4],[6, 8]]) - >>> z = 3 - >>> result = vectorized_func(x, y, z) - -The scalar argument ``z`` is transparently replicated 4 times. The input -arrays ``x`` and ``y`` are automatically converted into the right types (they -are of type ``numpy.dtype.int64`` but need to be ``numpy.dtype.int32`` and -``numpy.dtype.float32``, respectively) - -Sometimes we might want to explicitly exclude an argument from the vectorization -because it makes little sense to wrap it in a NumPy array. For instance, -suppose the function signature was - -.. code-block:: cpp - - double my_func(int x, float y, my_custom_type *z); - -This can be done with a stateful Lambda closure: - -.. code-block:: cpp - - // Vectorize a lambda function with a capture object (e.g. to exclude some arguments from the vectorization) - m.def("vectorized_func", - [](py::array_t x, py::array_t y, my_custom_type *z) { - auto stateful_closure = [z](int x, float y) { return my_func(x, y, z); }; - return py::vectorize(stateful_closure)(x, y); - } - ); - -In cases where the computation is too complicated to be reduced to -``vectorize``, it will be necessary to create and access the buffer contents -manually. The following snippet contains a complete example that shows how this -works (the code is somewhat contrived, since it could have been done more -simply using ``vectorize``). - -.. code-block:: cpp - - #include - #include - - namespace py = pybind11; - - py::array_t add_arrays(py::array_t input1, py::array_t input2) { - auto buf1 = input1.request(), buf2 = input2.request(); - - if (buf1.ndim != 1 || buf2.ndim != 1) - throw std::runtime_error("Number of dimensions must be one"); - - if (buf1.size != buf2.size) - throw std::runtime_error("Input shapes must match"); - - /* No pointer is passed, so NumPy will allocate the buffer */ - auto result = py::array_t(buf1.size); - - auto buf3 = result.request(); - - double *ptr1 = (double *) buf1.ptr, - *ptr2 = (double *) buf2.ptr, - *ptr3 = (double *) buf3.ptr; - - for (size_t idx = 0; idx < buf1.shape[0]; idx++) - ptr3[idx] = ptr1[idx] + ptr2[idx]; - - return result; - } - - PYBIND11_PLUGIN(test) { - py::module m("test"); - m.def("add_arrays", &add_arrays, "Add two NumPy arrays"); - return m.ptr(); - } - -.. seealso:: - - The file :file:`tests/test_numpy_vectorize.cpp` contains a complete - example that demonstrates using :func:`vectorize` in more detail. - -Functions taking Python objects as arguments -============================================ - -pybind11 exposes all major Python types using thin C++ wrapper classes. These -wrapper classes can also be used as parameters of functions in bindings, which -makes it possible to directly work with native Python types on the C++ side. -For instance, the following statement iterates over a Python ``dict``: - -.. code-block:: cpp - - void print_dict(py::dict dict) { - /* Easily interact with Python types */ - for (auto item : dict) - std::cout << "key=" << item.first << ", " - << "value=" << item.second << std::endl; - } - -Available types include :class:`handle`, :class:`object`, :class:`bool_`, -:class:`int_`, :class:`float_`, :class:`str`, :class:`bytes`, :class:`tuple`, -:class:`list`, :class:`dict`, :class:`slice`, :class:`none`, :class:`capsule`, -:class:`iterable`, :class:`iterator`, :class:`function`, :class:`buffer`, -:class:`array`, and :class:`array_t`. - -In this kind of mixed code, it is often necessary to convert arbitrary C++ -types to Python, which can be done using :func:`cast`: - -.. code-block:: cpp - - MyClass *cls = ..; - py::object obj = py::cast(cls); - -The reverse direction uses the following syntax: - -.. code-block:: cpp - - py::object obj = ...; - MyClass *cls = obj.cast(); - -When conversion fails, both directions throw the exception :class:`cast_error`. -It is also possible to call python functions via ``operator()``. - -.. code-block:: cpp - - py::function f = <...>; - py::object result_py = f(1234, "hello", some_instance); - MyClass &result = result_py.cast(); - -Keyword arguments are also supported. In Python, there is the usual call syntax: - -.. code-block:: python - - def f(number, say, to): - ... # function code - - f(1234, say="hello", to=some_instance) # keyword call in Python - -In C++, the same call can be made using: - -.. code-block:: cpp - - using pybind11::literals; // to bring in the `_a` literal - f(1234, "say"_a="hello", "to"_a=some_instance); // keyword call in C++ - -Unpacking of ``*args`` and ``**kwargs`` is also possible and can be mixed with -other arguments: - -.. code-block:: cpp - - // * unpacking - py::tuple args = py::make_tuple(1234, "hello", some_instance); - f(*args); - - // ** unpacking - py::dict kwargs = py::dict("number"_a=1234, "say"_a="hello", "to"_a=some_instance); - f(**kwargs); - - // mixed keywords, * and ** unpacking - py::tuple args = py::make_tuple(1234); - py::dict kwargs = py::dict("to"_a=some_instance); - f(*args, "say"_a="hello", **kwargs); - -Generalized unpacking according to PEP448_ is also supported: - -.. code-block:: cpp - - py::dict kwargs1 = py::dict("number"_a=1234); - py::dict kwargs2 = py::dict("to"_a=some_instance); - f(**kwargs1, "say"_a="hello", **kwargs2); - -.. seealso:: - - The file :file:`tests/test_python_types.cpp` contains a complete - example that demonstrates passing native Python types in more detail. The - file :file:`tests/test_callbacks.cpp` presents a few examples of calling - Python functions from C++, including keywords arguments and unpacking. - -.. _PEP448: https://www.python.org/dev/peps/pep-0448/ - -Using Python's print function in C++ -==================================== - -The usual way to write output in C++ is using ``std::cout`` while in Python one -would use ``print``. Since these methods use different buffers, mixing them can -lead to output order issues. To resolve this, pybind11 modules can use the -:func:`py::print` function which writes to Python's ``sys.stdout`` for consistency. - -Python's ``print`` function is replicated in the C++ API including optional -keyword arguments ``sep``, ``end``, ``file``, ``flush``. Everything works as -expected in Python: - -.. code-block:: cpp - - py::print(1, 2.0, "three"); // 1 2.0 three - py::print(1, 2.0, "three", "sep"_a="-"); // 1-2.0-three - - auto args = py::make_tuple("unpacked", true); - py::print("->", *args, "end"_a="<-"); // -> unpacked True <- - -Default arguments revisited -=========================== - -The section on :ref:`default_args` previously discussed basic usage of default -arguments using pybind11. One noteworthy aspect of their implementation is that -default arguments are converted to Python objects right at declaration time. -Consider the following example: - -.. code-block:: cpp - - py::class_("MyClass") - .def("myFunction", py::arg("arg") = SomeType(123)); - -In this case, pybind11 must already be set up to deal with values of the type -``SomeType`` (via a prior instantiation of ``py::class_``), or an -exception will be thrown. - -Another aspect worth highlighting is that the "preview" of the default argument -in the function signature is generated using the object's ``__repr__`` method. -If not available, the signature may not be very helpful, e.g.: - -.. code-block:: pycon - - FUNCTIONS - ... - | myFunction(...) - | Signature : (MyClass, arg : SomeType = ) -> NoneType - ... - -The first way of addressing this is by defining ``SomeType.__repr__``. -Alternatively, it is possible to specify the human-readable preview of the -default argument manually using the ``arg_t`` notation: - -.. code-block:: cpp - - py::class_("MyClass") - .def("myFunction", py::arg_t("arg", SomeType(123), "SomeType(123)")); - -Sometimes it may be necessary to pass a null pointer value as a default -argument. In this case, remember to cast it to the underlying type in question, -like so: - -.. code-block:: cpp - - py::class_("MyClass") - .def("myFunction", py::arg("arg") = (SomeType *) nullptr); - -Binding functions that accept arbitrary numbers of arguments and keywords arguments -=================================================================================== - -Python provides a useful mechanism to define functions that accept arbitrary -numbers of arguments and keyword arguments: - -.. code-block:: cpp - - def generic(*args, **kwargs): - # .. do something with args and kwargs - -Such functions can also be created using pybind11: - -.. code-block:: cpp - - void generic(py::args args, py::kwargs kwargs) { - /// .. do something with args - if (kwargs) - /// .. do something with kwargs - } - - /// Binding code - m.def("generic", &generic); - -(See ``tests/test_kwargs_and_defaults.cpp``). The class ``py::args`` -derives from ``py::list`` and ``py::kwargs`` derives from ``py::dict`` Note -that the ``kwargs`` argument is invalid if no keyword arguments were actually -provided. Please refer to the other examples for details on how to iterate -over these, and on how to cast their entries into C++ objects. - -.. warning:: - - Unlike Python, pybind11 does not allow combining normal parameters with the - ``args`` / ``kwargs`` special parameters. - -Partitioning code over multiple extension modules -================================================= - -It's straightforward to split binding code over multiple extension modules, -while referencing types that are declared elsewhere. Everything "just" works -without any special precautions. One exception to this rule occurs when -extending a type declared in another extension module. Recall the basic example -from Section :ref:`inheritance`. - -.. code-block:: cpp - - py::class_ pet(m, "Pet"); - pet.def(py::init()) - .def_readwrite("name", &Pet::name); - - py::class_(m, "Dog", pet /* <- specify parent */) - .def(py::init()) - .def("bark", &Dog::bark); - -Suppose now that ``Pet`` bindings are defined in a module named ``basic``, -whereas the ``Dog`` bindings are defined somewhere else. The challenge is of -course that the variable ``pet`` is not available anymore though it is needed -to indicate the inheritance relationship to the constructor of ``class_``. -However, it can be acquired as follows: - -.. code-block:: cpp - - py::object pet = (py::object) py::module::import("basic").attr("Pet"); - - py::class_(m, "Dog", pet) - .def(py::init()) - .def("bark", &Dog::bark); - -Alternatively, you can specify the base class as a template parameter option to -``class_``, which performs an automated lookup of the corresponding Python -type. Like the above code, however, this also requires invoking the ``import`` -function once to ensure that the pybind11 binding code of the module ``basic`` -has been executed: - -.. code-block:: cpp - - py::module::import("basic"); - - py::class_(m, "Dog") - .def(py::init()) - .def("bark", &Dog::bark); - -Naturally, both methods will fail when there are cyclic dependencies. - -Note that compiling code which has its default symbol visibility set to -*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the -ability to access types defined in another extension module. Workarounds -include changing the global symbol visibility (not recommended, because it will -lead unnecessarily large binaries) or manually exporting types that are -accessed by multiple extension modules: - -.. code-block:: cpp - - #ifdef _WIN32 - # define EXPORT_TYPE __declspec(dllexport) - #else - # define EXPORT_TYPE __attribute__ ((visibility("default"))) - #endif - - class EXPORT_TYPE Dog : public Animal { - ... - }; - - -Pickling support -================ - -Python's ``pickle`` module provides a powerful facility to serialize and -de-serialize a Python object graph into a binary data stream. To pickle and -unpickle C++ classes using pybind11, two additional functions must be provided. -Suppose the class in question has the following signature: - -.. code-block:: cpp - - class Pickleable { - public: - Pickleable(const std::string &value) : m_value(value) { } - const std::string &value() const { return m_value; } - - void setExtra(int extra) { m_extra = extra; } - int extra() const { return m_extra; } - private: - std::string m_value; - int m_extra = 0; - }; - -The binding code including the requisite ``__setstate__`` and ``__getstate__`` methods [#f3]_ -looks as follows: - -.. code-block:: cpp - - py::class_(m, "Pickleable") - .def(py::init()) - .def("value", &Pickleable::value) - .def("extra", &Pickleable::extra) - .def("setExtra", &Pickleable::setExtra) - .def("__getstate__", [](const Pickleable &p) { - /* Return a tuple that fully encodes the state of the object */ - return py::make_tuple(p.value(), p.extra()); - }) - .def("__setstate__", [](Pickleable &p, py::tuple t) { - if (t.size() != 2) - throw std::runtime_error("Invalid state!"); - - /* Invoke the in-place constructor. Note that this is needed even - when the object just has a trivial default constructor */ - new (&p) Pickleable(t[0].cast()); - - /* Assign any additional state */ - p.setExtra(t[1].cast()); - }); - -An instance can now be pickled as follows: - -.. code-block:: python - - try: - import cPickle as pickle # Use cPickle on Python 2.7 - except ImportError: - import pickle - - p = Pickleable("test_value") - p.setExtra(15) - data = pickle.dumps(p, 2) - -Note that only the cPickle module is supported on Python 2.7. The second -argument to ``dumps`` is also crucial: it selects the pickle protocol version -2, since the older version 1 is not supported. Newer versions are also fine—for -instance, specify ``-1`` to always use the latest available version. Beware: -failure to follow these instructions will cause important pybind11 memory -allocation routines to be skipped during unpickling, which will likely lead to -memory corruption and/or segmentation faults. - -.. seealso:: - - The file :file:`tests/test_pickling.cpp` contains a complete example - that demonstrates how to pickle and unpickle types using pybind11 in more - detail. - -.. [#f3] http://docs.python.org/3/library/pickle.html#pickling-class-instances - -Generating documentation using Sphinx -===================================== - -Sphinx [#f4]_ has the ability to inspect the signatures and documentation -strings in pybind11-based extension modules to automatically generate beautiful -documentation in a variety formats. The python_example repository [#f5]_ contains a -simple example repository which uses this approach. - -There are two potential gotchas when using this approach: first, make sure that -the resulting strings do not contain any :kbd:`TAB` characters, which break the -docstring parsing routines. You may want to use C++11 raw string literals, -which are convenient for multi-line comments. Conveniently, any excess -indentation will be automatically be removed by Sphinx. However, for this to -work, it is important that all lines are indented consistently, i.e.: - -.. code-block:: cpp - - // ok - m.def("foo", &foo, R"mydelimiter( - The foo function - - Parameters - ---------- - )mydelimiter"); - - // *not ok* - m.def("foo", &foo, R"mydelimiter(The foo function - - Parameters - ---------- - )mydelimiter"); - -.. [#f4] http://www.sphinx-doc.org -.. [#f5] http://github.com/pybind/python_example - -Evaluating Python expressions from strings and files -==================================================== - -pybind11 provides the :func:`eval` and :func:`eval_file` functions to evaluate -Python expressions and statements. The following example illustrates how they -can be used. - -Both functions accept a template parameter that describes how the argument -should be interpreted. Possible choices include ``eval_expr`` (isolated -expression), ``eval_single_statement`` (a single statement, return value is -always ``none``), and ``eval_statements`` (sequence of statements, return value -is always ``none``). - -.. code-block:: cpp - - // At beginning of file - #include - - ... - - // Evaluate in scope of main module - py::object scope = py::module::import("__main__").attr("__dict__"); - - // Evaluate an isolated expression - int result = py::eval("my_variable + 10", scope).cast(); - - // Evaluate a sequence of statements - py::eval( - "print('Hello')\n" - "print('world!');", - scope); - - // Evaluate the statements in an separate Python file on disk - py::eval_file("script.py", scope); - -Development of custom type casters -================================== - -In very rare cases, applications may require custom type casters that cannot be -expressed using the abstractions provided by pybind11, thus requiring raw -Python C API calls. This is fairly advanced usage and should only be pursued by -experts who are familiar with the intricacies of Python reference counting. - -The following snippets demonstrate how this works for a very simple ``inty`` -type that that should be convertible from Python types that provide a -``__int__(self)`` method. - -.. code-block:: cpp - - struct inty { long long_value; }; - - void print(inty s) { - std::cout << s.long_value << std::endl; - } - -The following Python snippet demonstrates the intended usage from the Python side: - -.. code-block:: python - - class A: - def __int__(self): - return 123 - - from example import print - print(A()) - -To register the necessary conversion routines, it is necessary to add -a partial overload to the ``pybind11::detail::type_caster`` template. -Although this is an implementation detail, adding partial overloads to this -type is explicitly allowed. - -.. code-block:: cpp - - namespace pybind11 { - namespace detail { - template <> struct type_caster { - public: - /** - * This macro establishes the name 'inty' in - * function signatures and declares a local variable - * 'value' of type inty - */ - PYBIND11_TYPE_CASTER(inty, _("inty")); - - /** - * Conversion part 1 (Python->C++): convert a PyObject into a inty - * instance or return false upon failure. The second argument - * indicates whether implicit conversions should be applied. - */ - bool load(handle src, bool) { - /* Extract PyObject from handle */ - PyObject *source = src.ptr(); - /* Try converting into a Python integer value */ - PyObject *tmp = PyNumber_Long(source); - if (!tmp) - return false; - /* Now try to convert into a C++ int */ - value.long_value = PyLong_AsLong(tmp); - Py_DECREF(tmp); - /* Ensure return code was OK (to avoid out-of-range errors etc) */ - return !(value.long_value == -1 && !PyErr_Occurred()); - } - - /** - * Conversion part 2 (C++ -> Python): convert an inty instance into - * a Python object. The second and third arguments are used to - * indicate the return value policy and parent object (for - * ``return_value_policy::reference_internal``) and are generally - * ignored by implicit casters. - */ - static handle cast(inty src, return_value_policy /* policy */, handle /* parent */) { - return PyLong_FromLong(src.long_value); - } - }; - } - }; - -Multiple Inheritance -==================== - -pybind11 can create bindings for types that derive from multiple base types -(aka. *multiple inheritance*). To do so, specify all bases in the template -arguments of the ``class_`` declaration: - -.. code-block:: cpp - - py::class_(m, "MyType") - ... - -The base types can be specified in arbitrary order, and they can even be -interspersed with alias types and holder types (discussed earlier in this -document)---pybind11 will automatically find out which is which. The only -requirement is that the first template argument is the type to be declared. - -There are two caveats regarding the implementation of this feature: - -1. When only one base type is specified for a C++ type that actually has - multiple bases, pybind11 will assume that it does not participate in - multiple inheritance, which can lead to undefined behavior. In such cases, - add the tag ``multiple_inheritance``: - - .. code-block:: cpp - - py::class_(m, "MyType", py::multiple_inheritance()); - - The tag is redundant and does not need to be specified when multiple base - types are listed. - -2. As was previously discussed in the section on :ref:`overriding_virtuals`, it - is easy to create Python types that derive from C++ classes. It is even - possible to make use of multiple inheritance to declare a Python class which - has e.g. a C++ and a Python class as bases. However, any attempt to create a - type that has *two or more* C++ classes in its hierarchy of base types will - fail with a fatal error message: ``TypeError: multiple bases have instance - lay-out conflict``. Core Python types that are implemented in C (e.g. - ``dict``, ``list``, ``Exception``, etc.) also fall under this combination - and cannot be combined with C++ types bound using pybind11 via multiple - inheritance. diff --git a/docs/advanced/cast/chrono.rst b/docs/advanced/cast/chrono.rst new file mode 100644 index 000000000..1213e2cf0 --- /dev/null +++ b/docs/advanced/cast/chrono.rst @@ -0,0 +1,81 @@ +Chrono +====== + +When including the additional header file :file:`pybind11/chrono.h` conversions +from C++11 chrono datatypes to python datetime objects are automatically enabled. +This header also enables conversions of python floats (often from sources such +as `time.monotonic()`, `time.perf_counter()` and `time.process_time()`) into +durations. + +An overview of clocks in C++11 +------------------------------ + +A point of confusion when using these conversions is the differences between +clocks provided in C++11. There are three clock types defined by the C++11 +standard and users can define their own if needed. Each of these clocks have +different properties and when converting to and from python will give different +results. + +The first clock defined by the standard is ``std::chrono::system_clock``. This +clock measures the current date and time. However, this clock changes with to +updates to the operating system time. For example, if your time is synchronised +with a time server this clock will change. This makes this clock a poor choice +for timing purposes but good for measuring the wall time. + +The second clock defined in the standard is ``std::chrono::steady_clock``. +This clock ticks at a steady rate and is never adjusted. This makes it excellent +for timing purposes, however the value in this clock does not correspond to the +current date and time. Often this clock will be the amount of time your system +has been on, although it does not have to be. This clock will never be the same +clock as the system clock as the system clock can change but steady clocks +cannot. + +The third clock defined in the standard is ``std::chrono::high_resolution_clock``. +This clock is the clock that has the highest resolution out of the clocks in the +system. It is normally a typedef to either the system clock or the steady clock +but can be its own independent clock. This is important as when using these +conversions as the types you get in python for this clock might be different +depending on the system. +If it is a typedef of the system clock, python will get datetime objects, but if +it is a different clock they will be timedelta objects. + +Conversions Provided +-------------------- + +.. rubric:: C++ to Python + +- ``std::chrono::system_clock::time_point`` → ``datetime.datetime`` + System clock times are converted to python datetime instances. They are + in the local timezone, but do not have any timezone information attached + to them (they are naive datetime objects). + +- ``std::chrono::duration`` → ``datetime.timedelta`` + Durations are converted to timedeltas, any precision in the duration + greater than microseconds is lost by rounding towards zero. + +- ``std::chrono::[other_clocks]::time_point`` → ``datetime.timedelta`` + Any clock time that is not the system clock is converted to a time delta. + This timedelta measures the time from the clocks epoch to now. + +.. rubric:: Python to C++ + +- ``datetime.datetime`` → ``std::chrono::system_clock::time_point`` + Date/time objects are converted into system clock timepoints. Any + timezone information is ignored and the type is treated as a naive + object. + +- ``datetime.timedelta`` → ``std::chrono::duration`` + Time delta are converted into durations with microsecond precision. + +- ``datetime.timedelta`` → ``std::chrono::[other_clocks]::time_point`` + Time deltas that are converted into clock timepoints are treated as + the amount of time from the start of the clocks epoch. + +- ``float`` → ``std::chrono::duration`` + Floats that are passed to C++ as durations be interpreted as a number of + seconds. These will be converted to the duration using ``duration_cast`` + from the float. + +- ``float`` → ``std::chrono::[other_clocks]::time_point`` + Floats that are passed to C++ as time points will be interpreted as the + number of seconds from the start of the clocks epoch. diff --git a/docs/advanced/cast/custom.rst b/docs/advanced/cast/custom.rst new file mode 100644 index 000000000..50b07db51 --- /dev/null +++ b/docs/advanced/cast/custom.rst @@ -0,0 +1,79 @@ +Custom type casters +=================== + +In very rare cases, applications may require custom type casters that cannot be +expressed using the abstractions provided by pybind11, thus requiring raw +Python C API calls. This is fairly advanced usage and should only be pursued by +experts who are familiar with the intricacies of Python reference counting. + +The following snippets demonstrate how this works for a very simple ``inty`` +type that that should be convertible from Python types that provide a +``__int__(self)`` method. + +.. code-block:: cpp + + struct inty { long long_value; }; + + void print(inty s) { + std::cout << s.long_value << std::endl; + } + +The following Python snippet demonstrates the intended usage from the Python side: + +.. code-block:: python + + class A: + def __int__(self): + return 123 + + from example import print + print(A()) + +To register the necessary conversion routines, it is necessary to add +a partial overload to the ``pybind11::detail::type_caster`` template. +Although this is an implementation detail, adding partial overloads to this +type is explicitly allowed. + +.. code-block:: cpp + + namespace pybind11 { namespace detail { + template <> struct type_caster { + public: + /** + * This macro establishes the name 'inty' in + * function signatures and declares a local variable + * 'value' of type inty + */ + PYBIND11_TYPE_CASTER(inty, _("inty")); + + /** + * Conversion part 1 (Python->C++): convert a PyObject into a inty + * instance or return false upon failure. The second argument + * indicates whether implicit conversions should be applied. + */ + bool load(handle src, bool) { + /* Extract PyObject from handle */ + PyObject *source = src.ptr(); + /* Try converting into a Python integer value */ + PyObject *tmp = PyNumber_Long(source); + if (!tmp) + return false; + /* Now try to convert into a C++ int */ + value.long_value = PyLong_AsLong(tmp); + Py_DECREF(tmp); + /* Ensure return code was OK (to avoid out-of-range errors etc) */ + return !(value.long_value == -1 && !PyErr_Occurred()); + } + + /** + * Conversion part 2 (C++ -> Python): convert an inty instance into + * a Python object. The second and third arguments are used to + * indicate the return value policy and parent object (for + * ``return_value_policy::reference_internal``) and are generally + * ignored by implicit casters. + */ + static handle cast(inty src, return_value_policy /* policy */, handle /* parent */) { + return PyLong_FromLong(src.long_value); + } + }; + }} // namespace pybind11::detail diff --git a/docs/advanced/cast/eigen.rst b/docs/advanced/cast/eigen.rst new file mode 100644 index 000000000..b83ca9af9 --- /dev/null +++ b/docs/advanced/cast/eigen.rst @@ -0,0 +1,50 @@ +Eigen +===== + +`Eigen `_ is C++ header-based library for dense and +sparse linear algebra. Due to its popularity and widespread adoption, pybind11 +provides transparent conversion support between Eigen and Scientific Python linear +algebra data types. + +Specifically, when including the optional header file :file:`pybind11/eigen.h`, +pybind11 will automatically and transparently convert + +1. Static and dynamic Eigen dense vectors and matrices to instances of + ``numpy.ndarray`` (and vice versa). + +2. Returned matrix expressions such as blocks (including columns or rows) and + diagonals will be converted to ``numpy.ndarray`` of the expression + values. + +3. Returned matrix-like objects such as Eigen::DiagonalMatrix or + Eigen::SelfAdjointView will be converted to ``numpy.ndarray`` containing the + expressed value. + +4. Eigen sparse vectors and matrices to instances of + ``scipy.sparse.csr_matrix``/``scipy.sparse.csc_matrix`` (and vice versa). + +This makes it possible to bind most kinds of functions that rely on these types. +One major caveat are functions that take Eigen matrices *by reference* and modify +them somehow, in which case the information won't be propagated to the caller. + +.. code-block:: cpp + + /* The Python bindings of these functions won't replicate + the intended effect of modifying the function arguments */ + void scale_by_2(Eigen::Vector3f &v) { + v *= 2; + } + void scale_by_2(Eigen::Ref &v) { + v *= 2; + } + +To see why this is, refer to the section on :ref:`opaque` (although that +section specifically covers STL data types, the underlying issue is the same). +The :ref:`numpy` sections discuss an efficient alternative for exposing the +underlying native Eigen types as opaque objects in a way that still integrates +with NumPy and SciPy. + +.. seealso:: + + The file :file:`tests/test_eigen.cpp` contains a complete example that + shows how to pass Eigen sparse and dense data types in more detail. diff --git a/docs/advanced/cast/functional.rst b/docs/advanced/cast/functional.rst new file mode 100644 index 000000000..5d0a01d13 --- /dev/null +++ b/docs/advanced/cast/functional.rst @@ -0,0 +1,113 @@ +Functional +########## + +The following features must be enabled by including :file:`pybind11/functional.h`. + + +Callbacks and passing anonymous functions +========================================= + +The C++11 standard brought lambda functions and the generic polymorphic +function wrapper ``std::function<>`` to the C++ programming language, which +enable powerful new ways of working with functions. Lambda functions come in +two flavors: stateless lambda function resemble classic function pointers that +link to an anonymous piece of code, while stateful lambda functions +additionally depend on captured variables that are stored in an anonymous +*lambda closure object*. + +Here is a simple example of a C++ function that takes an arbitrary function +(stateful or stateless) with signature ``int -> int`` as an argument and runs +it with the value 10. + +.. code-block:: cpp + + int func_arg(const std::function &f) { + return f(10); + } + +The example below is more involved: it takes a function of signature ``int -> int`` +and returns another function of the same kind. The return value is a stateful +lambda function, which stores the value ``f`` in the capture object and adds 1 to +its return value upon execution. + +.. code-block:: cpp + + std::function func_ret(const std::function &f) { + return [f](int i) { + return f(i) + 1; + }; + } + +This example demonstrates using python named parameters in C++ callbacks which +requires using ``py::cpp_function`` as a wrapper. Usage is similar to defining +methods of classes: + +.. code-block:: cpp + + py::cpp_function func_cpp() { + return py::cpp_function([](int i) { return i+1; }, + py::arg("number")); + } + +After including the extra header file :file:`pybind11/functional.h`, it is almost +trivial to generate binding code for all of these functions. + +.. code-block:: cpp + + #include + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + m.def("func_arg", &func_arg); + m.def("func_ret", &func_ret); + m.def("func_cpp", &func_cpp); + + return m.ptr(); + } + +The following interactive session shows how to call them from Python. + +.. code-block:: pycon + + $ python + >>> import example + >>> def square(i): + ... return i * i + ... + >>> example.func_arg(square) + 100L + >>> square_plus_1 = example.func_ret(square) + >>> square_plus_1(4) + 17L + >>> plus_1 = func_cpp() + >>> plus_1(number=43) + 44L + +.. warning:: + + Keep in mind that passing a function from C++ to Python (or vice versa) + will instantiate a piece of wrapper code that translates function + invocations between the two languages. Naturally, this translation + increases the computational cost of each function call somewhat. A + problematic situation can arise when a function is copied back and forth + between Python and C++ many times in a row, in which case the underlying + wrappers will accumulate correspondingly. The resulting long sequence of + C++ -> Python -> C++ -> ... roundtrips can significantly decrease + performance. + + There is one exception: pybind11 detects case where a stateless function + (i.e. a function pointer or a lambda function without captured variables) + is passed as an argument to another C++ function exposed in Python. In this + case, there is no overhead. Pybind11 will extract the underlying C++ + function pointer from the wrapped function to sidestep a potential C++ -> + Python -> C++ roundtrip. This is demonstrated in :file:`tests/test_callbacks.cpp`. + +.. note:: + + This functionality is very useful when generating bindings for callbacks in + C++ libraries (e.g. GUI libraries, asynchronous networking libraries, etc.). + + The file :file:`tests/test_callbacks.cpp` contains a complete example + that demonstrates how to work with callbacks and anonymous functions in + more detail. diff --git a/docs/advanced/cast/index.rst b/docs/advanced/cast/index.rst new file mode 100644 index 000000000..cce3a4747 --- /dev/null +++ b/docs/advanced/cast/index.rst @@ -0,0 +1,79 @@ +Casting data types +################## + +.. toctree:: + :maxdepth: 1 + + stl + functional + chrono + eigen + custom + +The following basic data types are supported out of the box (some may require +an additional extension header to be included). To pass other data structures +as arguments and return values, refer to the section on binding :ref:`classes`. + ++---------------------------------+--------------------------+-------------------------------+ +| Data type | Description | Header file | ++=================================+==========================+===============================+ +| ``int8_t``, ``uint8_t`` | 8-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int16_t``, ``uint16_t`` | 16-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int32_t``, ``uint32_t`` | 32-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``int64_t``, ``uint64_t`` | 64-bit integers | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``ssize_t``, ``size_t`` | Platform-dependent size | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``float``, ``double`` | Floating point types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``bool`` | Two-state Boolean type | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``char`` | Character literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``wchar_t`` | Wide character literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``const char *`` | UTF-8 string literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``const wchar_t *`` | Wide string literal | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::string`` | STL dynamic UTF-8 string | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::wstring`` | STL dynamic wide string | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::pair`` | Pair of two custom types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::tuple<...>`` | Arbitrary tuple of types | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::reference_wrapper<...>`` | Reference type wrapper | :file:`pybind11/pybind11.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::complex`` | Complex numbers | :file:`pybind11/complex.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::array`` | STL static array | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::vector`` | STL dynamic array | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::list`` | STL linked list | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::map`` | STL ordered map | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::unordered_map`` | STL unordered map | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::set`` | STL ordered set | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::unordered_set`` | STL unordered set | :file:`pybind11/stl.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::function<...>`` | STL polymorphic function | :file:`pybind11/functional.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::chrono::duration<...>`` | STL time duration | :file:`pybind11/chrono.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``std::chrono::time_point<...>``| STL date/time | :file:`pybind11/chrono.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::Matrix<...>`` | Eigen: dense matrix | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::Map<...>`` | Eigen: mapped memory | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ +| ``Eigen::SparseMatrix<...>`` | Eigen: sparse matrix | :file:`pybind11/eigen.h` | ++---------------------------------+--------------------------+-------------------------------+ diff --git a/docs/advanced/cast/stl.rst b/docs/advanced/cast/stl.rst new file mode 100644 index 000000000..bbd23732b --- /dev/null +++ b/docs/advanced/cast/stl.rst @@ -0,0 +1,154 @@ +STL containers +############## + +Automatic conversion +==================== + +When including the additional header file :file:`pybind11/stl.h`, conversions +between ``std::vector<>``, ``std::list<>``, ``std::set<>``, and ``std::map<>`` +and the Python ``list``, ``set`` and ``dict`` data structures are automatically +enabled. The types ``std::pair<>`` and ``std::tuple<>`` are already supported +out of the box with just the core :file:`pybind11/pybind11.h` header. + +The major downside of these implicit conversions is that containers must be +converted (i.e. copied) on every Python->C++ and C++->Python transition, which +can have implications on the program semantics and performance. Please read the +next sections for more details and alternative approaches that avoid this. + +.. note:: + + Arbitrary nesting of any of these types is possible. + +.. seealso:: + + The file :file:`tests/test_python_types.cpp` contains a complete + example that demonstrates how to pass STL data types in more detail. + +.. _opaque: + +Making opaque types +=================== + +pybind11 heavily relies on a template matching mechanism to convert parameters +and return values that are constructed from STL data types such as vectors, +linked lists, hash tables, etc. This even works in a recursive manner, for +instance to deal with lists of hash maps of pairs of elementary and custom +types, etc. + +However, a fundamental limitation of this approach is that internal conversions +between Python and C++ types involve a copy operation that prevents +pass-by-reference semantics. What does this mean? + +Suppose we bind the following function + +.. code-block:: cpp + + void append_1(std::vector &v) { + v.push_back(1); + } + +and call it from Python, the following happens: + +.. code-block:: pycon + + >>> v = [5, 6] + >>> append_1(v) + >>> print(v) + [5, 6] + +As you can see, when passing STL data structures by reference, modifications +are not propagated back the Python side. A similar situation arises when +exposing STL data structures using the ``def_readwrite`` or ``def_readonly`` +functions: + +.. code-block:: cpp + + /* ... definition ... */ + + class MyClass { + std::vector contents; + }; + + /* ... binding code ... */ + + py::class_(m, "MyClass") + .def(py::init<>) + .def_readwrite("contents", &MyClass::contents); + +In this case, properties can be read and written in their entirety. However, an +``append`` operation involving such a list type has no effect: + +.. code-block:: pycon + + >>> m = MyClass() + >>> m.contents = [5, 6] + >>> print(m.contents) + [5, 6] + >>> m.contents.append(7) + >>> print(m.contents) + [5, 6] + +Finally, the involved copy operations can be costly when dealing with very +large lists. To deal with all of the above situations, pybind11 provides a +macro named ``PYBIND11_MAKE_OPAQUE(T)`` that disables the template-based +conversion machinery of types, thus rendering them *opaque*. The contents of +opaque objects are never inspected or extracted, hence they *can* be passed by +reference. For instance, to turn ``std::vector`` into an opaque type, add +the declaration + +.. code-block:: cpp + + PYBIND11_MAKE_OPAQUE(std::vector); + +before any binding code (e.g. invocations to ``class_::def()``, etc.). This +macro must be specified at the top level (and outside of any namespaces), since +it instantiates a partial template overload. If your binding code consists of +multiple compilation units, it must be present in every file preceding any +usage of ``std::vector``. Opaque types must also have a corresponding +``class_`` declaration to associate them with a name in Python, and to define a +set of available operations, e.g.: + +.. code-block:: cpp + + py::class_>(m, "IntVector") + .def(py::init<>()) + .def("clear", &std::vector::clear) + .def("pop_back", &std::vector::pop_back) + .def("__len__", [](const std::vector &v) { return v.size(); }) + .def("__iter__", [](std::vector &v) { + return py::make_iterator(v.begin(), v.end()); + }, py::keep_alive<0, 1>()) /* Keep vector alive while iterator is used */ + // .... + +The ability to expose STL containers as native Python objects is a fairly +common request, hence pybind11 also provides an optional header file named +:file:`pybind11/stl_bind.h` that does exactly this. The mapped containers try +to match the behavior of their native Python counterparts as much as possible. + +The following example showcases usage of :file:`pybind11/stl_bind.h`: + +.. code-block:: cpp + + // Don't forget this + #include + + PYBIND11_MAKE_OPAQUE(std::vector); + PYBIND11_MAKE_OPAQUE(std::map); + + // ... + + // later in binding code: + py::bind_vector>(m, "VectorInt"); + py::bind_map>(m, "MapStringDouble"); + +Please take a look at the :ref:`macro_notes` before using the +``PYBIND11_MAKE_OPAQUE`` macro. + +.. seealso:: + + The file :file:`tests/test_opaque_types.cpp` contains a complete + example that demonstrates how to create and expose opaque types using + pybind11 in more detail. + + The file :file:`tests/test_stl_binders.cpp` shows how to use the + convenience STL container wrappers. diff --git a/docs/advanced/classes.rst b/docs/advanced/classes.rst new file mode 100644 index 000000000..4a423b578 --- /dev/null +++ b/docs/advanced/classes.rst @@ -0,0 +1,634 @@ +Classes +####### + +This section presents advanced binding code for classes and it is assumed +that you are already familiar with the basics from :doc:`/classes`. + +.. _overriding_virtuals: + +Overriding virtual functions in Python +====================================== + +Suppose that a C++ class or interface has a virtual function that we'd like to +to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is +given as a specific example of how one would do this with traditional C++ +code). + +.. code-block:: cpp + + class Animal { + public: + virtual ~Animal() { } + virtual std::string go(int n_times) = 0; + }; + + class Dog : public Animal { + public: + std::string go(int n_times) override { + std::string result; + for (int i=0; igo(3); + } + +Normally, the binding code for these classes would look as follows: + +.. code-block:: cpp + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_ animal(m, "Animal"); + animal + .def("go", &Animal::go); + + py::class_(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +However, these bindings are impossible to extend: ``Animal`` is not +constructible, and we clearly require some kind of "trampoline" that +redirects virtual calls back to Python. + +Defining a new type of ``Animal`` from within Python is possible but requires a +helper class that is defined as follows: + +.. code-block:: cpp + + class PyAnimal : public Animal { + public: + /* Inherit the constructors */ + using Animal::Animal; + + /* Trampoline (need one for each virtual function) */ + std::string go(int n_times) override { + PYBIND11_OVERLOAD_PURE( + std::string, /* Return type */ + Animal, /* Parent class */ + go, /* Name of function */ + n_times /* Argument(s) */ + ); + } + }; + +The macro :func:`PYBIND11_OVERLOAD_PURE` should be used for pure virtual +functions, and :func:`PYBIND11_OVERLOAD` should be used for functions which have +a default implementation. There are also two alternate macros +:func:`PYBIND11_OVERLOAD_PURE_NAME` and :func:`PYBIND11_OVERLOAD_NAME` which +take a string-valued name argument between the *Parent class* and *Name of the +function* slots. This is useful when the C++ and Python versions of the +function have different names, e.g. ``operator()`` vs ``__call__``. + +The binding code also needs a few minor adaptations (highlighted): + +.. code-block:: cpp + :emphasize-lines: 4,6,7 + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_ animal(m, "Animal"); + animal + .def(py::init<>()) + .def("go", &Animal::go); + + py::class_(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", &call_go); + + return m.ptr(); + } + +Importantly, pybind11 is made aware of the trampoline helper class by +specifying it as an extra template argument to :class:`class_`. (This can also +be combined with other template arguments such as a custom holder type; the +order of template types does not matter). Following this, we are able to +define a constructor as usual. + +Note, however, that the above is sufficient for allowing python classes to +extend ``Animal``, but not ``Dog``: see ref:`virtual_and_inheritance` for the +necessary steps required to providing proper overload support for inherited +classes. + +The Python session below shows how to override ``Animal::go`` and invoke it via +a virtual method call. + +.. code-block:: pycon + + >>> from example import * + >>> d = Dog() + >>> call_go(d) + u'woof! woof! woof! ' + >>> class Cat(Animal): + ... def go(self, n_times): + ... return "meow! " * n_times + ... + >>> c = Cat() + >>> call_go(c) + u'meow! meow! meow! ' + +Please take a look at the :ref:`macro_notes` before using this feature. + +.. note:: + + When the overridden type returns a reference or pointer to a type that + pybind11 converts from Python (for example, numeric values, std::string, + and other built-in value-converting types), there are some limitations to + be aware of: + + - because in these cases there is no C++ variable to reference (the value + is stored in the referenced Python variable), pybind11 provides one in + the PYBIND11_OVERLOAD macros (when needed) with static storage duration. + Note that this means that invoking the overloaded method on *any* + instance will change the referenced value stored in *all* instances of + that type. + + - Attempts to modify a non-const reference will not have the desired + effect: it will change only the static cache variable, but this change + will not propagate to underlying Python instance, and the change will be + replaced the next time the overload is invoked. + +.. seealso:: + + The file :file:`tests/test_virtual_functions.cpp` contains a complete + example that demonstrates how to override virtual functions using pybind11 + in more detail. + +.. _virtual_and_inheritance: + +Combining virtual functions and inheritance +=========================================== + +When combining virtual methods with inheritance, you need to be sure to provide +an override for each method for which you want to allow overrides from derived +python classes. For example, suppose we extend the above ``Animal``/``Dog`` +example as follows: + +.. code-block:: cpp + + class Animal { + public: + virtual std::string go(int n_times) = 0; + virtual std::string name() { return "unknown"; } + }; + class Dog : public class Animal { + public: + std::string go(int n_times) override { + std::string result; + for (int i=0; i class PyAnimal : public AnimalBase { + using AnimalBase::AnimalBase; // Inherit constructors + std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, AnimalBase, go, n_times); } + std::string name() override { PYBIND11_OVERLOAD(std::string, AnimalBase, name, ); } + }; + template class PyDog : public PyAnimal { + using PyAnimal::PyAnimal; // Inherit constructors + // Override PyAnimal's pure virtual go() with a non-pure one: + std::string go(int n_times) override { PYBIND11_OVERLOAD(std::string, DogBase, go, n_times); } + std::string bark() override { PYBIND11_OVERLOAD(std::string, DogBase, bark, ); } + }; + +This technique has the advantage of requiring just one trampoline method to be +declared per virtual method and pure virtual method override. It does, +however, require the compiler to generate at least as many methods (and +possibly more, if both pure virtual and overridden pure virtual methods are +exposed, as above). + +The classes are then registered with pybind11 using: + +.. code-block:: cpp + + py::class_> animal(m, "Animal"); + py::class_> dog(m, "Dog"); + py::class_> husky(m, "Husky"); + // ... add animal, dog, husky definitions + +Note that ``Husky`` did not require a dedicated trampoline template class at +all, since it neither declares any new virtual methods nor provides any pure +virtual method implementations. + +With either the repeated-virtuals or templated trampoline methods in place, you +can now create a python class that inherits from ``Dog``: + +.. code-block:: python + + class ShihTzu(Dog): + def bark(self): + return "yip!" + +.. seealso:: + + See the file :file:`tests/test_virtual_functions.cpp` for complete examples + using both the duplication and templated trampoline approaches. + +Extended trampoline class functionality +======================================= + +The trampoline classes described in the previous sections are, by default, only +initialized when needed. More specifically, they are initialized when a python +class actually inherits from a registered type (instead of merely creating an +instance of the registered type), or when a registered constructor is only +valid for the trampoline class but not the registered class. This is primarily +for performance reasons: when the trampoline class is not needed for anything +except virtual method dispatching, not initializing the trampoline class +improves performance by avoiding needing to do a run-time check to see if the +inheriting python instance has an overloaded method. + +Sometimes, however, it is useful to always initialize a trampoline class as an +intermediate class that does more than just handle virtual method dispatching. +For example, such a class might perform extra class initialization, extra +destruction operations, and might define new members and methods to enable a +more python-like interface to a class. + +In order to tell pybind11 that it should *always* initialize the trampoline +class when creating new instances of a type, the class constructors should be +declared using ``py::init_alias()`` instead of the usual +``py::init()``. This forces construction via the trampoline class, +ensuring member initialization and (eventual) destruction. + +.. seealso:: + + See the file :file:`tests/test_alias_initialization.cpp` for complete examples + showing both normal and forced trampoline instantiation. + +.. _custom_constructors: + +Custom constructors +=================== + +The syntax for binding constructors was previously introduced, but it only +works when a constructor with the given parameters actually exists on the C++ +side. To extend this to more general cases, let's take a look at what actually +happens under the hood: the following statement + +.. code-block:: cpp + + py::class_(m, "Example") + .def(py::init()); + +is short hand notation for + +.. code-block:: cpp + + py::class_(m, "Example") + .def("__init__", + [](Example &instance, int arg) { + new (&instance) Example(arg); + } + ); + +In other words, :func:`init` creates an anonymous function that invokes an +in-place constructor. Memory allocation etc. is already take care of beforehand +within pybind11. + +.. _classes_with_non_public_destructors: + +Non-public destructors +====================== + +If a class has a private or protected destructor (as might e.g. be the case in +a singleton pattern), a compile error will occur when creating bindings via +pybind11. The underlying issue is that the ``std::unique_ptr`` holder type that +is responsible for managing the lifetime of instances will reference the +destructor even if no deallocations ever take place. In order to expose classes +with private or protected destructors, it is possible to override the holder +type via a holder type argument to ``class_``. Pybind11 provides a helper class +``py::nodelete`` that disables any destructor invocations. In this case, it is +crucial that instances are deallocated on the C++ side to avoid memory leaks. + +.. code-block:: cpp + + /* ... definition ... */ + + class MyClass { + private: + ~MyClass() { } + }; + + /* ... binding code ... */ + + py::class_>(m, "MyClass") + .def(py::init<>) + +Implicit conversions +==================== + +Suppose that instances of two types ``A`` and ``B`` are used in a project, and +that an ``A`` can easily be converted into an instance of type ``B`` (examples of this +could be a fixed and an arbitrary precision number type). + +.. code-block:: cpp + + py::class_(m, "A") + /// ... members ... + + py::class_(m, "B") + .def(py::init()) + /// ... members ... + + m.def("func", + [](const B &) { /* .... */ } + ); + +To invoke the function ``func`` using a variable ``a`` containing an ``A`` +instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++ +will automatically apply an implicit type conversion, which makes it possible +to directly write ``func(a)``. + +In this situation (i.e. where ``B`` has a constructor that converts from +``A``), the following statement enables similar implicit conversions on the +Python side: + +.. code-block:: cpp + + py::implicitly_convertible(); + +.. note:: + + Implicit conversions from ``A`` to ``B`` only work when ``B`` is a custom + data type that is exposed to Python via pybind11. + +.. _static_properties: + +Static properties +================= + +The section on :ref:`properties` discussed the creation of instance properties +that are implemented in terms of C++ getters and setters. + +Static properties can also be created in a similar way to expose getters and +setters of static class attributes. It is important to note that the implicit +``self`` argument also exists in this case and is used to pass the Python +``type`` subclass instance. This parameter will often not be needed by the C++ +side, and the following example illustrates how to instantiate a lambda getter +function that ignores it: + +.. code-block:: cpp + + py::class_(m, "Foo") + .def_property_readonly_static("foo", [](py::object /* self */) { return Foo(); }); + +Operator overloading +==================== + +Suppose that we're given the following ``Vector2`` class with a vector addition +and scalar multiplication operation, all implemented using overloaded operators +in C++. + +.. code-block:: cpp + + class Vector2 { + public: + Vector2(float x, float y) : x(x), y(y) { } + + Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); } + Vector2 operator*(float value) const { return Vector2(x * value, y * value); } + Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; } + Vector2& operator*=(float v) { x *= v; y *= v; return *this; } + + friend Vector2 operator*(float f, const Vector2 &v) { + return Vector2(f * v.x, f * v.y); + } + + std::string toString() const { + return "[" + std::to_string(x) + ", " + std::to_string(y) + "]"; + } + private: + float x, y; + }; + +The following snippet shows how the above operators can be conveniently exposed +to Python. + +.. code-block:: cpp + + #include + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_(m, "Vector2") + .def(py::init()) + .def(py::self + py::self) + .def(py::self += py::self) + .def(py::self *= float()) + .def(float() * py::self) + .def("__repr__", &Vector2::toString); + + return m.ptr(); + } + +Note that a line like + +.. code-block:: cpp + + .def(py::self * float()) + +is really just short hand notation for + +.. code-block:: cpp + + .def("__mul__", [](const Vector2 &a, float b) { + return a * b; + }, py::is_operator()) + +This can be useful for exposing additional operators that don't exist on the +C++ side, or to perform other types of customization. The ``py::is_operator`` +flag marker is needed to inform pybind11 that this is an operator, which +returns ``NotImplemented`` when invoked with incompatible arguments rather than +throwing a type error. + +.. note:: + + To use the more convenient ``py::self`` notation, the additional + header file :file:`pybind11/operators.h` must be included. + +.. seealso:: + + The file :file:`tests/test_operator_overloading.cpp` contains a + complete example that demonstrates how to work with overloaded operators in + more detail. + +Pickling support +================ + +Python's ``pickle`` module provides a powerful facility to serialize and +de-serialize a Python object graph into a binary data stream. To pickle and +unpickle C++ classes using pybind11, two additional functions must be provided. +Suppose the class in question has the following signature: + +.. code-block:: cpp + + class Pickleable { + public: + Pickleable(const std::string &value) : m_value(value) { } + const std::string &value() const { return m_value; } + + void setExtra(int extra) { m_extra = extra; } + int extra() const { return m_extra; } + private: + std::string m_value; + int m_extra = 0; + }; + +The binding code including the requisite ``__setstate__`` and ``__getstate__`` methods [#f3]_ +looks as follows: + +.. code-block:: cpp + + py::class_(m, "Pickleable") + .def(py::init()) + .def("value", &Pickleable::value) + .def("extra", &Pickleable::extra) + .def("setExtra", &Pickleable::setExtra) + .def("__getstate__", [](const Pickleable &p) { + /* Return a tuple that fully encodes the state of the object */ + return py::make_tuple(p.value(), p.extra()); + }) + .def("__setstate__", [](Pickleable &p, py::tuple t) { + if (t.size() != 2) + throw std::runtime_error("Invalid state!"); + + /* Invoke the in-place constructor. Note that this is needed even + when the object just has a trivial default constructor */ + new (&p) Pickleable(t[0].cast()); + + /* Assign any additional state */ + p.setExtra(t[1].cast()); + }); + +An instance can now be pickled as follows: + +.. code-block:: python + + try: + import cPickle as pickle # Use cPickle on Python 2.7 + except ImportError: + import pickle + + p = Pickleable("test_value") + p.setExtra(15) + data = pickle.dumps(p, 2) + +Note that only the cPickle module is supported on Python 2.7. The second +argument to ``dumps`` is also crucial: it selects the pickle protocol version +2, since the older version 1 is not supported. Newer versions are also fine—for +instance, specify ``-1`` to always use the latest available version. Beware: +failure to follow these instructions will cause important pybind11 memory +allocation routines to be skipped during unpickling, which will likely lead to +memory corruption and/or segmentation faults. + +.. seealso:: + + The file :file:`tests/test_pickling.cpp` contains a complete example + that demonstrates how to pickle and unpickle types using pybind11 in more + detail. + +.. [#f3] http://docs.python.org/3/library/pickle.html#pickling-class-instances + +Multiple Inheritance +==================== + +pybind11 can create bindings for types that derive from multiple base types +(aka. *multiple inheritance*). To do so, specify all bases in the template +arguments of the ``class_`` declaration: + +.. code-block:: cpp + + py::class_(m, "MyType") + ... + +The base types can be specified in arbitrary order, and they can even be +interspersed with alias types and holder types (discussed earlier in this +document)---pybind11 will automatically find out which is which. The only +requirement is that the first template argument is the type to be declared. + +There are two caveats regarding the implementation of this feature: + +1. When only one base type is specified for a C++ type that actually has + multiple bases, pybind11 will assume that it does not participate in + multiple inheritance, which can lead to undefined behavior. In such cases, + add the tag ``multiple_inheritance``: + + .. code-block:: cpp + + py::class_(m, "MyType", py::multiple_inheritance()); + + The tag is redundant and does not need to be specified when multiple base + types are listed. + +2. As was previously discussed in the section on :ref:`overriding_virtuals`, it + is easy to create Python types that derive from C++ classes. It is even + possible to make use of multiple inheritance to declare a Python class which + has e.g. a C++ and a Python class as bases. However, any attempt to create a + type that has *two or more* C++ classes in its hierarchy of base types will + fail with a fatal error message: ``TypeError: multiple bases have instance + lay-out conflict``. Core Python types that are implemented in C (e.g. + ``dict``, ``list``, ``Exception``, etc.) also fall under this combination + and cannot be combined with C++ types bound using pybind11 via multiple + inheritance. diff --git a/docs/advanced/exceptions.rst b/docs/advanced/exceptions.rst new file mode 100644 index 000000000..348337916 --- /dev/null +++ b/docs/advanced/exceptions.rst @@ -0,0 +1,142 @@ +Exceptions +########## + +Built-in exception translation +============================== + +When C++ code invoked from Python throws an ``std::exception``, it is +automatically converted into a Python ``Exception``. pybind11 defines multiple +special exception classes that will map to different types of Python +exceptions: + +.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| + ++--------------------------------------+------------------------------+ +| C++ exception type | Python exception type | ++======================================+==============================+ +| :class:`std::exception` | ``RuntimeError`` | ++--------------------------------------+------------------------------+ +| :class:`std::bad_alloc` | ``MemoryError`` | ++--------------------------------------+------------------------------+ +| :class:`std::domain_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::invalid_argument` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::length_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::out_of_range` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`std::range_error` | ``ValueError`` | ++--------------------------------------+------------------------------+ +| :class:`pybind11::stop_iteration` | ``StopIteration`` (used to | +| | implement custom iterators) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::index_error` | ``IndexError`` (used to | +| | indicate out of bounds | +| | accesses in ``__getitem__``, | +| | ``__setitem__``, etc.) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::value_error` | ``ValueError`` (used to | +| | indicate wrong value passed | +| | in ``container.remove(...)`` | ++--------------------------------------+------------------------------+ +| :class:`pybind11::key_error` | ``KeyError`` (used to | +| | indicate out of bounds | +| | accesses in ``__getitem__``, | +| | ``__setitem__`` in dict-like | +| | objects, etc.) | ++--------------------------------------+------------------------------+ +| :class:`pybind11::error_already_set` | Indicates that the Python | +| | exception flag has already | +| | been initialized | ++--------------------------------------+------------------------------+ + +When a Python function invoked from C++ throws an exception, it is converted +into a C++ exception of type :class:`error_already_set` whose string payload +contains a textual summary. + +There is also a special exception :class:`cast_error` that is thrown by +:func:`handle::call` when the input arguments cannot be converted to Python +objects. + +Registering custom translators +============================== + +If the default exception conversion policy described above is insufficient, +pybind11 also provides support for registering custom exception translators. +To register a simple exception conversion that translates a C++ exception into +a new Python exception using the C++ exception's ``what()`` method, a helper +function is available: + +.. code-block:: cpp + + py::register_exception(module, "PyExp"); + +This call creates a Python exception class with the name ``PyExp`` in the given +module and automatically converts any encountered exceptions of type ``CppExp`` +into Python exceptions of type ``PyExp``. + +When more advanced exception translation is needed, the function +``py::register_exception_translator(translator)`` can be used to register +functions that can translate arbitrary exception types (and which may include +additional logic to do so). The function takes a stateless callable (e.g. a +function pointer or a lambda function without captured variables) with the call +signature ``void(std::exception_ptr)``. + +When a C++ exception is thrown, the registered exception translators are tried +in reverse order of registration (i.e. the last registered translator gets the +first shot at handling the exception). + +Inside the translator, ``std::rethrow_exception`` should be used within +a try block to re-throw the exception. One or more catch clauses to catch +the appropriate exceptions should then be used with each clause using +``PyErr_SetString`` to set a Python exception or ``ex(string)`` to set +the python exception to a custom exception type (see below). + +To declare a custom Python exception type, declare a ``py::exception`` variable +and use this in the associated exception translator (note: it is often useful +to make this a static declaration when using it inside a lambda expression +without requiring capturing). + + +The following example demonstrates this for a hypothetical exception classes +``MyCustomException`` and ``OtherException``: the first is translated to a +custom python exception ``MyCustomError``, while the second is translated to a +standard python RuntimeError: + +.. code-block:: cpp + + static py::exception exc(m, "MyCustomError"); + py::register_exception_translator([](std::exception_ptr p) { + try { + if (p) std::rethrow_exception(p); + } catch (const MyCustomException &e) { + exc(e.what()); + } catch (const OtherException &e) { + PyErr_SetString(PyExc_RuntimeError, e.what()); + } + }); + +Multiple exceptions can be handled by a single translator, as shown in the +example above. If the exception is not caught by the current translator, the +previously registered one gets a chance. + +If none of the registered exception translators is able to handle the +exception, it is handled by the default converter as described in the previous +section. + +.. seealso:: + + The file :file:`tests/test_exceptions.cpp` contains examples + of various custom exception translators and custom exception types. + +.. note:: + + You must call either ``PyErr_SetString`` or a custom exception's call + operator (``exc(string)``) for every exception caught in a custom exception + translator. Failure to do so will cause Python to crash with ``SystemError: + error return without exception set``. + + Exceptions that you do not plan to handle should simply not be caught, or + may be explicity (re-)thrown to delegate it to the other, + previously-declared existing exception translators. diff --git a/docs/advanced/functions.rst b/docs/advanced/functions.rst new file mode 100644 index 000000000..5c697b12c --- /dev/null +++ b/docs/advanced/functions.rst @@ -0,0 +1,263 @@ +Functions +######### + +Before proceeding with this section, make sure that you are already familiar +with the basics of binding functions and classes, as explained in :doc:`/basics` +and :doc:`/classes`. The following guide is applicable to both free and member +functions, i.e. *methods* in Python. + +Return value policies +===================== + +Python and C++ use wildly different ways of managing the memory and lifetime of +objects managed by them. This can lead to issues when creating bindings for +functions that return a non-trivial type. Just by looking at the type +information, it is not clear whether Python should take charge of the returned +value and eventually free its resources, or if this is handled on the C++ side. +For this reason, pybind11 provides a several `return value policy` annotations +that can be passed to the :func:`module::def` and :func:`class_::def` +functions. The default policy is :enum:`return_value_policy::automatic`. + +Return value policies can also be applied to properties, in which case the +arguments must be passed through the :class:`cpp_function` constructor: + +.. code-block:: cpp + + class_(m, "MyClass") + def_property("data" + py::cpp_function(&MyClass::getData, py::return_value_policy::copy), + py::cpp_function(&MyClass::setData) + ); + +The following table provides an overview of the available return value policies: + +.. tabularcolumns:: |p{0.5\textwidth}|p{0.45\textwidth}| + ++--------------------------------------------------+----------------------------------------------------------------------------+ +| Return value policy | Description | ++==================================================+============================================================================+ +| :enum:`return_value_policy::automatic` | This is the default return value policy, which falls back to the policy | +| | :enum:`return_value_policy::take_ownership` when the return value is a | +| | pointer. Otherwise, it uses :enum:`return_value::move` or | +| | :enum:`return_value::copy` for rvalue and lvalue references, respectively. | +| | See below for a description of what all of these different policies do. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::automatic_reference` | As above, but use policy :enum:`return_value_policy::reference` when the | +| | return value is a pointer. This is the default conversion policy for | +| | function arguments when calling Python functions manually from C++ code | +| | (i.e. via handle::operator()). You probably won't need to use this. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::take_ownership` | Reference an existing object (i.e. do not create a new copy) and take | +| | ownership. Python will call the destructor and delete operator when the | +| | object's reference count reaches zero. Undefined behavior ensues when the | +| | C++ side does the same. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::copy` | Create a new copy of the returned object, which will be owned by Python. | +| | This policy is comparably safe because the lifetimes of the two instances | +| | are decoupled. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::move` | Use ``std::move`` to move the return value contents into a new instance | +| | that will be owned by Python. This policy is comparably safe because the | +| | lifetimes of the two instances (move source and destination) are decoupled.| ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference` | Reference an existing object, but do not take ownership. The C++ side is | +| | responsible for managing the object's lifetime and deallocating it when | +| | it is no longer used. Warning: undefined behavior will ensue when the C++ | +| | side deletes an object that is still referenced and used by Python. | ++--------------------------------------------------+----------------------------------------------------------------------------+ +| :enum:`return_value_policy::reference_internal` | Indicates that the lifetime of the return value is tied to the lifetime | +| | of a parent object, namely the implicit ``this``, or ``self`` argument of | +| | the called method or property. Internally, this policy works just like | +| | :enum:`return_value_policy::reference` but additionally applies a | +| | ``keep_alive<0, 1>`` *call policy* (described in the next section) that | +| | prevents the parent object from being garbage collected as long as the | +| | return value is referenced by Python. This is the default policy for | +| | property getters created via ``def_property``, ``def_readwrite``, etc. | ++--------------------------------------------------+----------------------------------------------------------------------------+ + +.. warning:: + + Code with invalid return value policies might access unitialized memory or + free data structures multiple times, which can lead to hard-to-debug + non-determinism and segmentation faults, hence it is worth spending the + time to understand all the different options in the table above. + +One important aspect of the above policies is that they only apply to instances +which pybind11 has *not* seen before, in which case the policy clarifies +essential questions about the return value's lifetime and ownership. When +pybind11 knows the instance already (as identified by its type and address in +memory), it will return the existing Python object wrapper rather than creating +a new copy. + +.. note:: + + The next section on :ref:`call_policies` discusses *call policies* that can be + specified *in addition* to a return value policy from the list above. Call + policies indicate reference relationships that can involve both return values + and parameters of functions. + +.. note:: + + As an alternative to elaborate call policies and lifetime management logic, + consider using smart pointers (see the section on :ref:`smart_pointers` for + details). Smart pointers can tell whether an object is still referenced from + C++ or Python, which generally eliminates the kinds of inconsistencies that + can lead to crashes or undefined behavior. For functions returning smart + pointers, it is not necessary to specify a return value policy. + +.. _call_policies: + +Additional call policies +======================== + +In addition to the above return value policies, further `call policies` can be +specified to indicate dependencies between parameters. There is currently just +one policy named ``keep_alive``, which indicates that the +argument with index ``Patient`` should be kept alive at least until the +argument with index ``Nurse`` is freed by the garbage collector. Argument +indices start at one, while zero refers to the return value. For methods, index +``1`` refers to the implicit ``this`` pointer, while regular arguments begin at +index ``2``. Arbitrarily many call policies can be specified. When a ``Nurse`` +with value ``None`` is detected at runtime, the call policy does nothing. + +This feature internally relies on the ability to create a *weak reference* to +the nurse object, which is permitted by all classes exposed via pybind11. When +the nurse object does not support weak references, an exception will be thrown. + +Consider the following example: here, the binding code for a list append +operation ties the lifetime of the newly added element to the underlying +container: + +.. code-block:: cpp + + py::class_(m, "List") + .def("append", &List::append, py::keep_alive<1, 2>()); + +.. note:: + + ``keep_alive`` is analogous to the ``with_custodian_and_ward`` (if Nurse, + Patient != 0) and ``with_custodian_and_ward_postcall`` (if Nurse/Patient == + 0) policies from Boost.Python. + +.. seealso:: + + The file :file:`tests/test_keep_alive.cpp` contains a complete example + that demonstrates using :class:`keep_alive` in more detail. + +.. _python_objects_as_args: + +Python objects as arguments +=========================== + +pybind11 exposes all major Python types using thin C++ wrapper classes. These +wrapper classes can also be used as parameters of functions in bindings, which +makes it possible to directly work with native Python types on the C++ side. +For instance, the following statement iterates over a Python ``dict``: + +.. code-block:: cpp + + void print_dict(py::dict dict) { + /* Easily interact with Python types */ + for (auto item : dict) + std::cout << "key=" << item.first << ", " + << "value=" << item.second << std::endl; + } + +It can be exported: + +.. code-block:: cpp + + m.def("print_dict", &print_dict); + +And used in Python as usual: + +.. code-block:: pycon + + >>> print_dict({'foo': 123, 'bar': 'hello'}) + key=foo, value=123 + key=bar, value=hello + +For more information on using Python objects in C++, see :doc:`/advanced/pycpp/index`. + +Accepting \*args and \*\*kwargs +=============================== + +Python provides a useful mechanism to define functions that accept arbitrary +numbers of arguments and keyword arguments: + +.. code-block:: python + + def generic(*args, **kwargs): + ... # do something with args and kwargs + +Such functions can also be created using pybind11: + +.. code-block:: cpp + + void generic(py::args args, py::kwargs kwargs) { + /// .. do something with args + if (kwargs) + /// .. do something with kwargs + } + + /// Binding code + m.def("generic", &generic); + +The class ``py::args`` derives from ``py::tuple`` and ``py::kwargs`` derives +from ``py::dict``. Note that the ``kwargs`` argument is invalid if no keyword +arguments were actually provided. Please refer to the other examples for +details on how to iterate over these, and on how to cast their entries into +C++ objects. A demonstration is also available in +``tests/test_kwargs_and_defaults.cpp``. + +.. warning:: + + Unlike Python, pybind11 does not allow combining normal parameters with the + ``args`` / ``kwargs`` special parameters. + +Default arguments revisited +=========================== + +The section on :ref:`default_args` previously discussed basic usage of default +arguments using pybind11. One noteworthy aspect of their implementation is that +default arguments are converted to Python objects right at declaration time. +Consider the following example: + +.. code-block:: cpp + + py::class_("MyClass") + .def("myFunction", py::arg("arg") = SomeType(123)); + +In this case, pybind11 must already be set up to deal with values of the type +``SomeType`` (via a prior instantiation of ``py::class_``), or an +exception will be thrown. + +Another aspect worth highlighting is that the "preview" of the default argument +in the function signature is generated using the object's ``__repr__`` method. +If not available, the signature may not be very helpful, e.g.: + +.. code-block:: pycon + + FUNCTIONS + ... + | myFunction(...) + | Signature : (MyClass, arg : SomeType = ) -> NoneType + ... + +The first way of addressing this is by defining ``SomeType.__repr__``. +Alternatively, it is possible to specify the human-readable preview of the +default argument manually using the ``arg_v`` notation: + +.. code-block:: cpp + + py::class_("MyClass") + .def("myFunction", py::arg_v("arg", SomeType(123), "SomeType(123)")); + +Sometimes it may be necessary to pass a null pointer value as a default +argument. In this case, remember to cast it to the underlying type in question, +like so: + +.. code-block:: cpp + + py::class_("MyClass") + .def("myFunction", py::arg("arg") = (SomeType *) nullptr); diff --git a/docs/advanced/misc.rst b/docs/advanced/misc.rst new file mode 100644 index 000000000..2968f8ac1 --- /dev/null +++ b/docs/advanced/misc.rst @@ -0,0 +1,187 @@ +Miscellaneous +############# + +.. _macro_notes: + +General notes regarding convenience macros +========================================== + +pybind11 provides a few convenience macros such as +:func:`PYBIND11_MAKE_OPAQUE` and :func:`PYBIND11_DECLARE_HOLDER_TYPE`, and +``PYBIND11_OVERLOAD_*``. Since these are "just" macros that are evaluated +in the preprocessor (which has no concept of types), they *will* get confused +by commas in a template argument such as ``PYBIND11_OVERLOAD(MyReturnValue, myFunc)``. In this case, the preprocessor assumes that the comma indicates +the beginning of the next parameter. Use a ``typedef`` to bind the template to +another name and use it in the macro to avoid this problem. + + +Global Interpreter Lock (GIL) +============================= + +The classes :class:`gil_scoped_release` and :class:`gil_scoped_acquire` can be +used to acquire and release the global interpreter lock in the body of a C++ +function call. In this way, long-running C++ code can be parallelized using +multiple Python threads. Taking :ref:`overriding_virtuals` as an example, this +could be realized as follows (important changes highlighted): + +.. code-block:: cpp + :emphasize-lines: 8,9,33,34 + + class PyAnimal : public Animal { + public: + /* Inherit the constructors */ + using Animal::Animal; + + /* Trampoline (need one for each virtual function) */ + std::string go(int n_times) { + /* Acquire GIL before calling Python code */ + py::gil_scoped_acquire acquire; + + PYBIND11_OVERLOAD_PURE( + std::string, /* Return type */ + Animal, /* Parent class */ + go, /* Name of function */ + n_times /* Argument(s) */ + ); + } + }; + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + + py::class_ animal(m, "Animal"); + animal + .def(py::init<>()) + .def("go", &Animal::go); + + py::class_(m, "Dog", animal) + .def(py::init<>()); + + m.def("call_go", [](Animal *animal) -> std::string { + /* Release GIL before calling into (potentially long-running) C++ code */ + py::gil_scoped_release release; + return call_go(animal); + }); + + return m.ptr(); + } + + +Binding sequence data types, iterators, the slicing protocol, etc. +================================================================== + +Please refer to the supplemental example for details. + +.. seealso:: + + The file :file:`tests/test_sequences_and_iterators.cpp` contains a + complete example that shows how to bind a sequence data type, including + length queries (``__len__``), iterators (``__iter__``), the slicing + protocol and other kinds of useful operations. + + +Partitioning code over multiple extension modules +================================================= + +It's straightforward to split binding code over multiple extension modules, +while referencing types that are declared elsewhere. Everything "just" works +without any special precautions. One exception to this rule occurs when +extending a type declared in another extension module. Recall the basic example +from Section :ref:`inheritance`. + +.. code-block:: cpp + + py::class_ pet(m, "Pet"); + pet.def(py::init()) + .def_readwrite("name", &Pet::name); + + py::class_(m, "Dog", pet /* <- specify parent */) + .def(py::init()) + .def("bark", &Dog::bark); + +Suppose now that ``Pet`` bindings are defined in a module named ``basic``, +whereas the ``Dog`` bindings are defined somewhere else. The challenge is of +course that the variable ``pet`` is not available anymore though it is needed +to indicate the inheritance relationship to the constructor of ``class_``. +However, it can be acquired as follows: + +.. code-block:: cpp + + py::object pet = (py::object) py::module::import("basic").attr("Pet"); + + py::class_(m, "Dog", pet) + .def(py::init()) + .def("bark", &Dog::bark); + +Alternatively, you can specify the base class as a template parameter option to +``class_``, which performs an automated lookup of the corresponding Python +type. Like the above code, however, this also requires invoking the ``import`` +function once to ensure that the pybind11 binding code of the module ``basic`` +has been executed: + +.. code-block:: cpp + + py::module::import("basic"); + + py::class_(m, "Dog") + .def(py::init()) + .def("bark", &Dog::bark); + +Naturally, both methods will fail when there are cyclic dependencies. + +Note that compiling code which has its default symbol visibility set to +*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the +ability to access types defined in another extension module. Workarounds +include changing the global symbol visibility (not recommended, because it will +lead unnecessarily large binaries) or manually exporting types that are +accessed by multiple extension modules: + +.. code-block:: cpp + + #ifdef _WIN32 + # define EXPORT_TYPE __declspec(dllexport) + #else + # define EXPORT_TYPE __attribute__ ((visibility("default"))) + #endif + + class EXPORT_TYPE Dog : public Animal { + ... + }; + + + +Generating documentation using Sphinx +===================================== + +Sphinx [#f4]_ has the ability to inspect the signatures and documentation +strings in pybind11-based extension modules to automatically generate beautiful +documentation in a variety formats. The python_example repository [#f5]_ contains a +simple example repository which uses this approach. + +There are two potential gotchas when using this approach: first, make sure that +the resulting strings do not contain any :kbd:`TAB` characters, which break the +docstring parsing routines. You may want to use C++11 raw string literals, +which are convenient for multi-line comments. Conveniently, any excess +indentation will be automatically be removed by Sphinx. However, for this to +work, it is important that all lines are indented consistently, i.e.: + +.. code-block:: cpp + + // ok + m.def("foo", &foo, R"mydelimiter( + The foo function + + Parameters + ---------- + )mydelimiter"); + + // *not ok* + m.def("foo", &foo, R"mydelimiter(The foo function + + Parameters + ---------- + )mydelimiter"); + +.. [#f4] http://www.sphinx-doc.org +.. [#f5] http://github.com/pybind/python_example diff --git a/docs/advanced/pycpp/index.rst b/docs/advanced/pycpp/index.rst new file mode 100644 index 000000000..6885bdcff --- /dev/null +++ b/docs/advanced/pycpp/index.rst @@ -0,0 +1,13 @@ +Python C++ interface +#################### + +pybind11 exposes Python types and functions using thin C++ wrappers, which +makes it possible to conveniently call Python code from C++ without resorting +to Python's C API. + +.. toctree:: + :maxdepth: 2 + + object + numpy + utilities diff --git a/docs/advanced/pycpp/numpy.rst b/docs/advanced/pycpp/numpy.rst new file mode 100644 index 000000000..8b46b7c83 --- /dev/null +++ b/docs/advanced/pycpp/numpy.rst @@ -0,0 +1,299 @@ +.. _numpy: + +NumPy +##### + +Buffer protocol +=============== + +Python supports an extremely general and convenient approach for exchanging +data between plugin libraries. Types can expose a buffer view [#f2]_, which +provides fast direct access to the raw internal data representation. Suppose we +want to bind the following simplistic Matrix class: + +.. code-block:: cpp + + class Matrix { + public: + Matrix(size_t rows, size_t cols) : m_rows(rows), m_cols(cols) { + m_data = new float[rows*cols]; + } + float *data() { return m_data; } + size_t rows() const { return m_rows; } + size_t cols() const { return m_cols; } + private: + size_t m_rows, m_cols; + float *m_data; + }; + +The following binding code exposes the ``Matrix`` contents as a buffer object, +making it possible to cast Matrices into NumPy arrays. It is even possible to +completely avoid copy operations with Python expressions like +``np.array(matrix_instance, copy = False)``. + +.. code-block:: cpp + + py::class_(m, "Matrix") + .def_buffer([](Matrix &m) -> py::buffer_info { + return py::buffer_info( + m.data(), /* Pointer to buffer */ + sizeof(float), /* Size of one scalar */ + py::format_descriptor::format(), /* Python struct-style format descriptor */ + 2, /* Number of dimensions */ + { m.rows(), m.cols() }, /* Buffer dimensions */ + { sizeof(float) * m.rows(), /* Strides (in bytes) for each index */ + sizeof(float) } + ); + }); + +The snippet above binds a lambda function, which can create ``py::buffer_info`` +description records on demand describing a given matrix. The contents of +``py::buffer_info`` mirror the Python buffer protocol specification. + +.. code-block:: cpp + + struct buffer_info { + void *ptr; + size_t itemsize; + std::string format; + int ndim; + std::vector shape; + std::vector strides; + }; + +To create a C++ function that can take a Python buffer object as an argument, +simply use the type ``py::buffer`` as one of its arguments. Buffers can exist +in a great variety of configurations, hence some safety checks are usually +necessary in the function body. Below, you can see an basic example on how to +define a custom constructor for the Eigen double precision matrix +(``Eigen::MatrixXd``) type, which supports initialization from compatible +buffer objects (e.g. a NumPy matrix). + +.. code-block:: cpp + + /* Bind MatrixXd (or some other Eigen type) to Python */ + typedef Eigen::MatrixXd Matrix; + + typedef Matrix::Scalar Scalar; + constexpr bool rowMajor = Matrix::Flags & Eigen::RowMajorBit; + + py::class_(m, "Matrix") + .def("__init__", [](Matrix &m, py::buffer b) { + typedef Eigen::Stride Strides; + + /* Request a buffer descriptor from Python */ + py::buffer_info info = b.request(); + + /* Some sanity checks ... */ + if (info.format != py::format_descriptor::format()) + throw std::runtime_error("Incompatible format: expected a double array!"); + + if (info.ndim != 2) + throw std::runtime_error("Incompatible buffer dimension!"); + + auto strides = Strides( + info.strides[rowMajor ? 0 : 1] / sizeof(Scalar), + info.strides[rowMajor ? 1 : 0] / sizeof(Scalar)); + + auto map = Eigen::Map( + static_cat(info.ptr), info.shape[0], info.shape[1], strides); + + new (&m) Matrix(map); + }); + +For reference, the ``def_buffer()`` call for this Eigen data type should look +as follows: + +.. code-block:: cpp + + .def_buffer([](Matrix &m) -> py::buffer_info { + return py::buffer_info( + m.data(), /* Pointer to buffer */ + sizeof(Scalar), /* Size of one scalar */ + /* Python struct-style format descriptor */ + py::format_descriptor::format(), + /* Number of dimensions */ + 2, + /* Buffer dimensions */ + { (size_t) m.rows(), + (size_t) m.cols() }, + /* Strides (in bytes) for each index */ + { sizeof(Scalar) * (rowMajor ? m.cols() : 1), + sizeof(Scalar) * (rowMajor ? 1 : m.rows()) } + ); + }) + +For a much easier approach of binding Eigen types (although with some +limitations), refer to the section on :doc:`/advanced/cast/eigen`. + +.. seealso:: + + The file :file:`tests/test_buffers.cpp` contains a complete example + that demonstrates using the buffer protocol with pybind11 in more detail. + +.. [#f2] http://docs.python.org/3/c-api/buffer.html + +Arrays +====== + +By exchanging ``py::buffer`` with ``py::array`` in the above snippet, we can +restrict the function so that it only accepts NumPy arrays (rather than any +type of Python object satisfying the buffer protocol). + +In many situations, we want to define a function which only accepts a NumPy +array of a certain data type. This is possible via the ``py::array_t`` +template. For instance, the following function requires the argument to be a +NumPy array containing double precision values. + +.. code-block:: cpp + + void f(py::array_t array); + +When it is invoked with a different type (e.g. an integer or a list of +integers), the binding code will attempt to cast the input into a NumPy array +of the requested type. Note that this feature requires the +:file:``pybind11/numpy.h`` header to be included. + +Data in NumPy arrays is not guaranteed to packed in a dense manner; +furthermore, entries can be separated by arbitrary column and row strides. +Sometimes, it can be useful to require a function to only accept dense arrays +using either the C (row-major) or Fortran (column-major) ordering. This can be +accomplished via a second template argument with values ``py::array::c_style`` +or ``py::array::f_style``. + +.. code-block:: cpp + + void f(py::array_t array); + +The ``py::array::forcecast`` argument is the default value of the second +template parameter, and it ensures that non-conforming arguments are converted +into an array satisfying the specified requirements instead of trying the next +function overload. + +Structured types +================ + +In order for ``py::array_t`` to work with structured (record) types, we first need +to register the memory layout of the type. This can be done via ``PYBIND11_NUMPY_DTYPE`` +macro which expects the type followed by field names: + +.. code-block:: cpp + + struct A { + int x; + double y; + }; + + struct B { + int z; + A a; + }; + + PYBIND11_NUMPY_DTYPE(A, x, y); + PYBIND11_NUMPY_DTYPE(B, z, a); + + /* now both A and B can be used as template arguments to py::array_t */ + +Vectorizing functions +===================== + +Suppose we want to bind a function with the following signature to Python so +that it can process arbitrary NumPy array arguments (vectors, matrices, general +N-D arrays) in addition to its normal arguments: + +.. code-block:: cpp + + double my_func(int x, float y, double z); + +After including the ``pybind11/numpy.h`` header, this is extremely simple: + +.. code-block:: cpp + + m.def("vectorized_func", py::vectorize(my_func)); + +Invoking the function like below causes 4 calls to be made to ``my_func`` with +each of the array elements. The significant advantage of this compared to +solutions like ``numpy.vectorize()`` is that the loop over the elements runs +entirely on the C++ side and can be crunched down into a tight, optimized loop +by the compiler. The result is returned as a NumPy array of type +``numpy.dtype.float64``. + +.. code-block:: pycon + + >>> x = np.array([[1, 3],[5, 7]]) + >>> y = np.array([[2, 4],[6, 8]]) + >>> z = 3 + >>> result = vectorized_func(x, y, z) + +The scalar argument ``z`` is transparently replicated 4 times. The input +arrays ``x`` and ``y`` are automatically converted into the right types (they +are of type ``numpy.dtype.int64`` but need to be ``numpy.dtype.int32`` and +``numpy.dtype.float32``, respectively) + +Sometimes we might want to explicitly exclude an argument from the vectorization +because it makes little sense to wrap it in a NumPy array. For instance, +suppose the function signature was + +.. code-block:: cpp + + double my_func(int x, float y, my_custom_type *z); + +This can be done with a stateful Lambda closure: + +.. code-block:: cpp + + // Vectorize a lambda function with a capture object (e.g. to exclude some arguments from the vectorization) + m.def("vectorized_func", + [](py::array_t x, py::array_t y, my_custom_type *z) { + auto stateful_closure = [z](int x, float y) { return my_func(x, y, z); }; + return py::vectorize(stateful_closure)(x, y); + } + ); + +In cases where the computation is too complicated to be reduced to +``vectorize``, it will be necessary to create and access the buffer contents +manually. The following snippet contains a complete example that shows how this +works (the code is somewhat contrived, since it could have been done more +simply using ``vectorize``). + +.. code-block:: cpp + + #include + #include + + namespace py = pybind11; + + py::array_t add_arrays(py::array_t input1, py::array_t input2) { + auto buf1 = input1.request(), buf2 = input2.request(); + + if (buf1.ndim != 1 || buf2.ndim != 1) + throw std::runtime_error("Number of dimensions must be one"); + + if (buf1.size != buf2.size) + throw std::runtime_error("Input shapes must match"); + + /* No pointer is passed, so NumPy will allocate the buffer */ + auto result = py::array_t(buf1.size); + + auto buf3 = result.request(); + + double *ptr1 = (double *) buf1.ptr, + *ptr2 = (double *) buf2.ptr, + *ptr3 = (double *) buf3.ptr; + + for (size_t idx = 0; idx < buf1.shape[0]; idx++) + ptr3[idx] = ptr1[idx] + ptr2[idx]; + + return result; + } + + PYBIND11_PLUGIN(test) { + py::module m("test"); + m.def("add_arrays", &add_arrays, "Add two NumPy arrays"); + return m.ptr(); + } + +.. seealso:: + + The file :file:`tests/test_numpy_vectorize.cpp` contains a complete + example that demonstrates using :func:`vectorize` in more detail. diff --git a/docs/advanced/pycpp/object.rst b/docs/advanced/pycpp/object.rst new file mode 100644 index 000000000..8fc165d16 --- /dev/null +++ b/docs/advanced/pycpp/object.rst @@ -0,0 +1,96 @@ +Python types +############ + +Available wrappers +================== + +All major Python types are available as thin C++ wrapper classes. These +can also be used as function parameters -- see :ref:`python_objects_as_args`. + +Available types include :class:`handle`, :class:`object`, :class:`bool_`, +:class:`int_`, :class:`float_`, :class:`str`, :class:`bytes`, :class:`tuple`, +:class:`list`, :class:`dict`, :class:`slice`, :class:`none`, :class:`capsule`, +:class:`iterable`, :class:`iterator`, :class:`function`, :class:`buffer`, +:class:`array`, and :class:`array_t`. + +Casting back and forth +====================== + +In this kind of mixed code, it is often necessary to convert arbitrary C++ +types to Python, which can be done using :func:`py::cast`: + +.. code-block:: cpp + + MyClass *cls = ..; + py::object obj = py::cast(cls); + +The reverse direction uses the following syntax: + +.. code-block:: cpp + + py::object obj = ...; + MyClass *cls = obj.cast(); + +When conversion fails, both directions throw the exception :class:`cast_error`. + +Calling Python functions +======================== + +It is also possible to call python functions via ``operator()``. + +.. code-block:: cpp + + py::function f = <...>; + py::object result_py = f(1234, "hello", some_instance); + MyClass &result = result_py.cast(); + +Keyword arguments are also supported. In Python, there is the usual call syntax: + +.. code-block:: python + + def f(number, say, to): + ... # function code + + f(1234, say="hello", to=some_instance) # keyword call in Python + +In C++, the same call can be made using: + +.. code-block:: cpp + + using pybind11::literals; // to bring in the `_a` literal + f(1234, "say"_a="hello", "to"_a=some_instance); // keyword call in C++ + +Unpacking of ``*args`` and ``**kwargs`` is also possible and can be mixed with +other arguments: + +.. code-block:: cpp + + // * unpacking + py::tuple args = py::make_tuple(1234, "hello", some_instance); + f(*args); + + // ** unpacking + py::dict kwargs = py::dict("number"_a=1234, "say"_a="hello", "to"_a=some_instance); + f(**kwargs); + + // mixed keywords, * and ** unpacking + py::tuple args = py::make_tuple(1234); + py::dict kwargs = py::dict("to"_a=some_instance); + f(*args, "say"_a="hello", **kwargs); + +Generalized unpacking according to PEP448_ is also supported: + +.. code-block:: cpp + + py::dict kwargs1 = py::dict("number"_a=1234); + py::dict kwargs2 = py::dict("to"_a=some_instance); + f(**kwargs1, "say"_a="hello", **kwargs2); + +.. seealso:: + + The file :file:`tests/test_python_types.cpp` contains a complete + example that demonstrates passing native Python types in more detail. The + file :file:`tests/test_callbacks.cpp` presents a few examples of calling + Python functions from C++, including keywords arguments and unpacking. + +.. _PEP448: https://www.python.org/dev/peps/pep-0448/ diff --git a/docs/advanced/pycpp/utilities.rst b/docs/advanced/pycpp/utilities.rst new file mode 100644 index 000000000..ba0dbef88 --- /dev/null +++ b/docs/advanced/pycpp/utilities.rst @@ -0,0 +1,57 @@ +Utilities +######### + +Using Python's print function in C++ +==================================== + +The usual way to write output in C++ is using ``std::cout`` while in Python one +would use ``print``. Since these methods use different buffers, mixing them can +lead to output order issues. To resolve this, pybind11 modules can use the +:func:`py::print` function which writes to Python's ``sys.stdout`` for consistency. + +Python's ``print`` function is replicated in the C++ API including optional +keyword arguments ``sep``, ``end``, ``file``, ``flush``. Everything works as +expected in Python: + +.. code-block:: cpp + + py::print(1, 2.0, "three"); // 1 2.0 three + py::print(1, 2.0, "three", "sep"_a="-"); // 1-2.0-three + + auto args = py::make_tuple("unpacked", true); + py::print("->", *args, "end"_a="<-"); // -> unpacked True <- + +Evaluating Python expressions from strings and files +==================================================== + +pybind11 provides the :func:`eval` and :func:`eval_file` functions to evaluate +Python expressions and statements. The following example illustrates how they +can be used. + +Both functions accept a template parameter that describes how the argument +should be interpreted. Possible choices include ``eval_expr`` (isolated +expression), ``eval_single_statement`` (a single statement, return value is +always ``none``), and ``eval_statements`` (sequence of statements, return value +is always ``none``). + +.. code-block:: cpp + + // At beginning of file + #include + + ... + + // Evaluate in scope of main module + py::object scope = py::module::import("__main__").attr("__dict__"); + + // Evaluate an isolated expression + int result = py::eval("my_variable + 10", scope).cast(); + + // Evaluate a sequence of statements + py::eval( + "print('Hello')\n" + "print('world!');", + scope); + + // Evaluate the statements in an separate Python file on disk + py::eval_file("script.py", scope); diff --git a/docs/advanced/smart_ptrs.rst b/docs/advanced/smart_ptrs.rst new file mode 100644 index 000000000..97dc7ab70 --- /dev/null +++ b/docs/advanced/smart_ptrs.rst @@ -0,0 +1,149 @@ +Smart pointers +############## + +Unique pointers +=============== + +Given a class ``Example`` with Python bindings, it's possible to return +instances wrapped in C++11 unique pointers, like so + +.. code-block:: cpp + + std::unique_ptr create_example() { return std::unique_ptr(new Example()); } + +.. code-block:: cpp + + m.def("create_example", &create_example); + +In other words, there is nothing special that needs to be done. While returning +unique pointers in this way is allowed, it is *illegal* to use them as function +arguments. For instance, the following function signature cannot be processed +by pybind11. + +.. code-block:: cpp + + void do_something_with_example(std::unique_ptr ex) { ... } + +The above signature would imply that Python needs to give up ownership of an +object that is passed to this function, which is generally not possible (for +instance, the object might be referenced elsewhere). + +.. _smart_pointers: + +Reference-counting pointers +=========================== + +This section explains how to pass values that are wrapped in "smart" pointer +types with internal reference counting. For the simpler C++11 unique pointers, +refer to the previous section. + +The binding generator for classes, :class:`class_`, can be passed a template +type that denotes a special *holder* type that is used to manage references to +the object. If no such holder type template argument is given, the default for +a type named ``Type`` is ``std::unique_ptr``, which means that the object +is deallocated when Python's reference count goes to zero. + +It is possible to switch to other types of reference counting wrappers or smart +pointers, which is useful in codebases that rely on them. For instance, the +following snippet causes ``std::shared_ptr`` to be used instead. + +.. code-block:: cpp + + py::class_ /* <- holder type */> obj(m, "Example"); + +Note that any particular class can only be associated with a single holder type. + +To enable transparent conversions for functions that take shared pointers as an +argument or that return them, a macro invocation similar to the following must +be declared at the top level before any binding code: + +.. code-block:: cpp + + PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr); + +.. note:: + + The first argument of :func:`PYBIND11_DECLARE_HOLDER_TYPE` should be a + placeholder name that is used as a template parameter of the second + argument. Thus, feel free to use any identifier, but use it consistently on + both sides; also, don't use the name of a type that already exists in your + codebase. + +One potential stumbling block when using holder types is that they need to be +applied consistently. Can you guess what's broken about the following binding +code? + +.. code-block:: cpp + + PYBIND11_DECLARE_HOLDER_TYPE(T, std::shared_ptr); + + class Child { }; + + class Parent { + public: + Parent() : child(std::make_shared()) { } + Child *get_child() { return child.get(); } /* Hint: ** DON'T DO THIS ** */ + private: + std::shared_ptr child; + }; + + PYBIND11_PLUGIN(example) { + py::module m("example"); + + py::class_>(m, "Child"); + + py::class_>(m, "Parent") + .def(py::init<>()) + .def("get_child", &Parent::get_child); + + return m.ptr(); + } + +The following Python code will cause undefined behavior (and likely a +segmentation fault). + +.. code-block:: python + + from example import Parent + print(Parent().get_child()) + +The problem is that ``Parent::get_child()`` returns a pointer to an instance of +``Child``, but the fact that this instance is already managed by +``std::shared_ptr<...>`` is lost when passing raw pointers. In this case, +pybind11 will create a second independent ``std::shared_ptr<...>`` that also +claims ownership of the pointer. In the end, the object will be freed **twice** +since these shared pointers have no way of knowing about each other. + +There are two ways to resolve this issue: + +1. For types that are managed by a smart pointer class, never use raw pointers + in function arguments or return values. In other words: always consistently + wrap pointers into their designated holder types (such as + ``std::shared_ptr<...>``). In this case, the signature of ``get_child()`` + should be modified as follows: + +.. code-block:: cpp + + std::shared_ptr get_child() { return child; } + +2. Adjust the definition of ``Child`` by specifying + ``std::enable_shared_from_this`` (see cppreference_ for details) as a + base class. This adds a small bit of information to ``Child`` that allows + pybind11 to realize that there is already an existing + ``std::shared_ptr<...>`` and communicate with it. In this case, the + declaration of ``Child`` should look as follows: + +.. _cppreference: http://en.cppreference.com/w/cpp/memory/enable_shared_from_this + +.. code-block:: cpp + + class Child : public std::enable_shared_from_this { }; + + +Please take a look at the :ref:`macro_notes` before using this feature. + +.. seealso:: + + The file :file:`tests/test_smart_ptr.cpp` contains a complete example + that demonstrates how to work with custom reference-counting holder types + in more detail. diff --git a/docs/basics.rst b/docs/basics.rst index 339d55955..45272b7ed 100644 --- a/docs/basics.rst +++ b/docs/basics.rst @@ -60,6 +60,19 @@ command line. the tutorial and look at the test cases in the :file:`tests` directory, which exercise all features of pybind11. +Header and namespace conventions +================================ + +For brevity, all code examples assume that the following two lines are present: + +.. code-block:: cpp + + #include + + namespace py = pybind11; + +Some features may require additional headers, but those will be specified as needed. + Creating bindings for a simple function ======================================= @@ -93,6 +106,9 @@ a file named :file:`example.cpp` with the following contents: return m.ptr(); } +.. [#f1] In practice, implementation and binding code will generally be located + in separate files. + The :func:`PYBIND11_PLUGIN` macro creates a function that will be called when an ``import`` statement is issued from within Python. The next line creates a module named ``example`` (with the supplied docstring). The method @@ -235,79 +251,37 @@ The shorthand notation is also available for default arguments: // shorthand m.def("add2", &add, "i"_a=1, "j"_a=2); +Exporting variables +=================== + +To expose a value from C++, use the ``attr`` function to register it in a module +as shown below. Built-in types and general objects (more on that later) can be +converted using the function ``py::cast``. + +.. code-block:: cpp + + PYBIND11_PLUGIN(example) { + py::module m("example", "pybind11 example plugin"); + m.attr("the_answer") = py::cast(42); + m.attr("what") = py::cast("World"); + return m.ptr(); + } + +These are then accessible from Python: + +.. code-block:: pycon + + >>> import example + >>> example.the_answer + 42 + >>> example.what + 'World' + .. _supported_types: Supported data types ==================== -The following basic data types are supported out of the box (some may require -an additional extension header to be included). To pass other data structures -as arguments and return values, refer to the section on binding :ref:`classes`. - -+---------------------------------+--------------------------+-------------------------------+ -| Data type | Description | Header file | -+=================================+==========================+===============================+ -| ``int8_t``, ``uint8_t`` | 8-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int16_t``, ``uint16_t`` | 16-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int32_t``, ``uint32_t`` | 32-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``int64_t``, ``uint64_t`` | 64-bit integers | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``ssize_t``, ``size_t`` | Platform-dependent size | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``float``, ``double`` | Floating point types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``bool`` | Two-state Boolean type | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``char`` | Character literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``wchar_t`` | Wide character literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``const char *`` | UTF-8 string literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``const wchar_t *`` | Wide string literal | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::string`` | STL dynamic UTF-8 string | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::wstring`` | STL dynamic wide string | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::pair`` | Pair of two custom types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::tuple<...>`` | Arbitrary tuple of types | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::reference_wrapper<...>`` | Reference type wrapper | :file:`pybind11/pybind11.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::complex`` | Complex numbers | :file:`pybind11/complex.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::array`` | STL static array | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::vector`` | STL dynamic array | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::list`` | STL linked list | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::map`` | STL ordered map | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::unordered_map`` | STL unordered map | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::set`` | STL ordered set | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::unordered_set`` | STL unordered set | :file:`pybind11/stl.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::function<...>`` | STL polymorphic function | :file:`pybind11/functional.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::chrono::duration<...>`` | STL time duration | :file:`pybind11/chrono.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``std::chrono::time_point<...>``| STL date/time | :file:`pybind11/chrono.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::Matrix<...>`` | Eigen: dense matrix | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::Map<...>`` | Eigen: mapped memory | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ -| ``Eigen::SparseMatrix<...>`` | Eigen: sparse matrix | :file:`pybind11/eigen.h` | -+---------------------------------+--------------------------+-------------------------------+ - - -.. [#f1] In practice, implementation and binding code will generally be located - in separate files. +A large number of data types are supported out of the box and can be used +seamlessly as functions arguments, return values or with ``py::cast`` in general. +For a full overview, see the :doc:`advanced/cast/index` section. diff --git a/docs/index.rst b/docs/index.rst index ab9a1fc82..cedf65209 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -10,15 +10,36 @@ pybind11 --- Seamless operability between C++11 and Python Contents: .. toctree:: - :maxdepth: 2 + :maxdepth: 1 intro + changelog + +.. toctree:: + :caption: The Basics + :maxdepth: 2 + basics classes - advanced compiling + +.. toctree:: + :caption: Advanced Topics + :maxdepth: 2 + + advanced/functions + advanced/classes + advanced/exceptions + advanced/smart_ptrs + advanced/cast/index + advanced/pycpp/index + advanced/misc + +.. toctree:: + :caption: Extra Information + :maxdepth: 1 + + faq benchmark limitations - faq reference - changelog diff --git a/docs/reference.rst b/docs/reference.rst index 7edc43a2d..542259eba 100644 --- a/docs/reference.rst +++ b/docs/reference.rst @@ -215,9 +215,9 @@ Passing extra arguments to the def function .. function:: arg::arg(const char *name) -.. function:: template arg_t arg::operator=(const T &value) +.. function:: template arg_v arg::operator=(T &&value) -.. class:: template arg_t : public arg +.. class:: arg_v : public arg Represents a named argument with a default value