Classes ####### This section presents advanced binding code for classes and it is assumed that you are already familiar with the basics from :doc:`/classes`. .. _overriding_virtuals: Overriding virtual functions in Python ====================================== Suppose that a C++ class or interface has a virtual function that we'd like to to override from within Python (we'll focus on the class ``Animal``; ``Dog`` is given as a specific example of how one would do this with traditional C++ code). .. code-block:: cpp class Animal { public: virtual ~Animal() { } virtual std::string go(int n_times) = 0; }; class Dog : public Animal { public: std::string go(int n_times) override { std::string result; for (int i=0; igo(3); } Normally, the binding code for these classes would look as follows: .. code-block:: cpp PYBIND11_MODULE(example, m) { py::class_ animal(m, "Animal"); animal .def("go", &Animal::go); py::class_(m, "Dog", animal) .def(py::init<>()); m.def("call_go", &call_go); } However, these bindings are impossible to extend: ``Animal`` is not constructible, and we clearly require some kind of "trampoline" that redirects virtual calls back to Python. Defining a new type of ``Animal`` from within Python is possible but requires a helper class that is defined as follows: .. code-block:: cpp class PyAnimal : public Animal { public: /* Inherit the constructors */ using Animal::Animal; /* Trampoline (need one for each virtual function) */ std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE( std::string, /* Return type */ Animal, /* Parent class */ go, /* Name of function in C++ (must match Python name) */ n_times /* Argument(s) */ ); } }; The macro :func:`PYBIND11_OVERLOAD_PURE` should be used for pure virtual functions, and :func:`PYBIND11_OVERLOAD` should be used for functions which have a default implementation. There are also two alternate macros :func:`PYBIND11_OVERLOAD_PURE_NAME` and :func:`PYBIND11_OVERLOAD_NAME` which take a string-valued name argument between the *Parent class* and *Name of the function* slots, which defines the name of function in Python. This is required when the C++ and Python versions of the function have different names, e.g. ``operator()`` vs ``__call__``. The binding code also needs a few minor adaptations (highlighted): .. code-block:: cpp :emphasize-lines: 2,4,5 PYBIND11_MODULE(example, m) { py::class_ animal(m, "Animal"); animal .def(py::init<>()) .def("go", &Animal::go); py::class_(m, "Dog", animal) .def(py::init<>()); m.def("call_go", &call_go); } Importantly, pybind11 is made aware of the trampoline helper class by specifying it as an extra template argument to :class:`class_`. (This can also be combined with other template arguments such as a custom holder type; the order of template types does not matter). Following this, we are able to define a constructor as usual. Bindings should be made against the actual class, not the trampoline helper class. .. code-block:: cpp py::class_ animal(m, "Animal"); animal .def(py::init<>()) .def("go", &PyAnimal::go); /* <--- THIS IS WRONG, use &Animal::go */ Note, however, that the above is sufficient for allowing python classes to extend ``Animal``, but not ``Dog``: see :ref:`virtual_and_inheritance` for the necessary steps required to providing proper overload support for inherited classes. The Python session below shows how to override ``Animal::go`` and invoke it via a virtual method call. .. code-block:: pycon >>> from example import * >>> d = Dog() >>> call_go(d) u'woof! woof! woof! ' >>> class Cat(Animal): ... def go(self, n_times): ... return "meow! " * n_times ... >>> c = Cat() >>> call_go(c) u'meow! meow! meow! ' If you are defining a custom constructor in a derived Python class, you *must* ensure that you explicitly call the bound C++ constructor using ``__init__``, *regardless* of whether it is a default constructor or not. Otherwise, the memory for the C++ portion of the instance will be left uninitialized, which will generally leave the C++ instance in an invalid state and cause undefined behavior if the C++ instance is subsequently used. Here is an example: .. code-block:: python class Dachschund(Dog): def __init__(self, name): Dog.__init__(self) # Without this, undefind behavior may occur if the C++ portions are referenced. self.name = name def bark(self): return "yap!" Note that a direct ``__init__`` constructor *should be called*, and ``super()`` should not be used. For simple cases of linear inheritance, ``super()`` may work, but once you begin mixing Python and C++ multiple inheritance, things will fall apart due to differences between Python's MRO and C++'s mechanisms. Please take a look at the :ref:`macro_notes` before using this feature. .. note:: When the overridden type returns a reference or pointer to a type that pybind11 converts from Python (for example, numeric values, std::string, and other built-in value-converting types), there are some limitations to be aware of: - because in these cases there is no C++ variable to reference (the value is stored in the referenced Python variable), pybind11 provides one in the PYBIND11_OVERLOAD macros (when needed) with static storage duration. Note that this means that invoking the overloaded method on *any* instance will change the referenced value stored in *all* instances of that type. - Attempts to modify a non-const reference will not have the desired effect: it will change only the static cache variable, but this change will not propagate to underlying Python instance, and the change will be replaced the next time the overload is invoked. .. seealso:: The file :file:`tests/test_virtual_functions.cpp` contains a complete example that demonstrates how to override virtual functions using pybind11 in more detail. .. _virtual_and_inheritance: Combining virtual functions and inheritance =========================================== When combining virtual methods with inheritance, you need to be sure to provide an override for each method for which you want to allow overrides from derived python classes. For example, suppose we extend the above ``Animal``/``Dog`` example as follows: .. code-block:: cpp class Animal { public: virtual std::string go(int n_times) = 0; virtual std::string name() { return "unknown"; } }; class Dog : public Animal { public: std::string go(int n_times) override { std::string result; for (int i=0; i class PyAnimal : public AnimalBase { public: using AnimalBase::AnimalBase; // Inherit constructors std::string go(int n_times) override { PYBIND11_OVERLOAD_PURE(std::string, AnimalBase, go, n_times); } std::string name() override { PYBIND11_OVERLOAD(std::string, AnimalBase, name, ); } }; template class PyDog : public PyAnimal { public: using PyAnimal::PyAnimal; // Inherit constructors // Override PyAnimal's pure virtual go() with a non-pure one: std::string go(int n_times) override { PYBIND11_OVERLOAD(std::string, DogBase, go, n_times); } std::string bark() override { PYBIND11_OVERLOAD(std::string, DogBase, bark, ); } }; This technique has the advantage of requiring just one trampoline method to be declared per virtual method and pure virtual method override. It does, however, require the compiler to generate at least as many methods (and possibly more, if both pure virtual and overridden pure virtual methods are exposed, as above). The classes are then registered with pybind11 using: .. code-block:: cpp py::class_> animal(m, "Animal"); py::class_> dog(m, "Dog"); py::class_> husky(m, "Husky"); // ... add animal, dog, husky definitions Note that ``Husky`` did not require a dedicated trampoline template class at all, since it neither declares any new virtual methods nor provides any pure virtual method implementations. With either the repeated-virtuals or templated trampoline methods in place, you can now create a python class that inherits from ``Dog``: .. code-block:: python class ShihTzu(Dog): def bark(self): return "yip!" .. seealso:: See the file :file:`tests/test_virtual_functions.cpp` for complete examples using both the duplication and templated trampoline approaches. .. _extended_aliases: Extended trampoline class functionality ======================================= The trampoline classes described in the previous sections are, by default, only initialized when needed. More specifically, they are initialized when a python class actually inherits from a registered type (instead of merely creating an instance of the registered type), or when a registered constructor is only valid for the trampoline class but not the registered class. This is primarily for performance reasons: when the trampoline class is not needed for anything except virtual method dispatching, not initializing the trampoline class improves performance by avoiding needing to do a run-time check to see if the inheriting python instance has an overloaded method. Sometimes, however, it is useful to always initialize a trampoline class as an intermediate class that does more than just handle virtual method dispatching. For example, such a class might perform extra class initialization, extra destruction operations, and might define new members and methods to enable a more python-like interface to a class. In order to tell pybind11 that it should *always* initialize the trampoline class when creating new instances of a type, the class constructors should be declared using ``py::init_alias()`` instead of the usual ``py::init()``. This forces construction via the trampoline class, ensuring member initialization and (eventual) destruction. .. seealso:: See the file :file:`tests/test_virtual_functions.cpp` for complete examples showing both normal and forced trampoline instantiation. .. _custom_constructors: Custom constructors =================== The syntax for binding constructors was previously introduced, but it only works when a constructor of the appropriate arguments actually exists on the C++ side. To extend this to more general cases, pybind11 offers two different approaches: binding factory functions, and placement-new creation. Factory function constructors ----------------------------- It is possible to expose a Python-side constructor from a C++ function that returns a new object by value or pointer. For example, suppose you have a class like this: .. code-block:: cpp class Example { private: Example(int); // private constructor public: // Factory function: static Example create(int a) { return Example(a); } }; While it is possible to expose the ``create`` method to Python, it is often preferrable to expose it on the Python side as a constructor rather than a named static method. You can do this by calling ``.def(py::init(...))`` with the function reference returning the new instance passed as an argument. It is also possible to use this approach to bind a function returning a new instance by raw pointer or by the holder (e.g. ``std::unique_ptr``). The following example shows the different approaches: .. code-block:: cpp class Example { private: Example(int); // private constructor public: // Factory function - returned by value: static Example create(int a) { return Example(a); } // These constructors are publicly callable: Example(double); Example(int, int); Example(std::string); }; py::class_(m, "Example") // Bind the factory function as a constructor: .def(py::init(&Example::create)) // Bind a lambda function returning a pointer wrapped in a holder: .def(py::init([](std::string arg) { return std::unique_ptr(new Example(arg)); })) // Return a raw pointer: .def(py::init([](int a, int b) { return new Example(a, b); })) // You can mix the above with regular C++ constructor bindings as well: .def(py::init()) ; When the constructor is invoked from Python, pybind11 will call the factory function and store the resulting C++ instance in the Python instance. When combining factory functions constructors with :ref:`overriding_virtuals` there are two approaches. The first is to add a constructor to the alias class that takes a base value by rvalue-reference. If such a constructor is available, it will be used to construct an alias instance from the value returned by the factory function. The second option is to provide two factory functions to ``py::init()``: the first will be invoked when no alias class is required (i.e. when the class is being used but not inherited from in Python), and the second will be invoked when an alias is required. You can also specify a single factory function that always returns an alias instance: this will result in behaviour similar to ``py::init_alias<...>()``, as described in :ref:`extended_aliases`. The following example shows the different factory approaches for a class with an alias: .. code-block:: cpp #include class Example { public: // ... virtual ~Example() = default; }; class PyExample : public Example { public: using Example::Example; PyExample(Example &&base) : Example(std::move(base)) {} }; py::class_(m, "Example") // Returns an Example pointer. If a PyExample is needed, the Example // instance will be moved via the extra constructor in PyExample, above. .def(py::init([]() { return new Example(); })) // Two callbacks: .def(py::init([]() { return new Example(); } /* no alias needed */, []() { return new PyExample(); } /* alias needed */)) // *Always* returns an alias instance (like py::init_alias<>()) .def(py::init([]() { return new PyExample(); })) ; Low-level placement-new construction ------------------------------------ A second approach for creating new instances use C++ placement new to construct an object in-place in preallocated memory. To do this, you simply bind a method name ``__init__`` that takes the class instance as the first argument by pointer or reference, then uses a placement-new constructor to construct the object in the pre-allocated (but uninitialized) memory. For example, instead of: .. code-block:: cpp py::class_(m, "Example") .def(py::init()); you could equivalently write: .. code-block:: cpp py::class_(m, "Example") .def("__init__", [](Example &instance, int arg) { new (&instance) Example(arg); } ); which will invoke the constructor in-place at the pre-allocated memory. .. _classes_with_non_public_destructors: Non-public destructors ====================== If a class has a private or protected destructor (as might e.g. be the case in a singleton pattern), a compile error will occur when creating bindings via pybind11. The underlying issue is that the ``std::unique_ptr`` holder type that is responsible for managing the lifetime of instances will reference the destructor even if no deallocations ever take place. In order to expose classes with private or protected destructors, it is possible to override the holder type via a holder type argument to ``class_``. Pybind11 provides a helper class ``py::nodelete`` that disables any destructor invocations. In this case, it is crucial that instances are deallocated on the C++ side to avoid memory leaks. .. code-block:: cpp /* ... definition ... */ class MyClass { private: ~MyClass() { } }; /* ... binding code ... */ py::class_>(m, "MyClass") .def(py::init<>()) .. _implicit_conversions: Implicit conversions ==================== Suppose that instances of two types ``A`` and ``B`` are used in a project, and that an ``A`` can easily be converted into an instance of type ``B`` (examples of this could be a fixed and an arbitrary precision number type). .. code-block:: cpp py::class_(m, "A") /// ... members ... py::class_(m, "B") .def(py::init()) /// ... members ... m.def("func", [](const B &) { /* .... */ } ); To invoke the function ``func`` using a variable ``a`` containing an ``A`` instance, we'd have to write ``func(B(a))`` in Python. On the other hand, C++ will automatically apply an implicit type conversion, which makes it possible to directly write ``func(a)``. In this situation (i.e. where ``B`` has a constructor that converts from ``A``), the following statement enables similar implicit conversions on the Python side: .. code-block:: cpp py::implicitly_convertible(); .. note:: Implicit conversions from ``A`` to ``B`` only work when ``B`` is a custom data type that is exposed to Python via pybind11. .. _static_properties: Static properties ================= The section on :ref:`properties` discussed the creation of instance properties that are implemented in terms of C++ getters and setters. Static properties can also be created in a similar way to expose getters and setters of static class attributes. Note that the implicit ``self`` argument also exists in this case and is used to pass the Python ``type`` subclass instance. This parameter will often not be needed by the C++ side, and the following example illustrates how to instantiate a lambda getter function that ignores it: .. code-block:: cpp py::class_(m, "Foo") .def_property_readonly_static("foo", [](py::object /* self */) { return Foo(); }); Operator overloading ==================== Suppose that we're given the following ``Vector2`` class with a vector addition and scalar multiplication operation, all implemented using overloaded operators in C++. .. code-block:: cpp class Vector2 { public: Vector2(float x, float y) : x(x), y(y) { } Vector2 operator+(const Vector2 &v) const { return Vector2(x + v.x, y + v.y); } Vector2 operator*(float value) const { return Vector2(x * value, y * value); } Vector2& operator+=(const Vector2 &v) { x += v.x; y += v.y; return *this; } Vector2& operator*=(float v) { x *= v; y *= v; return *this; } friend Vector2 operator*(float f, const Vector2 &v) { return Vector2(f * v.x, f * v.y); } std::string toString() const { return "[" + std::to_string(x) + ", " + std::to_string(y) + "]"; } private: float x, y; }; The following snippet shows how the above operators can be conveniently exposed to Python. .. code-block:: cpp #include PYBIND11_MODULE(example, m) { py::class_(m, "Vector2") .def(py::init()) .def(py::self + py::self) .def(py::self += py::self) .def(py::self *= float()) .def(float() * py::self) .def(py::self * float()) .def("__repr__", &Vector2::toString); } Note that a line like .. code-block:: cpp .def(py::self * float()) is really just short hand notation for .. code-block:: cpp .def("__mul__", [](const Vector2 &a, float b) { return a * b; }, py::is_operator()) This can be useful for exposing additional operators that don't exist on the C++ side, or to perform other types of customization. The ``py::is_operator`` flag marker is needed to inform pybind11 that this is an operator, which returns ``NotImplemented`` when invoked with incompatible arguments rather than throwing a type error. .. note:: To use the more convenient ``py::self`` notation, the additional header file :file:`pybind11/operators.h` must be included. .. seealso:: The file :file:`tests/test_operator_overloading.cpp` contains a complete example that demonstrates how to work with overloaded operators in more detail. Pickling support ================ Python's ``pickle`` module provides a powerful facility to serialize and de-serialize a Python object graph into a binary data stream. To pickle and unpickle C++ classes using pybind11, two additional functions must be provided. Suppose the class in question has the following signature: .. code-block:: cpp class Pickleable { public: Pickleable(const std::string &value) : m_value(value) { } const std::string &value() const { return m_value; } void setExtra(int extra) { m_extra = extra; } int extra() const { return m_extra; } private: std::string m_value; int m_extra = 0; }; The binding code including the requisite ``__setstate__`` and ``__getstate__`` methods [#f3]_ looks as follows: .. code-block:: cpp py::class_(m, "Pickleable") .def(py::init()) .def("value", &Pickleable::value) .def("extra", &Pickleable::extra) .def("setExtra", &Pickleable::setExtra) .def("__getstate__", [](const Pickleable &p) { /* Return a tuple that fully encodes the state of the object */ return py::make_tuple(p.value(), p.extra()); }) .def("__setstate__", [](Pickleable &p, py::tuple t) { if (t.size() != 2) throw std::runtime_error("Invalid state!"); /* Invoke the in-place constructor. Note that this is needed even when the object just has a trivial default constructor */ new (&p) Pickleable(t[0].cast()); /* Assign any additional state */ p.setExtra(t[1].cast()); }); An instance can now be pickled as follows: .. code-block:: python try: import cPickle as pickle # Use cPickle on Python 2.7 except ImportError: import pickle p = Pickleable("test_value") p.setExtra(15) data = pickle.dumps(p, 2) Note that only the cPickle module is supported on Python 2.7. The second argument to ``dumps`` is also crucial: it selects the pickle protocol version 2, since the older version 1 is not supported. Newer versions are also fineā€”for instance, specify ``-1`` to always use the latest available version. Beware: failure to follow these instructions will cause important pybind11 memory allocation routines to be skipped during unpickling, which will likely lead to memory corruption and/or segmentation faults. .. seealso:: The file :file:`tests/test_pickling.cpp` contains a complete example that demonstrates how to pickle and unpickle types using pybind11 in more detail. .. [#f3] http://docs.python.org/3/library/pickle.html#pickling-class-instances Multiple Inheritance ==================== pybind11 can create bindings for types that derive from multiple base types (aka. *multiple inheritance*). To do so, specify all bases in the template arguments of the ``class_`` declaration: .. code-block:: cpp py::class_(m, "MyType") ... The base types can be specified in arbitrary order, and they can even be interspersed with alias types and holder types (discussed earlier in this document)---pybind11 will automatically find out which is which. The only requirement is that the first template argument is the type to be declared. It is also permitted to inherit multiply from exported C++ classes in Python, as well as inheriting from multiple Python and/or pybind-exported classes. There is one caveat regarding the implementation of this feature: When only one base type is specified for a C++ type that actually has multiple bases, pybind11 will assume that it does not participate in multiple inheritance, which can lead to undefined behavior. In such cases, add the tag ``multiple_inheritance`` to the class constructor: .. code-block:: cpp py::class_(m, "MyType", py::multiple_inheritance()); The tag is redundant and does not need to be specified when multiple base types are listed. .. _module_local: Module-local class bindings =========================== When creating a binding for a class, pybind by default makes that binding "global" across modules. What this means is that a type defined in one module can be returned from any module resulting in the same Python type. For example, this allows the following: .. code-block:: cpp // In the module1.cpp binding code for module1: py::class_(m, "Pet") .def(py::init()) .def_readonly("name", &Pet::name); .. code-block:: cpp // In the module2.cpp binding code for module2: m.def("create_pet", [](std::string name) { return new Pet(name); }); .. code-block:: pycon >>> from module1 import Pet >>> from module2 import create_pet >>> pet1 = Pet("Kitty") >>> pet2 = create_pet("Doggy") >>> pet2.name() 'Doggy' When writing binding code for a library, this is usually desirable: this allows, for example, splitting up a complex library into multiple Python modules. In some cases, however, this can cause conflicts. For example, suppose two unrelated modules make use of an external C++ library and each provide custom bindings for one of that library's classes. This will result in an error when a Python program attempts to import both modules (directly or indirectly) because of conflicting definitions on the external type: .. code-block:: cpp // dogs.cpp // Binding for external library class: py::class(m, "Pet") .def("name", &pets::Pet::name); // Binding for local extension class: py::class(m, "Dog") .def(py::init()); .. code-block:: cpp // cats.cpp, in a completely separate project from the above dogs.cpp. // Binding for external library class: py::class(m, "Pet") .def("get_name", &pets::Pet::name); // Binding for local extending class: py::class(m, "Cat") .def(py::init()); .. code-block:: pycon >>> import cats >>> import dogs Traceback (most recent call last): File "", line 1, in ImportError: generic_type: type "Pet" is already registered! To get around this, you can tell pybind11 to keep the external class binding localized to the module by passing the ``py::module_local()`` attribute into the ``py::class_`` constructor: .. code-block:: cpp // Pet binding in dogs.cpp: py::class(m, "Pet", py::module_local()) .def("name", &pets::Pet::name); .. code-block:: cpp // Pet binding in cats.cpp: py::class(m, "Pet", py::module_local()) .def("get_name", &pets::Pet::name); This makes the Python-side ``dogs.Pet`` and ``cats.Pet`` into distinct classes, avoiding the conflict and allowing both modules to be loaded. C++ code in the ``dogs`` module that casts or returns a ``Pet`` instance will result in a ``dogs.Pet`` Python instance, while C++ code in the ``cats`` module will result in a ``cats.Pet`` Python instance. This does come with two caveats, however: First, external modules cannot return or cast a ``Pet`` instance to Python (unless they also provide their own local bindings). Second, from the Python point of view they are two distinct classes. Note that the locality only applies in the C++ -> Python direction. When passing such a ``py::module_local`` type into a C++ function, the module-local classes are still considered. This means that if the following function is added to any module (including but not limited to the ``cats`` and ``dogs`` modules above) it will be callable with either a ``dogs.Pet`` or ``cats.Pet`` argument: .. code-block:: cpp m.def("pet_name", [](const pets::Pet &pet) { return pet.name(); }); For example, suppose the above function is added to each of ``cats.cpp``, ``dogs.cpp`` and ``frogs.cpp`` (where ``frogs.cpp`` is some other module that does *not* bind ``Pets`` at all). .. code-block:: pycon >>> import cats, dogs, frogs # No error because of the added py::module_local() >>> mycat, mydog = cats.Cat("Fluffy"), dogs.Dog("Rover") >>> (cats.pet_name(mycat), dogs.pet_name(mydog)) ('Fluffy', 'Rover') >>> (cats.pet_name(mydog), dogs.pet_name(mycat), frogs.pet_name(mycat)) ('Rover', 'Fluffy', 'Fluffy') It is possible to use ``py::module_local()`` registrations in one module even if another module registers the same type globally: within the module with the module-local definition, all C++ instances will be cast to the associated bound Python type. In other modules any such values are converted to the global Python type created elsewhere. .. note:: STL bindings (as provided via the optional :file:`pybind11/stl_bind.h` header) apply ``py::module_local`` by default when the bound type might conflict with other modules; see :ref:`stl_bind` for details. .. note:: The localization of the bound types is actually tied to the shared object or binary generated by the compiler/linker. For typical modules created with ``PYBIND11_MODULE()``, this distinction is not significant. It is possible, however, when :ref:`embedding` to embed multiple modules in the same binary (see :ref:`embedding_modules`). In such a case, the localization will apply across all embedded modules within the same binary. .. seealso:: The file :file:`tests/test_local_bindings.cpp` contains additional examples that demonstrate how ``py::module_local()`` works. Binding protected member functions ================================== It's normally not possible to expose ``protected`` member functions to Python: .. code-block:: cpp class A { protected: int foo() const { return 42; } }; py::class_(m, "A") .def("foo", &A::foo); // error: 'foo' is a protected member of 'A' On one hand, this is good because non-``public`` members aren't meant to be accessed from the outside. But we may want to make use of ``protected`` functions in derived Python classes. The following pattern makes this possible: .. code-block:: cpp class A { protected: int foo() const { return 42; } }; class Publicist : public A { // helper type for exposing protected functions public: using A::foo; // inherited with different access modifier }; py::class_(m, "A") // bind the primary class .def("foo", &Publicist::foo); // expose protected methods via the publicist This works because ``&Publicist::foo`` is exactly the same function as ``&A::foo`` (same signature and address), just with a different access modifier. The only purpose of the ``Publicist`` helper class is to make the function name ``public``. If the intent is to expose ``protected`` ``virtual`` functions which can be overridden in Python, the publicist pattern can be combined with the previously described trampoline: .. code-block:: cpp class A { public: virtual ~A() = default; protected: virtual int foo() const { return 42; } }; class Trampoline : public A { public: int foo() const override { PYBIND11_OVERLOAD(int, A, foo, ); } }; class Publicist : public A { public: using A::foo; }; py::class_(m, "A") // <-- `Trampoline` here .def("foo", &Publicist::foo); // <-- `Publicist` here, not `Trampoline`! .. note:: MSVC 2015 has a compiler bug (fixed in version 2017) which requires a more explicit function binding in the form of ``.def("foo", static_cast(&Publicist::foo));`` where ``int (A::*)() const`` is the type of ``A::foo``.