Redesigned virtual call mechanism and user-facing syntax (breaking change!)

Sergey Lyskov pointed out that the trampoline mechanism used to override
virtual methods from within Python caused unnecessary overheads when
instantiating the original (i.e. non-extended) class.

This commit removes this inefficiency, but some syntax changes were
needed to achieve this. Projects using this features will need to make a
few changes:

In particular, the example below shows the old syntax to instantiate a
class with a trampoline:

class_<TrampolineClass>("MyClass")
    .alias<MyClass>()
    ....

This is what should be used now:

class_<MyClass, std::unique_ptr<MyClass, TrampolineClass>("MyClass")
    ....

Importantly, the trampoline class is now specified as the *third*
argument to the class_ template, and the alias<..>() call is gone. The
second argument with the unique pointer is simply the default holder
type used by pybind11.

docs/advanced.rst

@@ -283,9 +283,8 @@ The binding code also needs a few minor adaptations (highlighted):
     PYBIND11_PLUGIN(example) {
         py::module m("example", "pybind11 example plugin");
 
-        py::class_<PyAnimal> animal(m, "Animal");
+        py::class_<Animal, std::unique_ptr<Animal>, PyAnimal /* <--- trampoline*/> animal(m, "Animal");
         animal
-            .alias<Animal>()
             .def(py::init<>())
             .def("go", &Animal::go);
 
@@ -297,10 +296,10 @@ The binding code also needs a few minor adaptations (highlighted):
         return m.ptr();
     }
 
-Importantly, the trampoline helper class is used as the template argument to
-:class:`class_`, and a call to :func:`class_::alias` informs the binding
-generator that this is merely an alias for the underlying type ``Animal``.
-Following this, we are able to define a constructor as usual.
+Importantly, pybind11 is made aware of the trampoline trampoline helper class
+by specifying it as the *third* template argument to :class:`class_`. The
+second argument with the unique pointer is simply the default holder type used
+by pybind11. Following this, we are able to define a constructor as usual.
 
 The Python session below shows how to override ``Animal::go`` and invoke it via
 a virtual method call.
@@ -321,12 +320,12 @@ a virtual method call.
 
 .. warning::
 
-    Both :func:`PYBIND11_OVERLOAD` and :func:`PYBIND11_OVERLOAD_PURE` are
-    macros, which means that they can get confused by commas in a template
-    argument such as ``PYBIND11_OVERLOAD(MyReturnValue<T1, T2>, myFunc)``. In
-    this case, the preprocessor assumes that the comma indicates the beginnning
-    of the next parameter. Use a ``typedef`` to bind the template to another
-    name and use it in the macro to avoid this problem.
+    The :func:`PYBIND11_OVERLOAD_*` calls are all just macros, which means that
+    they can get confused by commas in a template argument such as
+    ``PYBIND11_OVERLOAD(MyReturnValue<T1, T2>, myFunc)``. In this case, the
+    preprocessor assumes that the comma indicates the beginnning of the next
+    parameter. Use a ``typedef`` to bind the template to another name and use
+    it in the macro to avoid this problem.
 
 .. seealso::
 
@@ -369,9 +368,8 @@ be realized as follows (important changes highlighted):
     PYBIND11_PLUGIN(example) {
         py::module m("example", "pybind11 example plugin");
 
-        py::class_<PyAnimal> animal(m, "Animal");
+        py::class_<Animal, std::unique_ptr<Animal>, PyAnimal> animal(m, "Animal");
         animal
-            .alias<Animal>()
             .def(py::init<>())
             .def("go", &Animal::go);
 

docs/changelog.rst

@@ -5,9 +5,11 @@ Changelog
 
 1.8 (Not yet released)
 ----------------------
+* Redesigned virtual call mechanism and user-facing syntax (breaking change!)
 * Prevent implicit conversion of floating point values to integral types in
   function arguments
 * Transparent conversion of sparse and dense Eigen data types
+* ``std::vector<>`` type bindings analogous to Boost.Python's ``indexing_suite``
 * Fixed incorrect default return value policy for functions returning a shared
   pointer
 * Don't allow casting a ``None`` value into a C++ lvalue reference
@@ -16,10 +18,19 @@ Changelog
 * Extended ``str`` type to also work with ``bytes`` instances
 * Added ``[[noreturn]]`` attribute to ``pybind11_fail()`` to quench some
   compiler warnings
+* List function arguments in exception text when the dispatch code cannot find
+  a matching overload
 * Various minor ``iterator`` and ``make_iterator()`` improvements
+* Transparently support ``__bool__`` on Python 2.x and Python 3.x
+* Fixed issue with destructor of unpickled object not being called
 * Minor CMake build system improvements on Windows
 * Many ``mkdoc.py`` improvements (enumerations, template arguments, ``DOC()``
   macro accepts more arguments)
+* New ``pybind11::args`` and ``pybind11::kwargs`` types to create functions which
+  take an arbitrary number of arguments and keyword arguments
+* New syntax to call a Python function from C++ using ``*args`` and ``*kwargs``
+* Added an ``ExtraFlags`` template argument to the NumPy ``array_t<>`` wrapper. This
+  can be used to disable an enforced cast that may lose precision
 * Documentation improvements (pickling support, ``keep_alive``)
 
 1.7 (April 30, 2016)

example/example12.cpp

@@ -82,15 +82,11 @@ void runExample12Virtual(Example12 *ex) {
 }
 
 void init_ex12(py::module &m) {
-    /* Important: use the wrapper type as a template
-       argument to class_<>, but use the original name
-       to denote the type */
-    py::class_<PyExample12>(m, "Example12")
-        /* Declare that 'PyExample12' is really an alias for the original type 'Example12' */
-        .alias<Example12>()
+    /* Important: indicate the trampoline class PyExample12 using the third
+       argument to py::class_. The second argument with the unique pointer
+       is simply the default holder type used by pybind11. */
+    py::class_<Example12, std::unique_ptr<Example12>, PyExample12>(m, "Example12")
         .def(py::init<int>())
-        /* Copy constructor (not needed in this case, but should generally be declared in this way) */
-        .def(py::init<const PyExample12 &>())
         /* Reference original class in function definitions */
         .def("run", &Example12::run)
         .def("run_bool", &Example12::run_bool)

example/issues.cpp

@@ -42,8 +42,7 @@ void init_issues(py::module &m) {
         }
     };
 
-    py::class_<DispatchIssue> base(m2, "DispatchIssue");
-    base.alias<Base>()
+    py::class_<Base, std::unique_ptr<Base>, DispatchIssue>(m2, "DispatchIssue")
         .def(py::init<>())
         .def("dispatch", &Base::dispatch);
 
@@ -108,4 +107,28 @@ void init_issues(py::module &m) {
     // (no id): don't cast doubles to ints
     m2.def("expect_float", [](float f) { return f; });
     m2.def("expect_int", [](int i) { return i; });
+
+    // (no id): don't invoke Python dispatch code when instantiating C++
+    // classes that were not extended on the Python side
+    struct A {
+        virtual ~A() {}
+        virtual void f() { std::cout << "A.f()" << std::endl; }
+    };
+
+    struct PyA : A {
+        PyA() { std::cout << "PyA.PyA()" << std::endl; }
+
+        void f() override {
+            std::cout << "PyA.f()" << std::endl;
+            PYBIND11_OVERLOAD(void, A, f);
+        }
+    };
+
+    auto call_f = [](A *a) { a->f(); };
+
+	pybind11::class_<A, std::unique_ptr<A>, PyA>(m2, "A")
+	    .def(py::init<>())
+	    .def("f", &A::f);
+
+	 m2.def("call_f", call_f);
 }

example/issues.py

@@ -9,6 +9,7 @@ from example.issues import Placeholder, return_vec_of_reference_wrapper
 from example.issues import iterator_passthrough
 from example.issues import ElementList, ElementA, print_element
 from example.issues import expect_float, expect_int
+from example.issues import A, call_f
 import gc
 
 print_cchar("const char *")
@@ -55,3 +56,19 @@ except Exception as e:
     print("Failed as expected: " + str(e))
 
 print(expect_float(12))
+
+class B(A):
+    def __init__(self):
+        super(B, self).__init__()
+
+    def f(self):
+        print("In python f()")
+
+print("C++ version")
+a = A()
+call_f(a)
+
+print("Python version")
+b = B()
+call_f(b)
+

example/issues.ref

@@ -12,3 +12,9 @@ Failed as expected: Incompatible function arguments. The following argument type
     1. (int) -> int
     Invoked with: 5.2
 12.0
+C++ version
+A.f()
+Python version
+PyA.PyA()
+PyA.f()
+In python f()

include/pybind11/attr.h

@@ -65,6 +65,7 @@ enum op_type : int;
 struct undefined_t;
 template <op_id id, op_type ot, typename L = undefined_t, typename R = undefined_t> struct op_;
 template <typename... Args> struct init;
+template <typename... Args> struct init_alias;
 inline void keep_alive_impl(int Nurse, int Patient, handle args, handle ret);
 
 /// Internal data structure which holds metadata about a keyword argument

include/pybind11/pybind11.h

@@ -503,7 +503,7 @@ public:
 NAMESPACE_BEGIN(detail)
 /// Generic support for creating new Python heap types
 class generic_type : public object {
-    template <typename type, typename holder_type> friend class class_;
+    template <typename type, typename holder_type, typename type_alias> friend class class_;
 public:
     PYBIND11_OBJECT_DEFAULT(generic_type, object, PyType_Check)
 protected:
@@ -721,7 +721,7 @@ protected:
 };
 NAMESPACE_END(detail)
 
-template <typename type, typename holder_type = std::unique_ptr<type>>
+template <typename type, typename holder_type = std::unique_ptr<type>, typename type_alias = type>
 class class_ : public detail::generic_type {
 public:
     typedef detail::instance<type, holder_type> instance_type;
@@ -743,6 +743,11 @@ public:
         detail::process_attributes<Extra...>::init(extra..., &record);
 
         detail::generic_type::initialize(&record);
+
+        if (!std::is_same<type, type_alias>::value) {
+            auto &instances = pybind11::detail::get_internals().registered_types_cpp;
+            instances[std::type_index(typeid(type_alias))] = instances[std::type_index(typeid(type))];
+        }
     }
 
     template <typename Func, typename... Extra>
@@ -780,6 +785,12 @@ public:
         return *this;
     }
 
+    template <typename... Args, typename... Extra>
+    class_ &def(const detail::init_alias<Args...> &init, const Extra&... extra) {
+        init.template execute<type>(*this, extra...);
+        return *this;
+    }
+
     template <typename Func> class_& def_buffer(Func &&func) {
         struct capture { Func func; };
         capture *ptr = new capture { std::forward<Func>(func) };
@@ -856,11 +867,6 @@ public:
         return *this;
     }
 
-    template <typename target> class_ alias() {
-        auto &instances = pybind11::detail::get_internals().registered_types_cpp;
-        instances[std::type_index(typeid(target))] = instances[std::type_index(typeid(type))];
-        return *this;
-    }
 private:
     /// Initialize holder object, variant 1: object derives from enable_shared_from_this
     template <typename T>
@@ -959,9 +965,31 @@ private:
 
 NAMESPACE_BEGIN(detail)
 template <typename... Args> struct init {
-    template <typename Base, typename Holder, typename... Extra> void execute(pybind11::class_<Base, Holder> &class_, const Extra&... extra) const {
+    template <typename Base, typename Holder, typename Alias, typename... Extra,
+              typename std::enable_if<std::is_same<Base, Alias>::value, int>::type = 0>
+    void execute(pybind11::class_<Base, Holder, Alias> &class_, const Extra&... extra) const {
         /// Function which calls a specific C++ in-place constructor
-        class_.def("__init__", [](Base *instance, Args... args) { new (instance) Base(args...); }, extra...);
+        class_.def("__init__", [](Base *self, Args... args) { new (self) Base(args...); }, extra...);
+    }
+
+    template <typename Base, typename Holder, typename Alias, typename... Extra,
+              typename std::enable_if<!std::is_same<Base, Alias>::value &&
+                                       std::is_constructible<Base, Args...>::value, int>::type = 0>
+    void execute(pybind11::class_<Base, Holder, Alias> &class_, const Extra&... extra) const {
+        handle cl_type = class_;
+        class_.def("__init__", [cl_type](handle self, Args... args) {
+                if (self.get_type() == cl_type)
+                    new (self.cast<Base *>()) Base(args...);
+                else
+                    new (self.cast<Alias *>()) Alias(args...);
+            }, extra...);
+    }
+
+    template <typename Base, typename Holder, typename Alias, typename... Extra,
+              typename std::enable_if<!std::is_same<Base, Alias>::value &&
+                                      !std::is_constructible<Base, Args...>::value, int>::type = 0>
+    void execute(pybind11::class_<Base, Holder, Alias> &class_, const Extra&... extra) const {
+        class_.def("__init__", [](Alias *self, Args... args) { new (self) Alias(args...); }, extra...);
     }
 };