mirror of
https://github.com/pybind/pybind11.git
synced 2024-11-24 14:15:11 +00:00
97aa54fefa
This updates the compilation to always apply hidden visibility to resolve the issues with default visibility causing problems under debug compilations. Moreover using the cmake property makes it easier for a caller to override if absolutely needed for some reason. For `pybind11_add_module` we use cmake to set the property; for the targets, we append to compilation option to non-MSVC compilers.
274 lines
11 KiB
ReStructuredText
274 lines
11 KiB
ReStructuredText
Frequently asked questions
|
||
##########################
|
||
|
||
"ImportError: dynamic module does not define init function"
|
||
===========================================================
|
||
|
||
You are likely using an incompatible version of Python (for instance, the
|
||
extension library was compiled against Python 2, while the interpreter is
|
||
running on top of some version of Python 3, or vice versa).
|
||
|
||
"Symbol not found: ``__Py_ZeroStruct`` / ``_PyInstanceMethod_Type``"
|
||
========================================================================
|
||
|
||
See the first answer.
|
||
|
||
"SystemError: dynamic module not initialized properly"
|
||
======================================================
|
||
|
||
See the first answer.
|
||
|
||
The Python interpreter immediately crashes when importing my module
|
||
===================================================================
|
||
|
||
See the first answer.
|
||
|
||
CMake doesn't detect the right Python version
|
||
=============================================
|
||
|
||
The CMake-based build system will try to automatically detect the installed
|
||
version of Python and link against that. When this fails, or when there are
|
||
multiple versions of Python and it finds the wrong one, delete
|
||
``CMakeCache.txt`` and then invoke CMake as follows:
|
||
|
||
.. code-block:: bash
|
||
|
||
cmake -DPYTHON_EXECUTABLE:FILEPATH=<path-to-python-executable> .
|
||
|
||
Limitations involving reference arguments
|
||
=========================================
|
||
|
||
In C++, it's fairly common to pass arguments using mutable references or
|
||
mutable pointers, which allows both read and write access to the value
|
||
supplied by the caller. This is sometimes done for efficiency reasons, or to
|
||
realize functions that have multiple return values. Here are two very basic
|
||
examples:
|
||
|
||
.. code-block:: cpp
|
||
|
||
void increment(int &i) { i++; }
|
||
void increment_ptr(int *i) { (*i)++; }
|
||
|
||
In Python, all arguments are passed by reference, so there is no general
|
||
issue in binding such code from Python.
|
||
|
||
However, certain basic Python types (like ``str``, ``int``, ``bool``,
|
||
``float``, etc.) are **immutable**. This means that the following attempt
|
||
to port the function to Python doesn't have the same effect on the value
|
||
provided by the caller -- in fact, it does nothing at all.
|
||
|
||
.. code-block:: python
|
||
|
||
def increment(i):
|
||
i += 1 # nope..
|
||
|
||
pybind11 is also affected by such language-level conventions, which means that
|
||
binding ``increment`` or ``increment_ptr`` will also create Python functions
|
||
that don't modify their arguments.
|
||
|
||
Although inconvenient, one workaround is to encapsulate the immutable types in
|
||
a custom type that does allow modifications.
|
||
|
||
An other alternative involves binding a small wrapper lambda function that
|
||
returns a tuple with all output arguments (see the remainder of the
|
||
documentation for examples on binding lambda functions). An example:
|
||
|
||
.. code-block:: cpp
|
||
|
||
int foo(int &i) { i++; return 123; }
|
||
|
||
and the binding code
|
||
|
||
.. code-block:: cpp
|
||
|
||
m.def("foo", [](int i) { int rv = foo(i); return std::make_tuple(rv, i); });
|
||
|
||
|
||
How can I reduce the build time?
|
||
================================
|
||
|
||
It's good practice to split binding code over multiple files, as in the
|
||
following example:
|
||
|
||
:file:`example.cpp`:
|
||
|
||
.. code-block:: cpp
|
||
|
||
void init_ex1(py::module &);
|
||
void init_ex2(py::module &);
|
||
/* ... */
|
||
|
||
PYBIND11_MODULE(example, m) {
|
||
init_ex1(m);
|
||
init_ex2(m);
|
||
/* ... */
|
||
}
|
||
|
||
:file:`ex1.cpp`:
|
||
|
||
.. code-block:: cpp
|
||
|
||
void init_ex1(py::module &m) {
|
||
m.def("add", [](int a, int b) { return a + b; });
|
||
}
|
||
|
||
:file:`ex2.cpp`:
|
||
|
||
.. code-block:: cpp
|
||
|
||
void init_ex1(py::module &m) {
|
||
m.def("sub", [](int a, int b) { return a - b; });
|
||
}
|
||
|
||
:command:`python`:
|
||
|
||
.. code-block:: pycon
|
||
|
||
>>> import example
|
||
>>> example.add(1, 2)
|
||
3
|
||
>>> example.sub(1, 1)
|
||
0
|
||
|
||
As shown above, the various ``init_ex`` functions should be contained in
|
||
separate files that can be compiled independently from one another, and then
|
||
linked together into the same final shared object. Following this approach
|
||
will:
|
||
|
||
1. reduce memory requirements per compilation unit.
|
||
|
||
2. enable parallel builds (if desired).
|
||
|
||
3. allow for faster incremental builds. For instance, when a single class
|
||
definition is changed, only a subset of the binding code will generally need
|
||
to be recompiled.
|
||
|
||
"recursive template instantiation exceeded maximum depth of 256"
|
||
================================================================
|
||
|
||
If you receive an error about excessive recursive template evaluation, try
|
||
specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
|
||
culprit is generally the generation of function signatures at compile time
|
||
using C++14 template metaprogramming.
|
||
|
||
.. _`faq:hidden_visibility`:
|
||
|
||
"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
|
||
============================================================================================================
|
||
|
||
This error typically indicates that you are compiling without the required
|
||
``-fvisibility`` flag. pybind11 code internally forces hidden visibility on
|
||
all internal code, but if non-hidden (and thus *exported*) code attempts to
|
||
include a pybind type (for example, ``py::object`` or ``py::list``) you can run
|
||
into this warning.
|
||
|
||
To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
|
||
compiling pybind code.
|
||
|
||
As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
|
||
have been compiled under different versions of pybind itself, it is also
|
||
important that the symbols defined in one module do not clash with the
|
||
potentially-incompatible symbols defined in another. While Python extension
|
||
modules are usually loaded with localized symbols (under POSIX systems
|
||
typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
|
||
can be changed, but even if it isn't it is not always enough to guarantee
|
||
complete independence of the symbols involved when not using
|
||
``-fvisibility=hidden``.
|
||
|
||
Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
|
||
savings. (See the following section for more details).
|
||
|
||
|
||
.. _`faq:symhidden`:
|
||
|
||
How can I create smaller binaries?
|
||
==================================
|
||
|
||
To do its job, pybind11 extensively relies on a programming technique known as
|
||
*template metaprogramming*, which is a way of performing computation at compile
|
||
time using type information. Template metaprogamming usually instantiates code
|
||
involving significant numbers of deeply nested types that are either completely
|
||
removed or reduced to just a few instructions during the compiler's optimization
|
||
phase. However, due to the nested nature of these types, the resulting symbol
|
||
names in the compiled extension library can be extremely long. For instance,
|
||
the included test suite contains the following symbol:
|
||
|
||
.. only:: html
|
||
|
||
.. code-block:: none
|
||
|
||
__ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
|
||
|
||
.. only:: not html
|
||
|
||
.. code-block:: cpp
|
||
|
||
__ZN8pybind1112cpp_functionC1Iv8Example2JRNSt3__16vectorINS3_12basic_stringIwNS3_11char_traitsIwEENS3_9allocatorIwEEEENS8_ISA_EEEEEJNS_4nameENS_7siblingENS_9is_methodEA28_cEEEMT0_FT_DpT1_EDpRKT2_
|
||
|
||
which is the mangled form of the following function type:
|
||
|
||
.. code-block:: cpp
|
||
|
||
pybind11::cpp_function::cpp_function<void, Example2, std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&, pybind11::name, pybind11::sibling, pybind11::is_method, char [28]>(void (Example2::*)(std::__1::vector<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> >, std::__1::allocator<std::__1::basic_string<wchar_t, std::__1::char_traits<wchar_t>, std::__1::allocator<wchar_t> > > >&), pybind11::name const&, pybind11::sibling const&, pybind11::is_method const&, char const (&) [28])
|
||
|
||
The memory needed to store just the mangled name of this function (196 bytes)
|
||
is larger than the actual piece of code (111 bytes) it represents! On the other
|
||
hand, it's silly to even give this function a name -- after all, it's just a
|
||
tiny cog in a bigger piece of machinery that is not exposed to the outside
|
||
world. So we'll generally only want to export symbols for those functions which
|
||
are actually called from the outside.
|
||
|
||
This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
|
||
and Clang, which sets the default symbol visibility to *hidden*, which has a
|
||
tremendous impact on the final binary size of the resulting extension library.
|
||
(On Visual Studio, symbols are already hidden by default, so nothing needs to
|
||
be done there.)
|
||
|
||
In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
|
||
potential serious issues when loading multiple modules and is required for
|
||
proper pybind operation. See the previous FAQ entry for more details.
|
||
|
||
Another aspect that can require a fair bit of code are function signature
|
||
descriptions. pybind11 automatically generates human-readable function
|
||
signatures for docstrings, e.g.:
|
||
|
||
.. code-block:: none
|
||
|
||
| __init__(...)
|
||
| __init__(*args, **kwargs)
|
||
| Overloaded function.
|
||
|
|
||
| 1. __init__(example.Example1) -> NoneType
|
||
|
|
||
| Docstring for overload #1 goes here
|
||
|
|
||
| 2. __init__(example.Example1, int) -> NoneType
|
||
|
|
||
| Docstring for overload #2 goes here
|
||
|
|
||
| 3. __init__(example.Example1, example.Example1) -> NoneType
|
||
|
|
||
| Docstring for overload #3 goes here
|
||
|
||
|
||
In C++11 mode, these are generated at run time using string concatenation,
|
||
which can amount to 10-20% of the size of the resulting binary. If you can,
|
||
enable C++14 language features (using ``-std=c++14`` for GCC/Clang), in which
|
||
case signatures are efficiently pre-generated at compile time. Unfortunately,
|
||
Visual Studio's C++14 support (``constexpr``) is not good enough as of April
|
||
2016, so it always uses the more expensive run-time approach.
|
||
|
||
Working with ancient Visual Studio 2009 builds on Windows
|
||
=========================================================
|
||
|
||
The official Windows distributions of Python are compiled using truly
|
||
ancient versions of Visual Studio that lack good C++11 support. Some users
|
||
implicitly assume that it would be impossible to load a plugin built with
|
||
Visual Studio 2015 into a Python distribution that was compiled using Visual
|
||
Studio 2009. However, no such issue exists: it's perfectly legitimate to
|
||
interface DLLs that are built with different compilers and/or C libraries.
|
||
Common gotchas to watch out for involve not ``free()``-ing memory region
|
||
that that were ``malloc()``-ed in another shared library, using data
|
||
structures with incompatible ABIs, and so on. pybind11 is very careful not
|
||
to make these types of mistakes.
|