Attempting to mix py::module_local and non-module_local classes results
in some unexpected/undesirable behaviour:
- if a class is registered non-local by some other module, a later
attempt to register it locally fails. It doesn't need to: it is
perfectly acceptable for the local registration to simply override
the external global registration.
- going the other way (i.e. module `A` registers a type `T` locally,
then `B` registers the same type `T` globally) causes a more serious
issue: `A.T`'s constructors no longer work because the `self` argument
gets converted to a `B.T`, which then fails to resolve.
Changing the cast precedence to prefer local over global fixes this and
makes it work more consistently, regardless of module load order.
Types need `tp_name` set to a C-style string, but the current `strdup`
ends up with a leak (issue #977). This avoids the strdup by storing
the `std::string` in internals so that during interpreter shutdown it
will be properly destroyed.
This commit adds a `py::module_local` attribute that lets you confine a
registered type to the module (more technically, the shared object) in
which it is defined, by registering it with:
py::class_<C>(m, "C", py::module_local())
This will allow the same C++ class `C` to be registered in different
modules with independent sets of class definitions. On the Python side,
two such types will be completely distinct; on the C++ side, the C++
type resolves to a different Python type in each module.
This applies `py::module_local` automatically to `stl_bind.h` bindings
when the container value type looks like something global: i.e. when it
is a converting type (for example, when binding a `std::vector<int>`),
or when it is a registered type itself bound with `py::module_local`.
This should help resolve potential future conflicts (e.g. if two
completely unrelated modules both try to bind a `std::vector<int>`.
Users can override the automatic selection by adding a
`py::module_local()` or `py::module_local(false)`.
Note that this does mildly break backwards compatibility: bound stl
containers of basic types like `std::vector<int>` cannot be bound in one
module and returned in a different module. (This can be re-enabled with
`py::module_local(false)` as described above, but with the potential for
eventual load conflicts).
The builtin exception handler currently doesn't work across modules
under clang/libc++ for builtin pybind exceptions like
`pybind11::error_already_set` or `pybind11::stop_iteration`: under
RTLD_LOCAL module loading clang considers each module's exception
classes distinct types. This then means that the base exception
translator fails to catch the exceptions and the fall through to the
generic `std::exception` handler, which completely breaks things like
`stop_iteration`: only the `stop_iteration` of the first module loaded
actually works properly; later modules raise a RuntimeError with no
message when trying to invoke their iterators.
For example, two modules defined like this exhibit the behaviour under
clang++/libc++:
z1.cpp:
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z1, m) {
py::bind_vector<std::vector<long>>(m, "IntVector");
}
z2.cpp:
#include <pybind11/pybind11.h>
#include <pybind11/stl_bind.h>
namespace py = pybind11;
PYBIND11_MODULE(z2, m) {
py::bind_vector<std::vector<double>>(m, "FloatVector");
}
Python:
import z1, z2
for i in z2.FloatVector():
pass
results in:
Traceback (most recent call last):
File "zs.py", line 2, in <module>
for i in z2.FloatVector():
RuntimeError
This commit fixes the issue by adding a new exception translator each
time the internals pointer is initialized from python builtins: this
generally means the internals data was initialized by some other
module. (The extra translator(s) are skipped under libstdc++).
This adds the infrastructure for a separate test plugin for cross-module
tests. (This commit contains no tests that actually use it, but the
following commits do; this is separated simply to provide a cleaner
commit history).
This commit adds a PYBIND11_UNSHARED_STATIC_LOCALS macro that forces a
function to have hidden visibility under gcc and gcc-compatible
compilers. gcc, in particular, needs this to to avoid sharing static
local variables across modules (which happens even under a RTLD_LOCAL
dlopen()!). clang doesn't appear to have this issue, but the forced
visibility on internal pybind functions certainly won't hurt it and icc.
This updates the workaround from #862 to use this rather than the
version-specific template.
Currently types that are capable of conversion always call their convert
function when invoked with a `py::object` which is actually the correct
type. This means that code such as `py::cast<py::list>(obj)` and
`py::list l(obj.attr("list"))` make copies, which was an oversight
rather than an intentional feature.
While at first glance there might be something behind having
`py::list(obj)` make a copy (as it would in Python), this would be
inconsistent when you dig a little deeper because `py::list(l)`
*doesn't* make a copy for an existing `py::list l`, and having an
inconsistency within C++ would be worse than a C++ <-> Python
inconsistency.
It is possible to get around the copying using a
`reinterpret_borrow<list>(o)` (and this commit fixes one place, in
`embed.h`, that does so), but that seems a misuse of
`reinterpret_borrow`, which is really supposed to be just for dealing
with raw python-returned values, not `py::object`-derived wrappers which
are supposed to be higher level.
This changes the constructor of such converting types (i.e. anything
using PYBIND11_OBJECT_CVT -- `str`, `bool_`, `int_`, `float_`, `tuple`,
`dict`, `list`, `set`, `memoryview`) to reference rather than copy when
the check function passes.
It also adds an `object &&` constructor that is slightly more efficient
by avoiding an inc_ref when the check function passes.
The fix for #960 could result a type being registered multiple times if
its `__init__` is called multiple times. This can happen perfectly
ordinarily when python-side multiple inheritance is involved: for
example, with a diamond inheritance pattern with each intermediate
classes invoking the parent constructor.
With the change in #960, the multiple `__init__` calls meant
`register_instance` was called multiple times, but the deletion only
deleted it once. Thus, if a future instance of the same type was
allocated at the same location, pybind would pick it up as a registered
type.
This fixes the issue by tracking whether a value pointer has been
registered to avoid both double-registering it. (There's also a slight
optimization of not needing to do a registered_instances lookup when the
type is known not registered, but this is secondary).
`error_already_set` is more complicated than it needs to be, partly
because it manages reference counts itself rather than using
`py::object`, and partly because it tries to do more exception clearing
than is needed. This commit greatly simplifies it, and fixes#927.
Using `py::object` instead of `PyObject *` means we can rely on
implicit copy/move constructors.
The current logic did both a `PyErr_Clear` on deletion *and* a
`PyErr_Fetch` on creation. I can't see how the `PyErr_Clear` on
deletion is ever useful: the `Fetch` on creation itself clears the
error, so the only way doing a `PyErr_Clear` on deletion could do
anything if is some *other* exception was raised while the
`error_already_set` object was alive--but in that case, clearing some
other exception seems wrong. (Code that is worried about an exception
handler raising another exception would already catch a second
`error_already_set` from exception code).
The destructor itself called `clear()`, but `clear()` was a little bit
more paranoid that needed: it called `restore()` to restore the
currently captured error, but then immediately cleared it, using the
`PyErr_Restore` to release the references. That's unnecessary: it's
valid for us to release the references manually. This updates the code
to simply release the references on the three objects (preserving the
gil acquire).
`clear()`, however, also had the side effect of clearing the current
error, even if the current `error_already_set` didn't have a current
error (e.g. because of a previous `restore()` or `clear()` call). I
don't really see how clearing the error here can ever actually be
useful: the only way the current error could be set is if you called
`restore()` (in which case the current stored error-related members have
already been released), or if some *other* code raised the error, in
which case `clear()` on *this* object is clearing an error for which it
shouldn't be responsible.
Neither of those seem like intentional or desirable features, and
manually requesting deletion of the stored references similarly seems
pointless, so I've just made `clear()` an empty method and marked it
deprecated.
This also fixes a minor potential issue with the destruction: it is
technically possible for `value` to be null (though this seems likely to
be rare in practice); this updates the check to look at `type` which
will always be non-null for a `Fetch`ed exception.
This also adds error_already_set round-trip throw tests to the test
suite.
The instance registration for offset base types fails (under macOS, with
a segfault) in the presense of virtual base types. The issue occurs
when trying to `static_cast<Base *>(derived_ptr)` when `derived_ptr` has
been allocated (via `operator new`) but not initialized.
This commit fixes the issue by moving the addition to
`registered_instances` into `init_holder` rather than immediately after
value pointer allocation.
This also renames it to `init_instance` since it does more than holder
initialization now. (I also further renamed `init_holder_helper` to
`init_holder` since `init_holder` isn't used anymore).
Fixes#959.
Pre-C++17, std::pair can technically have an copy constructor even
though it can't actually be invoked without a compilation failure (due
to the underlying types being non-copyable). Most stls, including
libc++ since ~3.4, use the C++17 behaviour of not exposing an uncallable
copy constructor, but FreeBSD deliberately broke their libc++ to
preserve the nonsensical behaviour
(https://svnweb.freebsd.org/base?view=revision&revision=261801).
This updates pybind's internal `is_copy_constructible` to also detect
the std::pair case under pre-C++17.
This also everything (except for a couple cases in the internal version)
to use the internal `is_copy_constructible` rather than
`std::is_copy_constructible`.
This adds support for implicit conversions to bool from Python types
with `__bool__` (Python 3) or `__nonzero__` (Python 2) attributes, and
adds direct (i.e. non-converting) support for numpy bools.
If a class doesn't provide a `T::operator delete(void *)` but does have
a `T::operator delete(void *, size_t)` the latter is invoked by a
`delete someT`. Pybind currently only look for and call the former;
this commit adds detection and calling of the latter when the former
doesn't exist.
To fix a difficult-to-reproduce segfault on Python interpreter exit,
ensure that the tp_base field of a handful of new heap-types is
counted as a reference to that base type object.
This changes the pointer `cast()` in `PYBIND11_TYPE_CASTER` to recognize
the `take_ownership` policy: if casting a pointer with take-ownership,
the `cast()` now recalls `cast()` with a dereferenced rvalue (rather
than the previous code, which was always calling it with a const lvalue
reference), and deletes the pointer after the chained `cast()` is
complete.
This makes code like:
m.def("f", []() { return new std::vector<int>(100, 1); },
py::return_value_policy::take_ownership);
do the expected thing by taking over ownership of the returned pointer
(which is deleted once the chained cast completes).
PR #936 broke the ability to return a pointer to a stl container (and,
likewise, to a tuple) because the added deduced type matched a
non-const pointer argument: the pointer-accepting `cast` in
PYBIND11_TYPE_CASTER had a `const type *`, which is a worse match for a
non-const pointer than the universal reference template #936 added.
This changes the provided TYPE_CASTER cast(ptr) to take the pointer by
template arg (so that it will accept either const or non-const pointer).
It has two other effects: it slightly reduces .so size (because many
type casters never actually need the pointer cast at all), and it allows
type casters to provide their untemplated pointer `cast()` that will
take precedence over the templated version provided in the macro.
Currently select_cxx_standard(), which sets PYBIND11_CPP_STANDARD when
not externally set, is only called from pybind11_add_module(), but the
embed target setup (which runs unconditionally) makes use of
${PYBIND11_CPP_STANDARD}, which isn't set yet. This commit removes the
`select_cxx_standard` function completely and just always runs the
standard detection code.
This also tweaks the detection code to not bothering checking for the
`-std=c++11` flag when the `-std=c++14` detection succeeded.
In a Debug build, MSVC doesn't apply copy/move elision as often,
triggering a test failure. This relaxes the test count requirements
to let the test suite pass.
The value and holder iterator code had a past-the-end iterator
dereference. While of course invalid, the dereference didn't actually
cause any problems (which is why it wasn't caught before) because the
dereferenced value is never actually used and `vector` implementations
appear to allow dereferencing the past-the-end iterator. Under a MSVC
debug build, however, it fails a debug assertion and aborts.
This amends the iterator to just store and use a pointer to the vector
(rather than adding a second past-the-end iterator member), checking the
type index against the type vector size.
ICC was reporting that `try_direct_conversions()` cannot be `constexpr`
because `handle` is not a literal type. The fix removes `constexpr`
from the function since it isn't strictly needed.
This commit also suppresses new false positive warnings which mostly
appear in constexpr contexts (where the compiler knows conversions are
safe).
This updates the std::tuple, std::pair and `stl.h` type casters to
forward their contained value according to whether the container being
cast is an lvalue or rvalue reference. This fixes an issue where
subcaster casts were always called with a const lvalue which meant
nested type casters didn't have the desired `cast()` overload invoked.
For example, this caused Eigen values in a tuple to end up with a
readonly flag (issue #935) and made it impossible to return a container
of move-only types (issue #853).
This fixes both issues by adding templated universal reference `cast()`
methods to the various container types that forward container elements
according to the container reference type.
The std::pair caster can be written as a special case of the std::tuple
caster; this combines them via a base `tuple_caster` class (which is
essentially identical to the previous std::tuple caster).
This also removes the special empty tuple base case: returning an empty
tuple is relatively rare, and the base case still works perfectly well
even when the tuple types is an empty list.
When defining method from a member function pointer (e.g. `.def("f",
&Derived::f)`) we run into a problem if `&Derived::f` is actually
implemented in some base class `Base` when `Base` isn't
pybind-registered.
This happens because the class type is deduced from the member function
pointer, which then becomes a lambda with first argument this deduced
type. For a base class implementation, the deduced type is `Base`, not
`Derived`, and so we generate and registered an overload which takes a
`Base *` as first argument. Trying to call this fails if `Base` isn't
registered (e.g. because it's an implementation detail class that isn't
intended to be exposed to Python) because the type caster for an
unregistered type always fails.
This commit adds a `method_adaptor` function that rebinds a member
function to a derived type member function and otherwise (i.e. regular
functions/lambda) leaves the argument as-is. This is now used for class
definitions so that they are bound with type being registered rather
than a potential base type.
A closely related fix in this commit is to similarly update the lambdas
used for `def_readwrite` (and related) to bind to the class type being
registered rather than the deduced type so that registering a property
that resolves to a base class member similarly generates a usable
function.
Fixes#854, #910.
Co-Authored-By: Dean Moldovan <dean0x7d@gmail.com>
When casting to an unsigned type from a python 2 `int`, we currently
cast using `(unsigned long long) PyLong_AsUnsignedLong(src.ptr())`.
If the Python cast fails, it returns (unsigned long) -1, but then we
cast this to `unsigned long long`, which means we get 4294967295, but
because that isn't equal to `(unsigned long long) -1`, we don't detect
the failure.
This commit moves the unsigned casting into a `detail::as_unsigned`
function which, upon error, casts -1 to the final type, and otherwise
casts the return value to the final type to avoid the problematic double
cast when an error occurs.
The error most commonly shows up wherever `long` is 32-bits (e.g. under
both 32- and 64-bit Windows, and under 32-bit linux) when passing a
negative value to a bound function taking an `unsigned long`.
Fixes#929.
The added tests also trigger a latent segfault under PyPy: when casting
to an integer smaller than `long` (e.g. casting to a `uint32_t` on a
64-bit `long` architecture) we check both for a Python error and also
that the resulting intermediate value will fit in the final type. If
there is no conversion error, but we get a value that would overflow, we
end up calling `PyErr_ExceptionMatches()` illegally: that call is only
allowed when there is a current exception. Under PyPy, this segfaults
the test suite. It doesn't appear to segfault under CPython, but the
documentation suggests that it *could* do so. The fix is to only check
for the exception match if we actually got an error.
gcc 7 is now in debian testing ("buster"), with a proper stable upstream
release; this updates the associated travis-ci to use "buster" (rather
than "sid"), and removes the build from allow_failures.
This fixes#856. Instead of the weakref trick, the internals structure
holds an unordered_map from PyObject* to a vector of references. To
avoid the cost of the unordered_map lookup for objects that don't have
any keep_alive patients, a flag is added to each instance to indicate
whether there is anything to do.
Using `std::type_info::operator==` fails under libc++ because the .so
is loaded with RTLD_LOCAL. libc++ considers types under such .sos
distinct, and so comparing typeid() values directly isn't going to work.
This adds a custom hasher and equality class for the type lookup maps
when not under stdlibc++, and adds a `detail::same_type` function to
perform the equality test. It also converts a few pointer arguments to
const lvalue references, particularly since doing the pointer
comparison wasn't technically valid to being with (though in practice,
appeared to work everywhere).
This fixes#912.
Fixes a race condition when multiple threads try to acquire the GIL
before `detail::internals` have been initialized. `gil_scoped_release`
is now tasked with initializing `internals` (guaranteed single-threaded)
to ensure the safety of subsequent `acquire` calls from multiple threads.
Fixes the issue as described in the comments of commit e27ea47. This
just adds `enable_if_t<std::is_move_constructible<T>::value>` to
`make_move_constructor`. The change fixes MSVC and is harmless with
other compilers.
CLion slows to a crawl when evaluating the intricate `PYBIND11_NUMPY_DTYPE`
macro. This commit replaces the macro cascade with a simple `(void)0`
to ease IDE evaluation.
Jason Rhinelander
Ivan Smirnov
Dustin Spicuzza
Dean Moldovan
bennorth
Ivan Smirnov
Andreas Bergmeier
Bruce Merry