Zengtudor/pybind11
1
0
Fork 0
mirror of https://github.com/pybind/pybind11.git synced 2024-11-22 05:05:11 +00:00
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'master' into sh_merge_master

Browse Source
This commit is contained in:
Ralf W. Grosse-Kunstleve 2024-06-19 10:06:18 -07:00
commit 4068f10787
21 changed files with 590 additions and 218 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
.github/workflows/ci.yml vendored
View File

@ -1112,7 +1112,7 @@ jobs:
uses: jwlawson/actions-setup-cmake@v2.0
- name: Install ninja-build tool
uses: seanmiddleditch/gha-setup-ninja@v4
uses: seanmiddleditch/gha-setup-ninja@v5
- name: Run pip installs
run: |

231
docs/compiling.rst
View File

@ -3,15 +3,123 @@
Build systems
#############
For an overview of Python packaging including compiled packaging with a pybind11
example, along with a cookiecutter that includes several pybind11 options, see
the `Scientific Python Development Guide`_.
.. _Scientific Python Development Guide: https://learn.scientific-python.org/development/guides/packaging-compiled/
.. scikit-build-core:
Modules with CMake
==================
A Python extension module can be created with just a few lines of code:
.. code-block:: cmake
cmake_minimum_required(VERSION 3.15...3.29)
project(example LANGUAGES CXX)
set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(example example.cpp)
install(TARGET example DESTINATION .)
(You use the ``add_subdirectory`` instead, see the example in :ref:`cmake`.) In
this example, the code is located in a file named :file:`example.cpp`. Either
method will import the pybind11 project which provides the
``pybind11_add_module`` function. It will take care of all the details needed
to build a Python extension module on any platform.
To build with pip, build, cibuildwheel, uv, or other Python tools, you can
add a ``pyproject.toml`` file like this:
.. code-block:: toml
[build-system]
requires = ["scikit-build-core", "pybind11"]
build-backend = "scikit_build_core.build"
[project]
name = "example"
version = "0.1.0"
You don't need setuptools files like ``MANIFEST.in``, ``setup.py``, or
``setup.cfg``, as this is not setuptools. See `scikit-build-core`_ for details.
For projects you plan to upload to PyPI, be sure to fill out the ``[project]``
table with other important metadata as well (see `Writing pyproject.toml`_).
A working sample project can be found in the [scikit_build_example]_
repository. An older and harder-to-maintain method is in [cmake_example]_. More
details about our cmake support can be found below in :ref:`cmake`.
.. _scikit-build-core: https://scikit-build-core.readthedocs.io
.. [scikit_build_example] https://github.com/pybind/scikit_build_example
.. [cmake_example] https://github.com/pybind/cmake_example
.. _modules-meson-python:
Modules with meson-python
=========================
You can also build a package with `Meson`_ using `meson-python`_, if you prefer
that. Your ``meson.build`` file would look something like this:
.. _meson-example:
.. code-block:: meson
project(
'example',
'cpp',
version: '0.1.0',
default_options: [
'cpp_std=c++11',
],
)
py = import('python').find_installation(pure: false)
pybind11_dep = dependency('pybind11')
py.extension_module('example',
'example.cpp',
install: true,
dependencies : [pybind11_dep],
)
And you would need a ``pyproject.toml`` file like this:
.. code-block:: toml
[build-system]
requires = ["meson-python", "pybind11"]
build-backend = "mesonpy"
Meson-python *requires* your project to be in git (or mercurial) as it uses it
for the SDist creation. For projects you plan to upload to PyPI, be sure to fill out the
``[project]`` table as well (see `Writing pyproject.toml`_).
.. _Writing pyproject.toml: https://packaging.python.org/en/latest/guides/writing-pyproject-toml
.. _meson: https://mesonbuild.com
.. _meson-python: https://meson-python.readthedocs.io/en/latest
.. _build-setuptools:
Building with setuptools
========================
Modules with setuptools
=======================
For projects on PyPI, building with setuptools is the way to go. Sylvain Corlay
has kindly provided an example project which shows how to set up everything,
including automatic generation of documentation using Sphinx. Please refer to
the [python_example]_ repository.
For projects on PyPI, a historically popular option is setuptools. Sylvain
Corlay has kindly provided an example project which shows how to set up
everything, including automatic generation of documentation using Sphinx.
Please refer to the [python_example]_ repository.
.. [python_example] https://github.com/pybind/python_example
@ -21,11 +129,11 @@ To use pybind11 inside your ``setup.py``, you have to have some system to
ensure that ``pybind11`` is installed when you build your package. There are
four possible ways to do this, and pybind11 supports all four: You can ask all
users to install pybind11 beforehand (bad), you can use
:ref:`setup_helpers-pep518` (good, but very new and requires Pip 10),
:ref:`setup_helpers-setup_requires` (discouraged by Python packagers now that
PEP 518 is available, but it still works everywhere), or you can
:ref:`setup_helpers-copy-manually` (always works but you have to manually sync
your copy to get updates).
:ref:`setup_helpers-pep518` (good), ``setup_requires=`` (discouraged), or you
can :ref:`setup_helpers-copy-manually` (works but you have to manually sync
your copy to get updates). Third party packagers like conda-forge generally
strongly prefer the ``pyproject.toml`` method, as it gives them control over
the ``pybind11`` version, and they may apply patches, etc.
An example of a ``setup.py`` using pybind11's helpers:
@ -122,70 +230,41 @@ version number that includes the number of commits since your last tag and a
hash for a dirty directory. Another way to force a rebuild is purge your cache
or use Pip's ``--no-cache-dir`` option.
You also need a ``MANIFEST.in`` file to include all relevant files so that you
can make an SDist. If you use `pypa-build`_, that will build an SDist then a
wheel from that SDist by default, so you can look inside those files (wheels
are just zip files with a ``.whl`` extension) to make sure you aren't missing
files. `check-manifest`_ (setuptools specific) or `check-sdist`_ (general) are
CLI tools that can compare the SDist contents with your source control.
.. [Ccache] https://ccache.dev
.. [setuptools_scm] https://github.com/pypa/setuptools_scm
.. _setup_helpers-pep518:
PEP 518 requirements (Pip 10+ required)
---------------------------------------
Build requirements
------------------
If you use `PEP 518's <https://www.python.org/dev/peps/pep-0518/>`_
``pyproject.toml`` file, you can ensure that ``pybind11`` is available during
the compilation of your project. When this file exists, Pip will make a new
virtual environment, download just the packages listed here in ``requires=``,
and build a wheel (binary Python package). It will then throw away the
environment, and install your wheel.
With a ``pyproject.toml`` file, you can ensure that ``pybind11`` is available
during the compilation of your project. When this file exists, Pip will make a
new virtual environment, download just the packages listed here in
``requires=``, and build a wheel (binary Python package). It will then throw
away the environment, and install your wheel.
Your ``pyproject.toml`` file will likely look something like this:
.. code-block:: toml
[build-system]
requires = ["setuptools>=42", "pybind11>=2.6.1"]
requires = ["setuptools", "pybind11"]
build-backend = "setuptools.build_meta"
.. note::
The main drawback to this method is that a `PEP 517`_ compliant build tool,
such as Pip 10+, is required for this approach to work; older versions of
Pip completely ignore this file. If you distribute binaries (called wheels
in Python) using something like `cibuildwheel`_, remember that ``setup.py``
and ``pyproject.toml`` are not even contained in the wheel, so this high
Pip requirement is only for source builds, and will not affect users of
your binary wheels. If you are building SDists and wheels, then
`pypa-build`_ is the recommended official tool.
.. _PEP 517: https://www.python.org/dev/peps/pep-0517/
.. _cibuildwheel: https://cibuildwheel.readthedocs.io
.. _pypa-build: https://pypa-build.readthedocs.io/en/latest/
.. _setup_helpers-setup_requires:
Classic ``setup_requires``
--------------------------
If you want to support old versions of Pip with the classic
``setup_requires=["pybind11"]`` keyword argument to setup, which triggers a
two-phase ``setup.py`` run, then you will need to use something like this to
ensure the first pass works (which has not yet installed the ``setup_requires``
packages, since it can't install something it does not know about):
.. code-block:: python
try:
from pybind11.setup_helpers import Pybind11Extension
except ImportError:
from setuptools import Extension as Pybind11Extension
It doesn't matter that the Extension class is not the enhanced subclass for the
first pass run; and the second pass will have the ``setup_requires``
requirements.
This is obviously more of a hack than the PEP 518 method, but it supports
ancient versions of Pip.
.. _cibuildwheel: https://cibuildwheel.pypa.io
.. _pypa-build: https://build.pypa.io/en/latest/
.. _check-manifest: https://pypi.io/project/check-manifest
.. _check-sdist: https://pypi.io/project/check-sdist
.. _setup_helpers-copy-manually:
@ -231,32 +310,22 @@ the C++ source file. Python is then able to find the module and load it.
.. [cppimport] https://github.com/tbenthompson/cppimport
.. _cmake:
Building with CMake
===================
For C++ codebases that have an existing CMake-based build system, a Python
extension module can be created with just a few lines of code:
extension module can be created with just a few lines of code, as seen above in
the module section. Pybind11 currently supports a lower minimum if you don't
use the modern FindPython, though be aware that CMake 3.27 removed the old
mechanism, so pybind11 will automatically switch if the old mechanism is not
available. Please opt into the new mechanism if at all possible. Our default
may change in future versions. This is the minimum required:
.. code-block:: cmake
cmake_minimum_required(VERSION 3.5...3.29)
project(example LANGUAGES CXX)
add_subdirectory(pybind11)
pybind11_add_module(example example.cpp)
This assumes that the pybind11 repository is located in a subdirectory named
:file:`pybind11` and that the code is located in a file named :file:`example.cpp`.
The CMake command ``add_subdirectory`` will import the pybind11 project which
provides the ``pybind11_add_module`` function. It will take care of all the
details needed to build a Python extension module on any platform.
A working sample project, including a way to invoke CMake from :file:`setup.py` for
PyPI integration, can be found in the [cmake_example]_ repository.
.. [cmake_example] https://github.com/pybind/cmake_example
.. versionchanged:: 2.6
CMake 3.4+ is required.
@ -264,6 +333,7 @@ PyPI integration, can be found in the [cmake_example]_ repository.
.. versionchanged:: 2.11
CMake 3.5+ is required.
Further information can be found at :doc:`cmake/index`.
pybind11_add_module
@ -616,6 +686,13 @@ Building with Bazel
You can build with the Bazel build system using the `pybind11_bazel
<https://github.com/pybind/pybind11_bazel>`_ repository.
Building with Meson
===================
You can use Meson, which has support for ``pybind11`` as a dependency (internally
relying on our ``pkg-config`` support). See the :ref:`module example above <meson-example>`.
Generating binding code automatically
=====================================

6
docs/requirements.txt
View File

@ -269,7 +269,7 @@ sphinxcontrib-svg2pdfconverter==1.2.2 \
--hash=sha256:04ec767b55780a6b18d89cc1a8ada6d900c6efde9d1683abdb98a49b144465ca \
--hash=sha256:80a55ca61f70eae93efc65f3814f2f177c86ba55934a9f6c5022f1778b62146b
# via -r requirements.in
urllib3==2.2.1 \
--hash=sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d \
--hash=sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19
urllib3==2.2.2 \
--hash=sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472 \
--hash=sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168
# via requests

110
include/pybind11/detail/class.h
View File

@ -205,39 +205,40 @@ extern "C" inline PyObject *pybind11_meta_call(PyObject *type, PyObject *args, P
/// Cleanup the type-info for a pybind11-registered type.
extern "C" inline void pybind11_meta_dealloc(PyObject *obj) {
auto *type = (PyTypeObject *) obj;
auto &internals = get_internals();
with_internals([obj](internals &internals) {
auto *type = (PyTypeObject *) obj;
// A pybind11-registered type will:
// 1) be found in internals.registered_types_py
// 2) have exactly one associated `detail::type_info`
auto found_type = internals.registered_types_py.find(type);
if (found_type != internals.registered_types_py.end() && found_type->second.size() == 1
&& found_type->second[0]->type == type) {
// A pybind11-registered type will:
// 1) be found in internals.registered_types_py
// 2) have exactly one associated `detail::type_info`
auto found_type = internals.registered_types_py.find(type);
if (found_type != internals.registered_types_py.end() && found_type->second.size() == 1
&& found_type->second[0]->type == type) {
auto *tinfo = found_type->second[0];
auto tindex = std::type_index(*tinfo->cpptype);
internals.direct_conversions.erase(tindex);
auto *tinfo = found_type->second[0];
auto tindex = std::type_index(*tinfo->cpptype);
internals.direct_conversions.erase(tindex);
if (tinfo->module_local) {
get_local_internals().registered_types_cpp.erase(tindex);
} else {
internals.registered_types_cpp.erase(tindex);
}
internals.registered_types_py.erase(tinfo->type);
// Actually just `std::erase_if`, but that's only available in C++20
auto &cache = internals.inactive_override_cache;
for (auto it = cache.begin(), last = cache.end(); it != last;) {
if (it->first == (PyObject *) tinfo->type) {
it = cache.erase(it);
if (tinfo->module_local) {
get_local_internals().registered_types_cpp.erase(tindex);
} else {
++it;
internals.registered_types_cpp.erase(tindex);
}
}
internals.registered_types_py.erase(tinfo->type);
delete tinfo;
}
// Actually just `std::erase_if`, but that's only available in C++20
auto &cache = internals.inactive_override_cache;
for (auto it = cache.begin(), last = cache.end(); it != last;) {
if (it->first == (PyObject *) tinfo->type) {
it = cache.erase(it);
} else {
++it;
}
}
delete tinfo;
}
});
PyType_Type.tp_dealloc(obj);
}
@ -310,19 +311,20 @@ inline void traverse_offset_bases(void *valueptr,
}
inline bool register_instance_impl(void *ptr, instance *self) {
get_internals().registered_instances.emplace(ptr, self);
with_instance_map(ptr, [&](instance_map &instances) { instances.emplace(ptr, self); });
return true; // unused, but gives the same signature as the deregister func
}
inline bool deregister_instance_impl(void *ptr, instance *self) {
auto &registered_instances = get_internals().registered_instances;
auto range = registered_instances.equal_range(ptr);
for (auto it = range.first; it != range.second; ++it) {
if (self == it->second) {
registered_instances.erase(it);
return true;
return with_instance_map(ptr, [&](instance_map &instances) {
auto range = instances.equal_range(ptr);
for (auto it = range.first; it != range.second; ++it) {
if (self == it->second) {
instances.erase(it);
return true;
}
}
}
return false;
return false;
});
}
inline void register_instance(instance *self, void *valptr, const type_info *tinfo) {
@ -377,27 +379,32 @@ extern "C" inline int pybind11_object_init(PyObject *self, PyObject *, PyObject
}
inline void add_patient(PyObject *nurse, PyObject *patient) {
auto &internals = get_internals();
auto *instance = reinterpret_cast<detail::instance *>(nurse);
instance->has_patients = true;
Py_INCREF(patient);
internals.patients[nurse].push_back(patient);
with_internals([&](internals &internals) { internals.patients[nurse].push_back(patient); });
}
inline void clear_patients(PyObject *self) {
auto *instance = reinterpret_cast<detail::instance *>(self);
auto &internals = get_internals();
auto pos = internals.patients.find(self);
std::vector<PyObject *> patients;
if (pos == internals.patients.end()) {
pybind11_fail("FATAL: Internal consistency check failed: Invalid clear_patients() call.");
}
with_internals([&](internals &internals) {
auto pos = internals.patients.find(self);
if (pos == internals.patients.end()) {
pybind11_fail(
"FATAL: Internal consistency check failed: Invalid clear_patients() call.");
}
// Clearing the patients can cause more Python code to run, which
// can invalidate the iterator. Extract the vector of patients
// from the unordered_map first.
patients = std::move(pos->second);
internals.patients.erase(pos);
});
// Clearing the patients can cause more Python code to run, which
// can invalidate the iterator. Extract the vector of patients
// from the unordered_map first.
auto patients = std::move(pos->second);
internals.patients.erase(pos);
instance->has_patients = false;
for (PyObject *&patient : patients) {
Py_CLEAR(patient);
@ -664,10 +671,13 @@ inline PyObject *make_new_python_type(const type_record &rec) {
char *tp_doc = nullptr;
if (rec.doc && options::show_user_defined_docstrings()) {
/* Allocate memory for docstring (using PyObject_MALLOC, since
Python will free this later on) */
/* Allocate memory for docstring (Python will free this later on) */
size_t size = std::strlen(rec.doc) + 1;
#if PY_VERSION_HEX >= 0x030D0000
tp_doc = (char *) PyMem_MALLOC(size);
#else
tp_doc = (char *) PyObject_MALLOC(size);
#endif
std::memcpy((void *) tp_doc, rec.doc, size);
}

7
include/pybind11/detail/common.h
View File

@ -464,7 +464,7 @@ PYBIND11_WARNING_POP
});
}
\endrst */
#define PYBIND11_MODULE(name, variable) \
#define PYBIND11_MODULE(name, variable, ...) \
static ::pybind11::module_::module_def PYBIND11_CONCAT(pybind11_module_def_, name) \
PYBIND11_MAYBE_UNUSED; \
PYBIND11_MAYBE_UNUSED \
@ -473,7 +473,10 @@ PYBIND11_WARNING_POP
PYBIND11_CHECK_PYTHON_VERSION \
PYBIND11_ENSURE_INTERNALS_READY \
auto m = ::pybind11::module_::create_extension_module( \
PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name)); \
PYBIND11_TOSTRING(name), \
nullptr, \
&PYBIND11_CONCAT(pybind11_module_def_, name), \
##__VA_ARGS__); \
try { \
PYBIND11_CONCAT(pybind11_init_, name)(m); \
return m.ptr(); \

152
include/pybind11/detail/internals.h
View File

@ -19,6 +19,8 @@
#include "smart_holder_sfinae_hooks_only.h"
#include <exception>
#include <mutex>
#include <thread>
/// Tracks the `internals` and `type_info` ABI version independent of the main library version.
///
@ -169,15 +171,37 @@ struct override_hash {
}
};
using instance_map = std::unordered_multimap<const void *, instance *>;
// ignore: structure was padded due to alignment specifier
PYBIND11_WARNING_PUSH
PYBIND11_WARNING_DISABLE_MSVC(4324)
// Instance map shards are used to reduce mutex contention in free-threaded Python.
struct alignas(64) instance_map_shard {
std::mutex mutex;
instance_map registered_instances;
};
PYBIND11_WARNING_POP
/// Internal data structure used to track registered instances and types.
/// Whenever binary incompatible changes are made to this structure,
/// `PYBIND11_INTERNALS_VERSION` must be incremented.
struct internals {
#ifdef Py_GIL_DISABLED
std::mutex mutex;
#endif
// std::type_index -> pybind11's type information
type_map<type_info *> registered_types_cpp;
// PyTypeObject* -> base type_info(s)
std::unordered_map<PyTypeObject *, std::vector<type_info *>> registered_types_py;
std::unordered_multimap<const void *, instance *> registered_instances; // void * -> instance*
#ifdef Py_GIL_DISABLED
std::unique_ptr<instance_map_shard[]> instance_shards; // void * -> instance*
size_t instance_shards_mask;
#else
instance_map registered_instances; // void * -> instance*
#endif
std::unordered_set<std::pair<const PyObject *, const char *>, override_hash>
inactive_override_cache;
type_map<std::vector<bool (*)(PyObject *, void *&)>> direct_conversions;
@ -473,7 +497,8 @@ inline object get_python_state_dict() {
}
inline object get_internals_obj_from_state_dict(handle state_dict) {
return reinterpret_borrow<object>(dict_getitemstring(state_dict.ptr(), PYBIND11_INTERNALS_ID));
return reinterpret_steal<object>(
dict_getitemstringref(state_dict.ptr(), PYBIND11_INTERNALS_ID));
}
inline internals **get_internals_pp_from_capsule(handle obj) {
@ -485,6 +510,20 @@ inline internals **get_internals_pp_from_capsule(handle obj) {
return static_cast<internals **>(raw_ptr);
}
inline uint64_t round_up_to_next_pow2(uint64_t x) {
// Round-up to the next power of two.
// See https://graphics.stanford.edu/~seander/bithacks.html#RoundUpPowerOf2
x--;
x |= (x >> 1);
x |= (x >> 2);
x |= (x >> 4);
x |= (x >> 8);
x |= (x >> 16);
x |= (x >> 32);
x++;
return x;
}
/// Return a reference to the current `internals` data
PYBIND11_NOINLINE internals &get_internals() {
auto **&internals_pp = get_internals_pp();
@ -553,6 +592,16 @@ PYBIND11_NOINLINE internals &get_internals() {
internals_ptr->static_property_type = make_static_property_type();
internals_ptr->default_metaclass = make_default_metaclass();
internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass);
#ifdef Py_GIL_DISABLED
// Scale proportional to the number of cores. 2x is a heuristic to reduce contention.
auto num_shards
= static_cast<size_t>(round_up_to_next_pow2(2 * std::thread::hardware_concurrency()));
if (num_shards == 0) {
num_shards = 1;
}
internals_ptr->instance_shards.reset(new instance_map_shard[num_shards]);
internals_ptr->instance_shards_mask = num_shards - 1;
#endif // Py_GIL_DISABLED
}
return **internals_pp;
}
@ -613,13 +662,80 @@ inline local_internals &get_local_internals() {
return *locals;
}
#ifdef Py_GIL_DISABLED
# define PYBIND11_LOCK_INTERNALS(internals) std::unique_lock<std::mutex> lock((internals).mutex)
#else
# define PYBIND11_LOCK_INTERNALS(internals)
#endif
template <typename F>
inline auto with_internals(const F &cb) -> decltype(cb(get_internals())) {
auto &internals = get_internals();
PYBIND11_LOCK_INTERNALS(internals);
return cb(internals);
}
inline std::uint64_t mix64(std::uint64_t z) {
// David Stafford's variant 13 of the MurmurHash3 finalizer popularized
// by the SplitMix PRNG.
// https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html
z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9;
z = (z ^ (z >> 27)) * 0x94d049bb133111eb;
return z ^ (z >> 31);
}
template <typename F>
inline auto with_instance_map(const void *ptr,
const F &cb) -> decltype(cb(std::declval<instance_map &>())) {
auto &internals = get_internals();
#ifdef Py_GIL_DISABLED
// Hash address to compute shard, but ignore low bits. We'd like allocations
// from the same thread/core to map to the same shard and allocations from
// other threads/cores to map to other shards. Using the high bits is a good
// heuristic because memory allocators often have a per-thread
// arena/superblock/segment from which smaller allocations are served.
auto addr = reinterpret_cast<std::uintptr_t>(ptr);
auto hash = mix64(static_cast<std::uint64_t>(addr >> 20));
auto idx = static_cast<size_t>(hash & internals.instance_shards_mask);
auto &shard = internals.instance_shards[idx];
std::unique_lock<std::mutex> lock(shard.mutex);
return cb(shard.registered_instances);
#else
(void) ptr;
return cb(internals.registered_instances);
#endif
}
// Returns the number of registered instances for testing purposes. The result may not be
// consistent if other threads are registering or unregistering instances concurrently.
inline size_t num_registered_instances() {
auto &internals = get_internals();
#ifdef Py_GIL_DISABLED
size_t count = 0;
for (size_t i = 0; i <= internals.instance_shards_mask; ++i) {
auto &shard = internals.instance_shards[i];
std::unique_lock<std::mutex> lock(shard.mutex);
count += shard.registered_instances.size();
}
return count;
#else
return internals.registered_instances.size();
#endif
}
/// Constructs a std::string with the given arguments, stores it in `internals`, and returns its
/// `c_str()`. Such strings objects have a long storage duration -- the internal strings are only
/// cleared when the program exits or after interpreter shutdown (when embedding), and so are
/// suitable for c-style strings needed by Python internals (such as PyTypeObject's tp_name).
template <typename... Args>
const char *c_str(Args &&...args) {
auto &strings = get_internals().static_strings;
// GCC 4.8 doesn't like parameter unpack within lambda capture, so use
// PYBIND11_LOCK_INTERNALS.
auto &internals = get_internals();
PYBIND11_LOCK_INTERNALS(internals);
auto &strings = internals.static_strings;
strings.emplace_front(std::forward<Args>(args)...);
return strings.front().c_str();
}
@ -649,15 +765,18 @@ PYBIND11_NAMESPACE_END(detail)
/// pybind11 version) running in the current interpreter. Names starting with underscores
/// are reserved for internal usage. Returns `nullptr` if no matching entry was found.
PYBIND11_NOINLINE void *get_shared_data(const std::string &name) {
auto &internals = detail::get_internals();
auto it = internals.shared_data.find(name);
return it != internals.shared_data.end() ? it->second : nullptr;
return detail::with_internals([&](detail::internals &internals) {
auto it = internals.shared_data.find(name);
return it != internals.shared_data.end() ? it->second : nullptr;
});
}
/// Set the shared data that can be later recovered by `get_shared_data()`.
PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) {
detail::get_internals().shared_data[name] = data;
return data;
return detail::with_internals([&](detail::internals &internals) {
internals.shared_data[name] = data;
return data;
});
}
/// Returns a typed reference to a shared data entry (by using `get_shared_data()`) if
@ -665,14 +784,15 @@ PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) {
/// added to the shared data under the given name and a reference to it is returned.
template <typename T>
T &get_or_create_shared_data(const std::string &name) {
auto &internals = detail::get_internals();
auto it = internals.shared_data.find(name);
T *ptr = (T *) (it != internals.shared_data.end() ? it->second : nullptr);
if (!ptr) {
ptr = new T();
internals.shared_data[name] = ptr;
}
return *ptr;
return *detail::with_internals([&](detail::internals &internals) {
auto it = internals.shared_data.find(name);
T *ptr = (T *) (it != internals.shared_data.end() ? it->second : nullptr);
if (!ptr) {
ptr = new T();
internals.shared_data[name] = ptr;
}
return ptr;
});
}
PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE)

48
include/pybind11/detail/type_caster_base.h
View File

@ -217,12 +217,15 @@ inline detail::type_info *get_local_type_info(const std::type_index &tp) {
}
inline detail::type_info *get_global_type_info(const std::type_index &tp) {
auto &types = get_internals().registered_types_cpp;
auto it = types.find(tp);
if (it != types.end()) {
return it->second;
}
return nullptr;
return with_internals([&](internals &internals) {
detail::type_info *type_info = nullptr;
auto &types = internals.registered_types_cpp;
auto it = types.find(tp);
if (it != types.end()) {
type_info = it->second;
}
return type_info;
});
}
/// Return the type info for a given C++ type; on lookup failure can either throw or return
@ -253,15 +256,17 @@ PYBIND11_NOINLINE handle get_type_handle(const std::type_info &tp, bool throw_if
// Searches the inheritance graph for a registered Python instance, using all_type_info().
PYBIND11_NOINLINE handle find_registered_python_instance(void *src,
const detail::type_info *tinfo) {
auto it_instances = get_internals().registered_instances.equal_range(src);
for (auto it_i = it_instances.first; it_i != it_instances.second; ++it_i) {
for (auto *instance_type : detail::all_type_info(Py_TYPE(it_i->second))) {
if (instance_type && same_type(*instance_type->cpptype, *tinfo->cpptype)) {
return handle((PyObject *) it_i->second).inc_ref();
return with_instance_map(src, [&](instance_map &instances) {
auto it_instances = instances.equal_range(src);
for (auto it_i = it_instances.first; it_i != it_instances.second; ++it_i) {
for (auto *instance_type : detail::all_type_info(Py_TYPE(it_i->second))) {
if (instance_type && same_type(*instance_type->cpptype, *tinfo->cpptype)) {
return handle((PyObject *) it_i->second).inc_ref();
}
}
}
}
return handle();
return handle();
});
}
struct value_and_holder {
@ -506,16 +511,17 @@ PYBIND11_NOINLINE bool isinstance_generic(handle obj, const std::type_info &tp)
}
PYBIND11_NOINLINE handle get_object_handle(const void *ptr, const detail::type_info *type) {
auto &instances = get_internals().registered_instances;
auto range = instances.equal_range(ptr);
for (auto it = range.first; it != range.second; ++it) {
for (const auto &vh : values_and_holders(it->second)) {
if (vh.type == type) {
return handle((PyObject *) it->second);
return with_instance_map(ptr, [&](instance_map &instances) {
auto range = instances.equal_range(ptr);
for (auto it = range.first; it != range.second; ++it) {
for (const auto &vh : values_and_holders(it->second)) {
if (vh.type == type) {
return handle((PyObject *) it->second);
}
}
}
}
return handle();
return handle();
});
}
inline PyThreadState *get_thread_state_unchecked() {

4
include/pybind11/numpy.h
View File

@ -1553,7 +1553,9 @@ PYBIND11_NOINLINE void register_structured_dtype(any_container<field_descriptor>
auto tindex = std::type_index(tinfo);
numpy_internals.registered_dtypes[tindex] = {dtype_ptr, std::move(format_str)};
get_internals().direct_conversions[tindex].push_back(direct_converter);
with_internals([tindex, &direct_converter](internals &internals) {
internals.direct_conversions[tindex].push_back(direct_converter);
});
}
template <typename T, typename SFINAE>

129
include/pybind11/pybind11.h
View File

@ -1055,13 +1055,20 @@ protected:
- delegate translation to the next translator by throwing a new type of exception.
*/
auto &local_exception_translators
= get_local_internals().registered_exception_translators;
if (detail::apply_exception_translators(local_exception_translators)) {
return nullptr;
}
auto &exception_translators = get_internals().registered_exception_translators;
if (detail::apply_exception_translators(exception_translators)) {
bool handled = with_internals([&](internals &internals) {
auto &local_exception_translators
= get_local_internals().registered_exception_translators;
if (detail::apply_exception_translators(local_exception_translators)) {
return true;
}
auto &exception_translators = internals.registered_exception_translators;
if (detail::apply_exception_translators(exception_translators)) {
return true;
}
return false;
});
if (handled) {
return nullptr;
}
@ -1200,6 +1207,16 @@ struct handle_type_name<cpp_function> {
PYBIND11_NAMESPACE_END(detail)
// Use to activate Py_MOD_GIL_NOT_USED.
class mod_gil_not_used {
public:
explicit mod_gil_not_used(bool flag = true) : flag_(flag) {}
bool flag() const { return flag_; }
private:
bool flag_;
};
/// Wrapper for Python extension modules
class module_ : public object {
public:
@ -1300,7 +1317,11 @@ public:
``def`` should point to a statically allocated module_def.
\endrst */
static module_ create_extension_module(const char *name, const char *doc, module_def *def) {
static module_ create_extension_module(const char *name,
const char *doc,
module_def *def,
mod_gil_not_used gil_not_used
= mod_gil_not_used(false)) {
// module_def is PyModuleDef
// Placement new (not an allocation).
def = new (def)
@ -1320,6 +1341,11 @@ public:
}
pybind11_fail("Internal error in module_::create_extension_module()");
}
if (gil_not_used.flag()) {
#ifdef Py_GIL_DISABLED
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
#endif
}
// TODO: Should be reinterpret_steal for Python 3, but Python also steals it again when
// returned from PyInit_...
// For Python 2, reinterpret_borrow was correct.
@ -1399,15 +1425,16 @@ protected:
tinfo->default_holder = rec.default_holder;
tinfo->module_local = rec.module_local;
auto &internals = get_internals();
auto tindex = std::type_index(*rec.type);
tinfo->direct_conversions = &internals.direct_conversions[tindex];
if (rec.module_local) {
get_local_internals().registered_types_cpp[tindex] = tinfo;
} else {
internals.registered_types_cpp[tindex] = tinfo;
}
internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo};
with_internals([&](internals &internals) {
auto tindex = std::type_index(*rec.type);
tinfo->direct_conversions = &internals.direct_conversions[tindex];
if (rec.module_local) {
get_local_internals().registered_types_cpp[tindex] = tinfo;
} else {
internals.registered_types_cpp[tindex] = tinfo;
}
internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo};
});
if (rec.bases.size() > 1 || rec.multiple_inheritance) {
mark_parents_nonsimple(tinfo->type);
@ -1853,10 +1880,12 @@ public:
generic_type_initialize(record);
if (has_alias) {
auto &instances = record.module_local ? get_local_internals().registered_types_cpp
: get_internals().registered_types_cpp;
instances[std::type_index(typeid(type_alias))]
= instances[std::type_index(typeid(type))];
with_internals([&](internals &internals) {
auto &instances = record.module_local ? get_local_internals().registered_types_cpp
: internals.registered_types_cpp;
instances[std::type_index(typeid(type_alias))]
= instances[std::type_index(typeid(type))];
});
}
}
@ -2596,28 +2625,32 @@ keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret) {
inline std::pair<decltype(internals::registered_types_py)::iterator, bool>
all_type_info_get_cache(PyTypeObject *type) {
auto res = get_internals()
.registered_types_py
auto res = with_internals([type](internals &internals) {
return internals
.registered_types_py
#ifdef __cpp_lib_unordered_map_try_emplace
.try_emplace(type);
.try_emplace(type);
#else
.emplace(type, std::vector<detail::type_info *>());
.emplace(type, std::vector<detail::type_info *>());
#endif
});
if (res.second) {
// New cache entry created; set up a weak reference to automatically remove it if the type
// gets destroyed:
weakref((PyObject *) type, cpp_function([type](handle wr) {
get_internals().registered_types_py.erase(type);
with_internals([type](internals &internals) {
internals.registered_types_py.erase(type);
// TODO consolidate the erasure code in pybind11_meta_dealloc() in class.h
auto &cache = get_internals().inactive_override_cache;
for (auto it = cache.begin(), last = cache.end(); it != last;) {
if (it->first == reinterpret_cast<PyObject *>(type)) {
it = cache.erase(it);
} else {
++it;
// TODO consolidate the erasure code in pybind11_meta_dealloc() in class.h
auto &cache = internals.inactive_override_cache;
for (auto it = cache.begin(), last = cache.end(); it != last;) {
if (it->first == reinterpret_cast<PyObject *>(type)) {
it = cache.erase(it);
} else {
++it;
}
}
}
});
wr.dec_ref();
}))
@ -2822,7 +2855,11 @@ void implicitly_convertible() {
~set_flag() { flag = false; }
};
auto implicit_caster = [](PyObject *obj, PyTypeObject *type) -> PyObject * {
#ifdef Py_GIL_DISABLED
thread_local bool currently_used = false;
#else
static bool currently_used = false;
#endif
if (currently_used) { // implicit conversions are non-reentrant
return nullptr;
}
@ -2847,8 +2884,10 @@ void implicitly_convertible() {
}
inline void register_exception_translator(ExceptionTranslator &&translator) {
detail::get_internals().registered_exception_translators.push_front(
std::forward<ExceptionTranslator>(translator));
detail::with_internals([&](detail::internals &internals) {
internals.registered_exception_translators.push_front(
std::forward<ExceptionTranslator>(translator));
});
}
/**
@ -2858,8 +2897,11 @@ inline void register_exception_translator(ExceptionTranslator &&translator) {
* the exception.
*/
inline void register_local_exception_translator(ExceptionTranslator &&translator) {
detail::get_local_internals().registered_exception_translators.push_front(
std::forward<ExceptionTranslator>(translator));
detail::with_internals([&](detail::internals &internals) {
(void) internals;
detail::get_local_internals().registered_exception_translators.push_front(
std::forward<ExceptionTranslator>(translator));
});
}
/**
@ -3016,14 +3058,19 @@ get_type_override(const void *this_ptr, const type_info *this_type, const char *
/* Cache functions that aren't overridden in Python to avoid
many costly Python dictionary lookups below */
auto &cache = get_internals().inactive_override_cache;
if (cache.find(key) != cache.end()) {
bool not_overridden = with_internals([&key](internals &internals) {
auto &cache = internals.inactive_override_cache;
return cache.find(key) != cache.end();
});
if (not_overridden) {
return function();
}
function override = getattr(self, name, function());
if (override.is_cpp_function()) {
cache.insert(std::move(key));
with_internals([&](internals &internals) {
internals.inactive_override_cache.insert(std::move(key));
});
return function();
}

17
include/pybind11/pytypes.h
View File

@ -980,6 +980,23 @@ inline PyObject *dict_getitem(PyObject *v, PyObject *key) {
return rv;
}
inline PyObject *dict_getitemstringref(PyObject *v, const char *key) {
#if PY_VERSION_HEX >= 0x030D0000
PyObject *rv;
if (PyDict_GetItemStringRef(v, key, &rv) < 0) {
throw error_already_set();
}
return rv;
#else
PyObject *rv = dict_getitemstring(v, key);
if (rv == nullptr && PyErr_Occurred()) {
throw error_already_set();
}
Py_XINCREF(rv);
return rv;
#endif
}
// Helper aliases/functions to support implicit casting of values given to python
// accessors/methods. When given a pyobject, this simply returns the pyobject as-is; for other C++
// type, the value goes through pybind11::cast(obj) to convert it to an `object`.

32
include/pybind11/typing.h
View File

@ -63,6 +63,21 @@ class Callable<Return(Args...)> : public function {
using function::function;
};
template <typename T>
class Type : public type {
using type::type;
};
template <typename... Types>
class Union : public object {
using object::object;
};
template <typename T>
class Optional : public object {
using object::object;
};
PYBIND11_NAMESPACE_END(typing)
PYBIND11_NAMESPACE_BEGIN(detail)
@ -121,5 +136,22 @@ struct handle_type_name<typing::Callable<Return(Args...)>> {
+ const_name("], ") + make_caster<retval_type>::name + const_name("]");
};
template <typename T>
struct handle_type_name<typing::Type<T>> {
static constexpr auto name = const_name("type[") + make_caster<T>::name + const_name("]");
};
template <typename... Types>
struct handle_type_name<typing::Union<Types...>> {
static constexpr auto name = const_name("Union[")
+ ::pybind11::detail::concat(make_caster<Types>::name...)
+ const_name("]");
};
template <typename T>
struct handle_type_name<typing::Optional<T>> {
static constexpr auto name = const_name("Optional[") + make_caster<T>::name + const_name("]");
};
PYBIND11_NAMESPACE_END(detail)
PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE)

3
tests/cross_module_gil_utils.cpp
View File

@ -92,6 +92,9 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() {
if (m != nullptr) {
static_assert(sizeof(&gil_acquire) == sizeof(void *),
"Function pointer must have the same size as void*");
#ifdef Py_GIL_DISABLED
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
#endif
ADD_FUNCTION("gil_acquire_funcaddr", gil_acquire)
ADD_FUNCTION("gil_multi_acquire_release_funcaddr", gil_multi_acquire_release)
ADD_FUNCTION("gil_acquire_inner_custom_funcaddr",

3
tests/cross_module_interleaved_error_already_set.cpp
View File

@ -42,6 +42,9 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_interleaved_error_alrea
if (m != nullptr) {
static_assert(sizeof(&interleaved_error_already_set) == sizeof(void *),
"Function pointer must have the same size as void *");
#ifdef Py_GIL_DISABLED
PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED);
#endif
PyModule_AddObject(
m,
"funcaddr",

4
tests/eigen_tensor_avoid_stl_array.cpp
View File

@ -11,4 +11,6 @@
#include "test_eigen_tensor.inl"
PYBIND11_MODULE(eigen_tensor_avoid_stl_array, m) { eigen_tensor_test::test_module(m); }
PYBIND11_MODULE(eigen_tensor_avoid_stl_array, m, pybind11::mod_gil_not_used()) {
eigen_tensor_test::test_module(m);
}

2
tests/env.py
View File

@ -1,5 +1,6 @@
import platform
import sys
import sysconfig
import pytest
@ -9,6 +10,7 @@ WIN = sys.platform.startswith("win32") or sys.platform.startswith("cygwin")
CPYTHON = platform.python_implementation() == "CPython"
PYPY = platform.python_implementation() == "PyPy"
PY_GIL_DISABLED = bool(sysconfig.get_config_var("Py_GIL_DISABLED"))
def deprecated_call():

2
tests/pybind11_cross_module_tests.cpp
View File

@ -16,7 +16,7 @@
#include <numeric>
#include <utility>
PYBIND11_MODULE(pybind11_cross_module_tests, m) {
PYBIND11_MODULE(pybind11_cross_module_tests, m, py::mod_gil_not_used()) {
m.doc() = "pybind11 cross-module test module";
// test_local_bindings.py tests:

4
tests/pybind11_tests.cpp
View File

@ -58,7 +58,7 @@ void bind_ConstructorStats(py::module_ &m) {
// registered instances to allow instance cleanup checks (invokes a GC first)
.def_static("detail_reg_inst", []() {
ConstructorStats::gc();
return py::detail::get_internals().registered_instances.size();
return py::detail::num_registered_instances();
});
}
@ -75,7 +75,7 @@ const char *cpp_std() {
#endif
}
PYBIND11_MODULE(pybind11_tests, m) {
PYBIND11_MODULE(pybind11_tests, m, py::mod_gil_not_used()) {
m.doc() = "pybind11 test module";
// Intentionally kept minimal to not create a maintenance chore

2
tests/test_cmake_build/main.cpp
View File

@ -1,6 +1,6 @@
#include <pybind11/pybind11.h>
namespace py = pybind11;
PYBIND11_MODULE(test_cmake_build, m) {
PYBIND11_MODULE(test_cmake_build, m, py::mod_gil_not_used()) {
m.def("add", [](int i, int j) { return i + j; });
}

2
tests/test_embed/external_module.cpp
View File

@ -6,7 +6,7 @@ namespace py = pybind11;
* modules aren't preserved over a finalize/initialize.
*/
PYBIND11_MODULE(external_module, m) {
PYBIND11_MODULE(external_module, m, py::mod_gil_not_used()) {
class A {
public:
explicit A(int value) : v{value} {};

23
tests/test_pytypes.cpp
View File

@ -844,4 +844,27 @@ TEST_SUBMODULE(pytypes, m) {
m.def("annotate_iterator_int", [](const py::typing::Iterator<int> &) {});
m.def("annotate_fn",
[](const py::typing::Callable<int(py::typing::List<py::str>, py::str)> &) {});
m.def("annotate_type", [](const py::typing::Type<int> &) {});
m.def("annotate_union",
[](py::typing::List<py::typing::Union<py::str, py::int_, py::object>> l,
py::str a,
py::int_ b,
py::object c) -> py::typing::List<py::typing::Union<py::str, py::int_, py::object>> {
l.append(a);
l.append(b);
l.append(c);
return l;
});
m.def("union_typing_only",
[](py::typing::List<py::typing::Union<py::str>> &l)
-> py::typing::List<py::typing::Union<py::int_>> { return l; });
m.def("annotate_optional",
[](py::list &list) -> py::typing::List<py::typing::Optional<py::str>> {
list.append(py::str("hi"));
list.append(py::none());
return list;
});
}

25
tests/test_pytypes.py
View File

@ -955,3 +955,28 @@ def test_fn_annotations(doc):
doc(m.annotate_fn)
== "annotate_fn(arg0: Callable[[list[str], str], int]) -> None"
)
def test_type_annotation(doc):
assert doc(m.annotate_type) == "annotate_type(arg0: type[int]) -> None"
def test_union_annotations(doc):
assert (
doc(m.annotate_union)
== "annotate_union(arg0: list[Union[str, int, object]], arg1: str, arg2: int, arg3: object) -> list[Union[str, int, object]]"
)
def test_union_typing_only(doc):
assert (
doc(m.union_typing_only)
== "union_typing_only(arg0: list[Union[str]]) -> list[Union[int]]"
)
def test_optional_annotations(doc):
assert (
doc(m.annotate_optional)
== "annotate_optional(arg0: list) -> list[Optional[str]]"
)