diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2141414cb..ca1ed7564 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1112,7 +1112,7 @@ jobs: uses: jwlawson/actions-setup-cmake@v2.0 - name: Install ninja-build tool - uses: seanmiddleditch/gha-setup-ninja@v4 + uses: seanmiddleditch/gha-setup-ninja@v5 - name: Run pip installs run: | diff --git a/docs/compiling.rst b/docs/compiling.rst index cf97070a9..970bd5df8 100644 --- a/docs/compiling.rst +++ b/docs/compiling.rst @@ -3,15 +3,123 @@ Build systems ############# +For an overview of Python packaging including compiled packaging with a pybind11 +example, along with a cookiecutter that includes several pybind11 options, see +the `Scientific Python Development Guide`_. + +.. _Scientific Python Development Guide: https://learn.scientific-python.org/development/guides/packaging-compiled/ + +.. scikit-build-core: + +Modules with CMake +================== + +A Python extension module can be created with just a few lines of code: + +.. code-block:: cmake + + cmake_minimum_required(VERSION 3.15...3.29) + project(example LANGUAGES CXX) + + set(PYBIND11_FINDPYTHON ON) + find_package(pybind11 CONFIG REQUIRED) + + pybind11_add_module(example example.cpp) + install(TARGET example DESTINATION .) + +(You use the ``add_subdirectory`` instead, see the example in :ref:`cmake`.) In +this example, the code is located in a file named :file:`example.cpp`. Either +method will import the pybind11 project which provides the +``pybind11_add_module`` function. It will take care of all the details needed +to build a Python extension module on any platform. + +To build with pip, build, cibuildwheel, uv, or other Python tools, you can +add a ``pyproject.toml`` file like this: + +.. code-block:: toml + + [build-system] + requires = ["scikit-build-core", "pybind11"] + build-backend = "scikit_build_core.build" + + [project] + name = "example" + version = "0.1.0" + +You don't need setuptools files like ``MANIFEST.in``, ``setup.py``, or +``setup.cfg``, as this is not setuptools. See `scikit-build-core`_ for details. +For projects you plan to upload to PyPI, be sure to fill out the ``[project]`` +table with other important metadata as well (see `Writing pyproject.toml`_). + +A working sample project can be found in the [scikit_build_example]_ +repository. An older and harder-to-maintain method is in [cmake_example]_. More +details about our cmake support can be found below in :ref:`cmake`. + +.. _scikit-build-core: https://scikit-build-core.readthedocs.io + +.. [scikit_build_example] https://github.com/pybind/scikit_build_example + +.. [cmake_example] https://github.com/pybind/cmake_example + +.. _modules-meson-python: + +Modules with meson-python +========================= + +You can also build a package with `Meson`_ using `meson-python`_, if you prefer +that. Your ``meson.build`` file would look something like this: + +.. _meson-example: + +.. code-block:: meson + + project( + 'example', + 'cpp', + version: '0.1.0', + default_options: [ + 'cpp_std=c++11', + ], + ) + + py = import('python').find_installation(pure: false) + pybind11_dep = dependency('pybind11') + + py.extension_module('example', + 'example.cpp', + install: true, + dependencies : [pybind11_dep], + ) + + +And you would need a ``pyproject.toml`` file like this: + +.. code-block:: toml + + [build-system] + requires = ["meson-python", "pybind11"] + build-backend = "mesonpy" + +Meson-python *requires* your project to be in git (or mercurial) as it uses it +for the SDist creation. For projects you plan to upload to PyPI, be sure to fill out the +``[project]`` table as well (see `Writing pyproject.toml`_). + + +.. _Writing pyproject.toml: https://packaging.python.org/en/latest/guides/writing-pyproject-toml + +.. _meson: https://mesonbuild.com + +.. _meson-python: https://meson-python.readthedocs.io/en/latest + .. _build-setuptools: -Building with setuptools -======================== +Modules with setuptools +======================= -For projects on PyPI, building with setuptools is the way to go. Sylvain Corlay -has kindly provided an example project which shows how to set up everything, -including automatic generation of documentation using Sphinx. Please refer to -the [python_example]_ repository. +For projects on PyPI, a historically popular option is setuptools. Sylvain +Corlay has kindly provided an example project which shows how to set up +everything, including automatic generation of documentation using Sphinx. +Please refer to the [python_example]_ repository. .. [python_example] https://github.com/pybind/python_example @@ -21,11 +129,11 @@ To use pybind11 inside your ``setup.py``, you have to have some system to ensure that ``pybind11`` is installed when you build your package. There are four possible ways to do this, and pybind11 supports all four: You can ask all users to install pybind11 beforehand (bad), you can use -:ref:`setup_helpers-pep518` (good, but very new and requires Pip 10), -:ref:`setup_helpers-setup_requires` (discouraged by Python packagers now that -PEP 518 is available, but it still works everywhere), or you can -:ref:`setup_helpers-copy-manually` (always works but you have to manually sync -your copy to get updates). +:ref:`setup_helpers-pep518` (good), ``setup_requires=`` (discouraged), or you +can :ref:`setup_helpers-copy-manually` (works but you have to manually sync +your copy to get updates). Third party packagers like conda-forge generally +strongly prefer the ``pyproject.toml`` method, as it gives them control over +the ``pybind11`` version, and they may apply patches, etc. An example of a ``setup.py`` using pybind11's helpers: @@ -122,70 +230,41 @@ version number that includes the number of commits since your last tag and a hash for a dirty directory. Another way to force a rebuild is purge your cache or use Pip's ``--no-cache-dir`` option. +You also need a ``MANIFEST.in`` file to include all relevant files so that you +can make an SDist. If you use `pypa-build`_, that will build an SDist then a +wheel from that SDist by default, so you can look inside those files (wheels +are just zip files with a ``.whl`` extension) to make sure you aren't missing +files. `check-manifest`_ (setuptools specific) or `check-sdist`_ (general) are +CLI tools that can compare the SDist contents with your source control. + .. [Ccache] https://ccache.dev .. [setuptools_scm] https://github.com/pypa/setuptools_scm .. _setup_helpers-pep518: -PEP 518 requirements (Pip 10+ required) ---------------------------------------- +Build requirements +------------------ -If you use `PEP 518's `_ -``pyproject.toml`` file, you can ensure that ``pybind11`` is available during -the compilation of your project. When this file exists, Pip will make a new -virtual environment, download just the packages listed here in ``requires=``, -and build a wheel (binary Python package). It will then throw away the -environment, and install your wheel. +With a ``pyproject.toml`` file, you can ensure that ``pybind11`` is available +during the compilation of your project. When this file exists, Pip will make a +new virtual environment, download just the packages listed here in +``requires=``, and build a wheel (binary Python package). It will then throw +away the environment, and install your wheel. Your ``pyproject.toml`` file will likely look something like this: .. code-block:: toml [build-system] - requires = ["setuptools>=42", "pybind11>=2.6.1"] + requires = ["setuptools", "pybind11"] build-backend = "setuptools.build_meta" -.. note:: - - The main drawback to this method is that a `PEP 517`_ compliant build tool, - such as Pip 10+, is required for this approach to work; older versions of - Pip completely ignore this file. If you distribute binaries (called wheels - in Python) using something like `cibuildwheel`_, remember that ``setup.py`` - and ``pyproject.toml`` are not even contained in the wheel, so this high - Pip requirement is only for source builds, and will not affect users of - your binary wheels. If you are building SDists and wheels, then - `pypa-build`_ is the recommended official tool. - .. _PEP 517: https://www.python.org/dev/peps/pep-0517/ -.. _cibuildwheel: https://cibuildwheel.readthedocs.io -.. _pypa-build: https://pypa-build.readthedocs.io/en/latest/ - -.. _setup_helpers-setup_requires: - -Classic ``setup_requires`` --------------------------- - -If you want to support old versions of Pip with the classic -``setup_requires=["pybind11"]`` keyword argument to setup, which triggers a -two-phase ``setup.py`` run, then you will need to use something like this to -ensure the first pass works (which has not yet installed the ``setup_requires`` -packages, since it can't install something it does not know about): - -.. code-block:: python - - try: - from pybind11.setup_helpers import Pybind11Extension - except ImportError: - from setuptools import Extension as Pybind11Extension - - -It doesn't matter that the Extension class is not the enhanced subclass for the -first pass run; and the second pass will have the ``setup_requires`` -requirements. - -This is obviously more of a hack than the PEP 518 method, but it supports -ancient versions of Pip. +.. _cibuildwheel: https://cibuildwheel.pypa.io +.. _pypa-build: https://build.pypa.io/en/latest/ +.. _check-manifest: https://pypi.io/project/check-manifest +.. _check-sdist: https://pypi.io/project/check-sdist .. _setup_helpers-copy-manually: @@ -231,32 +310,22 @@ the C++ source file. Python is then able to find the module and load it. .. [cppimport] https://github.com/tbenthompson/cppimport + + .. _cmake: Building with CMake =================== For C++ codebases that have an existing CMake-based build system, a Python -extension module can be created with just a few lines of code: +extension module can be created with just a few lines of code, as seen above in +the module section. Pybind11 currently supports a lower minimum if you don't +use the modern FindPython, though be aware that CMake 3.27 removed the old +mechanism, so pybind11 will automatically switch if the old mechanism is not +available. Please opt into the new mechanism if at all possible. Our default +may change in future versions. This is the minimum required: -.. code-block:: cmake - cmake_minimum_required(VERSION 3.5...3.29) - project(example LANGUAGES CXX) - - add_subdirectory(pybind11) - pybind11_add_module(example example.cpp) - -This assumes that the pybind11 repository is located in a subdirectory named -:file:`pybind11` and that the code is located in a file named :file:`example.cpp`. -The CMake command ``add_subdirectory`` will import the pybind11 project which -provides the ``pybind11_add_module`` function. It will take care of all the -details needed to build a Python extension module on any platform. - -A working sample project, including a way to invoke CMake from :file:`setup.py` for -PyPI integration, can be found in the [cmake_example]_ repository. - -.. [cmake_example] https://github.com/pybind/cmake_example .. versionchanged:: 2.6 CMake 3.4+ is required. @@ -264,6 +333,7 @@ PyPI integration, can be found in the [cmake_example]_ repository. .. versionchanged:: 2.11 CMake 3.5+ is required. + Further information can be found at :doc:`cmake/index`. pybind11_add_module @@ -616,6 +686,13 @@ Building with Bazel You can build with the Bazel build system using the `pybind11_bazel `_ repository. +Building with Meson +=================== + +You can use Meson, which has support for ``pybind11`` as a dependency (internally +relying on our ``pkg-config`` support). See the :ref:`module example above `. + + Generating binding code automatically ===================================== diff --git a/docs/requirements.txt b/docs/requirements.txt index 36be9b781..293db6a06 100644 --- a/docs/requirements.txt +++ b/docs/requirements.txt @@ -269,7 +269,7 @@ sphinxcontrib-svg2pdfconverter==1.2.2 \ --hash=sha256:04ec767b55780a6b18d89cc1a8ada6d900c6efde9d1683abdb98a49b144465ca \ --hash=sha256:80a55ca61f70eae93efc65f3814f2f177c86ba55934a9f6c5022f1778b62146b # via -r requirements.in -urllib3==2.2.1 \ - --hash=sha256:450b20ec296a467077128bff42b73080516e71b56ff59a60a02bef2232c4fa9d \ - --hash=sha256:d0570876c61ab9e520d776c38acbbb5b05a776d3f9ff98a5c8fd5162a444cf19 +urllib3==2.2.2 \ + --hash=sha256:a448b2f64d686155468037e1ace9f2d2199776e17f0a46610480d311f73e3472 \ + --hash=sha256:dd505485549a7a552833da5e6063639d0d177c04f23bc3864e41e5dc5f612168 # via requests diff --git a/include/pybind11/detail/class.h b/include/pybind11/detail/class.h index 8308ba710..2b06fd5fe 100644 --- a/include/pybind11/detail/class.h +++ b/include/pybind11/detail/class.h @@ -205,39 +205,40 @@ extern "C" inline PyObject *pybind11_meta_call(PyObject *type, PyObject *args, P /// Cleanup the type-info for a pybind11-registered type. extern "C" inline void pybind11_meta_dealloc(PyObject *obj) { - auto *type = (PyTypeObject *) obj; - auto &internals = get_internals(); + with_internals([obj](internals &internals) { + auto *type = (PyTypeObject *) obj; - // A pybind11-registered type will: - // 1) be found in internals.registered_types_py - // 2) have exactly one associated `detail::type_info` - auto found_type = internals.registered_types_py.find(type); - if (found_type != internals.registered_types_py.end() && found_type->second.size() == 1 - && found_type->second[0]->type == type) { + // A pybind11-registered type will: + // 1) be found in internals.registered_types_py + // 2) have exactly one associated `detail::type_info` + auto found_type = internals.registered_types_py.find(type); + if (found_type != internals.registered_types_py.end() && found_type->second.size() == 1 + && found_type->second[0]->type == type) { - auto *tinfo = found_type->second[0]; - auto tindex = std::type_index(*tinfo->cpptype); - internals.direct_conversions.erase(tindex); + auto *tinfo = found_type->second[0]; + auto tindex = std::type_index(*tinfo->cpptype); + internals.direct_conversions.erase(tindex); - if (tinfo->module_local) { - get_local_internals().registered_types_cpp.erase(tindex); - } else { - internals.registered_types_cpp.erase(tindex); - } - internals.registered_types_py.erase(tinfo->type); - - // Actually just `std::erase_if`, but that's only available in C++20 - auto &cache = internals.inactive_override_cache; - for (auto it = cache.begin(), last = cache.end(); it != last;) { - if (it->first == (PyObject *) tinfo->type) { - it = cache.erase(it); + if (tinfo->module_local) { + get_local_internals().registered_types_cpp.erase(tindex); } else { - ++it; + internals.registered_types_cpp.erase(tindex); } - } + internals.registered_types_py.erase(tinfo->type); - delete tinfo; - } + // Actually just `std::erase_if`, but that's only available in C++20 + auto &cache = internals.inactive_override_cache; + for (auto it = cache.begin(), last = cache.end(); it != last;) { + if (it->first == (PyObject *) tinfo->type) { + it = cache.erase(it); + } else { + ++it; + } + } + + delete tinfo; + } + }); PyType_Type.tp_dealloc(obj); } @@ -310,19 +311,20 @@ inline void traverse_offset_bases(void *valueptr, } inline bool register_instance_impl(void *ptr, instance *self) { - get_internals().registered_instances.emplace(ptr, self); + with_instance_map(ptr, [&](instance_map &instances) { instances.emplace(ptr, self); }); return true; // unused, but gives the same signature as the deregister func } inline bool deregister_instance_impl(void *ptr, instance *self) { - auto ®istered_instances = get_internals().registered_instances; - auto range = registered_instances.equal_range(ptr); - for (auto it = range.first; it != range.second; ++it) { - if (self == it->second) { - registered_instances.erase(it); - return true; + return with_instance_map(ptr, [&](instance_map &instances) { + auto range = instances.equal_range(ptr); + for (auto it = range.first; it != range.second; ++it) { + if (self == it->second) { + instances.erase(it); + return true; + } } - } - return false; + return false; + }); } inline void register_instance(instance *self, void *valptr, const type_info *tinfo) { @@ -377,27 +379,32 @@ extern "C" inline int pybind11_object_init(PyObject *self, PyObject *, PyObject } inline void add_patient(PyObject *nurse, PyObject *patient) { - auto &internals = get_internals(); auto *instance = reinterpret_cast(nurse); instance->has_patients = true; Py_INCREF(patient); - internals.patients[nurse].push_back(patient); + + with_internals([&](internals &internals) { internals.patients[nurse].push_back(patient); }); } inline void clear_patients(PyObject *self) { auto *instance = reinterpret_cast(self); - auto &internals = get_internals(); - auto pos = internals.patients.find(self); + std::vector patients; - if (pos == internals.patients.end()) { - pybind11_fail("FATAL: Internal consistency check failed: Invalid clear_patients() call."); - } + with_internals([&](internals &internals) { + auto pos = internals.patients.find(self); + + if (pos == internals.patients.end()) { + pybind11_fail( + "FATAL: Internal consistency check failed: Invalid clear_patients() call."); + } + + // Clearing the patients can cause more Python code to run, which + // can invalidate the iterator. Extract the vector of patients + // from the unordered_map first. + patients = std::move(pos->second); + internals.patients.erase(pos); + }); - // Clearing the patients can cause more Python code to run, which - // can invalidate the iterator. Extract the vector of patients - // from the unordered_map first. - auto patients = std::move(pos->second); - internals.patients.erase(pos); instance->has_patients = false; for (PyObject *&patient : patients) { Py_CLEAR(patient); @@ -664,10 +671,13 @@ inline PyObject *make_new_python_type(const type_record &rec) { char *tp_doc = nullptr; if (rec.doc && options::show_user_defined_docstrings()) { - /* Allocate memory for docstring (using PyObject_MALLOC, since - Python will free this later on) */ + /* Allocate memory for docstring (Python will free this later on) */ size_t size = std::strlen(rec.doc) + 1; +#if PY_VERSION_HEX >= 0x030D0000 + tp_doc = (char *) PyMem_MALLOC(size); +#else tp_doc = (char *) PyObject_MALLOC(size); +#endif std::memcpy((void *) tp_doc, rec.doc, size); } diff --git a/include/pybind11/detail/common.h b/include/pybind11/detail/common.h index 22416b036..f0f3c50d3 100644 --- a/include/pybind11/detail/common.h +++ b/include/pybind11/detail/common.h @@ -464,7 +464,7 @@ PYBIND11_WARNING_POP }); } \endrst */ -#define PYBIND11_MODULE(name, variable) \ +#define PYBIND11_MODULE(name, variable, ...) \ static ::pybind11::module_::module_def PYBIND11_CONCAT(pybind11_module_def_, name) \ PYBIND11_MAYBE_UNUSED; \ PYBIND11_MAYBE_UNUSED \ @@ -473,7 +473,10 @@ PYBIND11_WARNING_POP PYBIND11_CHECK_PYTHON_VERSION \ PYBIND11_ENSURE_INTERNALS_READY \ auto m = ::pybind11::module_::create_extension_module( \ - PYBIND11_TOSTRING(name), nullptr, &PYBIND11_CONCAT(pybind11_module_def_, name)); \ + PYBIND11_TOSTRING(name), \ + nullptr, \ + &PYBIND11_CONCAT(pybind11_module_def_, name), \ + ##__VA_ARGS__); \ try { \ PYBIND11_CONCAT(pybind11_init_, name)(m); \ return m.ptr(); \ diff --git a/include/pybind11/detail/internals.h b/include/pybind11/detail/internals.h index 6af856a77..c7580728f 100644 --- a/include/pybind11/detail/internals.h +++ b/include/pybind11/detail/internals.h @@ -19,6 +19,8 @@ #include "smart_holder_sfinae_hooks_only.h" #include +#include +#include /// Tracks the `internals` and `type_info` ABI version independent of the main library version. /// @@ -169,15 +171,37 @@ struct override_hash { } }; +using instance_map = std::unordered_multimap; + +// ignore: structure was padded due to alignment specifier +PYBIND11_WARNING_PUSH +PYBIND11_WARNING_DISABLE_MSVC(4324) + +// Instance map shards are used to reduce mutex contention in free-threaded Python. +struct alignas(64) instance_map_shard { + std::mutex mutex; + instance_map registered_instances; +}; + +PYBIND11_WARNING_POP + /// Internal data structure used to track registered instances and types. /// Whenever binary incompatible changes are made to this structure, /// `PYBIND11_INTERNALS_VERSION` must be incremented. struct internals { +#ifdef Py_GIL_DISABLED + std::mutex mutex; +#endif // std::type_index -> pybind11's type information type_map registered_types_cpp; // PyTypeObject* -> base type_info(s) std::unordered_map> registered_types_py; - std::unordered_multimap registered_instances; // void * -> instance* +#ifdef Py_GIL_DISABLED + std::unique_ptr instance_shards; // void * -> instance* + size_t instance_shards_mask; +#else + instance_map registered_instances; // void * -> instance* +#endif std::unordered_set, override_hash> inactive_override_cache; type_map> direct_conversions; @@ -473,7 +497,8 @@ inline object get_python_state_dict() { } inline object get_internals_obj_from_state_dict(handle state_dict) { - return reinterpret_borrow(dict_getitemstring(state_dict.ptr(), PYBIND11_INTERNALS_ID)); + return reinterpret_steal( + dict_getitemstringref(state_dict.ptr(), PYBIND11_INTERNALS_ID)); } inline internals **get_internals_pp_from_capsule(handle obj) { @@ -485,6 +510,20 @@ inline internals **get_internals_pp_from_capsule(handle obj) { return static_cast(raw_ptr); } +inline uint64_t round_up_to_next_pow2(uint64_t x) { + // Round-up to the next power of two. + // See https://graphics.stanford.edu/~seander/bithacks.html#RoundUpPowerOf2 + x--; + x |= (x >> 1); + x |= (x >> 2); + x |= (x >> 4); + x |= (x >> 8); + x |= (x >> 16); + x |= (x >> 32); + x++; + return x; +} + /// Return a reference to the current `internals` data PYBIND11_NOINLINE internals &get_internals() { auto **&internals_pp = get_internals_pp(); @@ -553,6 +592,16 @@ PYBIND11_NOINLINE internals &get_internals() { internals_ptr->static_property_type = make_static_property_type(); internals_ptr->default_metaclass = make_default_metaclass(); internals_ptr->instance_base = make_object_base_type(internals_ptr->default_metaclass); +#ifdef Py_GIL_DISABLED + // Scale proportional to the number of cores. 2x is a heuristic to reduce contention. + auto num_shards + = static_cast(round_up_to_next_pow2(2 * std::thread::hardware_concurrency())); + if (num_shards == 0) { + num_shards = 1; + } + internals_ptr->instance_shards.reset(new instance_map_shard[num_shards]); + internals_ptr->instance_shards_mask = num_shards - 1; +#endif // Py_GIL_DISABLED } return **internals_pp; } @@ -613,13 +662,80 @@ inline local_internals &get_local_internals() { return *locals; } +#ifdef Py_GIL_DISABLED +# define PYBIND11_LOCK_INTERNALS(internals) std::unique_lock lock((internals).mutex) +#else +# define PYBIND11_LOCK_INTERNALS(internals) +#endif + +template +inline auto with_internals(const F &cb) -> decltype(cb(get_internals())) { + auto &internals = get_internals(); + PYBIND11_LOCK_INTERNALS(internals); + return cb(internals); +} + +inline std::uint64_t mix64(std::uint64_t z) { + // David Stafford's variant 13 of the MurmurHash3 finalizer popularized + // by the SplitMix PRNG. + // https://zimbry.blogspot.com/2011/09/better-bit-mixing-improving-on.html + z = (z ^ (z >> 30)) * 0xbf58476d1ce4e5b9; + z = (z ^ (z >> 27)) * 0x94d049bb133111eb; + return z ^ (z >> 31); +} + +template +inline auto with_instance_map(const void *ptr, + const F &cb) -> decltype(cb(std::declval())) { + auto &internals = get_internals(); + +#ifdef Py_GIL_DISABLED + // Hash address to compute shard, but ignore low bits. We'd like allocations + // from the same thread/core to map to the same shard and allocations from + // other threads/cores to map to other shards. Using the high bits is a good + // heuristic because memory allocators often have a per-thread + // arena/superblock/segment from which smaller allocations are served. + auto addr = reinterpret_cast(ptr); + auto hash = mix64(static_cast(addr >> 20)); + auto idx = static_cast(hash & internals.instance_shards_mask); + + auto &shard = internals.instance_shards[idx]; + std::unique_lock lock(shard.mutex); + return cb(shard.registered_instances); +#else + (void) ptr; + return cb(internals.registered_instances); +#endif +} + +// Returns the number of registered instances for testing purposes. The result may not be +// consistent if other threads are registering or unregistering instances concurrently. +inline size_t num_registered_instances() { + auto &internals = get_internals(); +#ifdef Py_GIL_DISABLED + size_t count = 0; + for (size_t i = 0; i <= internals.instance_shards_mask; ++i) { + auto &shard = internals.instance_shards[i]; + std::unique_lock lock(shard.mutex); + count += shard.registered_instances.size(); + } + return count; +#else + return internals.registered_instances.size(); +#endif +} + /// Constructs a std::string with the given arguments, stores it in `internals`, and returns its /// `c_str()`. Such strings objects have a long storage duration -- the internal strings are only /// cleared when the program exits or after interpreter shutdown (when embedding), and so are /// suitable for c-style strings needed by Python internals (such as PyTypeObject's tp_name). template const char *c_str(Args &&...args) { - auto &strings = get_internals().static_strings; + // GCC 4.8 doesn't like parameter unpack within lambda capture, so use + // PYBIND11_LOCK_INTERNALS. + auto &internals = get_internals(); + PYBIND11_LOCK_INTERNALS(internals); + auto &strings = internals.static_strings; strings.emplace_front(std::forward(args)...); return strings.front().c_str(); } @@ -649,15 +765,18 @@ PYBIND11_NAMESPACE_END(detail) /// pybind11 version) running in the current interpreter. Names starting with underscores /// are reserved for internal usage. Returns `nullptr` if no matching entry was found. PYBIND11_NOINLINE void *get_shared_data(const std::string &name) { - auto &internals = detail::get_internals(); - auto it = internals.shared_data.find(name); - return it != internals.shared_data.end() ? it->second : nullptr; + return detail::with_internals([&](detail::internals &internals) { + auto it = internals.shared_data.find(name); + return it != internals.shared_data.end() ? it->second : nullptr; + }); } /// Set the shared data that can be later recovered by `get_shared_data()`. PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) { - detail::get_internals().shared_data[name] = data; - return data; + return detail::with_internals([&](detail::internals &internals) { + internals.shared_data[name] = data; + return data; + }); } /// Returns a typed reference to a shared data entry (by using `get_shared_data()`) if @@ -665,14 +784,15 @@ PYBIND11_NOINLINE void *set_shared_data(const std::string &name, void *data) { /// added to the shared data under the given name and a reference to it is returned. template T &get_or_create_shared_data(const std::string &name) { - auto &internals = detail::get_internals(); - auto it = internals.shared_data.find(name); - T *ptr = (T *) (it != internals.shared_data.end() ? it->second : nullptr); - if (!ptr) { - ptr = new T(); - internals.shared_data[name] = ptr; - } - return *ptr; + return *detail::with_internals([&](detail::internals &internals) { + auto it = internals.shared_data.find(name); + T *ptr = (T *) (it != internals.shared_data.end() ? it->second : nullptr); + if (!ptr) { + ptr = new T(); + internals.shared_data[name] = ptr; + } + return ptr; + }); } PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/include/pybind11/detail/type_caster_base.h b/include/pybind11/detail/type_caster_base.h index 037c430a0..832d849bf 100644 --- a/include/pybind11/detail/type_caster_base.h +++ b/include/pybind11/detail/type_caster_base.h @@ -217,12 +217,15 @@ inline detail::type_info *get_local_type_info(const std::type_index &tp) { } inline detail::type_info *get_global_type_info(const std::type_index &tp) { - auto &types = get_internals().registered_types_cpp; - auto it = types.find(tp); - if (it != types.end()) { - return it->second; - } - return nullptr; + return with_internals([&](internals &internals) { + detail::type_info *type_info = nullptr; + auto &types = internals.registered_types_cpp; + auto it = types.find(tp); + if (it != types.end()) { + type_info = it->second; + } + return type_info; + }); } /// Return the type info for a given C++ type; on lookup failure can either throw or return @@ -253,15 +256,17 @@ PYBIND11_NOINLINE handle get_type_handle(const std::type_info &tp, bool throw_if // Searches the inheritance graph for a registered Python instance, using all_type_info(). PYBIND11_NOINLINE handle find_registered_python_instance(void *src, const detail::type_info *tinfo) { - auto it_instances = get_internals().registered_instances.equal_range(src); - for (auto it_i = it_instances.first; it_i != it_instances.second; ++it_i) { - for (auto *instance_type : detail::all_type_info(Py_TYPE(it_i->second))) { - if (instance_type && same_type(*instance_type->cpptype, *tinfo->cpptype)) { - return handle((PyObject *) it_i->second).inc_ref(); + return with_instance_map(src, [&](instance_map &instances) { + auto it_instances = instances.equal_range(src); + for (auto it_i = it_instances.first; it_i != it_instances.second; ++it_i) { + for (auto *instance_type : detail::all_type_info(Py_TYPE(it_i->second))) { + if (instance_type && same_type(*instance_type->cpptype, *tinfo->cpptype)) { + return handle((PyObject *) it_i->second).inc_ref(); + } } } - } - return handle(); + return handle(); + }); } struct value_and_holder { @@ -506,16 +511,17 @@ PYBIND11_NOINLINE bool isinstance_generic(handle obj, const std::type_info &tp) } PYBIND11_NOINLINE handle get_object_handle(const void *ptr, const detail::type_info *type) { - auto &instances = get_internals().registered_instances; - auto range = instances.equal_range(ptr); - for (auto it = range.first; it != range.second; ++it) { - for (const auto &vh : values_and_holders(it->second)) { - if (vh.type == type) { - return handle((PyObject *) it->second); + return with_instance_map(ptr, [&](instance_map &instances) { + auto range = instances.equal_range(ptr); + for (auto it = range.first; it != range.second; ++it) { + for (const auto &vh : values_and_holders(it->second)) { + if (vh.type == type) { + return handle((PyObject *) it->second); + } } } - } - return handle(); + return handle(); + }); } inline PyThreadState *get_thread_state_unchecked() { diff --git a/include/pybind11/numpy.h b/include/pybind11/numpy.h index 03abc8e77..05ef3918b 100644 --- a/include/pybind11/numpy.h +++ b/include/pybind11/numpy.h @@ -1553,7 +1553,9 @@ PYBIND11_NOINLINE void register_structured_dtype(any_container auto tindex = std::type_index(tinfo); numpy_internals.registered_dtypes[tindex] = {dtype_ptr, std::move(format_str)}; - get_internals().direct_conversions[tindex].push_back(direct_converter); + with_internals([tindex, &direct_converter](internals &internals) { + internals.direct_conversions[tindex].push_back(direct_converter); + }); } template diff --git a/include/pybind11/pybind11.h b/include/pybind11/pybind11.h index 11c69d547..b7c0e2810 100644 --- a/include/pybind11/pybind11.h +++ b/include/pybind11/pybind11.h @@ -1055,13 +1055,20 @@ protected: - delegate translation to the next translator by throwing a new type of exception. */ - auto &local_exception_translators - = get_local_internals().registered_exception_translators; - if (detail::apply_exception_translators(local_exception_translators)) { - return nullptr; - } - auto &exception_translators = get_internals().registered_exception_translators; - if (detail::apply_exception_translators(exception_translators)) { + bool handled = with_internals([&](internals &internals) { + auto &local_exception_translators + = get_local_internals().registered_exception_translators; + if (detail::apply_exception_translators(local_exception_translators)) { + return true; + } + auto &exception_translators = internals.registered_exception_translators; + if (detail::apply_exception_translators(exception_translators)) { + return true; + } + return false; + }); + + if (handled) { return nullptr; } @@ -1200,6 +1207,16 @@ struct handle_type_name { PYBIND11_NAMESPACE_END(detail) +// Use to activate Py_MOD_GIL_NOT_USED. +class mod_gil_not_used { +public: + explicit mod_gil_not_used(bool flag = true) : flag_(flag) {} + bool flag() const { return flag_; } + +private: + bool flag_; +}; + /// Wrapper for Python extension modules class module_ : public object { public: @@ -1300,7 +1317,11 @@ public: ``def`` should point to a statically allocated module_def. \endrst */ - static module_ create_extension_module(const char *name, const char *doc, module_def *def) { + static module_ create_extension_module(const char *name, + const char *doc, + module_def *def, + mod_gil_not_used gil_not_used + = mod_gil_not_used(false)) { // module_def is PyModuleDef // Placement new (not an allocation). def = new (def) @@ -1320,6 +1341,11 @@ public: } pybind11_fail("Internal error in module_::create_extension_module()"); } + if (gil_not_used.flag()) { +#ifdef Py_GIL_DISABLED + PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED); +#endif + } // TODO: Should be reinterpret_steal for Python 3, but Python also steals it again when // returned from PyInit_... // For Python 2, reinterpret_borrow was correct. @@ -1399,15 +1425,16 @@ protected: tinfo->default_holder = rec.default_holder; tinfo->module_local = rec.module_local; - auto &internals = get_internals(); - auto tindex = std::type_index(*rec.type); - tinfo->direct_conversions = &internals.direct_conversions[tindex]; - if (rec.module_local) { - get_local_internals().registered_types_cpp[tindex] = tinfo; - } else { - internals.registered_types_cpp[tindex] = tinfo; - } - internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo}; + with_internals([&](internals &internals) { + auto tindex = std::type_index(*rec.type); + tinfo->direct_conversions = &internals.direct_conversions[tindex]; + if (rec.module_local) { + get_local_internals().registered_types_cpp[tindex] = tinfo; + } else { + internals.registered_types_cpp[tindex] = tinfo; + } + internals.registered_types_py[(PyTypeObject *) m_ptr] = {tinfo}; + }); if (rec.bases.size() > 1 || rec.multiple_inheritance) { mark_parents_nonsimple(tinfo->type); @@ -1853,10 +1880,12 @@ public: generic_type_initialize(record); if (has_alias) { - auto &instances = record.module_local ? get_local_internals().registered_types_cpp - : get_internals().registered_types_cpp; - instances[std::type_index(typeid(type_alias))] - = instances[std::type_index(typeid(type))]; + with_internals([&](internals &internals) { + auto &instances = record.module_local ? get_local_internals().registered_types_cpp + : internals.registered_types_cpp; + instances[std::type_index(typeid(type_alias))] + = instances[std::type_index(typeid(type))]; + }); } } @@ -2596,28 +2625,32 @@ keep_alive_impl(size_t Nurse, size_t Patient, function_call &call, handle ret) { inline std::pair all_type_info_get_cache(PyTypeObject *type) { - auto res = get_internals() - .registered_types_py + auto res = with_internals([type](internals &internals) { + return internals + .registered_types_py #ifdef __cpp_lib_unordered_map_try_emplace - .try_emplace(type); + .try_emplace(type); #else - .emplace(type, std::vector()); + .emplace(type, std::vector()); #endif + }); if (res.second) { // New cache entry created; set up a weak reference to automatically remove it if the type // gets destroyed: weakref((PyObject *) type, cpp_function([type](handle wr) { - get_internals().registered_types_py.erase(type); + with_internals([type](internals &internals) { + internals.registered_types_py.erase(type); - // TODO consolidate the erasure code in pybind11_meta_dealloc() in class.h - auto &cache = get_internals().inactive_override_cache; - for (auto it = cache.begin(), last = cache.end(); it != last;) { - if (it->first == reinterpret_cast(type)) { - it = cache.erase(it); - } else { - ++it; + // TODO consolidate the erasure code in pybind11_meta_dealloc() in class.h + auto &cache = internals.inactive_override_cache; + for (auto it = cache.begin(), last = cache.end(); it != last;) { + if (it->first == reinterpret_cast(type)) { + it = cache.erase(it); + } else { + ++it; + } } - } + }); wr.dec_ref(); })) @@ -2822,7 +2855,11 @@ void implicitly_convertible() { ~set_flag() { flag = false; } }; auto implicit_caster = [](PyObject *obj, PyTypeObject *type) -> PyObject * { +#ifdef Py_GIL_DISABLED + thread_local bool currently_used = false; +#else static bool currently_used = false; +#endif if (currently_used) { // implicit conversions are non-reentrant return nullptr; } @@ -2847,8 +2884,10 @@ void implicitly_convertible() { } inline void register_exception_translator(ExceptionTranslator &&translator) { - detail::get_internals().registered_exception_translators.push_front( - std::forward(translator)); + detail::with_internals([&](detail::internals &internals) { + internals.registered_exception_translators.push_front( + std::forward(translator)); + }); } /** @@ -2858,8 +2897,11 @@ inline void register_exception_translator(ExceptionTranslator &&translator) { * the exception. */ inline void register_local_exception_translator(ExceptionTranslator &&translator) { - detail::get_local_internals().registered_exception_translators.push_front( - std::forward(translator)); + detail::with_internals([&](detail::internals &internals) { + (void) internals; + detail::get_local_internals().registered_exception_translators.push_front( + std::forward(translator)); + }); } /** @@ -3016,14 +3058,19 @@ get_type_override(const void *this_ptr, const type_info *this_type, const char * /* Cache functions that aren't overridden in Python to avoid many costly Python dictionary lookups below */ - auto &cache = get_internals().inactive_override_cache; - if (cache.find(key) != cache.end()) { + bool not_overridden = with_internals([&key](internals &internals) { + auto &cache = internals.inactive_override_cache; + return cache.find(key) != cache.end(); + }); + if (not_overridden) { return function(); } function override = getattr(self, name, function()); if (override.is_cpp_function()) { - cache.insert(std::move(key)); + with_internals([&](internals &internals) { + internals.inactive_override_cache.insert(std::move(key)); + }); return function(); } diff --git a/include/pybind11/pytypes.h b/include/pybind11/pytypes.h index 3d2d0c2da..f26c307a8 100644 --- a/include/pybind11/pytypes.h +++ b/include/pybind11/pytypes.h @@ -980,6 +980,23 @@ inline PyObject *dict_getitem(PyObject *v, PyObject *key) { return rv; } +inline PyObject *dict_getitemstringref(PyObject *v, const char *key) { +#if PY_VERSION_HEX >= 0x030D0000 + PyObject *rv; + if (PyDict_GetItemStringRef(v, key, &rv) < 0) { + throw error_already_set(); + } + return rv; +#else + PyObject *rv = dict_getitemstring(v, key); + if (rv == nullptr && PyErr_Occurred()) { + throw error_already_set(); + } + Py_XINCREF(rv); + return rv; +#endif +} + // Helper aliases/functions to support implicit casting of values given to python // accessors/methods. When given a pyobject, this simply returns the pyobject as-is; for other C++ // type, the value goes through pybind11::cast(obj) to convert it to an `object`. diff --git a/include/pybind11/typing.h b/include/pybind11/typing.h index bc275fc50..e0d0e45c4 100644 --- a/include/pybind11/typing.h +++ b/include/pybind11/typing.h @@ -63,6 +63,21 @@ class Callable : public function { using function::function; }; +template +class Type : public type { + using type::type; +}; + +template +class Union : public object { + using object::object; +}; + +template +class Optional : public object { + using object::object; +}; + PYBIND11_NAMESPACE_END(typing) PYBIND11_NAMESPACE_BEGIN(detail) @@ -121,5 +136,22 @@ struct handle_type_name> { + const_name("], ") + make_caster::name + const_name("]"); }; +template +struct handle_type_name> { + static constexpr auto name = const_name("type[") + make_caster::name + const_name("]"); +}; + +template +struct handle_type_name> { + static constexpr auto name = const_name("Union[") + + ::pybind11::detail::concat(make_caster::name...) + + const_name("]"); +}; + +template +struct handle_type_name> { + static constexpr auto name = const_name("Optional[") + make_caster::name + const_name("]"); +}; + PYBIND11_NAMESPACE_END(detail) PYBIND11_NAMESPACE_END(PYBIND11_NAMESPACE) diff --git a/tests/cross_module_gil_utils.cpp b/tests/cross_module_gil_utils.cpp index 7c20849dd..39112668a 100644 --- a/tests/cross_module_gil_utils.cpp +++ b/tests/cross_module_gil_utils.cpp @@ -92,6 +92,9 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_gil_utils() { if (m != nullptr) { static_assert(sizeof(&gil_acquire) == sizeof(void *), "Function pointer must have the same size as void*"); +#ifdef Py_GIL_DISABLED + PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED); +#endif ADD_FUNCTION("gil_acquire_funcaddr", gil_acquire) ADD_FUNCTION("gil_multi_acquire_release_funcaddr", gil_multi_acquire_release) ADD_FUNCTION("gil_acquire_inner_custom_funcaddr", diff --git a/tests/cross_module_interleaved_error_already_set.cpp b/tests/cross_module_interleaved_error_already_set.cpp index 3493a7e61..65a4463b8 100644 --- a/tests/cross_module_interleaved_error_already_set.cpp +++ b/tests/cross_module_interleaved_error_already_set.cpp @@ -42,6 +42,9 @@ extern "C" PYBIND11_EXPORT PyObject *PyInit_cross_module_interleaved_error_alrea if (m != nullptr) { static_assert(sizeof(&interleaved_error_already_set) == sizeof(void *), "Function pointer must have the same size as void *"); +#ifdef Py_GIL_DISABLED + PyUnstable_Module_SetGIL(m, Py_MOD_GIL_NOT_USED); +#endif PyModule_AddObject( m, "funcaddr", diff --git a/tests/eigen_tensor_avoid_stl_array.cpp b/tests/eigen_tensor_avoid_stl_array.cpp index eacc9e9bd..5aca66c5e 100644 --- a/tests/eigen_tensor_avoid_stl_array.cpp +++ b/tests/eigen_tensor_avoid_stl_array.cpp @@ -11,4 +11,6 @@ #include "test_eigen_tensor.inl" -PYBIND11_MODULE(eigen_tensor_avoid_stl_array, m) { eigen_tensor_test::test_module(m); } +PYBIND11_MODULE(eigen_tensor_avoid_stl_array, m, pybind11::mod_gil_not_used()) { + eigen_tensor_test::test_module(m); +} diff --git a/tests/env.py b/tests/env.py index 7eea5a3b3..4c60f4bbe 100644 --- a/tests/env.py +++ b/tests/env.py @@ -1,5 +1,6 @@ import platform import sys +import sysconfig import pytest @@ -9,6 +10,7 @@ WIN = sys.platform.startswith("win32") or sys.platform.startswith("cygwin") CPYTHON = platform.python_implementation() == "CPython" PYPY = platform.python_implementation() == "PyPy" +PY_GIL_DISABLED = bool(sysconfig.get_config_var("Py_GIL_DISABLED")) def deprecated_call(): diff --git a/tests/pybind11_cross_module_tests.cpp b/tests/pybind11_cross_module_tests.cpp index ad68e9a54..76f40bfa9 100644 --- a/tests/pybind11_cross_module_tests.cpp +++ b/tests/pybind11_cross_module_tests.cpp @@ -16,7 +16,7 @@ #include #include -PYBIND11_MODULE(pybind11_cross_module_tests, m) { +PYBIND11_MODULE(pybind11_cross_module_tests, m, py::mod_gil_not_used()) { m.doc() = "pybind11 cross-module test module"; // test_local_bindings.py tests: diff --git a/tests/pybind11_tests.cpp b/tests/pybind11_tests.cpp index 15cbf7454..3d2d84e77 100644 --- a/tests/pybind11_tests.cpp +++ b/tests/pybind11_tests.cpp @@ -58,7 +58,7 @@ void bind_ConstructorStats(py::module_ &m) { // registered instances to allow instance cleanup checks (invokes a GC first) .def_static("detail_reg_inst", []() { ConstructorStats::gc(); - return py::detail::get_internals().registered_instances.size(); + return py::detail::num_registered_instances(); }); } @@ -75,7 +75,7 @@ const char *cpp_std() { #endif } -PYBIND11_MODULE(pybind11_tests, m) { +PYBIND11_MODULE(pybind11_tests, m, py::mod_gil_not_used()) { m.doc() = "pybind11 test module"; // Intentionally kept minimal to not create a maintenance chore diff --git a/tests/test_cmake_build/main.cpp b/tests/test_cmake_build/main.cpp index e30f2c4b9..640449c31 100644 --- a/tests/test_cmake_build/main.cpp +++ b/tests/test_cmake_build/main.cpp @@ -1,6 +1,6 @@ #include namespace py = pybind11; -PYBIND11_MODULE(test_cmake_build, m) { +PYBIND11_MODULE(test_cmake_build, m, py::mod_gil_not_used()) { m.def("add", [](int i, int j) { return i + j; }); } diff --git a/tests/test_embed/external_module.cpp b/tests/test_embed/external_module.cpp index 5c482fe06..6564ecbef 100644 --- a/tests/test_embed/external_module.cpp +++ b/tests/test_embed/external_module.cpp @@ -6,7 +6,7 @@ namespace py = pybind11; * modules aren't preserved over a finalize/initialize. */ -PYBIND11_MODULE(external_module, m) { +PYBIND11_MODULE(external_module, m, py::mod_gil_not_used()) { class A { public: explicit A(int value) : v{value} {}; diff --git a/tests/test_pytypes.cpp b/tests/test_pytypes.cpp index f3709d40c..e5a318ad8 100644 --- a/tests/test_pytypes.cpp +++ b/tests/test_pytypes.cpp @@ -844,4 +844,27 @@ TEST_SUBMODULE(pytypes, m) { m.def("annotate_iterator_int", [](const py::typing::Iterator &) {}); m.def("annotate_fn", [](const py::typing::Callable, py::str)> &) {}); + m.def("annotate_type", [](const py::typing::Type &) {}); + + m.def("annotate_union", + [](py::typing::List> l, + py::str a, + py::int_ b, + py::object c) -> py::typing::List> { + l.append(a); + l.append(b); + l.append(c); + return l; + }); + + m.def("union_typing_only", + [](py::typing::List> &l) + -> py::typing::List> { return l; }); + + m.def("annotate_optional", + [](py::list &list) -> py::typing::List> { + list.append(py::str("hi")); + list.append(py::none()); + return list; + }); } diff --git a/tests/test_pytypes.py b/tests/test_pytypes.py index 38edfd999..72dac13b8 100644 --- a/tests/test_pytypes.py +++ b/tests/test_pytypes.py @@ -955,3 +955,28 @@ def test_fn_annotations(doc): doc(m.annotate_fn) == "annotate_fn(arg0: Callable[[list[str], str], int]) -> None" ) + + +def test_type_annotation(doc): + assert doc(m.annotate_type) == "annotate_type(arg0: type[int]) -> None" + + +def test_union_annotations(doc): + assert ( + doc(m.annotate_union) + == "annotate_union(arg0: list[Union[str, int, object]], arg1: str, arg2: int, arg3: object) -> list[Union[str, int, object]]" + ) + + +def test_union_typing_only(doc): + assert ( + doc(m.union_typing_only) + == "union_typing_only(arg0: list[Union[str]]) -> list[Union[int]]" + ) + + +def test_optional_annotations(doc): + assert ( + doc(m.annotate_optional) + == "annotate_optional(arg0: list) -> list[Optional[str]]" + )