Seamless operability between C++11 and Python
Go to file
Ralf W. Grosse-Kunstleve b58436afb6
[v2.12] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5368)
* Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5296)

* `self.__cpp_transporter__()` proof of concept: Enable passing C++ pointers across extensions even if the `PYBIND11_INTERNALS_VERSION`s do not match.

* Include cleanup (mainly to resolve PyPy build failures).

* Fix clang-tidy errors.

* Resolve `error: extra

* factor out platform_abi_id.h from internals.h (no functional changes)

* factor out internals_version.h from internals.h (no functional changes)

* Update CMakeLists.txt, tests/extra_python_package/test_files.py

* Revert "factor out internals_version.h from internals.h (no functional changes)"

This reverts commit 3ccea8cd43.

* Remove internals_version.h from CMakeLists.txt, tests/extra_python_package/test_files.py

* `.__cpp_transporter__()` implementation: compare `pybind11_platform_abi_id`, `cpp_typeid_name`

* Add PremiumTraveler

* Rename test_cpp_transporter_traveler_type.h -> test_cpp_transporter_traveler_types.h

* Expand tests: `PremiumTraveler`, `get_points()`

* Shuffle order of tests (no real changes).

* Move `__cpp_transporter__` lambda to `py::cpp_transporter()` regular function.

* Use `type_caster_generic::load(self)` instead of `cast<T *>(self)`

* Pass `const std::type_info *` via `py::capsule` (instead of `cpp_typeid_name`).

* Make platform_abi_id.h completely stand-alone.

* rename exo_planet.cpp -> exo_planet_pybind11.cpp

* Add exo_planet_c_api.cpp (incomplete).

* Fix silly oversight (wrong filename in `#include`).

* Resolve clang-tidy errors:

```
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:10:18: error: 'wrapGetLuggage' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   10 | static PyObject *wrapGetLuggage(PyObject *, PyObject *) { return PyUnicode_FromString("TODO"); }
      | ~~~~~~           ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:14:20: error: 'ThisMethodDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   14 | static PyMethodDef ThisMethodDef[]
      | ~~~~~~             ^
/__w/pybind11/pybind11/tests/exo_planet_c_api.cpp:17:27: error: 'ThisModuleDef' is a static definition in anonymous namespace; static is redundant here [readability-static-definition-in-anonymous-namespace,-warnings-as-errors]
   17 | static struct PyModuleDef ThisModuleDef = {
      | ~~~~~~                    ^
```

* Implement exo_planet_c_api GetLuggage(), GetPoints()

* Move new code from test_cpp_transporter_traveler_bindings.h to pybind11/detail/type_caster_base.h, under the name `class_dunder_cpp_transporter()`

* Fix oversight.

* Unconditionally add `__cpp_transporter__` method to all `py::class_` objects, but do not include that magic method in docstring signatures.

* Back out pybind11/detail/platform_abi_id.h for now. Maximizing reusability can be handled separately, later.

* Small cleanup.

* Restore and add to `test_call_cpp_transporter_*()`

* Ensure https://github.com/pybind/pybind11/issues/3788 does not bite again.

* `class_dunder_cpp_transporter()`: replace `obj.cast<std::string>()` with `std::string(obj)`

* Add (simple) copyright notices in all newly added files.

* Globally replace cpp_transporter with cpp_conduit

* style: pre-commit fixes

* IWYU fixes

* Rename `class_dunder_cpp_conduit()` -> `cpp_conduit_method()`

* Change `pybind11_platform_abi_id`, `pointer_kind` argument types from `str` to `bytes`.

This avoids the unicode decode/encode roundtrips:

* More robust (no decode/encode errors).

* Minor runtime optimization.

* Systematically rename `cap_cpp_type_info` -> `cpp_type_info_capsule` (no functional changes).

* Systematically replace `cpp_type_info_capsule` `name`: `"const std::type_info *"` -> `typeid(std::type_info).name()` (this IS a functional change).

This provides an extra layer of protection against C++ ABI mismatches:

* The first and most important layer is that the `PYBIND11_PLATFORM_ABI_ID`s must match between extensions.

* The second layer is that the `typeid(std::type_info).name()`s must match between extensions.

* Fix sort order accident in tests/CMakeLists.txt

* Apply suggestions from code review

Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

* style: pre-commit fixes

* refactor: rename to _pybind_conduit_v1_

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>

* Add test_home_planet_wrap_very_lonely_traveler(), test_exo_planet_pybind11_wrap_very_lonely_traveler()

* Resolve clang-tidy errors:

```
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:39:32: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
   10 |     py::class_<LonelyTraveler>(m, "LonelyTraveler");
      |                                ^
      |                                std::move( )
/__w/pybind11/pybind11/tests/test_cpp_conduit_traveler_bindings.h:43:52: error: parameter 'm' is passed by value and only copied once; consider moving it to avoid unnecessary copies [performance-unnecessary-value-param,-warnings-as-errors]
   43 |     py::class_<VeryLonelyTraveler, LonelyTraveler>(m, "VeryLonelyTraveler");
      |                                                    ^
      |                                                    std::move( )
```

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>

* Remove `from __future__ import annotations`

* Update Changelog

* Increment patch version number (v2.12.1)

* Revert "Increment patch version number (v2.12.1)"

This reverts commit 0999c2784b.

* Revert "Update Changelog"

This reverts commit 166ba04703.

---------

Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
2024-09-13 14:51:51 -07:00
.github docs: a few missed changes for 2.12 (#5074) 2024-03-27 19:24:54 -04:00
docs docs: a few missed changes for 2.12 (#5074) 2024-03-27 19:24:54 -04:00
include/pybind11 [v2.12] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5368) 2024-09-13 14:51:51 -07:00
pybind11 chore: prepare 2.12.0 (#5070) 2024-03-27 18:09:06 -04:00
tests [v2.12] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5368) 2024-09-13 14:51:51 -07:00
tools chore: prepare 2.12.0 (#5070) 2024-03-27 18:09:06 -04:00
.appveyor.yml Dropping MSVC 2015 (#3722) 2022-02-14 11:36:22 -08:00
.clang-format Final manual curation in preparation for global clang-formating (#3712) 2022-02-10 11:42:03 -08:00
.clang-tidy chore: use explicit defaulting in pyobject macros (#4017) 2022-07-20 12:02:20 -04:00
.cmake-format.yaml format: apply cmake-format 2020-07-30 20:27:55 -04:00
.codespell-ignore-lines [pre-commit.ci] pre-commit autoupdate (#4151) 2022-08-29 21:59:48 -07:00
.gitattributes style: pylint (#3720) 2022-02-15 17:48:33 -05:00
.gitignore First draft of Eigen::Tensor support (#4201) 2022-10-18 16:54:16 -07:00
.pre-commit-config.yaml ci: Ignore v1 updates for checkout (#5023) 2024-02-13 18:29:42 -05:00
.readthedocs.yml [ci skip] Adopt nanobind config. (#4792) 2023-08-15 07:02:54 -07:00
CMakeLists.txt [v2.12] Backport: Enable type-safe interoperability between different independent Python/C++ bindings systems. (#5368) 2024-09-13 14:51:51 -07:00
LICENSE docs: contrib/issue templates (#2377) 2020-08-17 10:14:23 -04:00
MANIFEST.in Create s Security Policy (#4671) 2023-05-23 10:05:25 -07:00
noxfile.py ci: add more versions of numpy/scipy/pypy (#4714) 2023-11-16 00:52:05 -05:00
pyproject.toml chore: move to ruff-format (#4912) 2023-11-07 23:42:54 -08:00
README.rst chore: prepare 2.12.0 (#5070) 2024-03-27 18:09:06 -04:00
SECURITY.md Create s Security Policy (#4671) 2023-05-23 10:05:25 -07:00
setup.cfg chore: 3.12 + cleanup (#4713) 2023-06-21 13:25:10 -04:00
setup.py chore: 3.12 + cleanup (#4713) 2023-06-21 13:25:10 -04:00

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

.. figure:: https://github.com/pybind/pybind11/raw/master/docs/pybind11-logo.png
   :alt: pybind11 logo

**pybind11 — Seamless operability between C++11 and Python**

|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |GitHub Discussions| |CI| |Build status|

|Repology| |PyPI package| |Conda-forge| |Python Versions|

`Setuptools example <https://github.com/pybind/python_example>`_
• `Scikit-build example <https://github.com/pybind/scikit_build_example>`_
• `CMake example <https://github.com/pybind/cmake_example>`_

.. start


**pybind11** is a lightweight header-only library that exposes C++ types
in Python and vice versa, mainly to create Python bindings of existing
C++ code. Its goals and syntax are similar to the excellent
`Boost.Python <http://www.boost.org/doc/libs/1_58_0/libs/python/doc/>`_
library by David Abrahams: to minimize boilerplate code in traditional
extension modules by inferring type information using compile-time
introspection.

The main issue with Boost.Python—and the reason for creating such a
similar project—is Boost. Boost is an enormously large and complex suite
of utility libraries that works with almost every C++ compiler in
existence. This compatibility has its cost: arcane template tricks and
workarounds are necessary to support the oldest and buggiest of compiler
specimens. Now that C++11-compatible compilers are widely available,
this heavy machinery has become an excessively large and unnecessary
dependency.

Think of this library as a tiny self-contained version of Boost.Python
with everything stripped away that isn't relevant for binding
generation. Without comments, the core header files only require ~4K
lines of code and depend on Python (3.6+, or PyPy) and the C++
standard library. This compact implementation was possible thanks to
some C++11 language features (specifically: tuples, lambda functions and
variadic templates). Since its creation, this library has grown beyond
Boost.Python in many ways, leading to dramatically simpler binding code in many
common situations.

Tutorial and reference documentation is provided at
`pybind11.readthedocs.io <https://pybind11.readthedocs.io/en/latest>`_.
A PDF version of the manual is available
`here <https://pybind11.readthedocs.io/_/downloads/en/latest/pdf/>`_.
And the source code is always available at
`github.com/pybind/pybind11 <https://github.com/pybind/pybind11>`_.


Core features
-------------


pybind11 can map the following core C++ features to Python:

- Functions accepting and returning custom data structures per value,
  reference, or pointer
- Instance methods and static methods
- Overloaded functions
- Instance attributes and static attributes
- Arbitrary exception types
- Enumerations
- Callbacks
- Iterators and ranges
- Custom operators
- Single and multiple inheritance
- STL data structures
- Smart pointers with reference counting like ``std::shared_ptr``
- Internal references with correct reference counting
- C++ classes with virtual (and pure virtual) methods can be extended
  in Python
- Integrated NumPy support (NumPy 2 requires pybind11 2.12+)

Goodies
-------

In addition to the core functionality, pybind11 provides some extra
goodies:

- Python 3.6+, and PyPy3 7.3 are supported with an implementation-agnostic
  interface (pybind11 2.9 was the last version to support Python 2 and 3.5).

- It is possible to bind C++11 lambda functions with captured
  variables. The lambda capture data is stored inside the resulting
  Python function object.

- pybind11 uses C++11 move constructors and move assignment operators
  whenever possible to efficiently transfer custom data types.

- It's easy to expose the internal storage of custom data types through
  Pythons' buffer protocols. This is handy e.g. for fast conversion
  between C++ matrix classes like Eigen and NumPy without expensive
  copy operations.

- pybind11 can automatically vectorize functions so that they are
  transparently applied to all entries of one or more NumPy array
  arguments.

- Python's slice-based access and assignment operations can be
  supported with just a few lines of code.

- Everything is contained in just a few header files; there is no need
  to link against any additional libraries.

- Binaries are generally smaller by a factor of at least 2 compared to
  equivalent bindings generated by Boost.Python. A recent pybind11
  conversion of PyRosetta, an enormous Boost.Python binding project,
  `reported <https://graylab.jhu.edu/Sergey/2016.RosettaCon/PyRosetta-4.pdf>`_
  a binary size reduction of **5.4x** and compile time reduction by
  **5.8x**.

- Function signatures are precomputed at compile time (using
  ``constexpr``), leading to smaller binaries.

- With little extra effort, C++ types can be pickled and unpickled
  similar to regular Python objects.

Supported compilers
-------------------

1. Clang/LLVM 3.3 or newer (for Apple Xcode's clang, this is 5.0.0 or
   newer)
2. GCC 4.8 or newer
3. Microsoft Visual Studio 2017 or newer
4. Intel classic C++ compiler 18 or newer (ICC 20.2 tested in CI)
5. Cygwin/GCC (previously tested on 2.5.1)
6. NVCC (CUDA 11.0 tested in CI)
7. NVIDIA PGI (20.9 tested in CI)

About
-----

This project was created by `Wenzel
Jakob <http://rgl.epfl.ch/people/wjakob>`_. Significant features and/or
improvements to the code were contributed by Jonas Adler, Lori A. Burns,
Sylvain Corlay, Eric Cousineau, Aaron Gokaslan, Ralf Grosse-Kunstleve, Trent Houliston, Axel
Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov, Johan Mabille, Tomasz Miąsko,
Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim
Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart.

We thank Google for a generous financial contribution to the continuous
integration infrastructure used by this project.


Contributing
~~~~~~~~~~~~

See the `contributing
guide <https://github.com/pybind/pybind11/blob/master/.github/CONTRIBUTING.md>`_
for information on building and contributing to pybind11.

License
~~~~~~~

pybind11 is provided under a BSD-style license that can be found in the
`LICENSE <https://github.com/pybind/pybind11/blob/master/LICENSE>`_
file. By using, distributing, or contributing to this project, you agree
to the terms and conditions of this license.

.. |Latest Documentation Status| image:: https://readthedocs.org/projects/pybind11/badge?version=latest
   :target: http://pybind11.readthedocs.org/en/latest
.. |Stable Documentation Status| image:: https://img.shields.io/badge/docs-stable-blue.svg
   :target: http://pybind11.readthedocs.org/en/stable
.. |Gitter chat| image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg
   :target: https://gitter.im/pybind/Lobby
.. |CI| image:: https://github.com/pybind/pybind11/workflows/CI/badge.svg
   :target: https://github.com/pybind/pybind11/actions
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/riaj54pn4h08xy40?svg=true
   :target: https://ci.appveyor.com/project/wjakob/pybind11
.. |PyPI package| image:: https://img.shields.io/pypi/v/pybind11.svg
   :target: https://pypi.org/project/pybind11/
.. |Conda-forge| image:: https://img.shields.io/conda/vn/conda-forge/pybind11.svg
   :target: https://github.com/conda-forge/pybind11-feedstock
.. |Repology| image:: https://repology.org/badge/latest-versions/python:pybind11.svg
   :target: https://repology.org/project/python:pybind11/versions
.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pybind11.svg
   :target: https://pypi.org/project/pybind11/
.. |GitHub Discussions| image:: https://img.shields.io/static/v1?label=Discussions&message=Ask&color=blue&logo=github
   :target: https://github.com/pybind/pybind11/discussions