Seamless operability between C++11 and Python
Go to file
Ralf W. Grosse-Kunstleve c557f9a3ad
[smart_holder] type_caster ODR guard (#4022)
* Insert type_caster_odr_guard<> (an empty struct to start with).

* Add odr_guard_registry() used in type_caster_odr_guard() default constructor.

* Add minimal_real_caster (from PR #3862) to test_async, test_buffers

* VERY MESSY SNAPSHOT of WIP, this was the starting point for cl/454658864, which has more changes on top.

* Restore original test_async, test_buffers from current smart_holder HEAD

* Copy from cl/454991845 snapshot Jun 14, 5:08 PM

* Cleanup of tests. Systematically insert `if (make_caster<T>::translation_unit_local) {`

* Small simplification of odr_guard_impl()

* WIP

* Add PYBIND11_SOURCE_FILE_LINE macro.

* Replace PYBIND11_TYPE_CASTER_UNIQUE_IDENTIFIER with PYBIND11_TYPE_CASTER_SOURCE_FILE_LINE, baked into PYBIND11_TYPE_CASTER macro.

* Add more PYBIND11_DETAIL_TYPE_CASTER_ACCESS_TRANSLATION_UNIT_LOCAL; resolves "unused" warning when compiling test_custom_type_casters.cpp

* load_type fixes & follow-on cleanup

* Strip ./ from source_file_line

* Add new tests to CMakeLists.txt, disable PYBIND11_WERROR

* Replace C++17 syntax. Compiles with Debian clang 13 C++11 mode, but fails to link. Trying GitHub Actions anyway to see if there are any platforms that support https://en.cppreference.com/w/cpp/language/tu_local before C++20. Note that Debian clang 13 C++17 works locally.

* Show C++ version along with ODR VIOLATION DETECTED message.

* Add source_file_line_basename()

* Introduce PYBIND11_TYPE_CASTER_ODR_GUARD_ON (but not set automatically).

* Minor cleanup.

* Set PYBIND11_TYPE_CASTER_ODR_GUARD_ON automatically.

* Resolve clang-tidy error.

* Compatibility with old compilers.

* Fix off-by-one in source_file_line_basename()

* Report PYBIND11_INTERNALS_ID & C++ Version from pytest_configure()

* Restore use of PYBIND11_WERROR

* Move cpp_version_in_use() from cast.h to pybind11_tests.cpp

* define PYBIND11_DETAIL_ODR_GUARD_IMPL_THROW_DISABLED true in test_odr_guard_1,2.cpp

* IWYU cleanup of detail/type_caster_odr_guard.h

* Replace `throw err;` to resolve clang-tidy error.

* Add new header filename to CMakeLists.txt, test_files.py

* Experiment: Try any C++17 compiler.

* Fix ifdef for pragma GCC diagnostic.

* type_caster_odr_guard_impl() cleanup

* Move type_caster_odr_guard to type_caster_odr_guard.h

* Rename test_odr_guard* to test_type_caster_odr_guard*

* Remove comments that are (now) more distracting than helpful.

* Mark tu_local_no_data_always_false operator bool as explicit (clang-tidy). See also: https://stackoverflow.com/questions/39995573/when-can-i-use-explicit-operator-bool-without-a-cast

* New PYBIND11_TYPE_CASTER_ODR_GUARD_STRICT option (current on by default).

* Add test_type_caster_odr_registry_values(), test_type_caster_odr_violation_detected_counter()

* Report UNEXPECTED: test_type_caster_odr_guard_2.cpp prevailed (but do not fail).

* Apply clang-tidy suggestion.

* Attempt to handle valgrind behavior.

* Another attempt to handle valgrind behavior.

* Yet another attempt to handle valgrind behavior.

* Trying a new direction: show compiler info & std for UNEXPECTED: type_caster_odr_violation_detected_count() == 0

* compiler_info MSVC fix. num_violations == 0 condition.

* assert pybind11_tests.compiler_info is not None

* Introduce `make_caster_intrinsic<T>`, to be able to undo the 2 changes from `load_type` to `load_type<T>`. This is to avoid breaking 2 `pybind11::detail::load_type()` calls found in the wild (Google global testing).

One of the breakages in the wild was: 0f0f600767/python/tensorstore/subscript_method.h (L61)

* Add test for stl.h / stl_bind.h mix.

Manually verified that the ODR guard detects the ODR violation:

```
C++ Info: Debian Clang 13.0.1 C++17 __pybind11_internals_v4_clang_libstdcpp_cxxabi1002_sh_def__
=========================================================== test session starts ============================================================
platform linux -- Python 3.9.12, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
...
================================================================= FAILURES =================================================================
_____________________________________________ test_type_caster_odr_violation_detected_counter ______________________________________________

    def test_type_caster_odr_violation_detected_counter():
        ...
        else:
>           assert num_violations == 1
E           assert 2 == 1
E             +2
E             -1

num_violations = 2

test_type_caster_odr_guard_1.py:51: AssertionError
========================================================= short test summary info ==========================================================
FAILED test_type_caster_odr_guard_1.py::test_type_caster_odr_violation_detected_counter - assert 2 == 1
======================================================= 1 failed, 5 passed in 0.08s ========================================================
```

* Eliminate need for `PYBIND11_DETAIL_TYPE_CASTER_ACCESS_TRANSLATION_UNIT_LOCAL` macro.

Copying code first developed by @amauryfa. I tried this at an earlier stage, but by itself this was insufficient. In the meantime I added in the TU-local mechanisms: trying again.

Passes local testing:
```
DISABLED std::system_error: ODR VIOLATION DETECTED: pybind11::detail::type_caster<mrc_ns::type_mrc>: SourceLocation1="/usr/local/google/home/rwgk/forked/pybind11/tests/test_type_caster_odr_guard_1.cpp:18", SourceLocation2="/usr/local/google/home/rwgk/forked/pybind11/tests/test_type_caster_odr_guard_2.cpp:19"
C++ Info: Debian Clang 13.0.1 C++17 __pybind11_internals_v4_clang_libstdcpp_cxxabi1002_sh_def__
=========================================================== test session starts ============================================================
platform linux -- Python 3.9.12, pytest-6.2.3, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /usr/local/google/home/rwgk/forked/pybind11/tests, configfile: pytest.ini
collected 6 items

test_type_caster_odr_guard_1.py::test_type_mrc_to_python PASSED
test_type_caster_odr_guard_1.py::test_type_mrc_from_python PASSED
test_type_caster_odr_guard_1.py::test_type_caster_odr_registry_values PASSED
test_type_caster_odr_guard_1.py::test_type_caster_odr_violation_detected_counter PASSED
test_type_caster_odr_guard_2.py::test_type_mrc_to_python PASSED
test_type_caster_odr_guard_2.py::test_type_mrc_from_python PASSED

============================================================ 6 passed in 0.01s =============================================================
```

* tu_local_descr with src_loc experiment

* clang-tidy suggested fixes

* Use source_file_line_from_sloc in type_caster_odr_guard_registry

* Disable type_caster ODR guard for __INTEL_COMPILER (see comment). Also turn off printf.

* Add missing include (discovered via google-internal testing).

* Work `scr_loc` into `descr`

* Use `TypeCasterType::name.sloc` instead of `source_file_line.sloc`

Manual re-verification:

```
+++ b/tests/test_type_caster_odr_guard_2.cpp
-    // m.def("pass_vector_type_mrc", mrc_ns::pass_vector_type_mrc);
+    m.def("pass_vector_type_mrc", mrc_ns::pass_vector_type_mrc);
```

```
>           assert num_violations == 1
E           assert 2 == 1

num_violations = 2

test_type_caster_odr_guard_1.py:51: AssertionError
```

* Fix small oversight (src_loc::here() -> src_loc{nullptr, 0}).

* Remove PYBIND11_DETAIL_TYPE_CASTER_ACCESS_TRANSLATION_UNIT_LOCAL macro completely.

* Remove PYBIND11_TYPE_CASTER_SOURCE_FILE_LINE macro completely. Some small extra cleanup.

* Minor tweaks looking at the PR with a fresh eye.

* src_loc comments

* Add new test_descr_src_loc & and fix descr.h `concat()` `src_loc` bug discovered while working on the test.

* Some more work on source code comments.

* Fully document the ODR violations in the ODR guard itself and introduce `PYBIND11_TYPE_CASTER_ODR_GUARD_ON_IF_AVAILABLE`

* Update comment (incl. mention of deadsnakes known to not work as intended).

* Use no-destructor idiom for type_caster_odr_guard_registry, as suggested by @laramiel

* Fix clang-tidy error: 'auto reg' can be declared as 'auto *reg' [readability-qualified-auto,-warnings-as-errors]

* WIP

* Revert "WIP" (tu_local_no_data_always_false_base experiment).

This reverts commit 31e8ac562f.

* Change `PYBIND11_TYPE_CASTER_ODR_GUARD_ON` to `PYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD`, based on a suggestion by @rainwoodman

* Improved `#if` determining `PYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD`, based on suggestion by @laramiel

* Make `descr::sloc` `const`, as suggested by @rainwoodman

* Rename macro to `PYBIND11_DETAIL_TYPE_CASTER_ODR_GUARD_IMPL_DEBUG`, as suggested by @laramiel

* Tweak comments some more (add "white hat hacker" analogy).

* Bring back `PYBIND11_CPP17` in determining `PYBIND11_ENABLE_TYPE_CASTER_ODR_GUARD`, to hopefully resolve most if not all of the many CI failures (89 failing, 32 successful: https://github.com/pybind/pybind11/runs/7430295771).

* Try another workaround for `__has_builtin`-related breakages (https://github.com/pybind/pybind11/runs/7430720321).

* Remove `defined(__has_builtin)` and subconditions.

* Update "known to not work" expectation in test and comment.

* `pytest.skip` `num_violations == 0` only `#ifdef __NO_INLINE__` (irrespective of the compiler)

* Systematically change all new `#ifdef` to `#if defined` (review suggestion).

* Bring back MSVC comment that got lost while experimenting.
2022-07-21 06:38:21 -07:00
.github Tracking ci.yml changes from master. 2022-07-09 21:50:08 -07:00
docs Merge branch 'master' into sh_merge_master 2022-07-09 21:47:49 -07:00
include/pybind11 [smart_holder] type_caster ODR guard (#4022) 2022-07-21 06:38:21 -07:00
pybind11 style: pylint (#3720) 2022-02-15 17:48:33 -05:00
tests [smart_holder] type_caster ODR guard (#4022) 2022-07-21 06:38:21 -07:00
tools fix(cmake): support cross-compiles with classic Python (#3959) 2022-05-20 09:38:29 -04:00
ubench Add/update type: ignore for new mypy version. 2022-04-12 16:13:14 -07:00
.appveyor.yml Dropping MSVC 2015 (#3722) 2022-02-14 11:36:22 -08:00
.clang-format Final manual curation in preparation for global clang-formating (#3712) 2022-02-10 11:42:03 -08:00
.clang-tidy enable two new clang-tidy checks (#3988) 2022-06-06 12:15:45 -04:00
.cmake-format.yaml format: apply cmake-format 2020-07-30 20:27:55 -04:00
.codespell-ignorelines [smart_holder] .def_readonly, .def_readwrite adaptors (continuation of PR #3581). (#3844) 2022-05-17 16:35:43 -07:00
.gitattributes style: pylint (#3720) 2022-02-15 17:48:33 -05:00
.gitignore style: pre-commit cleanup (#3111) 2021-07-14 16:49:13 -04:00
.pre-commit-config.yaml Merge branch 'master' into sh_merge_master 2022-07-12 18:37:15 -07:00
.readthedocs.yml Fix readthedocs build (#721) 2017-03-12 22:36:48 +01:00
CMakeLists.txt [smart_holder] type_caster ODR guard (#4022) 2022-07-21 06:38:21 -07:00
LICENSE docs: contrib/issue templates (#2377) 2020-08-17 10:14:23 -04:00
MANIFEST.in Merge branch 'master' into sh_merge_master_after_clang-format_etc 2022-02-14 14:36:08 -08:00
noxfile.py ci: fix PyPy (#3768) 2022-03-01 12:42:52 -05:00
pyproject.toml ci: drop win2016 (#3854) 2022-04-11 16:54:33 -04:00
README_smart_holder.rst Fixes based on newly added pre-submit checks. 2021-10-11 13:22:39 -07:00
README.rst Merge branch 'master' into sh_merge_master_after_clang-format_etc 2022-02-14 14:36:08 -08:00
setup.cfg Fix: 3.11 beta support (#3923) 2022-07-06 16:35:12 -04:00
setup.py chore: drop Python 3.5 (#3719) 2022-02-11 19:06:16 -05:00

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

.. figure:: https://github.com/pybind/pybind11/raw/master/docs/pybind11-logo.png
   :alt: pybind11 logo

**pybind11 — Seamless operability between C++11 and Python**

|Latest Documentation Status| |Stable Documentation Status| |Gitter chat| |GitHub Discussions| |CI| |Build status|

|Repology| |PyPI package| |Conda-forge| |Python Versions|

`Setuptools example <https://github.com/pybind/python_example>`_
• `Scikit-build example <https://github.com/pybind/scikit_build_example>`_
• `CMake example <https://github.com/pybind/cmake_example>`_

.. start

.. Note::

   This is the pybind11 **smart_holder** branch. Please refer to
   ``README_smart_holder.rst`` for branch-specific information.

**pybind11** is a lightweight header-only library that exposes C++ types
in Python and vice versa, mainly to create Python bindings of existing
C++ code. Its goals and syntax are similar to the excellent
`Boost.Python <http://www.boost.org/doc/libs/1_58_0/libs/python/doc/>`_
library by David Abrahams: to minimize boilerplate code in traditional
extension modules by inferring type information using compile-time
introspection.

The main issue with Boost.Python—and the reason for creating such a
similar project—is Boost. Boost is an enormously large and complex suite
of utility libraries that works with almost every C++ compiler in
existence. This compatibility has its cost: arcane template tricks and
workarounds are necessary to support the oldest and buggiest of compiler
specimens. Now that C++11-compatible compilers are widely available,
this heavy machinery has become an excessively large and unnecessary
dependency.

Think of this library as a tiny self-contained version of Boost.Python
with everything stripped away that isn't relevant for binding
generation. Without comments, the core header files only require ~4K
lines of code and depend on Python (3.6+, or PyPy) and the C++
standard library. This compact implementation was possible thanks to
some of the new C++11 language features (specifically: tuples, lambda
functions and variadic templates). Since its creation, this library has
grown beyond Boost.Python in many ways, leading to dramatically simpler
binding code in many common situations.

Tutorial and reference documentation is provided at
`pybind11.readthedocs.io <https://pybind11.readthedocs.io/en/latest>`_.
A PDF version of the manual is available
`here <https://pybind11.readthedocs.io/_/downloads/en/latest/pdf/>`_.
And the source code is always available at
`github.com/pybind/pybind11 <https://github.com/pybind/pybind11>`_.


Core features
-------------


pybind11 can map the following core C++ features to Python:

- Functions accepting and returning custom data structures per value,
  reference, or pointer
- Instance methods and static methods
- Overloaded functions
- Instance attributes and static attributes
- Arbitrary exception types
- Enumerations
- Callbacks
- Iterators and ranges
- Custom operators
- Single and multiple inheritance
- STL data structures
- Smart pointers with reference counting like ``std::shared_ptr``
- Internal references with correct reference counting
- C++ classes with virtual (and pure virtual) methods can be extended
  in Python

Goodies
-------

In addition to the core functionality, pybind11 provides some extra
goodies:

- Python 3.6+, and PyPy3 7.3 are supported with an implementation-agnostic
  interface (pybind11 2.9 was the last version to support Python 2 and 3.5).

- It is possible to bind C++11 lambda functions with captured
  variables. The lambda capture data is stored inside the resulting
  Python function object.

- pybind11 uses C++11 move constructors and move assignment operators
  whenever possible to efficiently transfer custom data types.

- It's easy to expose the internal storage of custom data types through
  Pythons' buffer protocols. This is handy e.g. for fast conversion
  between C++ matrix classes like Eigen and NumPy without expensive
  copy operations.

- pybind11 can automatically vectorize functions so that they are
  transparently applied to all entries of one or more NumPy array
  arguments.

- Python's slice-based access and assignment operations can be
  supported with just a few lines of code.

- Everything is contained in just a few header files; there is no need
  to link against any additional libraries.

- Binaries are generally smaller by a factor of at least 2 compared to
  equivalent bindings generated by Boost.Python. A recent pybind11
  conversion of PyRosetta, an enormous Boost.Python binding project,
  `reported <https://graylab.jhu.edu/Sergey/2016.RosettaCon/PyRosetta-4.pdf>`_
  a binary size reduction of **5.4x** and compile time reduction by
  **5.8x**.

- Function signatures are precomputed at compile time (using
  ``constexpr``), leading to smaller binaries.

- With little extra effort, C++ types can be pickled and unpickled
  similar to regular Python objects.

Supported compilers
-------------------

1. Clang/LLVM 3.3 or newer (for Apple Xcode's clang, this is 5.0.0 or
   newer)
2. GCC 4.8 or newer
3. Microsoft Visual Studio 2017 or newer
4. Intel classic C++ compiler 18 or newer (ICC 20.2 tested in CI)
5. Cygwin/GCC (previously tested on 2.5.1)
6. NVCC (CUDA 11.0 tested in CI)
7. NVIDIA PGI (20.9 tested in CI)

About
-----

This project was created by `Wenzel
Jakob <http://rgl.epfl.ch/people/wjakob>`_. Significant features and/or
improvements to the code were contributed by Jonas Adler, Lori A. Burns,
Sylvain Corlay, Eric Cousineau, Aaron Gokaslan, Ralf Grosse-Kunstleve, Trent Houliston, Axel
Huebl, @hulucc, Yannick Jadoul, Sergey Lyskov Johan Mabille, Tomasz Miąsko,
Dean Moldovan, Ben Pritchard, Jason Rhinelander, Boris Schäling, Pim
Schellart, Henry Schreiner, Ivan Smirnov, Boris Staletic, and Patrick Stewart.

We thank Google for a generous financial contribution to the continuous
integration infrastructure used by this project.


Contributing
~~~~~~~~~~~~

See the `contributing
guide <https://github.com/pybind/pybind11/blob/master/.github/CONTRIBUTING.md>`_
for information on building and contributing to pybind11.

License
~~~~~~~

pybind11 is provided under a BSD-style license that can be found in the
`LICENSE <https://github.com/pybind/pybind11/blob/master/LICENSE>`_
file. By using, distributing, or contributing to this project, you agree
to the terms and conditions of this license.

.. |Latest Documentation Status| image:: https://readthedocs.org/projects/pybind11/badge?version=latest
   :target: http://pybind11.readthedocs.org/en/latest
.. |Stable Documentation Status| image:: https://img.shields.io/badge/docs-stable-blue.svg
   :target: http://pybind11.readthedocs.org/en/stable
.. |Gitter chat| image:: https://img.shields.io/gitter/room/gitterHQ/gitter.svg
   :target: https://gitter.im/pybind/Lobby
.. |CI| image:: https://github.com/pybind/pybind11/workflows/CI/badge.svg
   :target: https://github.com/pybind/pybind11/actions
.. |Build status| image:: https://ci.appveyor.com/api/projects/status/riaj54pn4h08xy40?svg=true
   :target: https://ci.appveyor.com/project/wjakob/pybind11
.. |PyPI package| image:: https://img.shields.io/pypi/v/pybind11.svg
   :target: https://pypi.org/project/pybind11/
.. |Conda-forge| image:: https://img.shields.io/conda/vn/conda-forge/pybind11.svg
   :target: https://github.com/conda-forge/pybind11-feedstock
.. |Repology| image:: https://repology.org/badge/latest-versions/python:pybind11.svg
   :target: https://repology.org/project/python:pybind11/versions
.. |Python Versions| image:: https://img.shields.io/pypi/pyversions/pybind11.svg
   :target: https://pypi.org/project/pybind11/
.. |GitHub Discussions| image:: https://img.shields.io/static/v1?label=Discussions&message=Ask&color=blue&logo=github
   :target: https://github.com/pybind/pybind11/discussions