* Support free-threaded CPython (PEP 703)
Some additional locking is added in the free-threaded build when
`Py_GIL_DISABLED` is defined:
- Most accesses to internals are protected by a single mutex
- The registered_instances uses a striped lock to improve concurrency
Pybind11 modules can indicate support for running with the GIL disabled
by calling `set_gil_not_used()`.
* refactor: use PYBIND11_MODULE (#11)
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Address code review
* Suppress MSVC warning
* Changes from review
* style: pre-commit fixes
* `py::mod_gil_not_used()` suggestion.
* Update include/pybind11/pybind11.h
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
* ci: try Python 3.13t
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix: support Python 3.13t
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* fix: patch PyPy
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: one more int cast
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* tests: cleanup
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* refactor: use named constant in tests for immortal refcounts
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* docs: move comment about free threaded Python
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
---------
Signed-off-by: Henry Schreiner <henryschreineriii@gmail.com>
* Introduce `get_python_state_dict()`
* Conditional version bump for Python 3.12+
* Shuffle subexpressions to make the condition easier to understand (no change to logic).
* Make pybind11 ABI version 5 the minimum for Python 3.12+ (as suggested by @Lalaland)
* Add back condition for PYPY_VERSION, but keep it open for future PyPy versions.
* Fall back to simple `|| defined(PYPY_VERSION)`. `PY_VERSION_HEX` does not appear to be meaningful with PyPy.
* Use PyConfig_InitPythonConfig instead of PyConfig_InitIsolatedConfig
* add unit test for default python configuration
---------
Co-authored-by: Daniel Jacobs <daniel.jacobs@is4s.com>
* scoped_interpreter overloaded ctor: PyConfig param
* style: pre-commit fixes
* refact: some logics extracted into funcs (precheck_interpreter, _initialize_interpreter); config_guard
* style: pre-commit fixes
* refact: scoped_config, some funcs hidden in detail ns
* refact: macro PYBIND11_PYCONFIG_SUPPORT_PY_VERSION + undef
* feat: PYBIND11_PYCONFIG_SUPPORT_PY_VERSION set to 3.8
* tests: Custom PyConfig
* ci: python 3.6 -> 3.8
* ci: reverted py 38 back to 36; refact: initialize_interpreter overloads
* style: pre-commit fixes
* fix: readability-implicit-bool-conversion
* refact: each initialize_interpreter overloads in pybind11 ns
* Move `initialize_interpreter_pre_pyconfig()` into the `detail` namespace.
Move the `PYBIND11_PYCONFIG_SUPPORT_PY_VERSION_HEX` define down to where it is used for the first time, and check if it is defined already, so that it is possible to customize from the compilation command line, just in case there is some unforeseen issue for Python 3.8, 3.9, 3.10.
* tests: Add program dir to path, Custom PyConfig with argv
* refact: clang-formatted
* tests: Add-program-dir-to-path covers both scoped_interpreter overloads
* tests: Add-program-dir-to-path fixed
* tests: Add-program-dir-to-path py_version dependant validation
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
* Add option to force the use of the PYPY GIL scoped acquire/release logic to support nested gil access, see https://github.com/pybind/pybind11/issues/1276 and https://github.com/pytorch/pytorch/issues/83101
* Apply suggestions from code review
* Update CMakeLists.txt
* docs: update upgrade guide
* Update docs/upgrade.rst
* All bells & whistles.
* Add Reminder to common.h, so that we will not forget to purge `!WITH_THREAD` branches when dropping Python 3.6
* New sentence instead of semicolon.
* Temporarily pull in snapshot of PR #4246
* Add `test_release_acquire`
* Add more unit tests for nested gil locking
* Add test_report_builtins_internals_keys
* Very minor enhancement: sort list only after filtering.
* Revert change in docs/upgrade.rst
* Add test_multi_acquire_release_cross_module, while also forcing unique PYBIND11_INTERNALS_VERSION for cross_module_gil_utils.cpp
* Hopefully fix apparently new ICC error.
```
2022-10-28T07:57:54.5187728Z -- The CXX compiler identification is Intel 2021.7.0.20220726
...
2022-10-28T07:58:53.6758994Z icpc: remark #10441: The Intel(R) C++ Compiler Classic (ICC) is deprecated and will be removed from product release in the second half of 2023. The Intel(R) oneAPI DPC++/C++ Compiler (ICX) is the recommended compiler moving forward. Please transition to use this compiler. Use '-diag-disable=10441' to disable this message.
2022-10-28T07:58:54.5801597Z In file included from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/type_caster_base.h(15),
2022-10-28T07:58:54.5803794Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../cast.h(15),
2022-10-28T07:58:54.5805740Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/../attr.h(14),
2022-10-28T07:58:54.5809556Z from /home/runner/work/pybind11/pybind11/include/pybind11/detail/class.h(12),
2022-10-28T07:58:54.5812154Z from /home/runner/work/pybind11/pybind11/include/pybind11/pybind11.h(13),
2022-10-28T07:58:54.5948523Z from /home/runner/work/pybind11/pybind11/tests/cross_module_gil_utils.cpp(13):
2022-10-28T07:58:54.5949009Z /home/runner/work/pybind11/pybind11/include/pybind11/detail/../detail/internals.h(177): error #2282: unrecognized GCC pragma
2022-10-28T07:58:54.5949374Z PYBIND11_TLS_KEY_INIT(tstate)
2022-10-28T07:58:54.5949579Z ^
2022-10-28T07:58:54.5949695Z
```
* clang-tidy fixes
* Workaround for PYPY WIN exitcode None
* Revert "Temporarily pull in snapshot of PR #4246"
This reverts commit 23ac16e859150f27fda25ca865cabcb4444e0770.
* Another workaround for PYPY WIN exitcode None
* Clean up how the tests are run "run in process" Part 1: uniformity
* Clean up how the tests are run "run in process" Part 2: use `@pytest.mark.parametrize` and clean up the naming.
* Skip some tests `#if defined(THREAD_SANITIZER)` (tested with TSAN using the Google-internal toolchain).
* Run all tests again but ignore ThreadSanitizer exitcode 66 (this is less likely to mask unrelated ThreadSanitizer issues in the future).
* bug fix: missing common.h include before using `PYBIND11_SIMPLE_GIL_MANAGEMENT`
For the tests in the github CI this does not matter, because
`PYBIND11_SIMPLE_GIL_MANAGEMENT` is always defined from the command line,
but when monkey-patching common.h locally, it matters.
* if process.exitcode is None: assert t_delta > 9.9
* More sophisiticated `_run_in_process()` implementation, clearly reporting `DEADLOCK`, additionally exercised via added `intentional_deadlock()`
* Wrap m.intentional_deadlock in a Python function, for `ForkingPickler` compatibility.
```
> ForkingPickler(file, protocol).dump(obj)
E TypeError: cannot pickle 'PyCapsule' object
```
Observed with all Windows builds including mingw but not PyPy, and macos-latest with Python 3.9, 3.10, 3.11 but not 3.6.
* Add link to potential solution for WOULD-BE-NICE-TO-HAVE feature.
* Add `SKIP_IF_DEADLOCK = True` option, to not pollute the CI results with expected `DEADLOCK` failures while we figure out what to do about them.
* Add COPY-PASTE-THIS: gdb ... command (to be used for debugging the detected deadlock)
* style: pre-commit fixes
* Do better than automatic pre-commit fixes.
* Add `PYBIND11_SIMPLE_GIL_MANAGEMENT` to `pytest_report_header()` (so that we can easily know when harvesting deadlock information from the CI logs).
Co-authored-by: Arnim Balzer <arnim@seechange.ai>
Co-authored-by: Henry Schreiner <HenrySchreinerIII@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Add debug fprintf to test_interpreter.cpp
* Update `sys.path` from `PYTHONPATH` in Python >= 3.11 branch of `initialize_interpreter()`
* Use `config.isolated = 0; config.use_environment = 1;`
As suggsted by @vstinner here: https://github.com/pybind/pybind11/pull/4119#issuecomment-1219442853
* Add `TEST_CASE("PYTHONPATH is used to update sys.path")`
* Fix clang-tidy error.
* Use `_putenv_s()` under Windows.
* Fix clang-tidy error: argument name ... in comment does not match parameter name
* Remove slash from PYTHONPATH addition, to work around Windows slash-vs-backslash issue.
* Use `py::str(...)` instead of `.attr("__str__")` as suggested by @skylion007
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
* Clear local internals after finalizing interpreter
* Add descriptive comments
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* `#error BYE_BYE_GOLDEN_SNAKE`
* Removing everything related to 2.7 from ci.yml
* Commenting-out Centos7
* Removing `PYTHON: 27` from .appveyor.yml
* "PY2" removal, mainly from tests. C++ code is not touched.
* Systematic removal of `u` prefix from `u"..."` and `u'...'` literals. Collateral cleanup of a couple minor other things.
* Cleaning up around case-insensitive hits for `[^a-z]py.*2` in tests/.
* Removing obsolete Python 2 mention in compiling.rst
* Proper `#error` for Python 2.
* Using PY_VERSION_HEX to guard `#error "PYTHON 2 IS NO LONGER SUPPORTED.`
* chore: bump pre-commit
* style: run pre-commit for pyupgrade 3+
* tests: use sys.version_info, not PY
* chore: more Python 2 removal
* Uncommenting Centos7 block (PR #3691 showed that it is working again).
* Update pre-commit hooks
* Fix pre-commit hook
* refactor: remove Python 2 from CMake
* refactor: remove Python 2 from setup code
* refactor: simplify, better static typing
* feat: fail with nice messages
* refactor: drop Python 2 C++ code
* docs: cleanup for Python 3
* revert: intree
revert: intree
* docs: minor touchup to py2 statement
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
* Adding readability-qualified-auto to .clang-tidy
Ported from @henryiii's 287527f705
* fix: support Python < 3.6
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
* Download catch for MinGw
* Fix rest of MinGW
* fix: update catch to 2.13.5 to fix glibc 2.34 failures
Update the downloaded Catch version to 2.13.5, in order to fix build
failure on glibc 2.34:
```
In file included from /usr/include/signal.h:328,
from /tmp/pybind11/.nox/tests-3-9/tmp/tests/catch/catch.hpp:8030,
from /tmp/pybind11/tests/test_embed/catch.cpp:13:
/tmp/pybind11/.nox/tests-3-9/tmp/tests/catch/catch.hpp:10818:58: error: call to non-‘constexpr’ function ‘long int sysconf(int)’
10818 | static constexpr std::size_t sigStackSize = 32768 >= MINSIGSTKSZ ? 32768 : MINSIGSTKSZ;
| ^~~~~~~~~~~
In file included from /usr/include/python3.9/Python.h:36,
from /tmp/pybind11/include/pybind11/detail/common.h:215,
from /tmp/pybind11/include/pybind11/pytypes.h:12,
from /tmp/pybind11/include/pybind11/cast.h:13,
from /tmp/pybind11/include/pybind11/attr.h:13,
from /tmp/pybind11/include/pybind11/pybind11.h:13,
from /tmp/pybind11/include/pybind11/embed.h:12,
from /tmp/pybind11/tests/test_embed/catch.cpp:4:
/usr/include/unistd.h:640:17: note: ‘long int sysconf(int)’ declared here
640 | extern long int sysconf (int __name) __THROW;
| ^~~~~~~
In file included from /tmp/pybind11/tests/test_embed/catch.cpp:13:
/tmp/pybind11/.nox/tests-3-9/tmp/tests/catch/catch.hpp:10877:45: error: size of array ‘altStackMem’ is not an integral constant-expression
10877 | char FatalConditionHandler::altStackMem[sigStackSize] = {};
| ^~~~~~~~~~~~
```
The newest Catch version cannot be used yet because of regression:
https://github.com/catchorg/Catch2/pull/2364
* fix: add option for _ check, only define on pybind11
* Revert "fix: add option for _ check, only define on pybind11"
This reverts commit 86817db488.
* fix: only undef _ for catch cpp creation.
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
* override: Fix wrong caching of the overrides
There was a problem when the python type, which was stored in override
cache for C++ functions, was destroyed and the record wasn't removed from the
override cache. Therefor, dangling pointer was stored there. Then when the
memory was reused and new type was allocated at the given address and the
method with the same name (as previously stored in the cache) was actually
overridden in python, it would wrongly find it in the override cache for C++
functions and therefor override from python wouldn't be called.
The fix is to erase the type from the override cache when the type is destroyed.
* test: Pass by const ref instead of by value (clang-tidy)
* test: Rename classes and move to different files
Rename the classes and files so they're no too generic. Also, better place to
test the stuff is in test_virtual_functions.cpp/.py as we're basically testing
the virtual functions/trampolines.
* Add TODO for erasure code
* [pre-commit.ci] auto fixes from pre-commit.com hooks
for more information, see https://pre-commit.ci
Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Call PySys_SetArgv when initializing interpreter.
* Document argc/argv parameters in initialize_interpreter.
* Remove manual memory management from set_interpreter_argv in favor of smart pointers.
* Use size_t for indexers in set_interpreter_argv.
* Minimize macros for flow control in set_interpreter_argv.
* Fix 'unused variable' warning on Py2
* whitespace
* Define wide_char_arg_deleter outside set_interpreter_argv.
* Do sys.path workaround in C++ rather than eval.
* Factor out wchar conversion to a separate function.
* Restore widened_argv variable declaration.
* Fix undeclared widened_arg variable on some paths.
* Use delete[] to match new wchar_t[].
* Fix compiler errors
* Use PY_VERSION_HEX for a cleaner CVE-2008-5983 mode check.
* Fix typo
* Use explicit type for deleter so delete[] works cross-compiler.
* Always use PySys_SetArgvEx because pybind11 doesn't support pythons that don't include it.
* Remove pointless ternary operator.
* Use unique_ptr.reset instead of a second initialization.
* Rename add_program_dir_to_path parameter to clarify intent.
* Add defined() check before evaluating HAVE_BROKEN_MBSTOWCS.
* Apply clang-tidy fixes
* Pre-commit
* refactor: use const for set_interpreter_argv
* Try to fix const issue and allocate vector properly
* fix: copy strings on Python 2
* Applying clang-format-diff relative to master.
The only manual change is an added empty line between pybind11 and system `#include`s.
```
git diff -U0 --no-color master | python3 $HOME/clone/llvm-project/clang/tools/clang-format/clang-format-diff.py -p1 -style=file -i
```
Co-authored-by: Boris Staletic <boris.staletic@gmail.com>
Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
Co-authored-by: Henry Schreiner <henryschreineriii@gmail.com>
Co-authored-by: Ralf W. Grosse-Kunstleve <rwgk@google.com>
* Add py::raise_from to enable chaining exceptions on Python 3.3+
* Use 'raise from' in initialization
* Documenting the exact base version of _PyErr_FormatVFromCause, adding back `assert`s.
Co-authored-by: Dustin Spicuzza <dustin@virtualroadside.com>
* maint(clang-tidy) Improve code readability
* Fix minor typos
* Revert optimization that removed test case
* Fix comment formatting
* Revert another optimization to repro an issue
* Remove make_unique since it C++14 and newer only
* eformat comments
* Fix unsignedness of comparison
* Update comment
* Wrap PYBIND11_OVERLOAD_NAME and PYBIND11_OVERLOAD_PURE_NAME in do { ... } while (false), and resolve trailing semicolon
* Deprecate PYBIND11_OVERLOAD_* and get_overload in favor of PYBIND11_OVERRIDE_* and get_override
* Correct erroneous usage of 'overload' instead of 'override' in the implementation and internals
* Fix tests to use non-deprecated PYBIND11_OVERRIDE_* macros
* Update docs to use override instead of overload where appropriate, and add warning about deprecated aliases
* Add semicolons to deprecated PYBIND11_OVERLOAD macros to match original behavior
* Remove deprecation of PYBIND11_OVERLOAD_* macros and get_overload
* Add note to changelog and upgrade guide
* tests: keep source dir clean
* ci: make first build inplace
* ci: drop dev setting (wasn't doing anything)
* tests: warn if source directory is dirty
Catch v2 changed the `run(...)` signature to take a `char *argv[]`,
arguing partly that technically a `char *argv[]` type is the correct
`main()` signature rather than `const char *argv[]`.
Dropping the `const` here doesn't appear to cause any problems with
catch v1 (tested against both the cmake-downloaded 1.9.3 and Debian's
1.12.1 package) so we can follow suit.
* Fix segfault when reloading interpreter with external modules
When embedding the interpreter and loading external modules in that
embedded interpreter, the external module correctly shares its
internals_ptr with the one in the embedded interpreter. When the
interpreter is shut down, however, only the `internals_ptr` local to
the embedded code is actually reset to nullptr: the external module
remains set.
The result is that loading an external pybind11 module, letting the
interpreter go through a finalize/initialize, then attempting to use
something in the external module fails because this external module is
still trying to use the old (destroyed) internals. This causes
undefined behaviour (typically a segfault).
This commit fixes it by adding a level of indirection in the internals
path, converting the local internals variable to `internals **` instead
of `internals *`. With this change, we can detect a stale internals
pointer and reload the internals pointer (either from a capsule or by
creating a new internals instance).
(No issue number: this was reported on gitter by @henryiii and @aoloe).
In the latest MSVC in C++17 mode including Eigen causes warnings:
warning C4996: 'std::unary_negate<_Fn>': warning STL4008: std::not1(),
std::not2(), std::unary_negate, and std::binary_negate are deprecated in
C++17. They are superseded by std::not_fn(). You can define
_SILENCE_CXX17_NEGATORS_DEPRECATION_WARNING or
_SILENCE_ALL_CXX17_DEPRECATION_WARNINGS to acknowledge that you have
received this warning.
This disables 4996 for the Eigen includes.
Catch generates a similar warning for std::uncaught_exception, so
disable the warning there, too.
In both cases this is temporary; we can (and should) remove the warnings
disabling once new upstream versions of Eigen and Catch are available
that address the warning. (The Catch one, in particular, looks to be
fixed in upstream master, so will probably be fixed in the next (2.0.2)
release).
- For the debian/buster docker build (GCC 7/C++17) install and use the
system `catch` package; this also renames "COMPILER_PACKAGES" to
"EXTRA_PACKAGES" since it now contains a non-compiler package.
- Add a status message indicating the catch version being used for
compiling the embedded tests
- Simplify some bash code by using VAR+=" foo" to append (rather than
VAR="${VAR} foo"
- Fix CMAKE_INCLUDE_PATH appending: it was prepending the ':' but not
the existing $CMAKE_INCLUDE_PATH value and so would end up with
":/eigen-path" if CMAKE_INCLUDE_PATH was already set. (This wasn't
bug that was actually noticed since currently nothing else sets it).
Fixes a race condition when multiple threads try to acquire the GIL
before `detail::internals` have been initialized. `gil_scoped_release`
is now tasked with initializing `internals` (guaranteed single-threaded)
to ensure the safety of subsequent `acquire` calls from multiple threads.
Py_Finalize could potentially invoke code that calls `get_internals()`,
which could create a new internals object if one didn't exist.
`finalize_interpreter()` didn't catch this because it only used the
pre-finalize interpreter pointer status; if this happens, it results in
the internals pointer not being properly destroyed with the interpreter,
which leaks, and also causes a `get_internals()` under a future
interpreter to return an internals object that is wrong in various ways.
At this point, there is only a single test for interpreter basics.
Apart from embedding itself, having a C++ test framework will also
benefit the C++-side features by allowing them to be tested directly.
