pybind11/tests/test_thread.py

50 lines
1.0 KiB
Python
Raw Normal View History

from __future__ import annotations
import sys
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
import threading
import pytest
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
from pybind11_tests import thread as m
class Thread(threading.Thread):
def __init__(self, fn):
super().__init__()
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
self.fn = fn
self.e = None
def run(self):
try:
for i in range(10):
self.fn(i, i)
except Exception as e:
self.e = e
def join(self):
super().join()
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
if self.e:
raise self.e
@pytest.mark.skipif(sys.platform.startswith("emscripten"), reason="Requires threads")
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
def test_implicit_conversion():
a = Thread(m.test)
b = Thread(m.test)
c = Thread(m.test)
for x in [a, b, c]:
x.start()
for x in [c, b, a]:
x.join()
@pytest.mark.skipif(sys.platform.startswith("emscripten"), reason="Requires threads")
Fix thread safety for pybind11 loader_life_support (#3237) * Fix thread safety for pybind11 loader_life_support Fixes issue: https://github.com/pybind/pybind11/issues/2765 This converts the vector of PyObjects to either a single void* or a per-thread void* depending on the WITH_THREAD define. The new field is used by each thread to construct a stack of loader_life_support frames that can extend the life of python objects. The pointer is updated when the loader_life_support object is allocated (which happens before a call) as well as on release. Each loader_life_support maintains a set of PyObject references that need to be lifetime extended; this is done by storing them in a c++ std::unordered_set and clearing the references when the method completes. * Also update the internals version as the internal struct is no longer compatible * Add test demonstrating threading works correctly. It may be appropriate to run this under msan/tsan/etc. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update test to use lifetime-extended references rather than std::string_view, as that's a C++ 17 feature. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Make loader_life_support members private * Update version to dev2 * Update test to use python threading rather than concurrent.futures * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Remove unnecessary env in test * Remove unnecessary pytest in test * Use native C++ thread_local in place of python per-thread data structures to retain compatability * clang-format test_thread.cpp * Add a note about debugging the py::cast() error * thread_test.py now propagates exceptions on join() calls. * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove unused sys / merge * Update include order in test_thread.cpp * Remove spurious whitespace * Update comment / whitespace. * Address review comments * lint cleanup * Fix test IntStruct constructor. * Add explicit to constructor Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com>
2021-09-10 16:29:21 +00:00
def test_implicit_conversion_no_gil():
a = Thread(m.test_no_gil)
b = Thread(m.test_no_gil)
c = Thread(m.test_no_gil)
for x in [a, b, c]:
x.start()
for x in [c, b, a]:
x.join()