This is meant to aide in debugging hard-to-decipher pybind11 bugs (or surprises in behavior ;).
Tracing Python Executions
Generally, it's easiest to get a sense of where things are failing by using the trace module. Here's some code that can be copied+pasted. For examples here, assume these functions are defined in debug.py (Python 3.5+ only):
# -*- coding: utf-8 -*-
"""
Utilities that should be synchronized with:
https://drake.mit.edu/python_bindings.html#debugging-with-the-python-bindings
"""
def reexecute_if_unbuffered():
"""Ensures that output is immediately flushed (e.g. for segfaults).
ONLY use this at your entrypoint. Otherwise, you may have code be
re-executed that will clutter your console."""
import os
import sys
if os.environ.get("PYTHONUNBUFFERED") in (None, ""):
os.environ["PYTHONUNBUFFERED"] = "1"
argv = list(sys.argv)
if argv[0] != sys.executable:
argv.insert(0, sys.executable)
sys.stdout.flush()
os.execv(argv[0], argv)
def traced(func, ignoredirs=None):
"""Decorates func such that its execution is traced, but filters out any
Python code outside of the system prefix."""
import functools
import sys
import trace
if ignoredirs is None:
ignoredirs = ["/usr", sys.prefix]
tracer = trace.Trace(trace=1, count=0, ignoredirs=ignoredirs)
@functools.wraps(func)
def wrapped(*args, **kwargs):
return tracer.runfunc(func, *args, **kwargs)
return wrapped
Usage of this could look like the following:
import debug
import my_crazy_cool_module # Say this has bindings buried 3 levels deep.
@debug.traced
def main():
my_crazy_cool_module.do_something_weird()
if __name__ == "__main__":
debug.reexecute_if_unbuffered()
main()
Then you can see where a segfault happens along the actual trace of your code.
GDB on pybind11 unittests
The easiest way to do this is to use a debug build w/ a debug Python interpreter directly on pybind11 source code. Ideally, if you have an issue in your code base, you can make a min-reproduction as a failing test on the pybind11 source code.
Here's an example workflow of debugging a specific unittest on CPython 3.8 on Ubuntu 18.04; this assumes the following packages are installed:
sudo apt install cmake build-essential ninja-build python3.8-dev python3.8-venv python3.8-dbg
Then here's an example of running a unittest with GDB:
cd pybind11
python3.8-dbg -m venv ./venv
source ./venv/bin/activate
pip install -U pip wheel
pip install pytest
mkdir build && cd build
cmake .. -GNinja \
-DCMAKE_BUILD_TYPE=Debug \
-DPYTHON_EXECUTABLE=$(which python) \
-DPYBIND11_TEST_OVERRIDE=test_multiple_inheritance.cpp
# Get a sense of what is executed using the `-v` flag to ninja.
env PYTHONUNBUFFERED=1 PYTEST_ADDOPTS="-s -x" ninja -v pytest
# Now reformat it to your usage. For example:
src_dir=${PWD}/..
build_dir=${PWD}
( cd ${build_dir}/tests && gdb --args env PYTHONUNBUFFERED=1 python -m pytest -s -x ${src_dir}/tests/test_multiple_inheritance.py )
If you have a segfault, and are using something like debug.traced, you should see something like this (for this example, a fake segfault was injected):
../../tests/test_multiple_inheritance.py --- modulename: test_multiple_inheritance, funcname: test_failure_min
test_multiple_inheritance.py(14): class MI1(m.Base1, m.Base2):
--- modulename: test_multiple_inheritance, funcname: MI1
test_multiple_inheritance.py(14): class MI1(m.Base1, m.Base2):
test_multiple_inheritance.py(15): def __init__(self, i, j):
test_multiple_inheritance.py(19): MI1(1, 2)
--- modulename: test_multiple_inheritance, funcname: __init__
test_multiple_inheritance.py(16): m.Base1.__init__(self, i)
Program received signal SIGSEGV, Segmentation fault.
0x<crazyaddress> in pybind11::detail::get_type_info (type=0xdeed90) at ../include/pybind11/detail/type_caster_base.h:160
160 *value = 0xbadf00d;
(gdb)
GDB on C++ Programs with an Embedded Interpreter
Sometimes, it may be easier to test a program with an embedded interpreter.
TODO(eric): Add an example.
Code written here is licensed under pybind11's license (BSD-3) unless otherwise stated.