diff --git a/Debugging-segfaults-and-hard-to-decipher-pybind11-bugs.md b/Debugging-segfaults-and-hard-to-decipher-pybind11-bugs.md new file mode 100644 index 0000000..972ce27 --- /dev/null +++ b/Debugging-segfaults-and-hard-to-decipher-pybind11-bugs.md @@ -0,0 +1,115 @@ +This is meant to aide in debugging hard-to-decipher pybind11 bugs (or surprises in behavior ;). + +## Tracing Python Executions + +Generally, it's easiest to get a sense of where things are failing by using the `trace` module. Here's some code that can be copied+pasted. For examples here, assume these functions are defined in `debug.py`: + +```py +""" +Utilities that should be synchronized with: +https://drake.mit.edu/python_bindings.html#debugging-with-the-python-bindings +""" + + +def reexecute_if_unbuffered(): + """Ensures that output is immediately flushed (e.g. for segfaults). + ONLY use this at your entrypoint. Otherwise, you may have code be + re-executed that will clutter your console.""" + import os + import shlex + import sys + if os.environ.get("PYTHONUNBUFFERED") in (None, ""): + os.environ["PYTHONUNBUFFERED"] = "1" + argv = list(sys.argv) + if argv[0] != sys.executable: + argv.insert(0, sys.executable) + cmd = " ".join([shlex.quote(arg) for arg in argv]) + sys.stdout.flush() + os.execv(argv[0], argv) + + +def traced(func, ignoredirs=None): + """Decorates func such that its execution is traced, but filters out any + Python code outside of the system prefix.""" + import functools + import sys + import trace + if ignoredirs is None: + ignoredirs = ["/usr", sys.prefix] + tracer = trace.Trace(trace=1, count=0, ignoredirs=ignoredirs) + + @functools.wraps(func) + def wrapped(*args, **kwargs): + return tracer.runfunc(func, *args, **kwargs) + + return wrapped +``` + +Usage of this could look like the following: + +```sh +import my_crazy_cool_module # Say this has bindings buried 3 levels deep. + + +@debug.traced +def main(): + my_crazy_cool_module.do_something_weird() + +if __name__ == "__main__": + debug.reexecute_if_unbuffered() + main() +``` + +Then you can see where a segfault happens along the actual trace of your code. + +## GDB on pybind11 unittests + +The easiest way to do this is to use a debug build w/ a debug Python interpreter directly on pybind11 source code. Ideally, if you have an issue in your code base, you can make a min-reproduction as a failing test on the `pybind11` source code. + +Here's an example workflow of debugging a specific unittest on CPython 3.8 on Ubuntu 18.04; this assumes the following packages are installed: + +``` +sudo apt install cmake build-essential python3.8-dev python3.8-venv python3.8-dbg +``` + +Then here's an example of running a unittest with GDB: + +``` +cd pybind11 + +python3.8-dbg -m venv ./venv +source ./venv/bin/activate +pip install -U pip wheel +pip install pytest + +mkdir build && cd build +cmake .. -GNinja \ + -DCMAKE_BUILD_TYPE=Debug \ + -DPYTHON_EXECUTABLE=$(which python) \ + -DPYBIND11_TEST_OVERRIDE=test_multiple_inheritance.cpp + +# Get a sense of what is executed using the `-v` flag to ninja. +env PYTHONUNBUFFERED=1 PYTEST_ADDOPTS="-s -x" ninja -v pytest + +# Now reformat it to your usage. For example: +src_dir=${PWD}/.. +build_dir=${PWD} +( cd ${build_dir}/tests && gdb --args env PYTHONUNBUFFERED=1 python -m pytest -s -x ${src_dir}/tests/test_multiple_inheritance.py ) +``` + +If you have a segfault, and are using something like `debug.traced`, you should see something like this (for this example, a fake segfault was injected): +``` +../../tests/test_multiple_inheritance.py --- modulename: test_multiple_inheritance, funcname: test_failure_min +test_multiple_inheritance.py(14): class MI1(m.Base1, m.Base2): + --- modulename: test_multiple_inheritance, funcname: MI1 +test_multiple_inheritance.py(14): class MI1(m.Base1, m.Base2): +test_multiple_inheritance.py(15): def __init__(self, i, j): +test_multiple_inheritance.py(19): MI1(1, 2) + --- modulename: test_multiple_inheritance, funcname: __init__ +test_multiple_inheritance.py(16): m.Base1.__init__(self, i) + +Program received signal SIGSEGV, Segmentation fault. +0x in pybind11::detail::get_type_info (type=0xdeed90) at ../include/pybind11/detail/type_caster_base.h:160 +160 *value = 0xbadf00d; +(gdb) +``` \ No newline at end of file