binutils-gdb/gdb/python
Andrew Burgess 37f5e02ef5 gdb/python: don't use PyObject_IsInstance in gdbpy_is_color
The gdbpy_is_color function uses PyObject_IsInstance, and converts the
return from PyObject_IsInstance to a bool.

Unfortunately, PyObject_IsInstance can return -1, 0, or 1, for error,
failure, or success respectively.  When converting to a bool both -1
and 1 will convert to true.

Additionally, when PyObject_IsInstance returns -1 an error will be
set.

What this means is that, if gdbpy_is_color is called with a non
gdb.Color object, and the PyObject_IsInstance check raises an error,
then (a) GDB will continue as if the object is a gdb.Color object,
which is likely going to invoke undefined behaviour, see
gdbpy_get_color for example, and (b) when GDB eventually returns to
the Python interpreter, due to an error being set, we'll see:

  Python Exception <class 'SystemError'>: PyEval_EvalFrameEx returned a result with an error set
  Error occurred in Python: PyEval_EvalFrameEx returned a result with an error set

However, after the previous commit, gdb.Color can no longer be
sub-classed, this means that fixing the above problems is easy, we can
replace the PyObject_IsInstance check with a PyObject_TypeCheck, the
PyObject_TypeCheck function only returns 0 or 1, there's no -1 error
case.

It's also worth noting that PyObject_TypeCheck is the function that is
more commonly used within GDB's Python API implementation, include the
py-color.c use there were only 4 PyObject_IsInstance uses.  Of the
remaining 3, 2 are fine, and one other (in py-disasm.c) is also
wrong.  I'll address that in a separate patch.

There's also a new test included which exposes the above issue.

Approved-By: Tom Tromey <tom@tromey.com>
2025-04-23 15:46:22 +01:00
..
lib/gdb Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-all-events.def Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-arch.c gdb/python: add void_type () method to gdb.Architecture object 2025-02-04 13:56:49 +00:00
py-auto-load.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-block.c gdb/python: add subblocks property to gdb.Block 2025-02-04 13:56:49 +00:00
py-bpevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-breakpoint.c gdb/python: remove stray white space in error message 2025-04-22 17:50:55 +01:00
py-cmd.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-color.c gdb/python: don't use PyObject_IsInstance in gdbpy_is_color 2025-04-23 15:46:22 +01:00
py-color.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-connection.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-continueevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-dap.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-disasm.c gdb/python: stop using PyObject_IsInstance in py-disasm.c 2025-04-23 15:41:35 +01:00
py-event-types.def Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-event.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-event.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-events.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-evtregistry.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-evts.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-exitedevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-finishbreakpoint.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-frame.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-framefilter.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-function.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-gdb-readline.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-inferior.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-infevents.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-infthread.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-instruction.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-instruction.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-lazy-string.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-linetable.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-membuf.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-mi.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-micmd.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-newobjfileevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-objfile.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-param.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-prettyprint.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-progspace.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record-btrace.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record-btrace.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record-full.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record-full.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-record.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-ref.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-registers.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-signalevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-stopevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-stopevent.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-symbol.c gdb/python: convert gdb.Symbol to use gdbpy_registry 2025-03-19 21:12:53 +00:00
py-symtab.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-threadevent.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-tui.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-type.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-uiout.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-unwind.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-utils.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-value.c Clean up value_struct_elt_bitpos 2025-04-17 09:25:55 -06:00
py-varobj.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
py-xmethods.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
python-config.py Run isort 2024-04-02 11:21:39 -06:00
python-internal.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
python.c Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00
python.h Update copyright dates to include 2025 2025-04-08 10:54:39 -06:00