Extensions that raise exceptions in their `PyInit*` cause a crash in subinterpreters

[ZeroIntensity]

Crash report

What happened?

I found this while investigating #138045.

If a PyInit_ function raises an exception when imported from a subinterpreter, the process will crash. For example, if you modify xxsubtyppe.c to raise an exception in its PyInit and then import it from a subinterpreter, you'll see a crash:

diff --git a/Modules/xxsubtype.c b/Modules/xxsubtype.c
index a8a1417f40e..57a5ef75eb7 100644
--- a/Modules/xxsubtype.c
+++ b/Modules/xxsubtype.c
@@ -324,5 +324,6 @@ static struct PyModuleDef xxsubtypemodule = {
 PyMODINIT_FUNC
 PyInit_xxsubtype(void)
 {
-    return PyModuleDef_Init(&xxsubtypemodule);
+    PyErr_SetString(PyExc_TypeError, "whatever");
+    return NULL;
 }

from concurrent import interpreters

interp = interpreters.create()
interp.exec("import xxsubtype")  # Crash!

This isn't usually a problem for multi-phase extensions, because PyModuleDef_Init doesn't fail with an exception (except sometimes on the free-threaded build, but that's rare), but this does happen for single-phase extensions. Single-phase extension modules do not support loading in subinterpreters, but attempting to load them from a subinterpreter should raise an exception rather than crash the interpreter.

I believe the problem lies here:

cpython/Python/import.c

Lines 2178 to 2196 in d736349

	main_finally:
	/* Switch back to the subinterpreter. */
	if (switched) {
	assert(main_tstate != tstate);
	switch_back_from_main_interpreter(tstate, main_tstate, mod);
	/* Any module we got from the init function will have to be
	* reloaded in the subinterpreter. */
	mod = NULL;
	}
	/*****************************************************************/
	/* At this point we are back to the interpreter we started with. */
	/*****************************************************************/
	/* Finally we handle the error return from _PyImport_RunModInitFunc(). */
	if (rc < 0) {
	_Py_ext_module_loader_result_apply_error(&res, name_buf);
	goto error;
	}

The exception object raised by the PyInit_ function is always owned by the main interpreter, but the exception is raised from the subinterpreter, which breaks things. There's not a clean way to raise the exception in the calling interpreter, so I think the best solution is to print the exception as unraisable in the main interpreter, and then raise some sort of blanket exception in the calling interpreter.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Linux

Output from running 'python -VV' on the command line:

No response

Linked PRs

• gh-144601: Avoid sharing exception objects raised in a PyInit function across multiple interpreters #144602
• [3.14] gh-144601: Avoid sharing exception objects raised in a PyInit function across multiple interpreters (GH-144602) #144633