Enable ProactorEventLoop on windows for ipykernel

[NewUserHa]

see #1468

[ZupoLlask]

Hello @NewUserHa,

Thanks for your work here!

Dear @ianthomas23,

Is it possible to get this merged on the next release?

It is really important for Windows users, as current solution creates incompatibilities between libraries.

Thank you!

[ianthomas23]

@ZupoLlask It is not passing CI, so of course it will not be merged.

[ZupoLlask]

@NewUserHa Can you please have a look at this? TY

[NewUserHa]

I have no idea how this occur. this modification works in previous versions of ipython

[NewUserHa]

the tests/test_debugger.py::test_stop_on_breakpoint issue seems to be related to IOStream.flush timed out and pyzmq ioloop and proactoreventloop.
it may need help

[ianthomas23]

@NewUserHa There were some problems with the CI that I think I have now fixed, so can you rebase this on main to see what is still failing? I should be able to help next week with any further problems here.

[NewUserHa]

PR modified:

1. test_async.py: asyncio>asynclib. reason: was typo
2. zmqshell.py‎: warnings.filterwarnings("ignore",. the test test_async has no issue before changing to proactor eventloop. reason: because when user command is autowait ..., it is ran within the asyncio loop of ipykernel. after changing to proactoreventloop, autowait trio trio will prompt it need to be set running with guest mode. but there's no a good way to set that by dynamically detecting the user commands, and it works fine without setting that guest mode, except reciving the prompts from trio. therefore the fix is disabling the warning.
3. eventloops.py‎: blocking_poll(). reason: because ProactorEventLoop has no add_reader method and IOCP doesn't support ZMQ's FD directly, therefore use a thread to wait the poll() in a blocking way to make it work. the other solution is to warp the FD into a handle and use IocpProactor class 's handle wait_for_handle method to wait it, but it needs win32api to wrap. even though the latter method looks better but the former is more straightforward, so the fomer method is used.

current:

1. the fix of def loop_asyncio(kernel): of eventloops.py seemes to have no relation with the failed tests. whether it's fixed, the tests still pass. the loop_asyncio() seems to be not used or not covered.
2. the remained failed test test_stop_on_breakpoint, it seems that the debugpy did send the stopped msg, but the main loop of ipykernel seems to be unable to recive it because of the hread of the main loop of ipykernel is paused by debugpy. the IOStream.flush timed out can be seen after manually set the debug flag of ioPub class to True.
   but the selectoreventloop has no this issue.

[ZupoLlask]

Thank you @NewUserHa!

Let's wait and see if @ianthomas23 can help a bit here... 🙏

[ianthomas23]

Let's wait and see if @ianthomas23 can help a bit here... 🙏

I can, but it will be next week now.

[ianthomas23]

I've taken a look but I am not there is anything I can add here.

I notice that most of the eventloops testing is disabled on windows, e.g.

ipykernel/tests/test_eventloop.py

Lines 86 to 87 in 770afbe

	@windows_skip
	def test_asyncio_loop(kernel):

so this has evidently been a problematic area in the past.

[NewUserHa]

ipykernel/tests/test_eventloop.py

Lines 86 to 87 in 770afbe

	@windows_skip
	def test_asyncio_loop(kernel):

it should have no problem, since the difference between proactor and selector is add_reader() only mostly.

However, the debugpy can't debug ipyknerel and the test, reporting No module named ipykernel_launcher. it may be a issue of the test codes.
can you take a look at it? so I can debug why the eventloop is stucked and unable to recieve the "stopped" msg from debugpy and cause the failure of the CI test.

And the ipykernel's class IOPub starts a thread for tornado's class IOloop, while tornado >6.1 version starts a thread for proactor on windows as well, while pyzmq uses tornado's AsyncIOloop to start a thread also.
can you explain it a little? how many thread ipykernel will start, and is it starting a thread for IOloop and another thread for zmq internally? could ipykernel use the tornado's new class AsyncIOloop, since the old IOloop class is decrepted in its class's doc string, and its new class in tornado is right for solving the proactorloop issue of lacking add_reader(), by using selector in a thread already

[BoykoNeov]

@NewUserHa I dug into the test_stop_on_breakpoint deadlock you flagged — here's the root cause and a fix. (This is the area you already suspected: "could ipykernel use … selector in a thread already" — yes, almost.)

Root cause — ipykernel's read side, not debugpy

On a failing run debugpy does hit the breakpoint and write the stopped event; ipykernel's idle control loop just never wakes to read it. Mechanism:

• ProactorEventLoop has no native add_reader, so tornado drives a Proactor loop's zmq sockets through a helper "Tornado selector" thread that does select() then call_soon_threadsafe() to wake the loop. ipykernel's IOPub/control service loops each spawn one of these.
• ipykernel marks its own service threads debugger-exempt (is_pydev_daemon_thread / pydev_do_not_trace) but not tornado's helper thread.
• When debugpy suspends every thread at a breakpoint under sys.monitoring (Python ≥ 3.12, interpreter-global), that un-exempt helper freezes mid-wake — after select() returns but before call_soon_threadsafe() completes — so the control/debug read path never advances and the Proactor loop sits in the IOCP poll forever.
• On 3.11 (sys.settrace, per-thread) the helper isn't frozen → no repro. Matches the CI matrix exactly: Win 3.10/3.11 pass, 3.12/3.13/3.14/pypy fail.

(Proven on 3.13 with faulthandler: the wedged kernel's stack shows the "Tornado selector" thread frozen inside call_soon_threadsafe → do_wait_suspend; the 3.11 mirror shows it completing.)

Fix

Run the ipykernel service loops (control, IOPub, shell channel, subshells) on a SelectorEventLoop: it has a native add_reader, so no helper thread is spawned and there's nothing for the debugger to freeze. The main/user-code loop stays on Proactor, so your subprocess support (#1468) is preserved. Off-Windows it's a no-op (default loop is already selector-based). ~30 lines, 2 files, win32-guarded, no pydevd coupling.

Branch (ready to pull): https://github.com/BoykoNeov/ipykernel/tree/pr-1469 — single commit b142cae on top of your patch-2. Happy to open it as a PR into patch-2 if you'd like.

Result (Windows, local): full test_debugger.py on 3.13 Proactor goes from 5 breakpoint tests deadlocking (6 failed/5 passed, ~1014s) to 1 failed/10 passed, ~15s; identical on 3.11; test_kernel.py 24 passed/6 skipped; no new regressions.

One separate, minor thing (not fixed by the above — flagging it)

While bisecting I found test_attach_debug also flips under Proactor, but it's cosmetic, not a functional break. I verified the repl is fully functional under Proactor — a = 5 then a → '5', a + 100 → '105', namespace persists. The only difference is the first repl evaluate's DAP result field: empty under Selector, the actual value under Proactor (Proactor just surfaces it one evaluate earlier — arguably more correct). The main loop stays Proactor by design, so this fix doesn't change it. You'll probably want to update test_attach_debug's == "" assertion as part of this PR — it currently encodes the Selector-timing result, and editing it against current main would break CI under Selector, so it belongs with the Proactor switch.

[NewUserHa]

Thanks for your effort.
There are 3 issues:

Your commit changes all self.io_loop = IOLoop(make_current=False) to IOLoop(make_current=False, asyncio_loop=asyncio.SelectorEventLoop()), but IOLoop has no the asyncio_loop parameter, which is passed through anonymously and into AsyncIOLoop. Is this good to implement?

The BaseAsyncIOLoop of AsyncIOLoop in tornado already implemented
https://github.com/tornadoweb/tornado/blob/f491e4c1914be0ac6635a0eacb3c978d89eec4f1/tornado/platform/asyncio.py#L89-L106
can we re-ultilize that

Result (Windows, local): full test_debugger.py on 3.13 Proactor goes from 5 breakpoint tests deadlocking (6 failed/5 passed, ~1014s) to 1 failed/10 passed, ~15s; identical on 3.11; test_kernel.py 24 passed/6 skipped; no new regressions.

there is still a test failed, can you help take a look at that

[BoykoNeov]

Thanks @NewUserHa — good questions, taking them in turn.

1. asyncio_loop on IOLoop(...)

You're right that it flows through **kwargs and that the base IOLoop signature doesn't name it. But it isn't an accidental passthrough — the concrete loop tornado constructs on asyncio platforms, AsyncIOLoop, handles it explicitly:

# tornado/platform/asyncio.py — AsyncIOLoop.initialize
...
if "asyncio_loop" not in kwargs:
    kwargs["asyncio_loop"] = loop = asyncio.new_event_loop()
...
super().initialize(**kwargs)

i.e. it's designed to let the caller supply the loop and only creates its own when you don't. It's been public API since tornado 6.2:

.. versionchanged:: 6.2 — Support explicit asyncio_loop argument for specifying the asyncio loop to attach to, rather than always creating a new one with the default policy.

ipykernel's floor is tornado>=6.4.1, comfortably above 6.2, so relying on it is safe. Routing through IOLoop(...) is also the recommended path — referring to AsyncIOLoop directly is deprecated since 5.0 ("no longer necessary to refer to this class directly").

2. Reusing BaseAsyncIOLoop

We already are — IOLoop(asyncio_loop=...) is the public door into exactly that machinery. And the lines you linked are the heart of why the fix works:

# BaseAsyncIOLoop.initialize
self.selector_loop = asyncio_loop
if hasattr(asyncio, "ProactorEventLoop") and isinstance(
    asyncio_loop, asyncio.ProactorEventLoop
):
    self.selector_loop = AddThreadSelectorEventLoop(asyncio_loop)

That AddThreadSelectorEventLoop is the "Tornado selector" helper thread — the one that freezes mid-wake under the debugger and deadlocks the read path. Hand BaseAsyncIOLoop a SelectorEventLoop and selector_loop is asyncio_loop: the wrapper is never constructed, so there's no helper thread to freeze. So the fix isn't avoiding BaseAsyncIOLoop — it's using it with a loop that doesn't need the helper. (Feeding it a Proactor loop is precisely what reintroduces the deadlock.)

3. The remaining failed test

That's test_attach_debug — the cosmetic one I flagged in the comment above, not a new functional break. The repl is fully functional under Proactor (a = 5 then a → '5', namespace persists); the only difference is the first evaluate's DAP result field, which Proactor surfaces one evaluate earlier than Selector. The test is a single assertion:

# tests/test_debugger.py
assert reply["body"]["result"] == ""

which encodes the Selector timing. Since this PR makes the main loop Proactor, that assertion should move with it — editing it against current main (still Selector) would break CI the other way. Happy to push that update to my pr-1469 branch, using the value the Proactor run actually produces, so the suite is green under Proactor.

[NewUserHa]

Root cause — ipykernel's read side, not debugpy

On a failing run debugpy does hit the breakpoint and write the stopped event; ipykernel's idle control loop just never wakes to read it. Mechanism:

• ProactorEventLoop has no native add_reader, so tornado drives a Proactor loop's zmq sockets through a helper "Tornado selector" thread that does select() then call_soon_threadsafe() to wake the loop. ipykernel's IOPub/control service loops each spawn one of these.
• ipykernel marks its own service threads debugger-exempt (is_pydev_daemon_thread / pydev_do_not_trace) but not tornado's helper thread.
• When debugpy suspends every thread at a breakpoint under sys.monitoring (Python ≥ 3.12, interpreter-global), that un-exempt helper freezes mid-wake — after select() returns but before call_soon_threadsafe() completes — so the control/debug read path never advances and the Proactor loop sits in the IOCP poll forever.
• On 3.11 (sys.settrace, per-thread) the helper isn't frozen → no repro. Matches the CI matrix exactly: Win 3.10/3.11 pass, 3.12/3.13/3.14/pypy fail.

(Proven on 3.13 with faulthandler: the wedged kernel's stack shows the "Tornado selector" thread frozen inside call_soon_threadsafe → do_wait_suspend; the 3.11 mirror shows it completing.)

Thanks. good fix and analysis.

Your fix repleaces all service loops (including IOPub/control/iostream/etc.) to SelectorEventLoop, is it possible to keep as much as possible ProactorEventLoop for these service loops? considering ProactorEventLoop is superior than SelectorEventLoop in performance on Windows, although within ipykernel and zmq a SelectorEventLoop is enough for this usage scenarios.
Does changing iostream/IOPubThread toSelectorEventLoop affect the main loop? How is main loop for users affected by this

The #1532 also relates to this, please take a look with this pr.

Please open it as a PR into patch-2 includes covering the last remaining failed test
Thanks.

[BoykoNeov]

Opened it as a PR into patch-2: NewUserHa#1 — one commit (service loops → selector), rebased onto current patch-2, so the diff is just that.

On your questions:

Does moving IOPubThread / the service loops to selector touch the main loop? No — each service channel runs in its own thread with its own io_loop, so swapping those leaves the main/user-code loop on Proactor (#1468 stays intact). Confirmed it works: the subprocess kernel tests pass, including the asyncio→IOPub one.

Keep as much Proactor as possible? You could narrow it to just the control loop (the one on the debug read path) — but I'd lean against it. The other service loops only carry low-volume zmq traffic with no subprocess I/O, so Proactor buys them nothing measurable, and any loop left on Proactor keeps a freezable helper thread around under the debugger. Uniform selector for the service loops keeps the rule simple ("only the main loop is Proactor") at basically no cost.

The last failing test — test_attach_debug is already covered on current patch-2 by the debugpy >= 1.8.21 gate (#1524). Turns out it's a debugpy behavior change, not a loop thing, so my earlier "cosmetic Proactor timing" guess was off — nothing to change. Full test_debugger.py is green here with the fix.

Heads-up on testing: the debugger suite can't actually hit this deadlock right now — conftest.py forces selector and the debug tests use an in-process MockKernel on IOLoop.current() (no real Control/IOPub threads). Force it onto Proactor and the breakpoint tests pass with and without the fix, so a real regression test would need a subprocess-kernel-under-debugger harness. Probably a follow-up.

On #1532 — related but separate (dropping the dead loop_asyncio), so I kept it out of this PR. If that path goes, the os.name=='nt' poll branch from b48f2d2 goes with it (looks like it has an inverted while stop_event.is_set() and a calls_soon_threadsafe typo anyway), so removing is cleaner than fixing.

[NewUserHa]

Fixs: #1468

The PR passes all checks now. may you give it a review? @ianthomas23

After the PR, we now would be able to use asyncio.create_subprocess_* and the libraries depends on it in jupyter notebook on windows ProactorEventLoop.
It looks good to me. It now enables mainloop uses ProactorLoop now for user codes; While ipykernel uses SelectorLoop in their originally spanwed threads for IOPub/control/iostream/etc. in which zmq requires little loads.