servlet: fix UndertowTransportTest flakiness by eliminating container deadlocks#12908
Draft
AgraVator wants to merge 3 commits into
Draft
servlet: fix UndertowTransportTest flakiness by eliminating container deadlocks#12908AgraVator wants to merge 3 commits into
AgraVator wants to merge 3 commits into
Conversation
… thread parking deadlocks (grpc#12778) Fixes a long-standing flakiness bug in UndertowTransportTest.basicStream where the container worker thread would park for up to 1 minute in AsyncServletOutputStreamWriter.assureReadyAndDrainedTurnsFalse(), causing the test listener poll to time out after 10 seconds and fail with AssertionError: message expected. Root Cause: AsyncServletOutputStreamWriter previously assumed that readyAndDrained could only be true while onWritePossible() was actively executing and that any call to runOrBuffer() would return readyAndDrained back to false. However, when written payloads fit inside the servlet output buffer, outputStream.isReady() remains true after runOrBuffer() finishes, so runOrBuffer() exits leaving readyAndDrained == true. When Undertow subsequently dispatched WriteListener.onWritePossible() after async I/O write completion, onWritePossible() saw readyAndDrained == true and invoked assureReadyAndDrainedTurnsFalse(), parking the Undertow container thread via LockSupport.parkNanos(1 min). Because the application thread had already returned, no thread was running runOrBuffer() to clear readyAndDrained, leaving Undertow blocked for 60 seconds and causing the test timeout. Fix: 1. Removed assureReadyAndDrainedTurnsFalse() and LockSupport parking logic. A state of readyAndDrained == true with an empty writeChain and isReady() == true is a valid quiescent state; onWritePossible() verifies the queue is empty and exits cleanly without parking. 2. Safe state updates in runOrBuffer() when outputStream is not ready. Fixes grpc#12778
…tePossible non-blocking behavior (grpc#12778)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background & Problem Statement
UndertowTransportTest.basicStreamintermittently failed in CI with:Root Cause
AsyncServletOutputStreamWriterpreviously assumed thatreadyAndDrainedcould only remaintruewhileonWritePossible()was actively executing.However, when written payloads fit inside the servlet output buffer,
outputStream.isReady()remainstrueafterrunOrBuffer()finishes, sorunOrBuffer()exits leavingreadyAndDrained == true.When Undertow subsequently dispatched
WriteListener.onWritePossible()after async I/O write completion:onWritePossible()sawreadyAndDrained == true.assureReadyAndDrainedTurnsFalse(), which parked the Undertow container worker thread viaLockSupport.parkNanos(TimeUnit.MINUTES.toNanos(1)).runOrBuffer(), no thread was running to clearreadyAndDrained, leaving Undertow blocked for 60 seconds.AbstractTransportTest.basicStream()timed out after 10 seconds waiting atclientStreamListener.messageQueue.poll(), triggeringAssertionError: message expected.Solution
assureReadyAndDrainedTurnsFalse()andLockSupportparking:A state of
readyAndDrained == truewith an emptywriteChainandisReady() == trueis a valid quiescent state;onWritePossible()verifies the queue is empty and exits cleanly without parking worker threads.runOrBuffer():Updated state cleanup when
outputStreambecomes unready (isReady() == false), resolving concurrency races caught by Lincheck model checking.Added
AsyncServletOutputStreamWriterTestverifying non-blocking execution ofonWritePossible().Fixes #12778