Add Config.SoftStopTimeout for a cleaner way to shut a client down#1239
Open
Add Config.SoftStopTimeout for a cleaner way to shut a client down#1239
Config.SoftStopTimeout for a cleaner way to shut a client down#1239Conversation
A part of River code that's always bothered me is the graceful shutdown
example [1]. It's really, really complicated, and I think that
complicated orchestration is very likely a sign that our code isn't
quite right.
I initially tried putting the graceful shutdown code into a helper like
`river.GracefulShutdown`, but I found that also problematic because it's
not super clean to call, and even minor customizations would require a
user to copy out the entire function.
Playing with an LLM a bit, I came up with this alternative: build an
understanding of soft/hard stop into the client's default `Stop`
function by adding `Config.SoftStopTimeout`. By default, leaving this
value unconfigured keeps the existing behavior of River today where a
call to `Stop` waits unbounded time for jobs to finish working. Adding a
`SoftStopTimeout` value brings in functionality similar to the graceful
shutdown example: stopping proceeds normally waiting for jobs to finish,
but in case they don't inside of `SoftStopTimeout`, they're cancelled as
if `StopAndCancel` had been called.
The soft timeout is also respected for a cancelled `Start` context,
which makes the basic use of River for soft/hard stop very nice (you
don't even need to call `Stop` anymore):
// Use signal.NotifyContext to cancel the start context on SIGINT/SIGTERM.
// When the signal fires, the client initiates a soft stop. If running jobs
// don't finish within SoftStopTimeout, their contexts are automatically
// cancelled.
ctx, stop := signal.NotifyContext(ctx, syscall.SIGINT, syscall.SIGTERM)
defer stop()
if err := riverClient.Start(ctx); err != nil {
panic(err)
}
[1] https://github.com/riverqueue/river/blob/master/example_graceful_shutdown_test.go
c52de36 to
b7f599c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A part of River code that's always bothered me is the graceful shutdown
example [1]. It's really, really complicated, and I think that
complicated orchestration is very likely a sign that our code isn't
quite right.
I initially tried putting the graceful shutdown code into a helper like
river.GracefulShutdown, but I found that also problematic because it'snot super clean to call, and even minor customizations would require a
user to copy out the entire function.
Playing with an LLM a bit, I came up with this alternative: build an
understanding of soft/hard stop into the client's default
Stopfunction by adding
Config.SoftStopTimeout. By default, leaving thisvalue unconfigured keeps the existing behavior of River today where a
call to
Stopwaits unbounded time for jobs to finish working. Adding aSoftStopTimeoutvalue brings in functionality similar to the gracefulshutdown example: stopping proceeds normally waiting for jobs to finish,
but in case they don't inside of
SoftStopTimeout, they're cancelled asif
StopAndCancelhad been called.The soft timeout is also respected for a cancelled
Startcontext,which makes the basic use of River for soft/hard stop very nice (you
don't even need to call
Stopanymore):[1] https://github.com/riverqueue/river/blob/master/example_graceful_shutdown_test.go