Fix Postgres readiness race in FSM transitions#1147
Merged
Conversation
Three related fixes for a race where a keeper FSM transition could report a new state to the monitor while Postgres was still refusing TCP connections. 1. pg_setup_is_ready: accept PM_STATUS_STANDBY as "ready" PostgreSQL writes "standby" (not "ready") to postmaster.pid when hot_standby=on and the instance is accepting read-only connections but has not yet been promoted. pg_ctl -w treats both statuses as "server started"; pg_setup_is_ready previously only accepted "ready", causing it to loop indefinitely for hot standby nodes. Update the while condition, the early-exit break, and the return value to treat POSTMASTER_STATUS_READY and POSTMASTER_STATUS_STANDBY identically. Also fix the same hardcoded check in the pg_setup_init short-circuit inside pg_setup_wait_until_is_ready. 2. local_postgres_wait_until_ready: always wait for PM_STATUS Previously this function short-circuited via pg_is_running() (= pg_ctl status, which exits 0 as soon as the PID file exists) and skipped pg_setup_wait_until_is_ready when the process was already present. The PID file appears before the postmaster writes its ready/standby status, so callers could see pgIsRunning=true while the postmaster was still in the "starting" phase. Drop the pg_is_running pre-check entirely. pg_setup_wait_until_is_ready handles the "already running" case correctly: its first loop terminates immediately when get_pgpid finds a live pid, then the second loop checks pm_status. Every caller wants "accepting connections", not "process has a pid", so the short-circuit was never correct. 3. fsm_prepare_for_secondary: guard with ensure_postgres_service_is_running The CATCHINGUP -> SECONDARY transition assumed Postgres was ready because the prior WAIT_STANDBY -> CATCHINGUP transition started it. But fix #2 above was the primary path for closing that window; this call is an explicit belt-and-suspenders guard: if Postgres is somehow still in startup when fsm_prepare_for_secondary runs, the keeper retries the transition rather than reporting SECONDARY to the monitor prematurely.
143c3dd to
f23a090
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A keeper FSM transition could report a new state to the monitor while
Postgres was still refusing TCP connections. Three related fixes.
Root cause
local_postgres_wait_until_ready()calledpg_is_running()(=pg_ctl status, exits 0 as soon as the PID file exists) and short-circuited theactual readiness wait when the process was already present. The PID file
appears before the postmaster writes its status to that same file, so
callers could see
pgIsRunning=truewhile the postmaster was still in the"starting"phase — before TCP connections were accepted.Separately,
pg_setup_is_ready()only acceptedPM_STATUS_READYas "up",looping indefinitely on
PM_STATUS_STANDBY. PostgreSQL writes"standby"(not
"ready") topostmaster.pidwhenhot_standby=onand the instanceis accepting read-only connections but has not yet been promoted.
pg_ctl -wtreats both statuses as "server started";pg_setup_is_ready()did not.Changes
src/bin/common/pgsetup.c—pg_setup_is_ready()Accept
POSTMASTER_STATUS_STANDBYalongsidePOSTMASTER_STATUS_READY:update the
whilecondition, the early-exitbreak, and the return value.Also fix the same hardcoded
== POSTMASTER_STATUS_READYcheck in thepg_setup_initshort-circuit insidepg_setup_wait_until_is_ready().All call sites were audited: every caller either wants "is Postgres
accepting connections?" (yes for both statuses) or "is the process running?"
(also yes for both). Accepting
STANDBYis correct everywhere.src/bin/pg_autoctl/primary_standby.c—local_postgres_wait_until_ready()Drop the
pg_is_running()pre-check entirely.pg_setup_wait_until_is_ready()handles the "already running" case: its first poll loop terminates
immediately when
get_pgpidfinds a live pid, then the second loop checkspm_status. Every caller ofensure_postgres_service_is_running()wants"accepting connections", not "process has a pid", so the short-circuit was
never correct.
ensure_postgres_service_is_running_as_subprocess()retains its ownpg_is_running()call — that one detects whether Postgres was alreadyrunning before the status file was written, to decide whether to sleep
before the wait. It is doing something different and is unaffected.
src/bin/pg_autoctl/fsm_transition.c—fsm_prepare_for_secondary()Add
ensure_postgres_service_is_running()at the top of theCATCHINGUP → SECONDARYtransition. Postgres was started in the priorWAIT_STANDBY → CATCHINGUPtransition, but with the oldlocal_postgres_wait_until_ready()the readiness check could completebefore the TCP listener was open. This guard ensures the keeper retries the
transition rather than reporting
SECONDARYto the monitor prematurely.