Skip to content

Fix Postgres readiness race in FSM transitions#1147

Merged
dimitri merged 1 commit into
mainfrom
fix/postgres-readiness-race
Jul 13, 2026
Merged

Fix Postgres readiness race in FSM transitions#1147
dimitri merged 1 commit into
mainfrom
fix/postgres-readiness-race

Conversation

@dimitri

@dimitri dimitri commented Jul 13, 2026

Copy link
Copy Markdown
Contributor

A keeper FSM transition could report a new state to the monitor while
Postgres was still refusing TCP connections. Three related fixes.

Root cause

local_postgres_wait_until_ready() called pg_is_running() (= pg_ctl status, exits 0 as soon as the PID file exists) and short-circuited the
actual readiness wait when the process was already present. The PID file
appears before the postmaster writes its status to that same file, so
callers could see pgIsRunning=true while the postmaster was still in the
"starting" phase — before TCP connections were accepted.

Separately, pg_setup_is_ready() only accepted PM_STATUS_READY as "up",
looping indefinitely on PM_STATUS_STANDBY. PostgreSQL writes "standby"
(not "ready") to postmaster.pid when hot_standby=on and the instance
is accepting read-only connections but has not yet been promoted. pg_ctl -w treats both statuses as "server started"; pg_setup_is_ready() did not.

Changes

src/bin/common/pgsetup.cpg_setup_is_ready()

Accept POSTMASTER_STATUS_STANDBY alongside POSTMASTER_STATUS_READY:
update the while condition, the early-exit break, and the return value.
Also fix the same hardcoded == POSTMASTER_STATUS_READY check in the
pg_setup_init short-circuit inside pg_setup_wait_until_is_ready().

All call sites were audited: every caller either wants "is Postgres
accepting connections?" (yes for both statuses) or "is the process running?"
(also yes for both). Accepting STANDBY is correct everywhere.

src/bin/pg_autoctl/primary_standby.clocal_postgres_wait_until_ready()

Drop the pg_is_running() pre-check entirely. pg_setup_wait_until_is_ready()
handles the "already running" case: its first poll loop terminates
immediately when get_pgpid finds a live pid, then the second loop checks
pm_status. Every caller of ensure_postgres_service_is_running() wants
"accepting connections", not "process has a pid", so the short-circuit was
never correct.

ensure_postgres_service_is_running_as_subprocess() retains its own
pg_is_running() call — that one detects whether Postgres was already
running before the status file was written, to decide whether to sleep
before the wait. It is doing something different and is unaffected.

src/bin/pg_autoctl/fsm_transition.cfsm_prepare_for_secondary()

Add ensure_postgres_service_is_running() at the top of the
CATCHINGUP → SECONDARY transition. Postgres was started in the prior
WAIT_STANDBY → CATCHINGUP transition, but with the old
local_postgres_wait_until_ready() the readiness check could complete
before the TCP listener was open. This guard ensures the keeper retries the
transition rather than reporting SECONDARY to the monitor prematurely.

@dimitri dimitri self-assigned this Jul 13, 2026
@dimitri dimitri added the bug Something isn't working label Jul 13, 2026
Base automatically changed from fix/keeper-graceful-shutdown to main July 13, 2026 01:57
Three related fixes for a race where a keeper FSM transition could
report a new state to the monitor while Postgres was still refusing TCP
connections.

1. pg_setup_is_ready: accept PM_STATUS_STANDBY as "ready"

   PostgreSQL writes "standby" (not "ready") to postmaster.pid when
   hot_standby=on and the instance is accepting read-only connections
   but has not yet been promoted.  pg_ctl -w treats both statuses as
   "server started"; pg_setup_is_ready previously only accepted "ready",
   causing it to loop indefinitely for hot standby nodes.

   Update the while condition, the early-exit break, and the return
   value to treat POSTMASTER_STATUS_READY and POSTMASTER_STATUS_STANDBY
   identically.  Also fix the same hardcoded check in the pg_setup_init
   short-circuit inside pg_setup_wait_until_is_ready.

2. local_postgres_wait_until_ready: always wait for PM_STATUS

   Previously this function short-circuited via pg_is_running() (=
   pg_ctl status, which exits 0 as soon as the PID file exists) and
   skipped pg_setup_wait_until_is_ready when the process was already
   present.  The PID file appears before the postmaster writes its
   ready/standby status, so callers could see pgIsRunning=true while
   the postmaster was still in the "starting" phase.

   Drop the pg_is_running pre-check entirely.  pg_setup_wait_until_is_ready
   handles the "already running" case correctly: its first loop terminates
   immediately when get_pgpid finds a live pid, then the second loop checks
   pm_status.  Every caller wants "accepting connections", not "process
   has a pid", so the short-circuit was never correct.

3. fsm_prepare_for_secondary: guard with ensure_postgres_service_is_running

   The CATCHINGUP -> SECONDARY transition assumed Postgres was ready
   because the prior WAIT_STANDBY -> CATCHINGUP transition started it.
   But fix #2 above was the primary path for closing that window; this
   call is an explicit belt-and-suspenders guard: if Postgres is
   somehow still in startup when fsm_prepare_for_secondary runs, the
   keeper retries the transition rather than reporting SECONDARY to the
   monitor prematurely.
@dimitri dimitri force-pushed the fix/postgres-readiness-race branch from 143c3dd to f23a090 Compare July 13, 2026 02:00
@dimitri dimitri merged commit e975832 into main Jul 13, 2026
50 of 54 checks passed
@dimitri dimitri deleted the fix/postgres-readiness-race branch July 13, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant