Enable TCP keepalives to survive long jixia analysis phase#16
Merged
Conversation
The 'Load jixia data into PostgreSQL' step failed with: psycopg.OperationalError: consuming input failed: server closed the connection unexpectedly The DB connection is opened before the jixia analysis phase, which runs 30-60+ min of pure CPU work with zero DB traffic. GitHub Actions runners sit behind Azure's outbound NAT, whose default idle timeout (~4 min) silently drops the connection's NAT mapping long before analysis finishes, so the first write after that fails. Enable TCP keepalives (idle=30s, interval=10s, count=5) so the OS sends probe packets during the idle phase and the NAT mapping never expires.
There was a problem hiding this comment.
Pull request overview
This PR addresses intermittent PostgreSQL disconnects during the long, CPU-bound jixia analysis phase by enabling TCP keepalive probes on the existing psycopg connection so it stays active through Azure outbound NAT idle timeouts in GitHub Actions.
Changes:
- Configure psycopg connections in
databaseCLI entrypoint with TCP keepalive settings (keepalives_idle=30,keepalives_interval=10,keepalives_count=5). - Add inline rationale explaining the idle-connection/NAT-timeout failure mode and why keepalives prevent it.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
With the Heroku secret refreshed and #15 merged, the pipeline got further than ever — jixia processed all 492 modules cleanly — but the Load jixia data into PostgreSQL step then failed on the very first DB write:
Root cause
The DB connection is opened in
main()before dispatching to thejixiacommand, thenload_datarunsbatch_run_jixia— a purely CPU-bound analysis phase with zero DB traffic — for 30-60+ minutes (confirmed: 48 min this run) before the firstINSERT INTO module. GitHub Actions runners sit behind Azure's outbound NAT, whose default idle timeout is ~4 minutes. The NAT mapping for the idle DB connection is silently dropped long before analysis finishes, so the first write after that fails.What
Enable TCP keepalives on the connection (
keepalives_idle=30s, well under Azure's ~4 min timeout) so the OS sends periodic probe packets during the idle analysis phase, keeping the NAT mapping alive for the whole run.