[feat] Aggregate polling for retrieving job pending reason

[vkarak]

This PR improves the polling of jobs for the pending reason.

This is now done in a single command for all pending jobs. Two knobs are also exposed to users now as configuration options and environment variables:

1. slurm_job_cancel_reasons: This is a list of pending reasons that reframe will check and will cancel the job proactively.
2. slurm_pending_job_reason_poll_freq: This controls the frequency that pending jobs will be polled for their pending reasons (valid only for slurm backend).

Closes #3640.

[codecov]

Codecov Report

❌ Patch coverage is 50.84746% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.87%. Comparing base (4f021cf) to head (8881baf).

Files with missing lines	Patch %	Lines
reframe/core/schedulers/slurm.py	50.00%	27 Missing ⚠️
reframe/core/schedulers/__init__.py	33.33%	2 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3642      +/-   ##
===========================================
+ Coverage    91.69%   91.87%   +0.18%     
===========================================
  Files           62       62              
  Lines        13745    13755      +10     
===========================================
+ Hits         12603    12638      +35     
+ Misses        1142     1117      -25     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[jack-morrison]

Suggested change

	This option is relevant for the Slurm backends only.
	This option is relevant for the Slurm backend only.

[vkarak]

I would keep the plural here, because there are two Slurm backends: slurm and squeue.

[jack-morrison]

Are we dropping these default cancel reasons? Should they be made a part of slurm_job_cancel_reasons?

[vkarak]

I thought about it, but this list is not so accurate and to maintain it, we would have to deal with detailed Slurm semantics. For example, the Licenses reason is not a cancel reason in reality. Previously, we were poking Slurm to check if the reported nodes in the reason were indeed unavailable, which was not so efficient. That's why, I'm only included ReqNodeNotAvail reason as a cancel, which is indeed one, and leave any additional reason at user discretion.