Runs-on for linux-build-lib and linux-test (2X faster CI)
#20107
Conversation
github-actions
bot
added
documentation
|
Nice ! |
comphead
left a comment
comphead
left a comment
There was a problem hiding this comment.
Thanks @blaginin lets give it a try
Are you aware if any limits applies for AWS runners, I checked https://docs.github.com/en/actions/reference/limits but cannot say if it is for AWS runners
Yes, those limits do not apply to us. We will be bounded by the ASF AWS Account budget and service quotas, this will be on me and infra team to monitor I will merge this PR soon and we can start testing 🚀 |
|
I think it is a great idea to try one or two jobs initially and then slowly migrate over The biggest concern I have is if there is some difference between the normal runners (that will run on PRs) and the one that will run on main. |
Jefffrey
left a comment
Jefffrey
left a comment
There was a problem hiding this comment.
DataFusion has one of the largest runtimes across Apache projects
😅
|
If everyone is okay, I'm merging the PR |
Those on |
| linux-build-lib: | ||
| name: linux build test | ||
| runs-on: ubuntu-latest | ||
| runs-on: ${{ github.repository_owner == 'apache' && format('runs-on={0},family=m7a,cpu=16,image=ubuntu24-full-x64,extras=s3-cache,disk=large,tag=datafusion', github.run_id) || 'ubuntu-latest' }} |
There was a problem hiding this comment.
github.repository_owner == 'apache'
❤️
|
BTW I don't have data, but this feels like it made a major difference on CI time recently |
|
Yes, it is around twice faster now for the affected CI jobs! |
|
Thanks @blaginin I think we need the same for Comet, as its heavily tested. Appreciate if you can share how to start |
Of course! I'll switch the remaining DF actions and then happy to do the same with Comet. I'll also write a guide for transitioning :) |
thanks and FYI created apache/datafusion-comet#3404 |
|
FYI, noticed that for some reason when you rerun the action, it doesn't get picked up by the new runners 🤔 One important implication of that is that people who need their workflow to be approved won't be able to run the CI (since the first run is auto-cancelled as unapproved). ASF Infra team will have a debugging session with the Runs-on team Monday evening, and I also submitted a GitHub Support ticket. In the meanwhile, if that's blocking anyone, feel free to merge this #20170 and I'll roll it back hopefully on Monday 🙂 |
|
Found the issue! We had a very low on-demand quota on the account. We were bumping the spot quota, but reruns use on-demand, which is why they weren't managing to start. Submitted a request - hopefully everything should work fine today, and I'll switch over more workers |
|
https://github.com/apache/datafusion/actions/runs/21847381185/job/63104381656 reruns are working now 🙏 |
6 participants
Which issue does this PR close?
Related to #13813
Thanks to the infra team and @gmcdonald specifically, we now have the ability to use more powerful AWS-provided runners in our CI 🥳
DataFusion has one of the largest runtimes across Apache projects - that's why we're bringing those runners here first. Since we're first to test this, I think it's reasonable to do a gradual transition, so I updated the two most frequently failing actions to be hosted in AWS. The plan is to test that everything works fine and then transition the remaining actions.
What changes are included in this PR?
If the org is
apache, we'll now use ASF-provisioned runners in the ASF infra AWS account. Forks will not have access to those runners, so they will fall back to GitHub-provisioned ones.Are these changes tested?
Yes.
linux-build-libandlinux-testblaginin/datafusion#10Are there any user-facing changes?
No