[core] fix group offloading when using torchao#13276

sayakpaul wants to merge 8 commits intomainhuggingface/diffusers:mainfrom

fix-torchao-groupoffloadinghuggingface/diffusers:fix-torchao-groupoffloadingCopy head branch name to clipboard