feat: Add Modular Pipeline for Stable Diffusion 3 (SD3)#13324
feat: Add Modular Pipeline for Stable Diffusion 3 (SD3)#13324AlanPonnachan wants to merge 3 commits intohuggingface:mainfrom
Conversation
| class TestSD3Img2ImgModularPipelineFast(ModularPipelineTesterMixin): | ||
| pipeline_class = SD3ModularPipeline | ||
| pipeline_blocks_class = SD3AutoBlocks | ||
| pretrained_model_name_or_path = "hf-internal-testing/tiny-sd3-pipe" |
There was a problem hiding this comment.
the tests currently point to hf-internal-testing/tiny-sd3-pipe. So tests are failing as of now. I think this happens since /tiny-sd3-pipe lacks modular_model_index.json.
could someone on the team push a hf-internal-testing/tiny-sd3-modular testing repository with a modular_model_index.json?
Once that infrastructure is available on the Hub, I will update the pretrained_model_name_or_path and run my tests.
There was a problem hiding this comment.
I think for now, we could first load "hf-internal-testing/tiny-sd3-pipe" in a ModularPipeline and then save and push to a repository on the Hub. We can use it throughout and PR. Once the PR is close to merge, we can transfer the repo to the "hf-internal-testing" org. Does that work?
|
@AlanPonnachan thanks for this PR! Could you also provide some test code and sample outputs? |
sayakpaul
left a comment
sayakpaul
left a comment
There was a problem hiding this comment.
Thanks for getting started on this! I left some comments (majorly on the use of guidance).
| logger = logging.get_logger(__name__) | ||
|
|
||
|
|
||
| def retrieve_timesteps( |
There was a problem hiding this comment.
If we're using the one from here
let's directly import or add a "# Copied from ..." comment.
| return timesteps, num_inference_steps | ||
|
|
||
|
|
||
| def calculate_shift( |
| return timesteps, num_inference_steps | ||
|
|
||
|
|
||
| class SD3SetTimestepsStep(ModularPipelineBlocks): |
There was a problem hiding this comment.
Let's try to follow this semantic for the rest of the blocks as well.
| class SD3SetTimestepsStep(ModularPipelineBlocks): | |
| class StableDiffusion3SetTimestepsStep(ModularPipelineBlocks): |
| @property | ||
| def inputs(self) -> list[tuple[str, Any]]: | ||
| return[ | ||
| InputParam("joint_attention_kwargs"), |
There was a problem hiding this comment.
Let's provide a type hint here.
| return_dict=False, | ||
| )[0] | ||
|
|
||
| if block_state.do_classifier_free_guidance: |
There was a problem hiding this comment.
So, the guidance bit should be implemented using our guiders components. Refer to
as an example.
|
|
||
| logger = logging.get_logger(__name__) | ||
|
|
||
| def retrieve_latents( |
| return[OutputParam(name="processed_image")] | ||
|
|
||
| @staticmethod | ||
| def check_inputs(height, width, vae_scale_factor, patch_size): |
| return prompt_embeds.to(dtype=components.text_encoder_3.dtype, device=device) | ||
|
|
||
| @staticmethod | ||
| def _get_clip_prompt_embeds(components, prompt, device, clip_skip, clip_model_index): |
There was a problem hiding this comment.
Any chance we could use
? Perhaps with "# Copied from ... with self -> components"?2 participants
What does this PR do?
This PR introduces the modular architecture for Stable Diffusion 3 (SD3), implementing both Text-to-Image (T2I) and Image-to-Image (I2I) pipelines.
Key additions:
SD3ModularPipelineandSD3AutoBlocksto the dynamic modular pipeline resolver.BlockStateTestSD3ModularPipelineFastandTestSD3Img2ImgModularPipelineFasttest suites.Related issue: #13295
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@sayakpaul @asomoza