What Are the Effects of Directly Using NTK Rope for Extrapolation?

[kangyiyang]

I would like to inquire about the effects of directly using NTK Rope for extrapolation. Will it experience issues such as overlapping frames and multiple objects? Additionally, how does this compare to HiFlow? Does HiFlow encounter similar challenges? Thank you for your outstanding work, and I appreciate your assistance. Thank you!

[kangyiyang]

Additionally, do you have any recommendations for the parameter settings of alphas and betas?

[Bujiazi]

Hello @kangyiyang, thank you again for your interests in HiFlow 🌹. I'll address this question from multiple perspectives:

1. Challenges of Directly Using NTK Rope for High-Resolution Generation

Directly adopting NTK Rope for high resolution generation will also lead to generation collapse, a well-synthesized semantic start point is essential, where the Initialization Alignment in HiFlow plays a crucial role.

2. Effects of Other Components in HiFlow

Equipped with NTK Rope and Initialization Alignment, you are able to obtain a high-resolution generation result. However, its structure and detail appear to be corrupted and abnormal (low-fidelity, in other words). This is because, during the high-resolution generation process, the model is unable to maintain the structure of the low-resolution results on one hand, and on the other hand, it cannot synthesize reasonable details (since this has already far exceeded its training domain). That's why we further introduce Direction Alignment and Acceleration Alignment in HiFlow. They respectively maintain the structure and details of the high-resolution results, thereby further enhancing the generation quality.

3. Recommendations for Hyperparameter Settings

Our results are robust to parameter variations. A wide range of initial values and various decay strategies can achieve good results. However, please note that guidance scales should finally decay to 0.

If you have any other questions, please feel free to reach out! 😄

[kangyiyang]

Thank you for your reply.

[kangyiyang]

Hello,
I have a couple of follow-up questions and would be very grateful for your insights.
1.Regarding training-free extrapolation, is it a prerequisite for the base model to already possess strong inference capabilities at lower resolutions? Furthermore, in your opinion, what are the primary factors that limit the extrapolation performance of Diffusion Transformers (DiT)? Could it be related to the positional encoding, where the model has not encountered larger index values during its training phase?
2.You previously mentioned that the guidance scale should decay to zero towards the end. Could you please clarify this point? Specifically, are you referring to the guidance_scale_highres parameter, or something else?
I would be immensely grateful for any clarification you can provide.

[Bujiazi]

Hello @kangyiyang. Here are my humble insights on these issues:

1. Training-free Resolution Extrapolation Depends on the Quality of the Low-Resolution Trajectories/Results

Generally speaking, training-free high-resolution generation methods need to obtain guidance signals from the low-resolution synthesis trajectories or results (such as the proposed HiFlow, as well as previous outstanding works like I-Max, DemoFusion and DiffuseHigh, please refer to their papers or code implementations for detail). Therefore, the quality of the low-resolution trajectories or results is crucial for the following high-resolution generation process.

2. Main Factors Limiting DiT's Resolution Extrapolation Capability

Admittedly, the unseen positional encoding is one of the factors limiting DiT's resolution extrapolation capability. However, based on our explorations and observations from previous studies, too low self-attention temperature and too high a ratio of image tokens to text tokens are also key factors affecting its extrapolation performance.

3. Guidance Scale Decay Strategy

For training-free generation strategies, gradually weakening the control helps reduce the gap between the generated results and the samples in the original generation domain. This approach minimizes unnecessary artifacts and other instabilities. Here we refer to the direction guidance scale alpha and the acceleration guidance scale beta.

[kangyiyang]

Thank you for your response. I'm a bit confused regarding the factors affecting extrapolation. Could you please clarify what is meant by "too low self-attention temperature" and "too high a ratio of image tokens to text tokens"?

[Bujiazi]

Hello @kangyiyang. Let me provide further explanations:

Self-Attention Temperature

We already know that the attention matrix satisfies the softmax property. As the number of image tokens increases, the size of the matrix grows larger. However, the sum of each row in the matrix remains 1 (due to the softmax constraint). Consequently, the distribution of values in the matrix becomes more uniform, lacking distinct peaks and valleys. This is what we commonly refer to as a "low-temperature" attention matrix. This uniformity can hinder the model's ability to focus on specific regions of the image, leading to generation collapse.

Ratio of Image Tokens to Text Tokens

Taking Flux as an example, when we increase the image resolution, the number of image tokens grows quadratically, while the number of text tokens remains constant (usually 512). Therefore, the fixed number of text tokens has limited influence over the increasing number of image tokens, making it difficult to support a large number of image tokens effectively. The Ratio of Image Tokens to Text Tokens refers to the ratio between the number of image tokens and the number of text tokens.

[kangyiyang]

Thank you for your reply.