Question about the Element Type Constraint for LayoutPrompter

[Jackieam]

Thanks for sharing such an awesome research!
I have some questions about LayoutPrompter on the content aware layout generation task setting.
The code uses randomly selected labels (Element Type Constraint) on the test set as a query generation restriction (labels = random.choice(self.possible_labels)), but I tried it on GPT-4 API and could not achieve the results reported in the paper (That is, given different labels on the test set, the results are different.).
So I would like to know how the results reported in the paper generate the test set labels (or Element Type Constraint, like logo | text | text | text | underlay | text) that guide the LLM? Hope you response!

[kaoshizhou]

Hi,

Thanks for your interest in our work. Actually, the test set labels are obtained in the following way:

• We use ContentAwareProcessor to process all training samples. To achieve this, split is set to train. When processing training samples, training set labels are recorded in possible_labels.
• After processing training samples, split is set to test to produce test samples. Since PosterLayout does not include test labels, we randomly sample possible labels from training data (i.e., labels = random.choice(self.possible_labels)). And we use them to guide LLMs.

Hope this can help you, and feel free to ask me if you have any further questions.

[Jackieam]

Thank you very much for your reply!
I have another question regarding the underlay metrics. If the content constraint does not include the underlay element, should this sample be excluded when calculating the und_l and und_s values?

[kaoshizhou]

I think the answer is no. The two concepts are further clarified below:

• content constraint: It is a bounding box (the blue box in the figure below) represented by 4 coordinates (please refer to the code). To be more specific, it is obtained from the saliency map of the input canvas. For better readability, LayoutPrompter is prompted to place elements avoiding this area (code).
  

• underlay: It is one of the elements to be placed on the canvas, and often regarded as the background of the other elements, such as text and logo. Please refer to the figure for better understanding.
  

To summarize, the two are completely independent concepts. We didn't take the operation you mentioned. We just used the same evaluation metrics as in DSGAN.