Skip to main content
SpringerLink
Log in

GTLayout: Learning General Trees for Structured Grid Layout Generation

  • Conference paper
  • First Online:
Computational Visual Media (CVM 2024)

Part of the book series: Lecture Notes in Computer Science ((LNCS,volume 14593))

Included in the following conference series:

  • 112 Accesses

Abstract

Structured grid layouts are preferable in many scenarios of 2D visual content creation since their structures facilitate further layout editing. Multiple geometry-based methods can effectively create structured grid layouts but require user-provided constraints or rules. Existing data-driven approaches have achieved remarkable performance on layout generation, but fail to produce appropriate layout structures. We present GTLayout, a novel generative model for structured grid layout generation. We adopt general trees to represent structured grid layouts and exploit a recursive neural network (RvNN) for this generation task. Our model can handle grid layouts with varied structures and regular arrangements. Qualitative and quantitative experiments on public grid layout datasets show that our method outperforms several baselines in the tasks of layout reconstruction and layout generation, especially when the datasets contain a small number of samples. We also demonstrate that the structured layout space constructed by our method enables structure blending between structured layouts. We will release our code upon the acceptance of the paper.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Institutional subscriptions

References

  1. Arroyo, D.M., Postels, J., Tombari, F.: Variational transformer networks for layout generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13642–13652 (2021)

    Google Scholar 

  2. Baudisch, P., Cutrell, E., Hinckley, K., Eversole, A.: Snap-and-go: helping users align objects without the modality of traditional snapping. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 301–310 (2005)

    Google Scholar 

  3. Bier, E.A., Stone, M.C.: Snap-dragging. ACM SIGGRAPH. Comput. Graph. 20(4), 233–240 (1986)

    Article  Google Scholar 

  4. Chai, S., Zhuang, L., Yan, F.: Layoutdm: transformer-based diffusion model for layout generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18349–18358 (2023)

    Google Scholar 

  5. Dayama, N.R., Todi, K., Saarelainen, T., Oulasvirta, A.: Grids: interactive layout design with integer programming. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020)

    Google Scholar 

  6. Deka, B., et al.: Rico: a mobile app dataset for building data-driven design applications. In: Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pp. 845–854 (2017)

    Google Scholar 

  7. Dixon, M., Leventhal, D., Fogarty, J.: Content and hierarchy in pixel-based methods for reverse engineering interface structure. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 969–978 (2011)

    Google Scholar 

  8. Frisch, M., Kleinau, S., Langner, R., Dachselt, R.: Grids & guides: multi-touch layout and alignment tools. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1615–1618 (2011)

    Google Scholar 

  9. Goodfellow, I., et al.: Generative adversarial nets. In: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N., Weinberger, K. (eds.) Advances in Neural Information Processing Systems, vol. 27. Curran Associates, Inc. (2014). https://proceedings.neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf

  10. Gupta, K., Lazarow, J., Achille, A., Davis, L.S., Mahadevan, V., Shrivastava, A.: Layouttransformer: layout generation and completion with self-attention. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1004–1014 (2021)

    Google Scholar 

  11. Ho, J., Jain, A., Abbeel, P.: Denoising diffusion probabilistic models. Adv. Neural. Inf. Process. Syst. 33, 6840–6851 (2020)

    Google Scholar 

  12. Hui, M., Zhang, Z., Zhang, X., Xie, W., Wang, Y., Lu, Y.: Unifying layout generation with a decoupled diffusion model. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1942–1951 (2023)

    Google Scholar 

  13. Inoue, N., Kikuchi, K., Simo-Serra, E., Otani, M., Yamaguchi, K.: LayoutDM: discrete diffusion model for controllable layout generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10167–10176 (2023)

    Google Scholar 

  14. Jiang, Y., Du, R., Lutteroth, C., Stuerzlinger, W.: Orc layout: adaptive GUI layout with or-constraints. In: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–12 (2019)

    Google Scholar 

  15. Jiang, Y., Stuerzlinger, W., Lutteroth, C.: Reverseorc: reverse engineering of resizable user interface layouts with or-constraints. In: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–18 (2021)

    Google Scholar 

  16. Jiang, Z., et al.: Layoutformer++: conditional graphic layout generation via constraint serialization and decoding space restriction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18403–18412 (2023)

    Google Scholar 

  17. Jiang, Z., Sun, S., Zhu, J., Lou, J.G., Zhang, D.: Coarse-to-fine generative modeling for graphic layouts. In: AAAI 2022 (2022)

    Google Scholar 

  18. Jyothi, A.A., Durand, T., He, J., Sigal, L., Mori, G.: Layoutvae: stochastic scene layout generation from a label set. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9895–9904 (2019)

    Google Scholar 

  19. Kenton, J.D.M.W.C., Toutanova, L.K.: Bert: pre-training of deep bidirectional transformers for language understanding. In: Proceedings of NAACL-HLT, pp. 4171–4186 (2019)

    Google Scholar 

  20. Kikuchi, K., Otani, M., Yamaguchi, K., Simo-Serra, E.: Modeling visual containment for web page layout optimization. In: Computer Graphics Forum, vol. 40, pp. 33–44. Wiley Online Library (2021)

    Google Scholar 

  21. Kikuchi, K., Simo-Serra, E., Otani, M., Yamaguchi, K.: Constrained graphic layout generation via latent optimization. In: Proceedings of the 29th ACM International Conference on Multimedia, pp. 88–96 (2021)

    Google Scholar 

  22. Kingma, D.P., Ba, J.: Adam: a method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, 7–9 May 2015, Conference Track Proceedings (2015). http://arxiv.org/abs/1412.6980

  23. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (2013)

    Google Scholar 

  24. Kong, X., et al.: BLT: bidirectional layout transformer for controllable layout generation. In: Avidan, S., Brostow, G., Cissé, M., Farinella, G.M., Hassner, T. (eds.) Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, 23–27 October 2022, Proceedings, Part XVII, pp. 474–490. Springer, Cham (2022). https://doi.org/10.1007/978-3-031-19790-1_29

  25. Kuhn, H.W.: The Hungarian method for the assignment problem. Naval Res. Logist. Quart. 2(1–2), 83–97 (1955)

    Article  MathSciNet  Google Scholar 

  26. Lee, H.-Y., et al.: Neural design network: graphic layout generation with constraints. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, 23–28 August 2020, Proceedings, Part III, pp. 491–506. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58580-8_29

  27. Li, J., Yang, J., Hertzmann, A., Zhang, J., Xu, T.: Layoutgan: generating graphic layouts with wireframe discriminators. arXiv preprint arXiv:1901.06767 (2019)

  28. Li, J., Xu, K., Chaudhuri, S., Yumer, E., Zhang, H., Guibas, L.: Grass: generative recursive autoencoders for shape structures. ACM Trans. Graph. 36(4), 1–14 (2017)

    Google Scholar 

  29. Li, M., et al.: Grains: generative recursive autoencoders for indoor scenes. ACM Trans. Graph. 38(2), 1–16 (2019)

    Google Scholar 

  30. Lupton, E.: Thinking with Type: A Critical Guide for Designers, Writers, Editors, & Students. Chronicle Books (2014)

    Google Scholar 

  31. Manandhar, D., Ruta, D., Collomosse, J.: Learning structural similarity of user interface layouts using graph networks. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) Computer Vision – ECCV 2020: 16th European Conference, Glasgow, 23–28 August 2020, Proceedings, Part XXII, pp. 730–746. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58542-6_44

  32. Mo, K., et al.: Structurenet: hierarchical graph networks for 3d shape generation. arXiv preprint arXiv:1908.00575 (2019)

  33. Müller-Brockmann, J.: Grid systems in graphic design: a visual communication manual for graphic designers, typographers and three dimensional designers. Arthur Niggli (1996)

    Google Scholar 

  34. O’Donovan, P., Agarwala, A., Hertzmann, A.: Learning layouts for single-page graphic designs. IEEE Trans. Visual Comput. Graph. 20(8), 1200–1213 (2014)

    Article  Google Scholar 

  35. Patil, A.G., Ben-Eliezer, O., Perel, O., Averbuch-Elor, H.: Read: recursive autoencoders for document layout generation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 544–545 (2020)

    Google Scholar 

  36. Patil, A.G., Li, M., Fisher, M., Savva, M., Zhang, H.: Layoutgmn: neural graph matching for structural layout similarity. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11048–11057 (2021)

    Google Scholar 

  37. Raisamo, R., Räihä, K.J.: A new direct manipulation technique for aligning objects in drawing programs. In: Proceedings of the 9th Annual ACM Symposium on User Interface Software and Technology, pp. 157–164 (1996)

    Google Scholar 

  38. Scarselli, F., Gori, M., Tsoi, A.C., Hagenbuchner, M., Monfardini, G.: The graph neural network model. IEEE Trans. Neural Networks 20(1), 61–80 (2008)

    Article  Google Scholar 

  39. Socher, R., Lin, C.C., Manning, C., Ng, A.Y.: Parsing natural scenes and natural language with recursive neural networks. In: Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 129–136 (2011)

    Google Scholar 

  40. Swearngin, A., Wang, C., Oleson, A., Fogarty, J., Ko, A.J.: Scout: rapid exploration of interface layout alternatives through high-level design constraints. In: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pp. 1–13 (2020)

    Google Scholar 

  41. Vaswani, A., et al.: Attention is all you need. In: Guyon, I., et al. (eds.) Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf

  42. Xu, P., Fu, H., Tai, C.L., Igarashi, T.: GACA: group-aware command-based arrangement of graphic elements. In: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, pp. 2787–2795 (2015)

    Google Scholar 

  43. Xu, P., Li, Y., Yang, Z., Shi, W., Fu, H., Huang, H.: Hierarchical layout blending with recursive optimal correspondence. ACM Trans. Graph. (Proc. SIGGRAPH ASIA) 41(6), 249:1–249:15 (2022)

    Google Scholar 

  44. Xu, P., Yan, G., Fu, H., Igarashi, T., Tai, C.L., Huang, H.: Global beautification of 2d and 3d layouts with interactive ambiguity resolution. IEEE Trans. Visual Comput. Graph. 27(4), 2355–2368 (2019)

    Article  Google Scholar 

  45. Yamaguchi, K.: Canvasvae: learning to generate vector graphic documents. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5481–5489 (2021)

    Google Scholar 

  46. Yang, Y.L., Wang, J., Vouga, E., Wonka, P.: Urban pattern: layout design by hierarchical domain splitting. ACM Trans. Graph. 32(6), 1–12 (2013)

    Article  Google Scholar 

  47. Zeidler, C., Lutteroth, C., Sturzlinger, W., Weber, G.: The Auckland layout editor: an improved GUI layout specification process. In: Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology, pp. 343–352 (2013)

    Google Scholar 

  48. Zheng, X., Qiao, X., Cao, Y., Lau, R.W.: Content-aware generative modeling of graphic design layouts. ACM Trans. Graph. 38(4), 1–15 (2019)

    Article  Google Scholar 

  49. Zhong, X., Tang, J., Yepes, A.J.: Publaynet: largest dataset ever for document layout analysis. In: 2019 International Conference on Document Analysis and Recognition (ICDAR), pp. 1015–1022. IEEE (2019)

    Google Scholar 

  50. Zhu, C., Xu, K., Chaudhuri, S., Yi, R., Zhang, H.: Scores: shape composition with recursive substructure priors. ACM Trans. Graph. 37(6), 1–14 (2018)

    Article  Google Scholar 

Download references

Acknowledgments

This work was supported in parts by NSFC (62072316, U21B2023), NSF of Guangdong Province (2023A1515011297), DEGP Innovation Team (2022KCXTD025), Shenzhen Science and Technology Program (KQTD20210811090044003), and Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ).

Disclosure of Interests. The authors have no competing interests to declare that are relevant to the content of this article.

Author information

Authors and Affiliations

  1. Shenzhen University, Shenzhen, China

    Pengfei Xu, Weiran Shi, Xin Hu & Hui Huang

  2. City University of Hong Kong, Hong Kong, China

    Hongbo Fu

Authors
  1. Pengfei Xu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Weiran Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Xin Hu
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hongbo Fu
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Hui Huang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Pengfei Xu .

Editor information

Editors and Affiliations

  1. Victoria University of Wellington, Wellington, New Zealand

    Fang-Lue Zhang

  2. Ben-Gurion University, Be'er Sheva, Israel

    Andrei Sharf

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Singapore Pte Ltd.

About this paper

Check for updates. Verify currency and authenticity via CrossMark

Cite this paper

Xu, P., Shi, W., Hu, X., Fu, H., Huang, H. (2024). GTLayout: Learning General Trees for Structured Grid Layout Generation. In: Zhang, FL., Sharf, A. (eds) Computational Visual Media. CVM 2024. Lecture Notes in Computer Science, vol 14593. Springer, Singapore. https://doi.org/10.1007/978-981-97-2092-7_7

Download citation

  • DOI: https://doi.org/10.1007/978-981-97-2092-7_7

  • Published:

  • Publisher Name: Springer, Singapore

  • Print ISBN: 978-981-97-2091-0

  • Online ISBN: 978-981-97-2092-7

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation