LegoGPT: AI Model Turns Text into Blocky Dreams

12:05 PM | 12 May 2025

LegoGPT: AI Model Turns Text into Blocky Dreams

At last, an AI model we can really get behind: LegoGPT takes a text prompt and spits out a physically stable design.

However, before we get carried away and ask it to come up with a LEGO® Sistine Chapel, the tool can only generate designs that fit within a 20 x 20 x 20 grid using just eight basic brick types (1 x 1, 1 x 2, 1 x 4, 1 x 6, 1 x 8, 2 x 2, 2 x 4, and 2 x 6). There's no "nice part usage" here.

Lego-lization sample pics from Pun et al's paper: the first's input prompt is "Table featuring a flat rectangular surface over four evenly spaced legs" and the second is "Compact sofa with a geometric design" – click to enlarge

What you do get is a model that will take a text prompt and come up with a design that is possible to build using Lego bricks and, crucially, be physically stable.

The Carnegie Mellon University research [PDF] comes from Ava Pun, Kangle Deng, Ruixuan Liu, Deva Ramanan, Changliu Liu, and Jun-Yan Zhu.

The system works by generating a ShapeNetCore mesh from the entered prompt, then voxelizing it onto a 20 x 20 x 20 grid. "Legolization" then determines the brick layout.

"We augment each shape with multiple structural variations by randomizing the brick layout while preserving the overall shape," the team explained. Stability analysis is performed on each variation to remove anything that might fall apart.

Researcher Ava Pun told The Register: "We're dreaming of a future where making stuff becomes super personal! Imagine you just type what you want or show us a picture of a chair, and boom – we could actually make that product and ship it to you in just a week or two.

"Unfortunately, today's generative AIs cannot offer that. You can generate a cool image or a video of a chair, but the model does not know how things can be made in the real world, like what makes something stable or how parts fit together.

"To address this issue, we integrate physical laws and assembly constraints into generative models such as LLMs, enabling us to create objects that function in the real world. We explored this goal in the domain of brick assembly, as it is a widely available medium, and the results can be reproducible across different labs. We believe that our method could be applied to other manufacturing tasks. For example, users with specific ergonomic needs could use it to design custom furniture using a predefined set of parts."

While the ambition is to be applauded, and we've seen enough AI-generated imagery to know that real-world physics is all too frequently absent from what gets produced, it's a big jump from something that looks like it came from a childhood box of Lego bricks to shipping a finished product from a text prompt.

While acknowledging that "more refinement and human creativity" was needed to make models look impressive rather than blocky approximations, Pun said: "We see it being useful for inspiring new ideas and sketching out initial designs super fast. It could be a neat tool for creators to brainstorm and explore all sorts of different ideas in the early stage."

It's certainly a neat tool for brainstorming purposes, one area where AI currently shines. However, with its limited library and grid size, this first iteration is more of an example of what is possible. "In future work, we plan to expand the brick library to include a broader range of dimensions and brick types," Pun said.

"Our current system is still quite limited, as it only supports 20 x 20 x 20 dimensions, 20 object categories, and simple brick types. But we are working on expanding the system's capacity. Stay tuned."

We asked Lego what it thought of the research and a spokesperson said: "We're unable to comment at this time." ®