LegoGPT AI model turns text prompts into Lego designs

A team of computer scientists at Carnegie Mellon University has created LegoGPT, an AI model that can build real-life Lego structures from a single text prompt.

The team, led by Ava Pun, a PhD student at Carnegie Mellon University (CMU) and one of the study's coauthors, decided to train the AI model on Lego bricks because they are widely available and results can be easily reproduced in different labs.

However, the underlying idea for LegoGPT could have real-world applications, such as helping architects design structurally sound buildings or designers create custom furniture.

"We wanted to bring user imagination into the physical world," Pun told Dezeen. "Imagine if you could type in a prompt or show us a picture of a chair, and boom – we could actually make that product and ship it to you in a week or two."

Today, generative AI platforms like Midjourney and Dall-E 3 can generate fantastical castles or "aesthetically pleasing 3D objects" said Pun, but in order to build these designs, you would need to understand structural engineering and physics.

"The model does not know how things can be made in the real world, like what makes something stable or how parts fit together," Pun said.

To teach the model how to build Lego structures that won't topple over, the researchers trained it on a dataset of over 47,000 Lego structures and 28,000 unique 3D shapes. According to the team's research, the resulting model is capable of generating physically stable designs more than 98 per cent of the time.

Pun and the team fed in text prompts such as "straight-backed chair with square seat", which was then converted into a buildable Lego design.

The researchers focused on Lego bricks because they are relatable and familiar.

"As a child, I loved making Lego stop-motion videos with my brother," said Pun, adding that the team believes the model can help with other manufacturing tasks, as well.

"Architects could use it to draft ideas for buildings that are structurally stable, or people with specific ergonomic needs could use it to design custom furniture using a predefined set of parts," she said.

The team used robotic arms that respond to the digitally generated design in order to create the real-life Lego structures, but the structures can also be made by hand.

Since launching, LegoGPT has been used to generate around 10,000 Lego structures. The current version was not designed directly for amateur Lego builders (for example, there are no step-by-step instructions). But if there is enough continued demand, Pun said the team would be happy to improve the user experience.

For now, the objects you can create with are limited to 21 categories (including chairs, sofas, trains, and cars), but the researchers are planning to expand the number of categories to include a broader range of objects, dimensions, and brick types.

The team also used the model to generate several "textured" Lego designs from text prompts.

Since the code is available on Github, Pun said that anyone with an understanding of code can customise it to improve the UX.

"AI is a tool, and it is up to us humans to use it responsibly. We see our project being useful for brainstorming, inspiring new ideas, and sketching out initial designs super fast,” she said. "But it is not a replacement for human ingenuity, and it definitely needs a lot more refinement and human creativity to get the final model looking amazing."

Generative AI models such as ChatGPT have taken on a growing role in architecture.

Architect Tim Fu recently noted AI as a collaborator in a housing development design in Slovenia.

The post LegoGPT AI model turns text prompts into Lego designs appeared first on Dezeen.