Tool Reviews

New tool enables users to fix AI-generated 3D models and fabricate them exactly as they wish

MIT CSAIL and partners introduced InstructMesh, combining TRELLIS and GPT-4 to fix AI 3D models for practical printing.

New tool enables users to fix AI-generated 3D models and fabricate them exactly as they wish

Key Points

  • InstructMesh is an AI design tool from MIT CSAIL, Google, and Northeastern that combines TRELLIS 3D generation with GPT-4 to help users edit and fix AI-generated 3D models before printing.
  • It addresses a common failure of generative 3D printing: models may look correct visually but be structurally impractical, such as mugs that cannot hold coffee or robot shells that cannot function.
  • The interface lets novices revise models using natural language, image cues, and simple geometric editing, without requiring specialized 3D modeling or print-preparation knowledge.
  • In tests, nearly 80% of TRELLIS-recreated Thingiverse models had structural flaws, while novice users using InstructMesh successfully identified and corrected defects about 90% of the time.
  • Future work may integrate InstructMesh with AR, physics simulation, and newer TRELLIS systems to generate and fabricate customized, usable objects more quickly.

Meta description: InstructMesh combines TRELLIS and GPT-4 so novices can edit AI-generated 3D designs and fix structural flaws before printing household items or robots.

New tool enables users to fix AI-generated 3D models and fabricate them exactly as they wish

Tags: InstructMesh, TRELLIS, GPT-4, 3D printing, CSAIL

Featured image: Abstract, friendly illustration of a 3D mesh being edited through prompts, with no real people, logos, or text.

Researchers from MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), Google, and Northeastern University have introduced InstructMesh, an AI-driven design interface that lets people generate and edit 3D-printable objects before fabrication. For users who want household items, accessories, or robot shells that function as intended, the tool combines text, image, and geometry editing, with the work scheduled for presentation at the ACM Symposium on User Interface Software and Technology in November.

Why AI 3D printing often fails in practice

Many software tools follow a “what you see is what you get” rule, but generative systems for 3D printing can break it. The models can make an object that appears correct in a render while ignoring basic requirements such as whether a mug can contain coffee. The problem is not only visual: the output may be impractical in ways that undermine the item’s intended use. For people new to 3D design, fixing those defects can be difficult because conventional modeling software expects specialized knowledge of geometry and print preparation.

How InstructMesh combines generation and editing

The team built InstructMesh to connect two capabilities that usually sit apart. Microsoft’s TRELLIS system is used to create 3D models from text and image prompts, while GPT-4, the large language model (LLM) that supports ChatGPT, helps interpret what a user wants changed. In practice, a person can ask for a design, such as a pair of glasses, then highlight the part of the blueprint that needs refinement before 3D printing. The interface is meant to guide both experts and novices toward edits that preserve the object’s look while making it more usable.

New tool enables users to fix AI-generated 3D models and fabricate them exactly as they wish

Faraz Faruqi, the lead author on the paper, an SM ’22 and PhD ’26 graduate of MIT’s Department of Electrical Engineering and Computer Science, and a recent CSAIL affiliate, said the project aimed to combine the strengths of 3D generators with the reasoning ability of language models so people can make objects they actually want. He also said TRELLIS contributes strong shape generation because it has encountered many 3D models, while language models add text and image understanding.

From decorative mugs to wearable and robot designs

CSAIL researchers used InstructMesh to add personalized, creative twists to ordinary items. One design was a mug that looked as if a dragon were wrapped around it, with the dragon’s tail forming the handle. Another was a shiny blue whistle shaped like a shell. They also created glasses whose frame extended into butterfly wings above the lenses, and an octopus-like dispenser that could send liquid from each tentacle into several cups at once.

The approach also moved into less obvious applications. The team made a knee brace with a denim-like surface intended to match a patient’s jeans. They used InstructMesh to build enclosures for wireless components, including a “bristle bot” that resembles a colorful shrimp. The bot hides a motor inside; when switched on, it can slide across surfaces, behaving somewhat like a wind-up toy.

Testing flaws and user results

Before testing outside users, the researchers reported that InstructMesh could readily produce their own desired items. To understand whether the system could help ordinary users, they examined how TRELLIS handled existing designs. They asked it to recreate popular models from Thingiverse, a platform containing millions of 3D printable models. Nearly 80 percent of the recreated models were structurally flawed in some way, showing that generated geometry can look plausible but fail in real use.

CSAIL then asked novices to use InstructMesh to identify and correct those defects. The participants succeeded in both tasks around 90 percent of the time, according to expert review. The researchers concluded that people without formal modeling training could rely on visual and practical intuition when the software made editing accessible.

New tool enables users to fix AI-generated 3D models and fabricate them exactly as they wish

The tool works by operating inside the generative model’s latent space, the internal space where the system represents possible shapes. Instead of requiring users to manipulate vertices or surfaces directly, InstructMesh lets them describe problems in natural language and produces geometry changes they can review and approve. That scaffolding replaces much of the specialized 3D modeling work that previously required domain expertise.

In user sessions, participants made items such as phone stands and vases and said the process was easy to use. They found that InstructMesh enabled them to express a wide range of ideas, while sliders gave them more precision to make certain tweaks, such as enlarging or extruding a selected part of a model. In the researchers’ account, users got what they prompted for, could tweak designs where needed, and found that the printed items behaved as expected.

What comes next

Faruqi now works at Google and said he may soon integrate InstructMesh into an augmented reality (AR) platform. In that vision, a user would explain a need using the context of nearby objects, and the system would generate and rapidly print a matching item, such as a phone case that fits the style of a wallet.

The software could also add physics simulations to predict how a design would behave in use, including whether a bowl would break when dropped or which material might work best. The team also sees a possible integration with the more recent TRELLIS.2 to refine smaller features in generated models.

The paper’s senior author is Stefanie Mueller, an associate professor of electrical engineering and computer science and mechanical engineering at MIT and a member of CSAIL. Co-authors include Google researchers Ahmed Katary ’23, Fabian Manhardt, Vrushank Phadnis MEng ’13 and PhD ’20, Ruofei Du, and Federico Tombari. Northeastern University Assistant Professor Megan Hofmann is also a co-author, along with CSAIL colleagues Demircan Tas SM ’24 and SMArchS ’24, a PhD student in EECS and architecture; former visiting researcher Theresa Hradilak; Ning Zhang ’25, a graduate student in EECS; postdoc Jiaji Li; and Martin Nisser SM ’19 and PhD ’24. The work was supported in part by Google and the MIT-HPI Collaborative Research Program.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.