Tutorials

A comprehensive guide explains how to fine-tune 3D generation models.

Hugging Face scripts and Docker images let users fine-tune TRELLIS.2 on custom 3D assets, improving object geometry via low-rank adaptation.

A comprehensive guide explains how to fine-tune 3D generation models.

Key Takeaways

  • Hugging Face released scripts and container images to fine-tune TRELLIS.2, a popular 3D generation model, on custom 3D assets using either local Docker or Hugging Face Jobs.
  • The guide addresses a common problem in 3D generation: pretrained models may produce visually plausible objects but fail on precise category-specific geometry, such as shoe soles or furniture shapes.
  • The recommended approach uses low-rank adaptation (LoRA) to train only selected stages of the TRELLIS.2 pipeline, especially coarse structure and detailed geometry, while keeping material generation frozen.
  • LoRA is well suited for small datasets because it requires far fewer trainable parameters and less memory than full fine-tuning, while preserving the base model’s existing strengths.
  • The article demonstrates the workflow with 188 footwear objects from Google Scanned Objects, showing how targeted fine-tuning can improve category-specific shape accuracy without rebuilding the full 3D generation model.

A Hugging Face blog has published scripts and container images for fine-tuning TRELLIS.2, a popular 3D generation model, on a user’s own 3D assets. The tools are designed to run either on a local machine with Docker or on Hugging Face Jobs, a cloud service for containerized workloads. The article is useful for people who need a pretrained 3D model to behave better for one object category, such as footwear or furniture, without rebuilding the whole model.

A comprehensive guide explains how to fine-tune 3D generation models.

Why standard 3D generation is hard to control

3D generation matters in robotics, simulation, media, and entertainment. These fields often need generated objects to follow strict rules and use consistent representations. The blog notes that many file formats exist, including GLB and PLY, and many model representations exist, including neural radiance fields and 3D Gaussian splatting. Each has trade-offs.

TRELLIS introduced a unified latent representation, a compressed internal form that can be decoded into different 3D output formats. TRELLIS.2 improved the model’s representation of 3D structure. The missing piece, according to the blog, is a simple way to adapt either model to a specific task.

A footwear experiment shows the problem

The example is narrow: the team used a subset of the Google Scanned Objects dataset containing 188 footwear objects. In initial tests, the pretrained TRELLIS.2 model produced shoes that looked generally correct, but the soles were often wrong. The team wanted to test whether low-rank adaptation, a method that trains small adapter matrices while keeping original weights frozen, could improve sole geometry.

That example matters because a general-purpose 3D model may be good enough for visual plausibility but not for a category where exact shape is important. The blog says users may need accurate geometry for furniture models, and a pretrained model may produce insufficiently accurate geometry for that task.

The pipeline separates structure, detailed geometry, and materials. The team’s plan is to fine-tune only the components important for the task, and freeze the others. Choosing what to freeze is described as just as important as choosing what to train.

Which TRELLIS.2 stages are changed

TRELLIS.2 has five main stages: remove the image background, encode the image, predict the coarse structure, generate detailed geometry, and generate materials. The pipeline exports the result as a textured mesh.

BiRefNet removes the background from the input image. DINOv3-L then encodes the image into feature tokens that guide the 3D object.

The first trainable stage is the sparse-structure flow diffusion transformer, a transformer model used in diffusion-based generation. It predicts the object’s coarse shape as a latent of size 16³ × 8. A variational autoencoder decoder, a type of encoder-decoder model, changes that latent into a 64 × 64 × 64 occupancy grid. Each voxel in the grid marks whether that location is part of the object; a voxel is a pixel in three dimensions. The blog adds low-rank adapters here because this output guides the later geometry stage.

A comprehensive guide explains how to fine-tune 3D generation models.

TRELLIS.2 stores 3D objects in O-Voxel, a sparse voxel grid that keeps only voxels on the object’s surface. Each voxel stores local geometry, such as a mesh vertex, and material attributes. The source says a CPU can change a mesh into O-Voxel in seconds, and a GPU can convert O-Voxel back into a mesh in seconds.

Structured sparse latents from TRELLIS keep 3D data in sparse tensors, which makes computation faster. The blog says the model uses only approximately 10% of the voxels in the grid, so it does not spend compute on empty voxels. The shape flow and texture flow diffusion transformers are sparse transformers, while the sparse-structure flow diffusion transformer is dense.

The geometry stage uses the shape flow diffusion transformer. It takes the occupancy grid and DINOv3-L feature tokens to create a latent, which a variational autoencoder decoder converts into O-Voxel geometry at 512 × 512 × 512 resolution. The team adds low-rank adapters to this stage because it controls the object’s shape.

The material stage is frozen. The texture flow diffusion transformer uses the shape latent and DINOv3-L features to define surface materials and textures. A variational autoencoder decoder turns the latent into base color, metallic, roughness, and alpha values for each voxel. The blog says the dataset has diffuse color maps but lacks the separate material maps needed to train this stage. It also notes that the problem is shoe shape, and the base model already makes good colors and textures.

Finally, the pipeline combines geometry and material attributes into a textured mesh and exports it as a GLB file.

Why low-rank adaptation fits a small dataset

The Google Scanned Objects subset has 188 shoes out of 1,030 objects. For a dataset this small, the blog says full fine-tuning costs much more and can overwrite what the model already does well. Low-rank adaptation reduces the number of trainable parameters and the memory needed for their gradients and optimizer state.

The source uses a Hugging Face parameter-efficient fine-tuning library to add and train the adapters. Because TRELLIS.2 has a custom forward pass, the adapters are inserted directly into the model instead of relying on a standard wrapper.

The configuration trains approximately 9.3 million adapter parameters in total for the two targeted diffusion transformers. It uses rank 8 adapters in the attention and feed-forward layers of the sparse-structure flow diffusion transformer, and rank 16 adapters in the attention layers of the shape flow diffusion transformer. The rank sets the size of the adapter matrices; a higher rank lets the adapter learn more.

The training setup also limits data exposure. Of the 188 objects, 9 are held out for evaluation and 179 are used for training. Blender renders 16 views of each object, and each training sample uses one random view as the conditioning image. A batch size of 4 with 4 gradient accumulation steps gives an effective batch size of 16.

A comprehensive guide explains how to fine-tune 3D generation models.

The source keeps trainable adapter weights in 32-bit floating point for numerical stability and uses bfloat16 autocasting for the forward pass. Autocasting lets PyTorch use lower precision for some operations, which decreases memory use and does not change how the frozen backbone weights are stored.

How to run the pipeline

The blog supplies two Docker images: one for preprocessing and one for fine-tuning. They can be run on Hugging Face Jobs or locally. Before starting, users need a Hugging Face access token, a Hugging Face account that can access DINOv3 because it is a gated model, and uploaded meshes on Hugging Face Hub if they use their own dataset.

The preprocessing step converts GLB or OBJ meshes into the latent representations TRELLIS.2 uses for training and renders input images. TRELLIS.2’s frozen encoders convert the input meshes into training latents, and the blog says these latents are saved so later runs do not have to encode the meshes again. The preprocessed dataset is also saved so training runs can reuse it.

For Hugging Face Jobs, the preprocessing container renders the views and processes objects through the pipeline. For the 188 footwear objects, the full preprocessing run took approximately 1 hour on an a100-large job. A compact version of the preprocessing pipeline is also available in a companion repository.

The training container loads the prepared dataset and a configuration preset or file, and records training metrics with Trackio. It supports two modes: one for the sparse-structure flow diffusion transformer and one for the shape flow diffusion transformer. Under the hood, these modes reference different configuration files for each stage. If the --push-to option is set, the container uploads adapter checkpoints to Hugging Face Hub.

The same commands and flags are used locally with Docker, except for the Docker-specific options. The trained low-rank adapters are available in the repository suvadityamuk/trellis-2-finetune-gso.

What readers should watch next

The blog provides a practical path: convert meshes into training latents, render views, train adapters for the shape-related stages, and reuse the saved dataset. The most useful takeaway is the split between what changes and what stays fixed. For shape problems, the article points to the sparse-structure flow and shape flow stages. For material problems, it warns that mixed diffuse color maps can cause the model to learn lighting as surface color, and can risk catastrophic forgetting.

That distinction can help users decide whether to fine-tune a 3D model at all, and where to place the training effort. The footwear test case also gives a concrete way to check results: compare generated objects against the 9 held-out objects, and look closely at the details that matter most for the task, such as shoe soles.

Comments (0)

  1. No comments yet. Be the first to share what worked for you.

Leave a comment

Comments are reviewed before they appear. Your email address is not published.