Agentic VLMs for Procedural Material Generation
May 28, 2026
UCSD Undergraduate Engineering Research Symposium · Early Research Scholars Program
I worked on this year-long project in Prof. Manmohan Chandraker’s lab as part of a team of four undergraduates, mentored by a PhD student and a master’s student. We studied how to generate procedural materials in Blender from expert tutorial videos. The baseline pipeline can lose detail at two stages: merging tutorials in a single pass and condensing the combined procedure before generating code. We redesigned both stages, replacing the one-shot merge with iterative procedure composition and the single-pass compiler with progressive compilation through an Orchestrator–Generator–Verifier workflow.
I focused on inference-time scaling, designing the multi-step decomposition and step-level LLM verification between stages. I also ran an ablation study across four pipeline variants to evaluate the trade-off between quality and cost. We measured CLIP similarity, style loss, sliced Wasserstein distance, and graph edit distance, along with cost per run.
- The integrated pipeline achieved the highest average CLIP similarity of the four variants, scoring 0.847 compared with the one-shot baseline’s 0.831. It was also the only variant to outperform the baseline on every tested material.
- Redesigning only the encoding stage increased average CLIP similarity to 0.839 while reducing cost per run from $3.48 to $3.15—about 9% less. The full integrated pipeline cost $5.10 per run.
- The improvements varied by material: the encoding-only variant increased CLIP similarity over the baseline by 0.033 for steel and 0.035 for pineapple but reduced it by 0.043 for mahogany.
- Pineapple remained the most challenging material, with the lowest CLIP similarity across all four variants.
