Learning Visuomotor Policies for Robotic Manipulation

My thesis focuses on learning visuomotor policies for robotic manipulation in unstructured environments. One of the main challenges in robotic manipulation is building robust, generalist policies that can perceive, plan, and act based on both visual and language inputs. We address this by tackling three challenges: sim-to-real transfer, 3D point cloud-based language-guided policy learning, and policy generalization.

First, we propose a data-driven approach to optimize domain randomization parameters for sim-to-real transfer, using multi-object localization as a proxy task. This method improves the transferability of visuomotor policies trained in simulation to the real world.

Next, we introduce PolarNet and 3D-LOTUS, two architectures that leverage 3D point cloud encoders and multimodal transformers to fuse language and visual inputs for robotic manipulation. These models significantly outperform 2D-based baselines across multiple tasks in the RLBench benchmark and demonstrate strong real-world performance.

Finally, we present GemBench, a new benchmark designed to evaluate policy generalization across increasing task complexity, and 3D-LOTUS++, a modular policy that integrates large language models for high-level planning and vision-language models for object grounding. This system achieves state-of-the-art results on GemBench and the real robot, demonstrating capabilities in long-horizon and diverse manipulation tasks.