Integrated per-frame camera exposure, gain, and white-balance metadata into NeRF and 3DGS training loops to correct for photometric inconsistencies across multi-exposure captures. Implemented learned per-image appearance embeddings and HDR-aware loss terms that suppress floater artifacts caused by exposure-bracketed sequences. Applied to XR headset capture pipelines where auto-exposure creates large inter-frame intensity swings, significantly improving mesh extraction quality evaluated against LiDAR ground truth.

Integrated CLIP and DINOv2 features into live XR headset reconstructions, enabling natural-language spatial queries — text prompt → highlighted 3D region — without additional annotations or fine-tuning. Feature vectors were baked directly into reconstructed mesh vertices for on-device query latency. Demonstrated on a live XR headset demo with voice-based queries over pre-reconstructed indoor scenes.

Designed and implemented a block-wise NeRF pipeline for reconstructing large multi-room indoor scenes that exceed the capacity of a single neural radiance field model. The scene was partitioned into spatially overlapping blocks with shared margin regions to ensure seamless boundary continuity at block interfaces. Each block was independently optimized; the resulting per-block meshes were stitched into a unified watertight surface mesh using a boundary-aware merging strategy. Targeted XR headset capture scenarios with RGB-D input and LiDAR-based quality evaluation across scenes spanning tens of meters.

Led a 5-person team adapting Neural SDF methods for room-scale XR scenes on RGB-D data within Nerfstudio and Sdfstudio — evaluating and extending Neuralangelo, NeuS, MonoSDF, and VolSDF. Integrated depth priors, surface normal priors, and HDR-aware priors derived from camera metadata (exposure, gain) into both NeRF and Gaussian Splatting variants (SuGaR, 2DGS, Gaussian Opacity Fields). Achieved <4 cm Chamfer distance against LiDAR ground truth in 60-second reconstructions of 7×7 m scenes. Also applied feed-forward 3D foundation models (DUSt3R, MASt3R, Spann3R) to live XR headset camera streams — replacing per-scene NeRF optimization with a single forward pass yielding metric-scale dense point clouds, fused with depth sensor data to fill hardware blind spots on dark or specular surfaces.

Proposed and co-authored a novel eye tracking method using event cameras (Prophesee) and coded differential illumination, achieving kilohertz-rate gaze estimation — orders of magnitude faster than frame-based trackers. The method was concurrently patented as US 11,176,367 (event-camera eye surface mapping, 12 citations) and US 11,853,473 (differential illumination for corneal glint detection, 6 citations).

Reconstructed photorealistic 3D eye surface geometry — iris, sclera, and the transparent, refractive cornea — from multi-camera near-infrared light-dome captures using NeRF. The corneal surface is invisible to standard photogrammetry; specialized modeling of its specular and refractive properties recovered the 3D geometry that generates NIR glints. The resulting eye model served as the substrate for all downstream sensor simulation work. Former colleagues subsequently published this pipeline as arXiv:2503.16742 (Lin et al., 2025); the work is now fully public and describes the infrastructure built during my Meta tenure prior to my departure.

Built a sensor simulation framework over reconstructed 3D eye models, enabling virtual placement of cameras, photodiodes, fringe projectors, and OCT systems at arbitrary positions and predicting their real-world response from first principles — ray-traced radiance, spectral sensitivity, shot noise, and dark current. Supported systematic hardware design-space exploration: evaluating dozens of LED arrangements and camera placements in simulation before committing to physical manufacture. Calibrated simulation-to-real gaps via bundle adjustment over anthropomorphic phantoms in Ceres Solver, jointly optimizing sensor parameters and 3D geometry.

Implemented the full LiDAR–camera tightly-coupled fusion pipeline from the PhD thesis as a live real-time demo running on NVIDIA Jetson TK1 — demonstrating that the measurement-level fusion algorithm is feasible on constrained embedded hardware. Built during a Volkswagen ERL internship as a direct continuation of the UCSC dissertation work. Included CUDA-accelerated point cloud processing, feature tracking, and on-device visualization of fused depth and velocity estimates.

Extended the tightly-coupled LiDAR–camera fusion framework from velocity estimation to full 3D shape recovery of moving objects. The joint formulation simultaneously optimizes ego-motion, object trajectory, and surface geometry — recovering metric 3D structure of vehicles in traffic from a single moving observer. Implemented using SfM, SLAM, and bundle adjustment pipelines. Published at IEEE Intelligent Transportation Systems Conference 2017; research sponsored by Volkswagen ERL.

Core PhD contribution: tightly-coupled (measurement-level) fusion of spinning LiDAR and monocular camera for simultaneous velocity and 3D shape estimation of moving vehicles in autonomous driving scenes. Unlike loosely-coupled approaches that fuse detections, this method fuses raw LiDAR point correspondences and image feature tracks jointly in a bundle-adjustment framework. Achieved accurate velocity estimation without requiring explicit dynamic object segmentation. Published at IEEE Intelligent Vehicles Symposium 2017; research sponsored by Volkswagen ERL.

Developed a learning-based approach to the LiDAR–camera extrinsic calibration problem using a Siamese neural network that learns a joint embedding of LiDAR reflectance and camera intensity patches. The network regresses the 6-DOF rigid transform between sensor frames, eliminating the need for manual correspondences or calibration targets. Outperformed classical checkerboard-based calibration in robustness to partial observations and varying scene structure.

Designed and built the hardware and software integration pipeline for the Valeo LiDAR sensor into a Volkswagen test vehicle. Designed a custom CAD model for the sensor mount and integration bracket. Built the data streaming pipeline synchronizing LiDAR point cloud acquisition with in-vehicle measurement buses (CAN), enabling time-aligned LiDAR–camera–CAN capture for autonomous driving research datasets.

Co-invented and implemented a camera-based lane detection system for the early identification of exit-only and shared lanes from a moving vehicle. Used monocular camera imagery to classify lane types robustly under varied lighting and occlusion conditions. Work issued as US Patent 10,025,996 (23 citations).

Master's thesis on estimating accurate optical flow in the presence of spatially-varying motion blur — a regime where standard flow methods fail due to the non-uniform point-spread function introduced by per-pixel motion during exposure. Developed a blur-aware flow estimator and published at the 10th International Symposium on Visual Computing (ISVC 2014). Advised by Peyman Milanfar (UCSC / Google Research).