Research Scientist Intern, Low-Power Scene Reconstruction for AR/VR

Job description

Future AR and VR devices occupy ever-smaller, glasses-like form factors and thus require ever-tighter power budgets. At the same time, we require higher-quality passthrough rendering and scene understanding. In particular, depth estimation, which reconstructs scene geometry in real time from sensor signals, is one of the biggest power consumers in AR and VR devices. However, most existing work targets datacenter GPUs, or at best mobile silicon with an order of magnitude more power headroom than what glasses form factors allow. Reality Labs is Meta's innovation engine for next-generation AR/VR and wearable technology and XR Graphics is involved in building the next-generation imaging and reconstruction stack that closes this gap. We are looking for a Research Scientist Intern to work on low-power real-time reconstruction of scene geometry for AR/VR. Today, VR passthrough, hand interaction, and environment understanding run on separate pipelines, each tuned for different accuracy, latency and power target. We are interested in deduplicating the compute (and thus reducing power): - What is the right depth estimation and processing strategy, from the ground up, for a glasses power budget? - Can we train a single lightweight scene representation that can serve passthrough, hand occlusions, and background occlusions simultaneously? These are open questions, and you would help define the approach. This is a research internship. You will own a genuinely open problem, choose the methods, build the evaluation, and present your results, with support for publication where the work warrants it.