OmniRobotHome
A Multi-Camera Home Platform for
Real-Time Human-Robot Interaction
OmniRobotHome
Human motion, in 3D.
Markerless motion capture in a furnished home.
Loading the room…
A Multi-Camera Home Platform for
Real-Time Human-Robot Interaction
OmniRobotHome
Markerless motion capture in a furnished home.
Loading the room…
Robots that assist people must respond promptly and appropriately to human activity, which requires low-latency processing, accurate perception, and anticipation of human motion. These requirements are especially difficult to meet in living environments, where perception must cover wide areas, remain robust to occlusion, and capture natural behavior without wearable devices. We present OmniRobotHome, a testbed that addresses these challenges in a fully furnished home that supports natural everyday activity. Its 48 hardware-synchronized cameras provide room-wide coverage, enabling accurate markerless perception under occlusion. A distributed perception pipeline processes these streams in parallel and delivers persistent human and object states to robot controllers with low latency. We demonstrate OmniRobotHome through two human-robot interaction studies: proactive yielding for safety and nonverbal intent recognition for assistance. Through these studies, we analyze how camera coverage, state latency, multi-view pose, hand, and gaze estimation, and behavior-memory-based forecasting affect overall task performance. Our results highlight the value of OmniRobotHome as a testbed for studying human-robot assistance in real living environments.
Hover a marker to see each component in action, anchored to where it lives in the room — all perceived in one unified world frame.
The robot predicts human motion and moves aside before getting in the way.
Using recent motion and patterns learned from activity in this room, the model predicts several plausible future movements. The robot checks these predictions and moves aside if it may get in the person’s way.
The user looks at the mustard bottle and points to where it should go. The robot picks it up and places it there.
The user looks at the mustard bottle to select the object.
The camera array tracks the user’s body and hands in 3D across the room. The system estimates gaze from head and torso pose, and pointing from arm and fingertip positions. It matches these directions to objects and destinations in a shared 3D coordinate frame. The robot then uses the object’s current position and orientation to plan its grasp.
Compare body and hand pose tracking with 38, 16, 8, and 4 cameras. Each view shows the same motion.
Auto cycles through the camera counts. Compare all four shows them side by side.
Each dot represents a point in the room. Its color shows how many cameras cover that point.
Markerless 3D pose tracking across the living space.
Stereo cameras provide depth for markerless tracking of an object’s position and orientation. The robot uses the tracked pose to position its grasp.
The system tracks people’s 3D poses in real time during everyday activities. The monitor shows the live tracking results.
The robot arms perform household tasks using live 6D object poses and a library of object-relative grasps. Collision-aware motion planning guides their movements.
@misc{lee2026omnirobothomemulticamerahomeplatform,
title={OmniRobotHome: A Multi-Camera Home Platform for Real-Time Human-Robot Interaction},
author={Junyoung Lee and Inhee Lee and Kyungwon Cho and Jeonghwan Kim and Mingi Choi and Sookwan Han and Lee Chae-Yeon and Wonjung Woo and Gunhee Kim and Jisoo Kim and Jeonghyeon Na and Hanbyul Joo},
year={2026},
eprint={2604.28197},
archivePrefix={arXiv},
primaryClass={cs.RO},
url={https://arxiv.org/abs/2604.28197},
}