The Multimedia Operating Systems and Networking (MONET) Research Group, led by Professor Klara Nahrstedt in the Siebel School of Computing and Data Science at the University of Illinois at Urbana-Champaign, is engaged in research across a broad range of topics, including distributed multimedia systems, multi-agent systems, large language models/vision language models reasoning and IoT. The group has multiple on-going and published projects spanning diverse aspects not limited to algorithms, energy, system efficiency and multimedia applications.
Lingzhi Zhao- Trinity: Exploiting Latency Sensitivity to Improve Quality of Experience on Cloud VR Gaming
In this talk, I will present Trinity, a novel and practical solution that enables dynamic game streaming from cloud servers to virtual reality (VR) headsets. Achieving the stringent motion-to-photon (MTP) latency requirement of under 25ms presents a critical challenge in VR game streaming. Existing approaches, such as frame pre-fetching, primarily support specific actions (e.g., head movement), resulting in an MTP latency discrepancy with other actions subject to round-trip delays, thereby degrading overall user quality of experience (QoE).
To bridge the MTP latency gap, Trinity proposes a novel approach grounded in the observation that player interactions exhibit heterogeneous latency requirements. Specifically, we classify user inputs into three distinct motion categories: head motion, hand motion, and body motion. By aligning latency levels with users' perceptual sensitivity to different types of motion and strategically introducing controlled delays to the hand motion, Trinity mitigates the perceptual impact of latency discrepancy across motions. We evaluate Trinity with a highly latency-sensitive first-person shooting game streamed from a remote server to a VR headset. Real-world user studies demonstrate that Trinity delivers a user experience comparable to a local streaming setup, despite operating under network-induced delays.
Mingyuan Wu- miVirtualSeat: Semantics-aware Content Distribution for Immersive Meeting Environments
The pandemic has greatly reinforced the need for virtual meetings in both work-related and social settings. It has also highlighted the dire lack of video conferencing tools that can simulate the rich immersive experience of in-person meetings, the current tools often leading to “zoom fatigue’’ caused by having to interact over an unnatural communication medium. This collaborative project brings together investigators from University of Illinois, Urbana-Champaign, University of Massachusetts, Amherst, and New Jersey Institute of Technology to research, build, and evaluate a distributed system, called miVirtualSeat, that more closely simulates the immersive experience of in-person meetings, including physical and virtual participants in a physical meeting space.
The project is focused on key research challenges in providing an immersive meeting experience where physical and virtual participants interact with each other in a physical meeting room. Some participants are virtually present in the physical meeting room, but physically located in remote sites with only limited compute and network resources. The challenges are (a) detecting, tracking, and localizing distributed physical and virtual 360-degree avatars and objects in the joint immersive scene in real-time, (b) reducing the bandwidth and latency of delivering integrated and synchronized 360-degree, volumetric, and 2D/3D video, and ambisonics audio, and (c) ensuring good quality-of-experience in the form of natural interaction between physical and virtual participant.
This seminar will be hybrid, refreshments will be available for in-person attendees.