Digital twins are being defined as digital replications of living as well as nonliving entities that enable data to be seamlessly transmitted between the physical and virtual worlds. Digital twins facilitate the means to monitor, understand, and optimize the functions of all physical entities and for humans to provide continuous feedback to improve quality of life and well-being. My research work is under this research umbrella to create a digital twin by 3D reconstruction, add sensory information (including emotion, physiological signals, stiffness property), interact through augmented reality, and enable its intelligence. In this presentation, I will talk about the different aspects of the digital twin in my research (as follows), followed by a discussion on its open research questions leading to my future work.
1) 3D Reconstruction for Creating a Digital Twin
To achieve a high precision digital twin model, a widely-used RGB-D camera was evaluated to clarify its accuracy distribution and depth entropy. Based on the evaluation results, a linear trilateration method was proposed to improve the depth accuracy by utilizing multiple sensors' measurements from different viewing angles. After having the baseline, a nonlinear trilateration approach and a geometric trilateration approach were proposed respectively for further improving the accuracy, which eventually leads to a vivid digital twin.
2) Monitoring the Digital Twin
After having the 3D digital twin model of a human body, tracking its movement was feasible. A most-likelihood estimation approach capable of occlusion cases was proposed which can accurately track the digital twin. Moreover, rich features, like emotion and sitting posture of the digital twin were also added for better interaction.
3) Interacting with the Digital Twin
The 3D model of the digital twin was visualized by HoloLens with different layers of information. To improve the user experience, not only hand gesture (gesture vocabulary was defined based on our elicitation study on human performance and memorability) but also haptic rendering was put together to enhance the feelings for effectively interacting with the digital twin.
