Virtual reality multitracking interaction with 3D characters for immersive training applications

This thesis implements a Virtual Reality (VR) Multitracking interaction with 3D Characters resulting in an Immersive dance training application using the Occulus Rift HMD system and the Leap Motion hand tracking Controller. The player, who is placed in the main scene, is trained by the 3D white light scanned human character in a virtual environment which is designed and developed in Unity3D. Realistic
animations were applied to the character using a complex Inverse Kinematics (IK) system applied to the rigged 3D Human Model. Characters are rigged with custom skeletons using 3D geometrical bones and IK solvers. Motion Capture data were retargetted so that animations of the 3D character are produced. The Motion Capture data, derived from the tracking device Kinect v2, were applied to the rigged 3D model to create realistic dance animations. Rendering optimization techniques such as Baked GI, were applied in the scene to reduce the GPU rendering load and to increase the frame rate per second (FPS) of the VR Application.

The project features a tutorial guide process, so that the users familiarize themselves with the user interface (UI) being prepared for their VR dance training. During this process, the users are allowed to modify the sound volume and restart
the training through the Settings UI panel. The 3D character is standing just below the UI tutorial guide in front of the users guiding them in relation to the training functionality. This procedure allows the users to practice their actions by interacting with UI panels using gestures such as thumb and waving hands, combining touches and pinch actions. Users with VR experience can skip this pre-training procedure and continue with the main Dance Tutorial, where they can select from the main UI
tutorial panel, one out of three basic options. The training process starts with the first option named ”Basic Steps” of the Salsa Latin dance which includes the ”Basic A”, the ”Side Steps” and the ”Right Turn” dance routine. By selecting the ”Interaction” option, the users can practice dance by leading the 3D model to execute the ”Side Steps” and the ”Turn” animation. Finally, the users can select the last ”Dance” option, where they can improvise and dance, leading the 3D character to basic ”Side Steps” and ”Turn” while they execute the basic steps they have learned, during the first option.