scripts, etc.

3.2 MR framework operation for character simulation

The software architecture is composed of multiple software components called services, as

their responsibilities are clearly defined. They have to take care of rendering of 3D simulation

scenes and sound, processing inputs from the external VR devices, animation of the 3D models

and in particular complex animation of virtual human models including skeleton animation and

respective skin and cloth deformation. They are also responsible for maintenance of the consistent

simulation and interactive scenario state that can be modified with python scripts at runtime.

To keep good performance, the system utilized four threads. One thread is used to manage

the updates of all the services that we need to compute, such as human animation, cloth simulation

or voice (sound) management. A second thread is used for the 3D renderer, who obtains information

from the current scenegraph about the objects that must be drawn as well as the image

received from the camera. It will change the model view matrix accordingly to the value provide

by the tracker. The third thread has the responsibility of capturing and tracking images. The last

thread is the python interpreter, which allows us to create scripts for manipulating our application

at the system level, such as generating behaviors for the human actions (key-frame animation,

voice, navigation).

Fig. 2 VHD++ AR Framework Overview

The AR system presented in Fig. 2 features immersive real-time interactive simulation supplied

with proper information in course of the simulation. That is why content components are

much diversified and thus their development is extremely laborious process involving long and

complex data processing pipelines, multiple recording technologies, various design tools and

custom made software. The various 3D models to be included in the virtual environments like

virtual human or auxiliary objects have to be created manually by 3D designers. The creation of

virtual humans require to record motion captured data for realistic skeletal animations as well as

a database of real gestures for facial animations. Sound environments, including voice acting,

need to be recorded in advance based on the story-board. For each particular scenario, dedicated

system configuration data specifying system operational parameters, parameters of the physical

environment and parameters of the VR devices used have to be defined as well as scripts defining

atomic behaviors of simulation elements, in particular virtual humans. These scripts can

modify any data in use by the current simulation in real-time. This allows us to continue running