This project will be leaded by Fraunhofer IGD and several research centers and enterprises working in the field of virtual and augmented reality will be participating as well. The aim of this project is to improve lightweight near-to-the-eye displays and tiled stereoscopic large size displays.

The improvements on the hardware level consist in developing a unique stereoscopic head mounted display (HMD) using emerging display technology such as OLEDs. For tiled stereoscopic large screen displays improved calibration techniques will be developed to ease and accelerate their use. On the software level improvements comprise the fidelity of the content to be displayed (rendering quality), the interfacing between the user and the displays through innovative 2D/3D interaction techniques for mixed realities and advanced tracking systems. The achievements of IMPROVE are integrated into a collaborative mixed reality product development environment, showcased and evaluated in two application scenarios: collaborative product design in the car industry and architectural design.

Graphitech’s role has been mainly based on novel interaction techniques. After the phase of design, just concluded, the project is moving towards the release of the first prototype. The following sections illustrate the current state of the project.

Goals of the Project

The goals of the IMPROVE project can be summarized as following:

• To improve technologies for large scale tiled displays. Large tiled screen displays are tessellated displays combining together many off-the-shelf beamers aiming at higher resolution and brightness. The tiling entails some problems such as geometric and color calibration. Within IMPROVE new calibration techniques are developed to ease the use of tiled large screen projection screens.
   
• To enhance the realism of the displayed virtual objects, especially in mixed reality scenes through seamlessly melting together real and virtual objects by light acquisition and physically-based light simulation techniques.
   
• To improve user interaction with those displays through new interaction metaphors and tracking approaches (e.g. large area tracking for mobile users).
   
• To integrate and demonstrate large scale displays with near-to-the eye displays in innovative collaborative mixed reality product development for industrial users.

Such achievements have been applied to two prototypes specifically designed for the two scenarios of relevance of the project, the architectural and automotive scenario, illustrated in the following sections.

Architectural Scenarios

In the outdoor setup, the architect has to be able to sketch a building and its surroundings on-site in order to capture a first design and the parameters of the real construction site environment. Then, the architect uses Improve to review and modify his/her design solution indoor by exchanging design data with 3rd party solutions (ArchiCAD/VectorWorks). A further scenario allows for a collaborative outdoor review and presentation of the design prototypes, taking annotations of various kinds within the model. The architect is to be supported by construction verification functionalities like validating the logical and spatial structure building structure or daylight simulation according to the global positioning and orientation.

Automotive Scenario

The automotive design review process has to include the functional and aesthetical analysis of a prototype model and serves as a platform for the creation of alternative design solutions based on an existing virtual prototype. Two visualization systems are used: the Head Mounted Display and a Tiled Visualization Display.
The design prototypes are constructed using a traditional CAS/CAD approach, interrupted by incremental design review sessions, during which the design surfaces and their materials are evaluated and potentially slightly modified. The proposed surfaces have to be verified with respect to the visual quality using diagnostic shaders. Likewise it must be possible to verify that surfaces can be manufactured without any loss in quality by superimposing simulation data (stamping analysis). The car designer then can inspect and mark critical and problematic areas in real-time.

Results

The achievements of IMPROVE are being integrated into a collaborative mixed reality product development environment, showcased and evaluated in two application scenarios: collaborative product design in the car industry and architectural design. Recently the final user test for the automotive scenario was undertaken at Elasis, Naples which is one of the IMPROVE partners.

The final test has seen the simulation of a design review session. The set-up deployed for the test has seen two users collaborating with their IMPROVE system, sharing the same virtual session. This is shown on their personal interaction devices as well as onto a PowerWall in front of them. Each group of two users had to use the two set-ups deployed for the test. These were respectively a PC connected to a touchscreen, a 3-axis gyroscope and accelerometer or a PC connected to a tablet. Each user could interact with the system through advanced GUI, gesture or voice.

A powerwall, an additional instance of the application, served as the main display for the group of reviewers and represented at all times the view as the peer reviewer saw it.

Collaboration

Following the peer reviewer paradigm, one master reviewer was enabled to lead the reviewing session with respect to navigation. A local and a shared mode allowed the user both to follow the peer reviewing process, but also to leave the design review session temporarily for exploring the model separately from the group and to take e.g. annotations on the model. All information from the user is shared through propagating them (annotation/change of materials etc.) to the other users in the reviewing group via a communication backbone based on XmlBlaster.

Customizable Multimodal interaction

One of the main peculiarity if the system is in that it allows for a fully customizable GUI and interaction dialogue from outside the application to tailor it for the specific task at hand of the user.
Modalities are bound together via a graphically editable bidirectional interaction graph which allows the specification of domain keywords as nodes. Attributes on edges, present within the graph, specify how to advance in the interaction dialogue with respect to each modality.

These attributes comprise gestures like gesture="circle, rectangle", speech input like speech=”node, wordnet” and “dialog” with respect to the hierarchical ring menu. Mixed modalities are usable for instance like seen below:

create(voice)-annotation(dialog)-pick(tap, gesture)

or

store(dialog)-view(voice)- rectangle(rectangle, gesture)

Configurations for the various modalities (specifically voice) are generated from the designed, user-tailored interaction graph.

The appropriate action in the application is invoked by matching these keywords with attributes assigned to the application-defined behavior.
As a further benefit, the user could specify which interaction scheme is most efficient for his needs. Furthermore, the gestures were not only used for identifying the appropriate edges on the graph. In fact they were used as well for location-based interaction such as picking, selecting via their geometric properties. The user was thus enabled to find his\her personal way of working with the application.

Superhand

Given the high priority of a non-fatigue navigation through the scene and around the reviewed car model, we used the Xsens-Motiontracker (see img) for controlling the virtual camera in a convenient and natural way. The Xsens MT delivers accelerations and orientation along the 3 axis of rotation of the tracker attached to the back of the hand.



The user could navigate the scene by bending his arm. Navigation modes supported were fly, pan and tilt. To switch between navigation modes the user can speak the command, press the relevant button or simply shake gently his/her hands. In fact a fast movement of the hand to the right or left would trigger a change of the active navigation scheme whereas the natural orientation of the hand would orient and control the virtual camera. The system would provide adequate feedback by speaking out the new navigation mode.

The users who undertook the testing session at Naples found it to be effective over their traditional ways of navigation and they have clearly stated that this required less fatigue since it only involved bending the hand without the need for holding their forearm up (see picture). Furthermore this way movements to achieve a desired view on the reviewed model are reduced to a bare minimum.

Partners