CERTH designed a series of Augmented Reality games that require either a tablet, a dynamometer or Augmented Reality glasses. Each game records data such as the duration of the game and the level of difficulty. The analysis of the data can provide information about the user’s hand stability, grip strength, reflexes, cognitive problems or short-term memory. The collected data provide real-time information on the user, thus enabling personalized recommendations about the user’s cognitive or physical state that will help delay the onset of frailty. The Gravity Ball, the Floating Archery Target and the Memory AR games are described below.
1. Gravity Ball
Gravity Ball is a tablet game designed by CERTH. It is an Augmented Reality (AR) game, since it involves the user interaction both with the tablet and with physical objects in order to play. The game trains physical aspects of the player, such as balance and coordination skills, as well as cognitive skills, such as visual perception and puzzle solving. The game contains levels of increasing difficulty to challenge the player in terrains that are more difficult to solve. The game records data related to the duration of the game, the level of difficulty, and data related to the trajectory followed by the ball. Analysis of such data can provide information about the user's hand stability and cognitive problems that might be present.
The user plays the game by holding and moving a piece of paper, called the "marker", in front of the tablet's camera. When viewed through the tablet's screen, the marker is rendered as a terrain with a hole and obstacles around it. When the user moves the marker, the terrain moves as well, following the position and orientation of the marker. When the game starts, a ball drops into the terrain and the goal is to tilt the terrain so that the ball passes through the obstacles and falls within the hole. The game is usually played with the tablet being held by a stand, so the user can easily move the marker around. See a video here.
To make the rendered terrain follow the movement of the marker, a special black-and-white pattern is printed on the marker paper. The pattern acts as a bar code, but also allows the tablet to understand the position and orientation of the marker, by measuring the perspective distortion of the pattern when viewed through the tablet's camera. This is a characteristic of Augmented Reality technologies.
2. Floating Archery Target
Floating Archery Target is another Augmented Reality (AR) game developed by CERTH, with the goal to monitor and exercise the player's strength as well as visual perception skills. The game requires that the user wears a pair of AR glasses. These glasses are worn as regular glasses, allowing the user to view the surroundings, but "augmenting" the surroundings with virtual objects, seen only through the AR glasses. The game also requires the user to hold a dynamometer, which is a device designed to measure the user's grip strength.
To play the game, the user wears the AR glasses and holds the dynamometer. Virtual archery targets are placed within the surroundings by the game. The user moves his/her head around until he/she finds one of these targets. Then, the user aims at the target using the dynamometer and puts strength on the dynamometer grip to release the arrow. If the user wants to aim better, a red cross appears on top of the dynamometer, showing where the arrow will land. Once the user hits a target, the target disappears and the user moves on to find the next one.
The AR glasses are designed so that they continuously track their position and orientation by using embedded accelerometers and compasses. Knowing their position and orientation within the surrounding environment allows them to project upon the glasses images of 3D virtual objects as if they were positioned at specified locations and orientations in the surroundings. In this manner, they "augment" the reality around the user. The red cross appearing on top of the dynamometer is another example of such an augmentation. To track the position of the dynamometer within the surroundings, a special marker (a piece of paper with a black-and-white pattern on it) is put on the dynamometer. Measuring the distortion of this marker when viewed through the AR glasses allows the glasses to know its relevant position and orientation, and thus render custom objects (the red cross) on top of it.
The data recorded by the game include the game duration, the positions of the targets, the position of the dynamometer, the trajectories followed by the user, the grip strength, etc. Combined, they can provide useful information about the user's grip strength and reflexes, as well as about the user's visual perception.
3. Memory AR
Memory AR is an Augmented Reality (AR) game developed by CERTH, aiming at monitoring and exercising the player's cognitive skills. It involves the use of Augmented Reality glasses, which is a special equipment able to project virtual objects in the actual surroundings of the player.
Memory AR is a memory and reflex game. The goal is to quickly find a number of objects in the player's surroundings and then remember the order of discovery. To play the game, the user wears the AR glasses. Through the glasses, the user can see his/her surroundings, but they are "augmented" with the introduction of some virtual objects, which are only seen through the glasses. These objects, e.g. a teapot or a cup, are positioned in pre-defined positions within the surroundings. The first goal of the user is to find them, by moving the user's head around while wearing the glasses. Once an object is found, it is removed from the list and the user moves on to the next object. When all objects have been found, the user is then requested to list them in the order they were found.
The AR glasses have embedded sensors, such as accelerometers and compasses, which allow them to constantly be aware of their position and orientation within the surroundings. Knowing this, they are able to render virtual objects on the glasses and make them appear as they would be seen if they existed in the real world at the specified positions and orientations.
The game has levels of increasing difficulty in order to make the tasks more challenging as the game progresses (more items to find, less available time, etc.). As the user plays the game, data are recorded about the game duration, the positions of the objects, the orientation of the user and the success in remembering the objects found, which can provide valuable information about the user's reflexes and short- term memory.