Understanding VR and Hand Tracking

What is Hand Tracking?

Hand tracking has become an integral feature within recent generations of immersive technology and Virtual Reality (VR) devices. With the widespread adoption of this feature, engineers and developers are faced with a list of opportunities and challenges to tackle, most of which are related to human usage. 

So what is hand tracking? It is a feature that enables users of VR devices to use their hands as an applicable input method, instead of using the VR device’s controllers. Users can perform tasks or actions in the VR environment by making simple gestures with their hands, such as pinching, pulling, pushing, or dragging.

The central aspect to the successful portrayal of realism in a VR environment is the extent to which a user feels immersed in the experience. This means a fully articulated tracking system and hardware, which will be able to determine where the user’s hands are in virtual space relative to the real world positions, and determine what every finger and joint is doing as quickly and accurately as possible. This will facilitate natural interactions, and give users a heightened sense of presence and a more engaging experience.

A number of VR Head Mounted Displays (HMD) now support hand tracking natively. VR explorers and enthusiasts can find great examples of fully articulated hand tracking in the HoloLens 2, the Leap Motion, the Oculus Quest, as well as the smartphone Software Development Kit (SDK) ManoMotion.

HoloLens 2

HoloLens 2 is the second iteration of Microsoft’s revolutionary HMD. Microsoft has branded the HoloLens 2 as a Mixed Reality headset, but this label could be confusing to some as they may see it more as an Augmented Reality device.

Nevertheless, it is a self-contained device that was officially launched in 2019 that retains the features of its predecessor while bringing some upgrades to the table with more computing power, better sensors, a longer battery life, and a bigger Field of View (FOV) which enables the user to see more information. The bigger visor on the HoloLens 2 allows for an eye tracking feature, enabled by cameras and sensors that are located near the nose ridge directed towards the user’s eyes.

The HoloLens does not have any dedicated controllers like other HMDs, instead it is controlled by hand and by voice commands. There are a number of new, and more intuitive hand gestures that users can use for easier content manipulation. 

Oculus Quest 2

The Oculus Quest 2 is a standalone HMD that promises to expand on the experience of its predecessor with a processor that is three generations newer, enhanced ergonomics for a lighter and smaller headset, a higher resolution display brought on by a new design for the lens, and redesigned motion controllers. These Touch Controllers’s movements are tracked using the sensors located on the headset itself. 

The newest feature for the Quest 2 is the controller-free hand tracking. The feature enables its users to use their hands as a viable input method by making simple gestures such as pinching, dragging, or holding to perform certain tasks or actions within the environment. The software incorporates Machine Learning (ML), creating a group of points that map the user’s fingers and knuckles allowing the computer to determine the position of the user’s fingers, which in turn represents movement to an accurate degree in the environment.

Click here to book your 15-minute consultation, and see how hand-tracking and haptics can deliver measurable results to your digital training.

Leap Motion

The Leap Motion Controller is a small device that connects to a computer and enables it’s users to manipulate digital objects with hand gestures and motions. Functioning as an addition to other hardware, the Leap Motion device adds a new way to interact with the digital world. Giving users the ability to play games and create designs with programs designed to interpret hand gestures.

The Leap Motion device uses an infrared scanner and sensor to map and track the human hand. The information gathered is then used in real time to create a digital version of the user’s hands, so they can be used to manipulate digital objects.

Interacting with Hand Tracking

There are three ways a user can interact with the virtual environment and three ways the system can interpret hand pose and gestures. VR software will use ML algorithms for hand pose estimations. These include: Discriminative Approach, Generative Approach, and the Hybrid Approach.

The discriminative approach sees that the images are extracted and examined by the HMD system to determine the movement of hands on a frame by frame basis, and run that against a hand gesture database. This database can be created using ML and a training system to recognise the different gestures.

The generative approach employs existing 3D models of the hands and manipulates the model into a gesture that best corresponds to the user’s hand gesture inputs.

The hybrid method combines both of the previous approaches to rely on the frame by frame tracking along with reliable capabilities of re-initialisation from using the hand models.

With the hand models in place and successfully tracking gestures and hand positions the user can interact with the virtual environment by way of Direct Manipulation, Hand Beams, or Gesture Recognition.

Direct manipulation is where users can reach out with their hands to interact with objects in the VR environment. Objects are made to react as they would in reality, and can be an easy and intuitive way to control the virtual world.

Users can pick up objects, press buttons, and activate User Interface (UI) components as if they were interacting with a touchscreen. Users will find that they will need to be in close proximity with objects to directly interact with them using this model.

A hand beam is a VR design concept where users can emit a beam from their hand to reach an object at a distance. Then the user would perform gestures to exert control over that object from afar. These controls can include picking up distant objects, flipping or rotating an object, or pushing buttons. Users may even experience some Haptic Feedback at the beginning of interaction or during interaction with the object.

The VR design concept of gesture recognition is where the HMD device analyses the hand and finger postures of the user, then triggering a corresponding reaction. An example of this would include a system that allows a user to take a screenshot by bringing the tips of their thumbs toward the forefinger from the opposite hand to create a square shape to imitate looking through the frame of a camera. It is worth developers including gestures in their designs when the inherent context of a gesture matches the outcome.

The Future of Hand Tracking

Hand tracking presents tremendous opportunities from enterprise to education. For now there are still various kinks to be worked out, but as VR and hand tracking technology becomes integrated into the modern world, more companies will want to take advantage of the potential.

Wireless will forever be the future, and what controller-free hand tracking technology offers is a deeper level of immersion. When users have to deal with just a screen on an HMD, without controllers or too many wires and they can see their hands reacting in real time, complete immersion can be achieved.