What is Hand Tracking?
Hand tracking has become an integral feature within recent generations of immersive technology and Virtual Reality (VR) devices. With the widespread adoption of this feature, engineers and developers are faced with a list of opportunities and challenges to tackle, most of which are related to human usage.
So what is hand tracking? It is a feature that enables users of VR devices to use their hands as an applicable input method, instead of using the VR device’s controllers. Users can perform tasks or actions in the VR environment by making simple gestures with their hands, such as pinching, pulling, pushing, or dragging.
The central aspect to the successful portrayal of realism in a VR environment is the extent to which a user feels immersed in the experience. This means a fully articulated tracking system and hardware, which will be able to determine where the user’s hands are in virtual space relative to the real world positions, and determine what every finger and joint is doing as quickly and accurately as possible. This will facilitate natural interactions, and give users a heightened sense of presence and a more engaging experience.
A number of VR Head Mounted Displays (HMD) now support hand tracking natively. VR explorers and enthusiasts can find great examples of fully articulated hand tracking in the HoloLens 2, the Leap Motion, the Oculus Quest, as well as the smartphone Software Development Kit (SDK) ManoMotion.
HoloLens 2 is the second iteration of Microsoft’s revolutionary HMD. Microsoft has branded the HoloLens 2 as a Mixed Reality headset, but this label could be confusing to some as they may see it more as an Augmented Reality device.
Nevertheless, it is a self-contained device that was officially launched in 2019 that retains the features of its predecessor while bringing some upgrades to the table with more computing power, better sensors, a longer battery life, and a bigger Field of View (FOV) which enables the user to see more information. The bigger visor on the HoloLens 2 allows for an eye tracking feature, enabled by cameras and sensors that are located near the nose ridge directed towards the user’s eyes.
The HoloLens does not have any dedicated controllers like other HMDs, instead it is controlled by hand and by voice commands. There are a number of new, and more intuitive hand gestures that users can use for easier content manipulation.
Oculus Quest 2
The Oculus Quest 2 is a standalone HMD that promises to expand on the experience of its predecessor with a processor that is three generations newer, enhanced ergonomics for a lighter and smaller headset, a higher resolution display brought on by a new design for the lens, and redesigned motion controllers. These Touch Controllers’s movements are tracked using the sensors located on the headset itself.
The newest feature for the Quest 2 is the controller-free hand tracking. The feature enables its users to use their hands as a viable input method by making simple gestures such as pinching, dragging, or holding to perform certain tasks or actions within the environment. The software incorporates Machine Learning (ML), creating a group of points that map the user’s fingers and knuckles allowing the computer to determine the position of the user’s fingers, which in turn represents movement to an accurate degree in the environment.