HAPPIE Haptics For Innovate UK – VR Tactile Feedback

Introduction

Haptic Authoring Pipeline for the Production of Immersive Experiences, or HAPPIE for short, was an Innovate UK funded Research and Development project that Sliced Bread Animation was more than excited to tackle in the later months of 2019 through to 2021. The aim of the research was to build projects that would give a user a tactile sensation or experience as a method of interacting with a computer device or a number of objects in a virtual environment. Being an R&D project, there was plenty of scope to play around!

Over the number of commercial projects we have worked on, we have found that there is a certain lack of ability to create rapid workflows for pharmaceutical marketing, which prohibits pharmaceutical companies from being able to explain their treatment’s Mechanism of Action with patients and clinicians.

We came up with a concept to create a Virtual Reality (VR) haptic story interface, that would allow marketers to develop and design their own marketing stories in an immersive virtual space using an extensive library of objects, a drag-and-drop haptic interface, and an easy to use playback and review system to be able to share the final experience with peers and patients.

So what were the challenges?

Software

Aside from the high price tag – and the opinion that VR can cause nausea and headaches – as a barrier to entry for a number of people to get acquainted with virtual experiences, is that only a small percentage of the population have interacted with VR software. This means there may be some users in the target demographic for the HAPPIE project that may not know the User Experience and User Interaction design languages for VR products.

The main challenge for the project was in finding the right platform to build on the requirements set out during the planning phase. The software needed to be robust enough to produce a virtual environment across several platforms, such as Windows, iOS and Android, if needed. We found that working with Unity’s game engine was more than adequate for previous projects and was more than capable to handle the intricate demands for this project too.

Hardware

It was a requirement of the HAPPIE project, that the hardware involved in the development be capable of administering haptic feedback for interactions and scenarios. We needed to find Haptic Gloves that would give our users this type of feedback when possible. During our search we found several haptic glove devices that provided haptic feedback, as well as force-feedback. Which could utilise motors in the device to restrict finger movements past a certain point. Enabling user interactions with virtual items to take that next step into immersiveness.

These haptic glove hardware components needed to be able to communicate with the Unity3D software, and be able to integrate well with the VR headset.

We found that the SenseGloves met these specifications, and also provided provisions for motion tracking. Which, pardon the pun, went hand in hand with our desire to target entry-level and low-tech users as this would be the most intuitive way to interact with the virtual environment.

User Testing

Due to the unforeseen circumstances that came with COVID-19, we found it particularly difficult to organise for user testing sessions that didn’t take place outside of our respective homes. Though these home test scenarios included entry-level users, the feedback quickly became repetitive as the users became accustomed to the project.

One of the major hurdles the users came across was the dichotomy between the virtual and real world interactions. Where they found themselves being very much immersed in the virtual world, they found themselves suddenly being disconnected from the immersion when the force-feedback wasn’t also applied to other parts of the body. They pointed out that seeing their virtual hands passing through the collision points of 3D objects in some interactions, while being able to pick up other objects would often remove them from the simulation.

Solutions & Findings

Video games and VR products often employ a tutorial sequence or an onboarding process to educate its users on how to control and understand what to expect from the product. To solve the problem of targeting entry-level users who may never have handled VR products before, we decided to integrate an onboarding sequence into the HAPPIE project. The aim was to encourage these users in an environment that was comfortable and familiar while intuitively learning the haptic gestures and UI interactivity.

The environment we designed was one almost every user would be familiar with: a home environment. A studio apartment complete with living area and kitchen/dining area. The onboarding sequence asked the user to aid in making a cup of tea, something most users may be accustomed to doing almost everyday. The haptic gestures that we created were designed to enable the user to navigate through the VT environment and interact with objects and UI elements. This included forming a hand into a finger-gun, pointing towards the destination, and bringing the thumb down to instantly be transported to the desired location. To interact with the UI elements, the user was asked to form a fist then twist the wrist, opening up the fist upwards to reveal our main control panel floating above their palm.

The control panel allowed the user to interact with a library of 3D objects that they could pull into the environment and then place where they pleased. If the user ever found an object wasn’t needed anymore, they had another gesture at their disposal. They could hold the object and throw it over their shoulder to remove it from the environment entirely.

Though most users found these interactions intuitive and easy enough to understand and perform, there was a recurring issue when introducing these gestures during the onboarding session. Some users pointed out that they were unsure of how far down they should move their thumb when aiming to teleport, while others questioned how fast they should open their fist to reveal the main control panel. To solve these issues we spent some time fine-tuning the gestures to find the best positions to trigger a haptic vibration to notify the user when they had successfully performed the gesture.

Benefits & Limitations

There is an initial curiosity mixed with concern from users when they first put on the SenseGlove hardware, but because the gloves allow users to use their own hands as the basis for VR interactions and controls, users find that they are immersed into the virtual world a lot quicker than when using conventional VR controllers. With the conventional controllers come buttons, and joysticks that an entry-level user may find confusing and intimidating.

The limitation of certain gloves similar to the SenseGlove is that, while they give force-feedback in the fingers, users may expect force-feedback as well in the rest of the body. As one would expect from real world interactions with objects. Rather, users find themselves pushing past objects they would otherwise collide with. This can quickly remove a user from the immersion of the VR experience.

We have found that utilising the onboarding sequence was of a great benefit to the user in getting them to understand and get comfortable with the HAPPIE project. These days most applications, games and softwares come with a form of onboarding process. Whether it be a tutorial, or a series of pop-up UI texts that guide the user through and around the software, it is vital for user retention to get them comfortable with the product early on. This will limit the time users spend in frustration, and instead increase the time users spend with the product.

The Healthcare and Pharmaceutical sector are some of just many industries that would benefit tremendously from VR products and experiences. Not only could it aid in the rapid development of prototypes, or help students with practical learning without much reciprocation but it could also aid in sharing treatment processes to patients.

Conclusion

The HAPPIE project was an Innovate UK Research and Development project which required its participants to create a product that integrated haptic and tactile sensations into computer devices or VR experiences. Due to the fact that we began the development of the project with intuitive interactions in mind, we were able to create a final product that users were able to pick up quite quickly. Though there were plenty of hurdles to overcome, our research has given us a number of much needed insights when it comes to designing and developing virtual experiences for users, of which we are more than excited to carry into any future projects with similar requirements.