Yes in the latest Star Wars flick “The Mandalorian” we see Baby Yoda grasp entities and people from afar with a simple motion of his hands and his mind-bending haptic abilities that, in Baby Yoda’s world, is a reality – in our world on earth a future possibility (think Neurolink). Even space exploration, whether it be to the Moon or Mars, in the not so distant future will require a remotely operated, AI enabled, virtual reality surgical suite. Think of the technology visuals in the space exploration fiction movies like “Passengers” or “Elysium” as possible realities in the future.
Meanwhile as COVID 19 brings real life activity to a halt in our world, post pandemic there will be new expectations of preparedness to provide access to future healthcare diagnostics and treatments – remotely, rapidly and virtually to address inevitable future pandemics and disasters.
What this will look like will be quite different than what we know of today!
In Episode 2, we chat with Dr. Pierre Boulanger about the proxy haptic of force feedback touch that will enable full body (mannequin and device) feedback in the virtual reality training for surgical teams.
Pierre talks about the pilot study on the use of proxy haptic (touch) feedback as compared to the use of current virtual reality systems which have no proxy haptic (touch) feedback.
So how close is proxy haptic touch to reality? What other applications does Pierre envision as possibilities to improve the healthcare system for society?
Pierre shares his insights on how virtual reality with haptic force feedback (full body) can improve team training to include surgeons, operating room nurses, anesthetists, and radiologists.
Go on, use your imagination – you might be just the one to make a difference to improving training for what will be a new paradigm of healthcare solutions!
Dr. Boulanger’s Blog
In our laboratory, we created a prototype proxy haptic system and then did a pilot study to determine the effects of our system for simple surgical training tasks. The goal of this pilot study was to determine if people think our system is more believable than standard VR by asking them to do simple tasks such as moving the patient’s hand or pointing a syringe to specific target locations.
In the pilot project, we compared the results of using our system against a standard VR system with no haptic feedback (i.e., wand controller only). In the end, our results show that the amount of time the users need to complete the pointing tasks is longer in the system with proxy haptics than using the VR wand. This makes sense as in the virtual world objects are ghosts and one can move freely without collisions.
In the proxy haptic world, objects are real and the laws of physics do apply. Our results also show that from a user perspective, our system is more believable and closer to the real world than the standard VR interface. This system is an early prototype, and in our next project, we intend to demonstrate that one can train surgical teams to perform complex surgical procedures such as endoscopic surgery.
About Multi-Sensory Virtual Reality
In recent years and with the advancement of technologies, the applications of Virtual Reality (VR) have been used in many fields. Technologies such as high-resolution digital displays, GPUs and CPUs, are now able to render complex virtual worlds in real-time. Modern VR systems can create high-quality VR environments that can be used in numerous applications such as entertainment, education, and medical training.
VR has been used in surgical training to either replace more expensive training techniques (such as cadavers) or serve alongside additional training techniques to increase a surgeon’s skills. As VR technologies evolve, one can trick the brain into thinking that a user is seeing and touching real-world objects. To trick the brain at a believable level, one needs to be able to generate feedback to multiple human senses (vision, haptic, sound) that are realistic and consistent with behaviors in the real world. Because of these requirements, many systems have been proposed to deal with the multi-sensory outputs VR needs to produce in order to be useful for real-world applications such as surgical training. Visual perception can be dealt with very well using modern Head-Mounted Displays (HMDs) and advanced rendering software.
Haptic perception however is still at its infancy and has not reached the same level of realism that HMDs can provide. Many haptic systems use force-feedback devices to create a sense of touch, most of them require to hold a wand attached to a small robot or wear a glove that provides forces or friction to a user’s hand movement. The problem with standard haptic devices is that they do not provide haptic feedback to all parts of the body hence reducing the sense of immersion and making their use impractical in complex simulations.
About Dr. Pierre Boulanger PhD
Pierre is currently the Director of the Advanced Man Machine Interface Laboratory (AMMI) as well as the Scientific Director of the SERVIER Virtual Cardiac Centre in the Mazankowski Heart Institute. In 2013, Dr. Boulanger was awarded the CISCO chair in healthcare solutions, a 10 years investment by CISCO systems in the development of new IT technologies for healthcare in Canada.
Dr. Boulanger received his in Engineering Physics and his Masters in Physics Laval University, and his Ph.D. in Electrical Engineering from the University of Montreal. Dr. Boulanger cumulates more than 35 years of experience in 3D computer vision, rapid product development, and the applications of virtual reality systems to medicine and industrial manufacturing. Dr. Boulanger worked for 18 years at the National Research Council of Canada as a senior research officer where his primary research interest was in 3D computer vision, rapid product development, and virtualized reality systems. He now has a double appointment as a professor at the University of Alberta Department of Computing Science and at the Department of Radiology and Diagnostic Imaging.
