Holograms: Breathing Effect

Been awhile, I've been busy with a new job for the last 9 months. We are currently working on a product that utilizes Unity, the Hololens, and augmented reality. While there are a lot of pieces to this project, there was one "problem space" we stumbled across that I believe is worth sharing with the community. I will state that since this project is still a work in progress and propitiatory, I am not able to show images of the actual results we are using.

The Problem Space: How do we affect the physical world through holograms?

The Process: When we first approached this issue, we did extensive research online. We read technical docs for the Hololens, reached out to online communities, and spoke to others in the industry. Everyone told us it was impossible to have a hologram affect the real world. For this project, I wanted an effect that makes movement under the skin, like lungs expanding and contracting. The roadblocks here were that we couldn't use animated models, but rather needed an effect that would always cause this distortion. This would allow to scan ANY person with the Hololens and cause distortion, based off markers.

I started off with two capsules, basic shapes to represent the lungs. Since we need this effect to actually impact the world behind it I established the primary functions that the effect would have to have. First, it would need to be able to expand and contract, mimicking actual lung movement. Second, these would need to be public variables that could be altered based on incoming data. Third, there needed to be some sort of texture to the effect that will add to the overall distortion.

I broke this up into two main functions, movement and distortion. For the expanding and contracting effect I decided to make a script I could attach to my capsules. Now I have mainly an art background, so don't be intimidated by any code I implemented; its fairly simplistic. Essentially, this script has three public variables that allow you to control its grow size, rate of growth, and pause time (lungs have a slight pause in between expanding and contracting so this was necessary).

That's it. That simple script gives you these variables in the inspector and they can be fine-tuned for specific purposes. Max Size lets you put in a positive or negative value. Be warned though, this assumes the scale of the object is 1,1,1. So any scaling should be done prior to Unity. Grow Factor is the speed at which the object will grow. Last, Wait Time indicates a delay before the object shrinks back to its original size. This can be set to 0 if not needed, depending on the function.

After applying this script I determined it worked quite well. I could have two separate lungs (left and right) and apply different variables so one behaves "normal" while the other indicates a tension pneumothorax. It allows for a multitude of different behaviors depending on how you play with the settings. This handles the movement function of the effect, but we still need to tackle distortion.

In order to heavily distort things viewed through the effect, I created a custom shader. As shaders go, its nothing terribly complicated.

This shader works by grabbing a snapshot of the textures behind it and then running a distortion function on them, supplied by the user. So the output for this shader allows for two maps, an RGB Tint Color and a Normal Map. These maps will add to the distortion of things behind them.

You can plug in the desired maps for the normal and tint desired. There is also a public function slider that lets you control the impact of the distortion. This shader worked great in combination with the movement script already attached. It distorted the objects behind it, and the expanding and contracting of the effect lead to the appearance of expanding and contracting on other objects. I was able to successfully test in Unity and in VR on the HTC Vive. However, the real challenge was about to begin: how do I transfer this to a system that will affect the real world around it? 

The short answer is, it isn't possible. There are a few exceptions. You could mount a forward facing camera on the Vive, and then overlay the holograms on that captured environment. This will however cause a good deal of lag, so we didn't consider it as an option. So we were left with coming up with a way of making it appear to affect the environment. At this point, I got sucked down a rabbit-hole. I turned to one of our developers (a not-artist, if you will) and we were able to work together, by combining the functions I had with one of his own creation. He supplied a shader that would hide all objects rendered behind it. This way the affect wont be seen on other objects in the scene. Here is that shader. 

Next, we flattened the capsules from the effect so they wouldn't collide with anything else around them. Then I placed an object behind the effect that has a transparent shader. So that object isn't visible in AR, but exists and thus allows for the effect to be seen. By placing this object behind the effect, it was clearly visible in AR. Here is the same effect applied to a different asset. You can see the distortion happening inside the highlighted area. This will also expand and contract when played.

So ultimately, we found a cheat that allowed us to push forward and have a successful effect that distorts things in the real world. It was a huge challenge, took several weeks of work, but ultimately it was very rewarding to get it working. Again, I apologize I can't currently show images, but after the product has shipped I can come back and add those in.