This is a concept Cortana appliance I've built. It's basically what I imagined Microsoft's version of Alexa or Google Home would be like if they were to use the holographic AI sidekick from the Halo franchise. While still a work in progress, I just wanted to show how it's coming along.

So, this a Pepper's Ghost hologram that uses Windows 10's native Cortana experience. It's all powered by a Windows 10 device with 4GB of RAM and a built-in Arduino used for controlling the platform lights. There's a portable USB monitor on the top that reflects on 3 panes of mirror glass. The enclosure itself is a custom design I modeled. I then 3D printed each part of the assembly in PLA. There's also an omnidirectional microphone that sits on top of the unit, and a decent mini speaker built into Cortana's pedestal on the base.

The software it runs is essentially in two parts: A Unity 3D app that presents and animates Cortana in three different camera angles and communicates to a second app: a proxy service that parses data going to and from the native Cortana web service. The proxy app renders the HTML from Cortana result queries which is then presented in the Unity app. It also features real-time face tracking with the front-facing camera which moves the rendered camera perspective relative to a single viewer's head position - this makes it look a bit more 3 dimensional when facing it. The app also controls an LED strip surrounding the edge of the platform.

 

Cortana's animations were recorded with the help of my lovely wife who regrettably endured many takes of motion capture in our living room. I used dual Kinects to record her motions and applied them to the Cortana model in Unity. Other animations would include an array of screens that Cortana brings forth with her hand motions. Each screen displays a unique mock UI loop and each loop plays on a random screen at start. This makes the screen animation semi unique for each query. I have more animation ideas for query processing, but have not gotten around to it.

The query's answer card is rendered and presented on one of the screens and is flipped around to the viewer. This was done by first rendering the html in a headless browser, then modifying its content to look good on a dark background.

Knowing when the native Cortana was being displayed and hidden was quite the challenge. Because Cortana is a UWP app, its sandbox design prevents Window Handle api calls like Win32 apps, locking me out from reading its UI content. I ultimately settled on monitoring a single pixel color in the taskbar to determine when Cortana's UI was present.

As you can imagine, since this is all using the native Windows 10 Cortana, this device can be used not only for knowledge based queries, but also for home automation and music tasks too. I can extend functionality by simply building UWP apps that can fulfill Cortana integration and use them accordingly. I can perhaps demonstrate this in the future, but that's it for now

Here's a gallery containing additional build photos:

Cortana Hologram Concept

For direct inquiries regarding this project, reach me at cortanahologram@unt1tled.com 

Press and Feedback