A new fun and intuitive way to learn the piano using Augmented Reality
A first playable demo of Magic Keys for the Oculus Quest will be coming soon
OverviewMagic Keys is an Augmented Reality/Mixed Reality app for the Oculus Quest that is supposed to teach you simple piano pieces and techniques within just minutes in a playful manner. Similarly to Guitar Hero, the notes of any piece you would like to play approach to the according keys in real time, making it possible to play along pieces without any prior practice or experience. I started working on this project about two years ago, intially intended to be a small AR app running on a tablet. After it turned out that it could be of actual use, I ported the application to the HoloLens alongside with some major improvements. I continued to add features and tried to enhance the visualization of the music but I was greatly limited by the small field-of-view of the HoloLens.
In early 2019, I finally got access to a brand-new Magic Leap One, which was a huge step forward and enabled me to add some cool new features, including in-app feedback on note hits or misses (using MIDI input and output). In 2021, the Oculus Quest released its Passthrough API, opening up the possiblity of camera-based AR using their very affordable VR headset. Although the see-through camera technology is still rough and in its early days, it looks very promising.I continue to work on this project since the ultimate goal is to develop a fully-featured Virtual Piano Teacher.