MAGIC KEYS

Mixed Reality Piano Learning

Magic Keys is a Mixed Reality piano learning app using a fun and innovative way of learning the instrument. With a very accessible 3D notation system, similar to the likes of Guitar Hero, users can play along simple pieces within minutes without having to read sheet music. Using Mixed Reality technology the notes are directly projected on top of the real-world piano keyboard providing a seamless interface to the music. On top of that Magic Keys aims to become a fully-featured virtual piano teacher including a learning schedule and interactive lessons for learning technique, music theory, improvisation and much more.

My Vision

My goal with Magic Keys is to lower the barrier of entry for anyone wishing to learn the piano. I want to provide an innovative new way of learning or just having fun. Learning and playing the piano has been one of the most meaningful and rewarding activites I have done in my life. More and more people wish to learn the piano as an adult, however, for many the beginning seems too daunting to even try - from reading sheet music to navigating the 88 keys to coordinating left and right hand. My mission is to keep all these people enganged in following their dream and give them a fun platform to explore this wonderful instrument.

I am a game developer and a passionate pianist. When I first got into AR I was immediately hooked and couldn't think of a better use-case than piano learning. Combining my two passions this felt like a perfect match and I went on to spend the next few years developing prototypes for different hardware and researching digital music education.

Dominik Hackl, Creator of Magic Keys
Thanks for subscribing!

My Journey

I started working on this project in late 2017, initially as a prototype for a university course on Augmented Reality. Since I did not own any AR hardware at this time, I developed the first prototype for Android tablet using marker-based tracking. Even though the prototype was very rough and required the user to hover the tablet over the keyboard, it was still already clear that this could be of actual use.

I got access to the HoloLens, back then the only major AR headset available, and ported the application alongside with some major improvements. I continued to add features and tried to enhance the visualization of the music but I was greatly limited by the small field-of-view of the HoloLens back then.

In early 2019, I finally got access to a brand-new Magic Leap One, which was a huge step forward and enabled me to add some cool new features, including in-app feedback on note hits or misses (using MIDI input and output). The app at this point was a great proof-of-concept and was well-received by many testers. However, the low availablity of the Magic Leap One kept me from turning it into a full product which can be publicly released.

In 2021, Oculus released its Passthrough API for the Quest, opening up the possiblity of camera-based AR using their very affordable VR headset. Although the see-through camera technology is still in its early days, it looks very promising and finally provides an easy access to Mixed Reality.

The Virtual Piano Teacher

My long-term goal for this project has always been to turn it into a virtual environment for home piano learning. In the last years more and more people have started to learn the piano in self-study due to the increasing number of online resources including high-quality YouTube video lessons but also online learning apps and platforms. However, with the power of Mixed Reality and the sensory capabilities of modern devices like the Quest, virtual piano teaching can be brought to a whole new level. There are so many potential ways to enrich the piano learning experience but here are a few ideas:

  • Learning notes and the piano keyboard
    Simple exercises or games to recognize the note names and match them with the proper keys could increase the early-stage learning rate.
  • Technical exercises
    A whole lot of enjoyable and still pianistically valuable exercises could be realized. By adding a level system, students would be even more aware of their progress and more likely to remain motivated.
  • Dictionary of chords, scales etc.
    Players could look up all possible chords and scales and would be able to see them highlighted right on top of the keyboard.
  • Theory Lessons
    With AR and a 3D notation system music theory lessons could be designed in a very visual and interactive way.

Can't wait to build the future of music education 🎹