In the course of designing my NUI musical instrument, I've encountered a number of valuable lessons about NUI design. This blog assignment covers some of the specific things I've learned, divided into the categories of "characteristics and limitations of specific NUI input technologies," "Low-level NUI design," and "High-level NUI design."
NUI Input Technology
My instrument uses a Leap Motion device for input, which detects hand positions, poses, and gestures. The strength of the Leap is that a lot of the groundwork for hand recognition has already been laid by the Leap team. The latest Leap SDK provides a lot of control for recognizing hand positions, down to the level of each finger bone. In my design, I use different hand positions to determine the behavior of the instrument, and some of the more subtle hand position fallbacks would have not been possible under the old SDK. I haven't implemented any gesture controls yet, but it's nice to know that, if I wanted to, I could choose from a library of predefined gestures including circles and swipes.
The biggest limitation of the Leap that I've encountered is the size of the "input box," or the region in which hands can be detected. Because it only extends about two feet above the device, I had trouble finding a range of musical notes that were far enough apart to allow distinct note selection without being such a small range that real song playing was impossible.
Low-Level NUI Design
My instrument design uses only a few primitives, which makes implementation much simpler: vertical position controls pitch, horizontal position (toward and away from the musician) controls volume, and the number of fingers extended enables continuous or half-step mode.
The most significant low-level design concern I have dealt with is the registration/continuation/termination process. In my design, registration and termination are handled by passing the hand through the "zero-volume" plane. No note will play until the hand passes into the "active" zone, and when it is removed from the zone, the note is stopped. This caused some problems in early designs because the note failed to terminate if the hand exited the input box to the left or right instead of passing through the zero-volume plane.
High-level NUI Design
Much of the high-level design in my instrument remains to be implemented. The finished version will include a "beginner" and "advanced" mode to allow users to learn the basics of the instrument and progressively add more complex interactions. The GUI component of my interface will also include the visual metaphors of volume gauges and indicate pitch using colored bars.