For my grad school thesis, I’m designing an intro-level music education app. I’m operating within the techno/hip-hop paradigm, with an Afrocentric rhythm-oriented approach. Electronic dance music production software had brought me much joy over the years, joy that I’m eager to spread to more people. I firmly believe that everyone is a potential musician, and that the right interface can draw beginners in and motivate them. So as I ponder this project, I’m naturally giving a lot of thought to electronic music interfaces, both software and hardware. And because all interfaces on a screen necessarily involve some music visualization, I’ve been exploring that too. For example, here’s a particularly attractive music interface/visualization, the pitch correction program Melodyne:
Earlier this summer I took Advanced Computer Music Composition, which included a lot of history of the twentieth century avant-gardists. While these people have had a lot of not-so-wonderful ideas about music, they have done a lot of interesting experiments with novel interfaces.
Graphical notation
Long before there was music software, music visualization was done on paper. The most widely used music visualization method today is still western music notation.
But standard notation isn’t the only way to visually represent music. Composers have been using alternative schemes for centuries, including various forms of graphic notation. Iannis Xenakis was one of the major innovators in the field. Though the music itself is fairly unbearable, Xenakis’ initial sketch for “Metastasis” is a beautiful work of art in its own right.
Brian Eno’s “Music For Airports” included a graphical score on the back of the album cover.
WFMU has a nifty gallery showing more graphic notation.
Drawing music
The graphic score concept has been worked directly into a few actual electronic music interfaces. The ANS synthesizer, invented by Evgeny Murzin in 1938, translated drawings directly into audio. The initials stand for Alexander Nikolayevich Scriabin, the mystical synaesthetic composer to whom Murzin dedicated his invention. Read more about it on Boing Boing.
The Fairlight CMI let you draw audio waveforms freehand on a screen using its clunky light pen, which you could then play back on the keyboard.
The idea of drawing on an electronic surface and having it make sound is attractive. Yet the idea hasn’t widely taken hold, even though you could do such a thing with any laptop, tablet or smartphone. Why aren’t we all drawing music? I guess it’s because there’s no existing instinctive relationship between drawing and sound, so as intellectually exciting as the idea is, in musical practice it doesn’t connect.
The idea of controlling electronic sounds using graphics also suffers from score-centrism. Writing and drawing come from different parts of the brain than singing and dancing, where music originates. It’s counterintuitive to abstractly conceive the music in visual form first, and only hear it afterwards. Besides, visual appeal and musical appeal are two different animals. Gérard Pape said it best:
Sometimes the pages that are the most attractive visually don’t sound well – beauty on a visual level does not necessarily correspond to interest on a sonic or musical level.
And why should anyone expect otherwise? Being disappointed that a Picasso doesn’t translate into good music is like being disappointed that it doesn’t taste good.
Keyboard-centrism
The keyboard has dominated electronic music interfaces since the Telharmonium, and for good reason. It’s a straightforward and intuitive mapping of one key to one pitch, and the idea of hitting the key harder to play the sound louder mirrors the familiar analog world. The MIDI standard has entrenched the keyboard metaphor even more strongly, and its “piano roll” sequencing paradigm translates effortlessly to and from standard notation.
The keyboard paradigm also extends into environments where there’s no keyboard involved. Here’s the scale entry interface in Auto-tune.
Alternatives to the keyboard
Electronic instruments have been based on non-keyboard instruments like the saxophone or violin, and even the accordion.
Like the keyboard, these interfaces stick close to analog reality in their parameters and mappings. It doesn’t take much imagination to have your MIDI violin map fingerboard position to pitch.
Once you’ve got the eyes, ears and fingertips involved, is that a complete interface? Maybe not. I like the story of Paul DeMarinis‘ touch-sensitive guitar controller. Each version of his instrument kept moving closer and closer to the construction of actual electric guitars. DeMarinis refers to the smell of the leather straps as evoking the rock and roll fantasy so integral to the electric guitar experience. How many electronic interface designers think about smell? More of them should.
What if you want to leave the acoustic instrument paradigm entirely? That means having to determine the mappings from gesture to sound entirely from scratch, and this is where you run into some major challenges. The past century has seen a lot of fascinating experiments in non-traditional control schemes from the theremin on down, but none of those devices have posed much of a challenge to keyboard-centrism.
Here’s a profoundly non-keyboard-centric interface, the Buchla 200e analog synth.
If it looks confusing, it is. You need to know a fair amount about signal flow and the fundamentals of analog audio to make sense of it. It gives you incredible control over every nuance of its sound, but the learning curve is seriously steep.
Morton Subotnick, Buchla co-designer and staunch opponent of keyboard-centrism, has conducted a variety of experiments with nontraditional interfaces. His most adventurous method is to use one sound’s envelope to control the envelope of a different sound.
The possibilities of this approach are endless, but the results are fairly impenetrable to listeners. How are you supposed to know what sound is controlling what? One exception is Subotnick’s use of his voice to control envelopes on the Buchla. There, the relationship between his bodily movement and the Buchla’s response is more intelligible.
Joan La Barbara’s use of the Interactor system to control a video camera with her voice is another example of the movement-result relationship problem. The mapping between a particular dimension of her voice and a particular camera move is totally arbitrary. There’s a similar problem with Laetitia Sonami’s Lady’s Glove. It, too, is a fascinating idea, but people keep developing glove interfaces and they keep failing to catch on. Again, the problem is the arbitrariness of the mappings. Can a community of shared practice develop around one of these arbitrary mapping schemes? If not, it’ll necessarily be limited to an interesting one-off experiment.
Without reference to the analog world, how are musicians and listeners supposed to understand the relationship between a particular gesture and a particular sound? Morton Subotnick laments that when the gestures in his stage pieces became too abstracted from the sounds they triggered, it would have made no difference if he just played a tape of the desired sounds and had the performer mime along.
There’s a steep learning curve facing any musician wanting to use a non-analog-centric interface. You have to learn the mappings before any expression is possible, and then the audience has to go through the same learning process before the music will be fully relatable. It’s no wonder that interface designers keep returning to drums and keyboards. For example, the ubiquitous Akai MPC model, mimicked in many devices since, is essentially a drum for the fingertips. The relationship between hitting a pad and hearing a sample played back is obvious and appeals to our intuition.
Motion capture and haptic feedback
Motion-capture systems like the Lady’s Glove suffer from the lack of haptic feedback. They don’t push back when you push them. A guitar gives strong haptic feedback — when you bend a string, it requires more pressure, and as the string digs into your fingertip, you get an instant read on how hard you’re pushing. Motion capture works fine in dance-oriented video games, where the player is imitating a specific set of movements. As a music generation method, however, motion is largely an unsolved problem. There’s no physical pushback to your movements, and no set of conventions to hinge your learning or expectations around. Each system needs to be learned conceptually from the ground up, with no bodily guidance from the system itself. This is a long distance to travel before you can get down to expressing emotions.
Robert Moog devised a super cool multiple-touch-sensitive keyboard that not only prefigures the iPad, it offers the major advantage of haptic feedback. The touchscreen is too incorporeal, not as much as the theremin or motion capture, but still not as satisfying as hitting keys or drum pads. There are experiments underway to make touchscreens haptic, using slight electric charges in the surface to stimulate nerves in the fingertips and create the feeling of roughness, different materials and surfaces. I’m encouraged by this development. I like the iPad as a music interface, but the eyes and ears aren’t enough for a completely satisfying expressive tool. There has to be touch too.
Tod Machover’s hypercello tries to bridge a familiar instrument paradigm with futuristic motion sensing. The hypercello maps the cellist’s movements to sound parameters at many different levels of granularity. For example, ten different bowing intensity levels can produce ten different sounds entirely. A single cellist could control a vast array of electronic sounds on many dimensions simultaneously. But Machover expresses a crucial misgiving about his idea:
I’ve often thought about the model of the one-man band on stage and been a bit scared of that as a comic image. But it should be possible to extract that amount of information from a single player on a single instrument.
It certainly is possible, but is it worth it? The one-man band image is well-chosen. Why do we find these guys ridiculous instead of impressive? I think it comes from the intrinsically social role of music. A key part of the pleasure of hearing an ensemble lies in experiencing (or imagining) their interrelationships. We see a one-man band and subconsciously wonder why the dude couldn’t find any other people to make music with.
Maybe a motion capture standard will eventually catch on the way MIDI did, and Microsoft Kinect-like controllers will become as ubiquitous as keyboards. I’m not holding my breath. There’s no instinctive causal relationship between movement and sound generation. Usually it’s the sound that inspires the movement.
The novelty problem
Dorothy Martirano describes getting nonviolin sounds out of a MIDI violin as “exciting.” My first experience with MIDI guitar was similarly thrilling; being able to bend notes on a piano had a delightful dreamlike quality. But the novelty of upending expected sound relationships wears off quickly. The challenge is to find an emotional language for one instrument sounding like another. Jerry Garcia played some nice flute from his MIDI guitar, but for the most part, the idea still mostly hinges on its “gee whiz” quality more than actual musical expressiveness. The same goes for the more wacky MIDI interfaces, like the drumpants. They’re fun, but I’ll only get interested when pants prove to be a better percussion control system than regular old drum pads.
Video game controllers offer a way forward
The past couple of generations have grown up controlling software using novel interfaces in video games. Thumbsticks, buttons, touchpads and screens, and most recently motion controls are part of daily life for most Americans under 40 and quite a few over. Game controllers are quickly making their way into electronic music, helped along by music-specific games like Guitar Hero. The surprise of seeing a Wiimote onstage is rapidly wearing off, and I expect such controllers to eventually become commonplace in music. It’s no accident that music and games share the verb “to play.” The more they converge, the happier our musical culture will be.
Quotations come from Electric Sound by Joel Chadabe