Game controllers as musical instruments

This is a picture of my electronic funk-soul-R&B band doing a show. From left to right, it’s Nicole Bishop, me and Barbara Singer. We were the whole band for that show. I did all the beats, samples and keyboards from my computer using a video game controller.

Here’s a screenshot of the program that the game controller is connected to.

The outer space background is my desktop image and isn’t part of the program itself. But maybe it should be.

Hear the game controller in action on the synth in this track:

Take The 2-3 Train by ethanhein

The software maps the buttons and knobs on the controller to different MIDI values. I can play one octave of each of a few different scales (blues, major, harmonic and melodic minor, diminished) in all twelve keys. I can scroll through the circle of fifths with the controller’s D-pad. It’s set so that my left index and middle fingers control the root and third of the scale, my right index and middle control the fourth and fifth, and my right thumb reaches the rest of the scale tones. With the thumb sticks I can control pitch bend, modulation and other parameters, depending on which software instrument is dialed up.

The controller plays anything that any other MIDI instrument can play, not just synthesizers. I can map any batch of recorded sounds to the buttons. It’s fun loading bells or speech samples or bird calls onto it and playing them through heavy delay over a beat.

The controller interface software was written by Ben Lacker in Max/MSP. It works with any USB video game controller, but it was specifically designed for the one in the screenshot, a Logitech Dual Action Gamepad.

I mostly played guitar in my bands through my twenties, using lots of digital delay and other high-tech effects. As my sound got more electronic I started using a keyboard hooked up to my laptop. For a while I was carrying around a Korg 49, which has a bunch of cool drum pads and control knobs in addition to a half-piano’s worth of keybs. It was way more controller than I needed. I felt kind of like a chump carrying such a big instrument around just to play one note while twiddling a knob for the entire song. Part of the motivation to set up the game controller was to be able to have the same control scheme on a device I could more easily carry around on the subway.

The Korg 49’s keys and drum pads are pressure-sensitive. The game controller isn’t. Its buttons have only has two settings, on and off. It offers no control of dynamics at all. This limitation has turned out to be mostly a good thing for live situations, and even for home sequencing. For samples especially, it sounds better to mix everything to a nice balance and then be forced to keep it that way. It moves my complete focus to rhythm. I can pitch bend or filter with the thumbsticks for expressiveness when I need it.

I’ve tried a few other game controllers with the MIDI interface program. Some of them show potential. The most intriguing one is the Dance Dance Revolution pad. It would be perfect if it didn’t map itself to strange MIDI parameters by default. Out of the box, half the buttons don’t do anything useful, and I don’t have the programming mojo to fix it. Maybe in the future I’ll get it ironed out. It could be like a customizable, more ergonomic version of the giant ground piano in Big, as seen in this extremely bootleg Youtube video.

The Guitar Hero and Rock Band controllers have potential too, but they don’t have as many buttons or parameters as the Logitech pad. Same with the Taiko Drum Master controller. This is nothing against any of these controllers in their original contexts, where they work great. I haven’t gotten to try DJ Hero but I expect it’ll be a similar deal. The Nintendo Wii controller is becoming the game controller of choice for futuristic computer musicians. I haven’t used one for anything except games yet, but there are some cool-looking things on my list. Specifically, I’m looking forward to experimenting with Wii Loop Machine and Wii Drum High. There are also some groovy-looking things for the Game Boy DS, like Korg DS-10 and Nitrotracker. For all of the above plus iPhone there’s a thing called Osculator that looks fun.