I'm a computer music researcher. What I liked about both disciplines was that I could be truly creative. The act of programming, I believe, can be and should be an extremely creative endeavor; not only do you get the wonder and the joy of building things, but you get to craft, build, and then express. That's also why I love music. But with music, there's an emotional aspect that touches people in a universal way.

Our apps are panoplies of both [music and programming]. Everything from graphics, numerical analysis, programming, design, networking—all of these things go into every one of our apps. On the musical side, we are exploring the rules of Western tonality, how we can bend and break them, emotionally and psychologically, and how we can convey that to other people. Both fields are extremely rich. At some point in grad school it occurred to me that I could combine the two. And that's how I started at computer music research [as a student] at Princeton.

I started as a professor at Stanford in September of 2007, first in music and eventually in computer science. When I first came to Stanford I had absolutely no plans to found a company.

Right after I started, I met Jeff Smith. He had been the CEO of several startup companies and was going back to Stanford for music composition; he's an excellent pianist and a great composer.

That spring I was actually working on an extension of my dissertation at Princeton, a music programming language called ChucK. I knew that the fact that the iPhone is always connected would make it a truly revolutionary computing platform. It's the most personal computing platform we have today: it's the first piece of technology I use in the morning, and it's the last one I use before going to sleep at night. It's my alarm clock, my time waster, my motivator: in fact, the alarm clock isn't actually what wakes me up. It's only when I check my email—and panic—that I finally get out of bed. That level of intimacy made it clear that [the iPhone] aligned with my mission as a researcher. I had to do it.

It's not, entirely. We tried an experimental mobile phone orchestra [using Nokia N95s] around the same time that the iPhone came out. I was really pondering the super-smartphone, the modern smartphone. I was realizing that we might be at an inflection point of computing. We have these powerful compact devices that are intimate and personal.

But with the iPhone, we have multi-touch, an onboard speaker, CPU, GPU, mic, networking, and GPS. There is also this minimalism; the form factor is already there. A real, live ocarina is really no bigger than an iPhone. It's one of the smallest, simplest instruments we know of. (See Figure 11-2, a real ocarina.)

Figure 11.2. A modern ocarina. The earliest of the instruments are believed to date back 15,000 years. Because they do not rely on their length to produce their tone as flutes do, but instead create their sound from the resonance of the whole cavity, there is no standard length for an ocarina, and the placement of the holes—there are usually between four and twelve—is irrelevant and variable. (Photo credit: http://de.wikipedia.com)

We got to start fresh with iPhone. The design process here [at Smule] is one that doesn't involve cramming or porting. We do not want to take an existing experience and put it on the phone. We try to have the discipline to say, "What is available that we can use?" It's an "inside out" design approach. We didn't try to cram a large instrument into the phone; instead, the process was more looking for a sweet spot where we wouldn't have to sacrifice something. In the process, we realized this is more than a miniature computer. It's a thing unto itself. (Figures 11-3 and 11-4 are early mockups of Ocarina and show how Wang and his engineers mapped musical notes to the button layout.)

Figure 11.3. Ocarina uses just four finger "holes" but can produce a range of musical notes.

Figure 11.4. Mapping the interaction design behind Ocarina.

We can use ChucK to synthesize sound and to design sounds and interactions. It offers a really graphic turnaround of how a sound might turn out. There's actually a mini ChucK engine running on each phone. The microphone and buttons generate signals that ChucK analyzes with signal processing, extracting information like how hard are you blowing, then deciding to generate sound that is rendered out the speaker. ChucK handles both the interaction and the sound. (Figure 11-5 shows the ChucK operator symbols).

Figure 11.5. the ChucK operator symbols.

We built on what we knew from doing laptop orchestras at Stanford and Princeton, where we were essentially building completely new instruments on the laptop. In our instruments, there are equal doses of computer and human; the same is true for both the laptop orchestra and the iPhone.

Building the ChucK engine involved writing a ton of code. I am very much a C and C++ programmer, but transitioning was straightforward. A lot of our audio work on the iPhone has ended up being in C++; in the iPhone SDK you can write all C languages in the same project or even same file, so it was pretty easy to mix to three.

Listing 11-1 is a simple project in ChucK that allows a user to dither some user-generated audio input, perhaps from an iPhone microphone.

// dither.ck
// demo of dithering
//
// can use any UGen as source in play_with_dither()
// qbits : number of bits to quantize to
// do_dither : whether to dither
//
// gewang

// patch
Impulse imp => dac;

// sine wave generator
SinOsc s => blackhole;
220 => s.freq;

// go
play_with_dither( s, 2::second, 6, false );
play_with_dither( s, 2::second, 6, true );
.5::second => now;

play_with_dither( s, 2::second, 5, false );
play_with_dither( s, 2::second, 5, true );
.5::second => now;

play_with_dither( s, 2::second, 4, false );
play_with_dither( s, 2::second, 4, true );
.5::second => now;

// dither
fun void play_with_dither( UGen src, dur T, int qbits, int do_dither )
{
    // sanity check
    if( qbits <= 0 || qbits > 24 )
    {
        <<< "quantization bits out of range (1-24)", "" >>>;
        return;
    }

    // loop
    float sample;
    int quantized;
    (1 << 24) => int max;
    while( T > 0::second )
    {
        // get the last sample
        src.last() => sample;
        // quantize it
        if( do_dither ) // dither
            ((sample + Std.rand2f(0,Math.pow(2,-qbits))) * max) $ int => quantized;
        else // no dither
            (sample * max) $ int => quantized;

        // throw away extra resolution
        quantized >> (24-qbits) << (24-qbits) => quantized;
        // cast it back for playback
        (quantized $ float) / max => imp.next;
        // advance time
        1::samp => now;
        // decrement
        1::samp -=> T;
    }
}

The nice thing about computers is their generality. With laptop orchestra, we can build instruments that are tailored for experts or people who don't have much traditional training. There's a really nice spectrum of instruments. Every time we add a new piece to the orchestra, we redesign the whole thing [to be consonant]. The curse is that we have to start over every time, but we have a fundamental say in how the instrument works and sounds. Basically, we learned what was possible along a massive spectrum.

The instruments in laptop orchestra are meant for people who have experience with laptop orchestra, not necessarily experience with real instruments; we have both novices and expert musicians in that play with us. Some instruments pretty much deconstruct and reconstruct the actions of a traditional instrument by modeling the interactions of all parts of the instrument, and connecting them to parts of sound generating algorithms. Others are more bizarre. Ultimately, all the instruments are out of this world. The way you "play" them also varies. On one you might bow your finger across the track pad while manipulating the wind with thekeys.

One app that I really loved was Fieldrunners. It's one of the earliest games. It was clearly built with such quality, tender love, and care. These soldiers come marching [at your gate] and your job is to shoot them down. The graphics are inviting, the interaction is fluid, and it's my number-one most played game on the iPhone. That's saying a lot, since it's the first and last piece of technology I use during my day. Whether I am waiting in line, or just have some downtime, the phone is right there.

A game or a tool—I'm not sure what [they are]. There is a game element to them. I'd say it's a new type of computer-mediated social musical instrument. But it's built on this belief that everyone is inherently creative, and that by setting the conditions right, we can unlock creativity in anyone. Music is great for this, because it's hard to find someone who hates music. That's the social component. There's also the spontaneity; the sound is synthesized right on the phone, its not prerecorded—that's another social component. People can actually play an instrument for the first time on the iPhone. And people are doing it; we've had 20 million shared performances and over a million downloads. If you told me a year ago that a million people would be blowing into their phones to create music, I would have thought it was a real stretch.

Sonic Lighter was the first app. You can light it by flicking your fingers, or by holding it up to another phone—social "lighting." One phone emits a sound and another phone hears it.

If there's one app that influences the rest, it's Sonic Lighter. We had the globe in there, so you could see lighters all over the world. Leaf Trombone (pictured in early mock-up drawings in Figure 11-6) is definitely our most sophisticated app to date; it has really taken social experience to a new level. You can give each other feedback on your playing. We've had more than 400,000 judging sessions now.

Figure 11.6. Early drawings of Leaf Trombone showed the competition globe featuring prominently.

We try to make them absolutely as accessible to as many people as possible, but without losing the possibility for virtuosity. We don't want to make it just for expert musicians, but we want the expertise to be the goal. If someone practices at an instrument, they can get better at that it; we try to make our apps the same way. We try to cover all the bases here. There's a YouTube video of one user playing Oh Shenandoah, and she really plays it beautifully. We were like, "Wow."

It's not all about playing, though. Sometimes it's about listening. We had a review from a soldier in Iraq; in the few nights he has off, he said he takes out his iPhone and listens to music with Ocarina, and that it's really peaceful in the midst of all this chaos. To see someone from Japan play Amazing Grace, or hear the Star Wars theme from Jamaica—there's something poignant about it. (Figure 11-7 shows the Smule globe as it appears in Ocarina, which allows users to listen to other players all over the world.)

Figure 11.7. The Smule globe as it appears in Ocarina.

Well, we named the company Sonic Mule after my favorite sci-fi characters, the Mule from Isaac Asimov's Foundation Series. He's not the nicest of characters; he could use special powers to influence millions of people. In some ways that's what we want from our apps. We want to bring what we can do with computers and music to a wide audience to change the way people play music, listen to music, and interact through the vehicles of expressive audio.

Basically, we're trying to build things that people didn't know they liked to do.