Solutions in this chapter:
The NXT features an internal speaker and the hardware necessary to drive it, thus making your robot capable of producing sounds. Those familiar with the sound system on the RCX robot will be pleasantly surprised with the NXT’s superior sound system and wider range of capabilities. Perhaps most significantly, it is possible to make your NXT speak, thus providing a very simple and direct way for it to communicate with you. Do not underestimate the sound features the NXT provides. Aside from the convenience of easily getting information from your robot, which will help in testing and debugging your programs, sounds and music are a fun way to give your robots a more defined personality. The NXT also features a sound sensor, allowing a certain amount of two-way aural communication between you and your robot.
The topic of playing sounds and music with the NXT is more closely related to programming than to building techniques. However, when you are dealing with robotics, the two matters are seldom separable. For some of the robots described in the second part of this book, sounds are an important component in their interface with the external world; for others, sounds are an interesting addition that enriches their behavior.
If you are not familiar with musical terminology or audio file formats, you might find topics in this chapter a bit complex. But the prize is worth the effort, because the techniques explained here open exciting new opportunities in your robot world. You will discover how to use simple tones, how to write melodies, how to control your robot with sound, and even how to convert digital audio files into sound effects that you can incorporate into your program!
The NXT features an internal speaker. There is little evidence of it on the outside: The NXT has four very small slits on the sides from which the sound emanates. You can alter the volume of the speaker via the Settings interface. The sound system of the NXT is designed to be accessed from your program; you have full control over the frequency (pitch) and the duration of the notes.
The following examples, written in RobotC, include four basic instructions on how to produce sounds, called ClearSound, Play Sound, PlaySoundFile, PlaylmmediateTone, and Play Tone. Using the Play Sound statement, the NXT can output one of nine predefined sound patterns, such as a blip, a double beep, or a short sequence of tones:
PlaySound(soundBlip); PlaySound(soundBeepBeep); PlaySound (soundFastUpwardTon.es) ; PlaySound(soundDownwardTones);
The Play Tone command plays a single note of a given pitch (in hertz) and duration (in hundredths of a second). The following statement plays a tone of 262 hertz for half a second:
PlayTone(262,50);
The NXT is capable of reproducing any frequency from 31 hertz to more than 18,000 hertz; however, you will usually limit yourself to the frequencies which correspond to the musical notes (see the table in Appendix C). All the programming languages built over the LEGO firmware offer this same feature, and most of the others include some kind of more or less sophisticated control over sound.
Sounds are the most immediate way your NXT has to inform you about a specific situation. There is, of course, the display, but it’s not always in sight, especially when your robot is running across the room! There’s also the ability to log data to a file on the NXT, but to use the file you would have to access it from your PC. Sounds, on the other hand, can be emitted by the robot without interrupting any other activities, and you can hear them even if the robot is out of sight or far away.
Through simple sound patterns, you can make your robot inform you that an operation has ended, something has gone wrong, its batteries are low, and much more. It can acknowledge the push of a button, or tell you it’s waiting for specific input from you. A sound emission can even be used to communicate with nonhuman robot fans, as in the case of Katherine Anderson’s SnackBot (see Appendix A), which fills a bowl with dog food before emitting two high-pitched tones to let the family pet know that his dinner is ready.
Sometimes a sound pattern can give your creatures a specific character. Could you imagine a silent reproduction of the famous R2-D2 droid from the Star Wars saga?
Music can enrich the personality of your robot even more than tone sequences. A wrestling robot probably appears more resolute if, while facing its opponents, it plays Wagner’s “Ride of the Valkyries” rather than a Chopin piano sonata or nothing at all. And any sort of dancing robot, without musical accompaniment, becomes just a robot that is swinging its arms and moving around.
One of the easiest ways to play music on your NXT is to use a piano tool, which allows you to simply click on a graphical keyboard to choose the notes you want the NXT to play. To access the piano tool in NXT-G, create a sound block and select Tone; the piano tool will appear in the configuration panel. For a more sophisticated tool, including a transposition function, try the piano tool in the Bricx Command Center application created by Mark Overmars and maintained by John Hansen (see Appendix A). This will create an NXT Melody file (with an .rmd extension) which you can use in programs you write for the NXT. Note that you also can use NXT Melody files in NXT-G sound blocks, but you should save them in the appropriate directory with an .rso extension in order to get the filename to appear in the list of possible sounds that the sound block can play.
A piano tool is a good way to create a short melody. However, if you want to code a longer piece, you may find using the RobotC PlayTone statement to be more efficient. Every note in the song requires two attributes: pitch and duration— the first expressed by a frequency and the second by a time. You must introduce delays between the notes to let the CPU wait out the note’s duration before playing the next note.
PlayTone(440,50) ; waitlMsec (50) ; PlayTone(220,100) ;
waitlMsec(100) ;
In this example, the NXT plays an A (440 hertz) for half a second, waits for the note to finish, and then plays another A (220 hertz) one octave below the previous note for one second.
The NXT is limited to playing a single note at a time; thus, we say it’s a monophonk device. It is not capable of playing chords, which require two or more notes played at the same time, but you can adjust note timing to get various effects. In our previous example, the duration of the first note filled the entire interval before the second note, thus producing a legato effect. You can just as easily get a staccato effect—shortening the duration of the note inside the interval produced by the waitlMsec statement—by introducing a pause with no sound between the two notes:
PlayTone(440,10); waitlMsec (50); PlayTone(220,100);waitlMsec (100);
Coding a melody by hand is a long and tedious task. What happens if when you’re finished, you discover that the execution is faster or slower than what you intended? Unfortunately, you’d have to go back and change all the time intervals. A better approach takes advantage of a feature that all textual programming environments offer: the definition of constants. Using constants, you can make all the intervals relative to a specific duration that controls the execution speed:
#define BEAT 50 PlayTone(440, BEAT); waitlMsec(BEAT); PlayTone(220, 2*BEAT);waitlMsec(2*BEAT);
The preceding code behaves exactly like our first example, but you’ll see that by having defined a constant, the code is clearer and easier to maintain, simply by changing the value of BEAT to change the overall speed. We can extend the usage of constants to include note frequencies as well, making our code more readable:
#define BEAT 50 #define A3 220 #define A4 440 PlayTone(A3, BEAT); waitlMsec(BEAT); P1ayTone(A4, 2*BEAT) ; waitlMsec(2*BEAT);
You can also patiently define a table of constants for all the notes, so you can reuse them in many different programs:
#define Cl 33 #define Csl 35 #define D1 37 #define Dsl 39 //... #define C4 262 #define Cs4 277 //... #define B8 7902
The preceding table represents, for example, the D# note as Ds(D sharp) because most languages don’t allow the use of special symbols such as # in the names of constants and variables. Don’t worry about the length of this table, because constants get resolved by the compiler and don’t change the length of your actual code or the space it takes up in memory.
Creating a soundtrack for your robot is a typical example of where multitasking proves to be really helpful. You will typically enclose your song in a separate task, starting and stopping it from the main task, as required by the situation.
If the preceding instructions for creating music for your NXT seem too involved, or if you’re not familiar enough with music concepts to use them, don’t despair: You can still play music on your NXT and even make it speak by using tools that convert different types of sound files into code that your NXT can understand.
The Musical Instruments Digital Interface (MIDI) is a complex standard that includes communication protocols between instruments and computers, hardware connections, and storage formats. A MIDI file is a song stored in a file according to the format defined by this standard.
MIDI files have achieved incredible success among professionals, amateurs, and instrument manufacturers, and they are by far the most preferred way for musicians to exchange songs. For this reason, you can easily find virtually any song you’re looking for already stored in a MIDI file.
But what is a MIDI file? It is simply a sequence of notes to play, their duration, their intensity, and, of course, a code that denotes the instrument to be used. Thus, a MIDI file is not an audio file. It does not contain digital music such as CDs, WAV files, MP3 files, or other common audio formats. Rather, it contains instructions for a player (either a human being or a machine) to reproduce the song, almost a score, to be performed by actual musicians. And as with a real score, the result rests heavily on who actually performs it. For MIDI files, this means that the output depends on the device which renders the music: With a professional MIDI expander, you can get impressive results, whereas execution of the notes by a low-end PC audio card will probably be very poor. What makes MIDI files so interesting to musicians is that they are easy to read and edit (with special programs) in terms of standard musical notation.
So, the key question is whether there’s a way you can render MIDI files with the NXT. Though you cannot import them directly to the NXT, there’s a very nice utility that can convert any MIDI file into the proper code: MIDIBatch, a free conversion utility that is part of the Bricx Command Center project. This utility runs on both Windows and Mac OS X machines and produces NXT Melody files, which have an .rmd extension.
Before we provide details regarding how to use MIDIBatch and what it can do for you, there’s another characteristic of MIDI files you must be aware of. The notes inside a MIDI file are grouped into channels, and each channel is assigned to the instrument meant to reproduce those notes. For example, channel 1 could be assigned to an Acoustic Piano, channel 2 to a Bass Guitar, channel 3 to a Nylon String Guitar, and so on. Channel 10 is always assigned to Drums, and channel 4 is usually, but not always, assigned to the melody line— that is, the notes sung by the vocalist or played by the leading instrument. As explained earlier, the NXT has monophonic sound capabilities and cannot reproduce more than one note at a time, so you have to carefully choose the notes it plays.
Before you start converting a MIDI file into code, we suggest you explore using specific software to see which channel could better render the idea of the song. Many commercial products are capable of manipulating MIDI files in almost every possible way, but you don’t actually need all the power and complexity they provide. The Internet is crammed with freeware and shareware programs perfectly suitable for the task of identifying the best single channel to be converted into instructions for the NXT.You open your MIDI file with the editor, mute all the channels except one in turn, and decide which one to use. If you feel at ease with the MIDI editor, you can cut away some notes from the selected channel, because you probably don’t need the whole song, only a chunk of it (the part that contains the refrain or main theme). If you do this through editing, you will save the modified MIDI file, of course.
Note
You can save a lot of work if you find a MIDI file meant to be used as a ring tone. These typically have sound reproduction limits very similar to those of the NXT.
Now you’re ready to use MIDIBatch. MIDIBatch can convert many files at a time into the NXT Melody format. To do this, open a MIDIBatch window and type the name of the file folder where the MIDI files are stored in the Input Directory box. Choose a directory for the output, whether to convert all channels or just the first one, and click Convert. Converting all of the channels is usually not a good idea: The result will be almost unrecognizable. To hear your NXT Melody file before you download it to your NXT, you can use another Bricx Command Center utility called RMDPlayer. If you’re satisfied with the result, download the .rmd file to your NXT for use in your programs. Using the RobotC PlaySoundFile statement is one way to do this:
PlaySoundFile("pachelbel_canon.rmd");Another handy utility that is part of the Bricx Command Center is WAV2RSO, which is an application that converts WAV files into the NXT sound file format and vice versa. Unlike MIDI files, WAV files contain digitized audio ready to be executed. If you are familiar with graphics file formats, you can think of MIDI files as vector graphics, whereas WAV files resemble raster graphics.
There are some limitations to be aware of when converting WAV files into RSO files for use on your NXT. The first is that the resulting RSO file will likely be huge by NXT standards. For example, a WAV file created using text-to-speech software with just the word Hello, when converted to an RSO file, may take up 20 KB. Just a couple of seconds of a recorded song could require hundreds of kilobytes of space. Converting WAV files to RSO files does decrease their size by approximately half, but space is still a major consideration. In fact, WAV2RSO does not allow WAV files larger than 64 KB to be converted, likely because the memory capacity of the NXT is only about 130.7 KB in total. You want to save a little room for your program!
Another limitation is the volume at which an RSO file is played. You may find that playing an RSO file while the NXT’s motors run is pointless, because the whir of the motors drowns out the sound coming from the small speaker. This is less of a problem with .rmd files, so if you want your robot to move and sing at the same time stick with NXT Melody files rather than converted WAV files.
Despite these limitations, the ability to play sound effects and to hear your robot “talk” makes WAV2RSO a great tool for enhancing your robot’s character. Look for WAV files that emulate sci-fi sound effects such as laser guns, jump sparks, and buzzing. Seek out free text-to-speech conversion software on the Web and create WAV files of phrases you want your robot to say. Then, use WAV2RSO (which you can use in a manner very similar to MIDIBatch) to convert these files into RSO files. Upload the RSO files to your NXT, use them in your programs, and watch your robot come alive.
Aside from making sounds, the NXT is also capable of detecting sounds and reacting to them. When the sound sensor is attached, the NXT can “hear” sounds happening around it and determine the decibel level of the sounds. The NXT cannot distinguish different tones, but it can tell the difference between a soft noise and a loud one. So, for example, you can’t program it to start when you say “Start” and stop when you say “Stop,” but you can program it to adjust its speed (for example) according to the level of sound that it is detecting.
When the sound sensor is attached to the NXT, the robot constantly polls the sensor for data. The sound data can be normalized to a percent value, so the value of the sound sensor can range from 0 to 100. RobotC provides access to this value via the Sensor Values array. It also provides a configuration wizard for creating the necessary constants to help you access this array. For example, if the sound sensor is plugged into sensor port 2, the wizard will place this code at the beginning of the RobotC program:
const tSensors soundSensor = (tSensors) S2;
Now the soundSensor constant provides an index into the SensorValue array, and this value can be assigned to the motors:
while (true) {
motor[motorB] = SensorValue[soundSensor];
motor[motorC] = SensorValue[soundSensor];
}You can use the sound sensor to make one robot appear to control another. For example, program Robot 1 to play a sound file with a voice saying “Follow me” and have Robot 2 move forward when it detects a certain level of sound. Keep in mind that the noise of Robot l’s motors may also be enough to set Robot 2 in motion, depending on how close together the robots are and what level of sound you’ve programmed Robot 2 to respond to. In the following example, Robot 2 waits until the sound sensor value reaches 15 percent before following Robot 1:
while (true) {
waitlOMsec(100);
currentValue = SensorValue[sound];
if (currValue >=15) {
motor[motorB] = 100;
motor[motorC] = 100;
}Through some skilled manipulation of NXT-G, Sivan Toledo has created a clap counter for the NXT. Using sample sound data gathered by the NXT, Toledo was able to determine a typical pattern of values that his NXT would detect when he clapped. Using this, he enabled the robot to count the number of claps it detected and show that number using the NXT graphical display. The NXT-G code for the clap counter is available on Toledo’s Web site (see Appendix A).
The purpose of this short journey into the sound system of the NXT was to show that, despite its limitations, it’s still an invaluable resource. It can support you in debugging, return information in the form of sounds of different patterns or frequencies, and complete the personality of your robots.
RobotC offers two commands to control the sound system: PlaySound to perform predefined sound patterns, and Play Tone to play any note of a desired pitch for the desired duration. Whereas Play Sound is suitable for most user interface needs, Play Tone offers finer control and lets you create melodies.
Thanks to the work of independent developers, you can convert some of the most common digital audio formats straight into either RSO files or NXT Melody files. Considering the hardware limitations of the NXT, MIDI files translate very well and are the ideal candidates to provide your robots with a musical soundtrack. The conversion of WAV files is somewhat problematic because of the large size of a typical WAV file. Nevertheless, this sound format can equip your robot with amazing sound effects, including speech.
The sound sensor makes the NXT capable of responding to noises, though it can distinguish sounds based only on their decibel levels and not on the nature of the sound. Still, the possibilities for utilizing the sound sensor are broad and the field is young. Sivan Toledo’s clap counter hints at future work that could include teaching the NXT to detect certain sound patterns and respond in different ways.