How to sonify data in your C or C++ application, using VSS.
Camille Goudeseune
1. Overview
This document explains how to perform "data sonification" in your application.
It assumes that your application is written in C or in C++,
and that you have a copy of VSS available,
version 3.1 dated 1/1/1999 or later.
It assumes that you've managed to get basic sound running
as described here.
Sonifying data here means making sounds which:
- offer feedback for user actions (clicking buttons, waving the wand);
- reflect the state of the application ("data-driven" sound);
- alert the user to changes of state in the application (warnings).
2. Where will sound improve my application?
2.1 A checklist.
- Does the application have several different "modes" it can be in?
Play different background sounds to remind the user which mode they're in.
- Do the GUI controls take effort to point at or adjust?
Trigger short sounds to give feedback for selecting and releasing
a control; maybe, play a continuous controllable sound while the
control is being adjusted.
- Do emergencies arise, where the user of your application needs to be
interrupted from their main task to attend to the abnormality?
Have the emergency trigger an alarm sound in addition to any visual
alert: the sound will capture the user's attention more strongly.
2.2 Should I use audio or video for a particular message?
Use auditory presentation if: |
Use visual presentation if: |
The message is simple. |
The message is complex. |
The message is short. |
The message is long. |
The message will not be referred to later. |
The message will be referred to later. |
The message deals with events in time. |
The message deals with events in space. |
The message calls for immediate action. |
The message does not call for immediate action. |
Derived from:
B.H. Deatherage, "Auditory and Other Sensory Forms of Information
Presentation." In Van Cott and Kinkade (Eds.), Human Engineering
Guide to Equipment Design, Revised Ed., Washington: U.S. Gov't Printing
Office, 1972, p. 124.
Two other general references:
Kantowitz and Sorkin, Human Factors: Understanding People-System
Relationships. NY: Wiley, 1983.
Sanders and McCormick, Human Factors in Engineering and Design,
6th ed. NY: McGraw-Hill, 1987.
You have to be looking at a visual message to get it;
audio messages you'll get no matter where you're looking.
2.3 Sounds for particular VR tasks.
Several ideas for VR issues in this section came from the paper:
NCSA Technical report TR032,
Human Factors in Virtual Environments for the Visual Analysis of Scientific Data,
by M. Pauline Baker and Christopher D. Wickens, 1995.
(Available here.)
Searching through a data set.
VR worlds.
Many copies of the same sound.
Localizing sounds.
2.4 Using VSS with the CAVE
Before starting up the whole CAVE library, it makes sense to verify that the
run-time environment is intact. Also, note that CAVEExit() actually exits
the program, so don't put any code after it. This leads to the following order:
- BeginSoundServer()
- AUDinit()
- CAVEConfigure(), CAVEMalloc(), CAVEInit(), CAVEDisplay()
- main loop: AUDupdate()'s
- (if needed:) AUDterminate()
- EndSoundServer()
- CAVEExit()
You may prefer to call BeginSoundServer() after
CAVEConfigure(), not before, in order to automatically pick up
the environment variable $SOUNDSERVER which the cave configuration
files can specify for you. Note carefully how this works:
- The cave machine's system.caverc file may specify a default value
for $SOUNDSERVER.
- If your application's own .caverc file doesn't
specify a different value, system.caverc's value will be passed
to your cave application by the call to CAVEConfigure().
- If you need a value other than the default (you're running VSS on
a different machine this week, say), then your own .caverc must
specify putenv SOUNDSERVER machinename. (The "putenv" is required!)
- We discourage specifying $SOUNDSERVER manually in this context
of .caverc files; behavior may be erratic, particularly across different
versions of the CAVE libraries.
- If all else fails, add debugging statements to your application like
"printf("$SOUNDSERVER is currently <%s>\n\n", getenv("SOUNDSERVER"));.
If you use "-L" to specify directories when linking,
it's a good idea to put any VSS-specific directories before
CAVE directories. Otherwise you might link with an unintended copy of libsnd.a
from a CAVE directory.
3. How do I try out the various sonifications?
- Start up VSS.
- Change to the directory
sonifications.
- For continuous sounds, type one of the following commands:
audpanel continuous_probe.ap continuous_probe.aud
audpanel continuous_gurgle.ap continuous_gurgle.aud
audpanel continuous_vowel.ap continuous_vowel.aud
audpanel continuous_whistle.ap continuous_whistle.aud
audpanel continuous_dimple.ap continuous_dimple.aud
audpanel continuous_wind.ap continuous_wind.aud
audpanel continuous_puttputt.ap continuous_puttputt.aud
To adjust the sound, wiggle the sliders of audpanel.
To reset or end the sound, click RESET or EXIT.
(You can find the program audpanel in the directory where
VSS is installed.)
- For triggered sounds, type one of the following commands:
audpanel triggered_swoop.ap triggered_swoop.aud
audpanel triggered_beep.ap triggered_beep.aud
To adjust the sound, wiggle the sliders as before.
Either click HIT to trigger sounds one at a time,
or click SENDING to generate a stream of sounds
(use the Interval
slider to adjust the duration between successive sounds, but
don't make the duration too short).
To reset or end the sound, click RESET or EXIT.
5. How do I add one of these sonifications to my application?
First of all, get the basic VSS structure in place:
#include "vssClient.h", BeginSoundServer(), EndSoundServer(),
AUDinit(), makefile, .aud file.
5.1 How do I add a beep, swoop, or other controllable triggered sound to my application?
- Note the name of the sound's message group.
Note the sound's number of parameters, their meaning, and their
suggested range. Here:
- "Beep";
Pitch (40-4000), Brightness (0-10),
Loudness (0-1), Duration (0.05-2).
- "Swoop";
Start Pitch (40-4000), End Pitch (40-4000),
Brightness (0-10), Loudness (0-1), Duration (0.05-2).
These numbers come from the .ap file, the lines called "minValues"
and "maxValues".
- Find the place in your code where you want the triggered sound
to be played.
- Add the following function call to your code:
{ float myArray[4] = { ... };
AUDupdateSimple("Beep", 4, myArray); }
The array of floats should contain four values
corresponding to the sound you want to hear. (If you're using
a swoop instead of a beep, the function call would be:
{ float myArray[5] = { ... };
AUDupdateSimple("Swoop", 5, myArray); }
and similarly for other triggered sounds.
5.2 How do I add a controllable continuous sound to my application?
- Note the names of the message groups needed to start, modify, and end
the sound. We use
continuous_probe.aud
and
continuous_probe.ap
as an example here:
"ButtonDown", "WandMove", "ButtonUp" respectively.
- Note the parameters of each message group. Typically, the message groups
used to start and to modify the sound will have the same parameters
(so you don't need to send a WandMove nanoseconds after the ButtonDown).
Here:
- "ButtonDown": Loudness (0-.5), Pitch (7-9), Width (.001-.15)
- "WandMove": Loudness (0-.5), Pitch (7-9), Width (.001-.15)
- "ButtonUp": (no parameters)
These numbers come from the .ap file, the lines called "minValues"
and "maxValues" for each message group.
- Find the place in your code where you want the continuous sound to start.
- Add the following function call to your code:
AUDupdateSimple("ButtonDown", 3, array_of_three_floats);
The array of floats should contain three values
corresponding to the sound you want to hear.
- Repeat these two steps for where you want the sound to stop, and for
any places you want to modify the sound.
If a message group has zero parameters, as with ButtonUp here,
use this function call:
AUDupdateSimple("ButtonUp", 0, NULL);
6. A fancy example including C code: pinball
This is a simple physical model that
generates interesting patterns of discrete events and continuous changes
of parameters. A ball moves around a playing field with three kinds of
obstacles; friction and gravity affect the ball's motion as well. The
first kind of obstacle is a simple wall, which reflects the ball in the
usual way. The second kind is an accelerator, which increases the ball's
speed while maintaining its direction. The third kind is a randomizer,
which preserves the ball's net speed but randomly changes its direction.
Run this example by starting VSS,
going to the sonifications/pinball directory
and typing one of the following:
- pinball simplest.aud
- pinball 2fm.aud
- pinball 2fm_goodcollisions.aud
- pinball addsyn.aud
(You might have to type make first to create the file pinball.)
It works best if your terminal window is 80 columns wide
and at least 40 columns high.
Press q or the escape key to quit.
Obstacles, friction, and gravity are implemented by the functions
constrainmotion() and moveball()
in the file pinball.c.
For details on how the playing field works, refer to the manpage for
the "curses" library (man curses). This graphics
library works on any Unix machine.
When the ball collides with an obstacle, an appropriate sound sample
is played using the SampleActor.
Well, that's oversimplified.
When the ball collides with an obstacle, what really happens is
that a
variable (fHitBumper, for example) is incremented by some amount (4).
Every frame (time step) of the simulation, this variable is decremented by 1.
The sound sample is played only on a "rising edge" of this variable --
when it is now nonzero but was zero in the previous frame.
We do this to avoid playing several overlapping sound samples in a very
short space of time, if the ball hits several of the same kind of obstacle
in quick succession. This is computationally more efficient for the sound
server, and avoids the danger of hard clipping from playing too many
sounds at once. The code for this is the first part of
sendaudiodata() in pinball.c.
A better design would use an actual timer instead of just counting frames.
The frame rates of other graphics libraries (GLUT, CAVE, etc.) may not
be as steady as that of the curses library!
The second part of sendaudiodata() deals with continuous sounds
derived from the ball's motion, in particular its absolute speed vBall,its height by and its vertical speed dy. vBall
is mapped to three
different parameters which are stored in the array speedarray
and passed
to the message group mSpeed for use by a pair of FmActors
in the file
2fm.aud.
Similarly, by and dy are mapped to
values in the array
ballArray which is sent to the message group mBallY.
These blocks of
code are enclosed in "if (fabs(x-xPrev) > threshhold)" tests.
When such a test fails (for example, if x==xPrev),
it prevents the corresponding AUDupdateSimple() from being called.
This reduces the number of messages being sent to VSS,
which gives VSS more time to actually compute sound.
Commenting out the line "#define SOUND" at the beginning
will compile pinball.c without its audio code,
so it can still run if a sound server is not available for some reason.
Four separate .aud files are provided, to illustrate how different
sounds can be made without recompiling your application
(or, if you code it right, even without restarting it!).
The code in pinball.c sends the same messages to each .aud file.
- 2fm.aud
-
This .aud file defines three message groups for the sound-sample playback
from ball collisions. In each case, the message group defines a
SampleActor, preloads a particular sample file into memory, sets its
default amplitude, and adds a single message to the message group (to play
that sample file, of course).
The "Speed of ball" message group is
continuous, not discrete. It uses two notes of an FmActor, slightly
detuned. The three parameters *0 *1 *2 map to amplitude, index of
modulation, and carrier/modulator ratio respectively, of both notes.
The "Height and vertical speed of ball" message group uses a
single FmActor
note. The first parameter *0 adjusts the note's carrier frequency
to make
the high-low swooping sound, and the second parameter *1
adjusts the index of modulation to make the sound more or less penetrating.
- 2fm_goodcollisions.aud
-
This is a slight variant of 2fm.aud.
The message groups for ball collisions now use the speed of the ball to adjust
the volume of the sound produced. Instead of a simple
"SetAmp hB 1;", they use the first argument passed in
AUDupdateSimple() as follows: "AddMessage mBumper SetAmp hB *0;".
This .aud file also dispenses with the
"dentist's drill" sound produced by the note nBallY.
It sounds the nicest of all the .aud files provided with this example.
- simplest.aud
-
This stripped-down version plays only a single soundfile for all collisions,
and sonifies only the height of the ball.
- addsyn.aud
-
This maps the ball's height and vertical velocity to the pitch and brightness
of an additive-synthesis tone (AddActor).
Collisions, instead of triggering soundfiles,
cause noise bursts to be played with the NoiseActor.
Accelerator collisions actually play a rising-pitch additive-synthesis note.
Walls play a short noise burst, decaying at the end. Randomizers play a
longer, more irregular noise burst by choosing a small modulation value.
7. How can I "debug" or try out my sounds, without
running my whole application?
Use audpanel.
This program, together with a .ap file, acts as a stub or simulator of your
whole application. Its buttons and sliders let you exercise your .aud file
directly.
The .ap file acts as the "glue" between the .aud file's message groups
and audpanel's sliders.
Your own application doesn't use a .ap file: its AUDupdate...() calls connect
directly to the .aud file's message groups.
8. How do I effectively combine several sounds at once?
Playing many sounds at once is nontrivial.
- It uses more CPU power. If you exceed what your CPU can
provide, you'll hear interruptions in the sound, a
crunchy fast stuttering. You can check if VSS's CPU usage is near 100% with a
CPU meter (gr_osview on SGI, Task Manager in Windows NT).
One cure is to run VSS with fewer channels or at a lower sampling rate.
- The total number of sounds may get too loud. You can only
get so loud before distortion called "hard clipping" occurs.
This sounds like a really bad electric guitar.
You can confirm that hard clipping is happening by noticing if the
sound returns to normal when you turn down the slider in VSS's control panel
(or the volume slider in SGI's "apanel").
Quick fix: start VSS with the option -soft
which replaces the nasty hard clipping with soft clipping.
Sledgehammer fix: start VSS with the option -limit
which prevents hard clipping entirely, no matter how hard you try.
The tradeoff is that quieter sounds may become too quiet.
(Technical note: this is a "fast-knee slow-release limiter".)
A more permanent cure is to reduce the number of simultaneous sounds,
and/or reduce how loud the individual sounds are. The numbers specifying
these things may be in the .aud file, or they may sent from
your application via AUDupdate.
Even if everything's playing okay, it may still be hard to hear
individual things. There are a few simple orchestration tricks
to keep individual sounds intelligible and recognizable.
Advanced exercise: with good headphones, listen to densely
engineered post-1985 popular music (especially dance and techno)
and you'll hear all of the following methods at work.
- If you have at least two speakers, you can use the
SetPan
command to put different sounds at different positions in the stereo field.
(In a 4-speaker setup, the sounds will spread all around like "surround sound.")
Anywhere in a .aud file where you see BeginSound, you can append
the phrase SetPan x to "pan" or move the sound from
far left to far right as x varies from -1 to 1.
(The term "pan" comes from "pan pot", an abbreviation of
the audio engineering term "panoramic potentiometer" which is itself
panoramic.)
- If you're using the 8-speaker system in the CAVE, the
SetXYZ
command will let you use distance and elevation as well to distinguish one
sound from another. (Or just use both SetPan and SetDistance.)
SetXYZ will do the right thing with fewer speakers, too,
so you can still hear what's going on with two speakers.
- Use different frequency ranges for different sounds. You can
do this by ear like Beethoven and Rimsky-Korsakov, or you can
see what frequencies your sounds have by starting VSS with the option
-graphspectrum. VSS also has
other
command-line flags which do various things.
- One sound at a time.
If you have several continuous sounds playing loud all the time,
it's like a party with too many drunks. Entertaining but not that
informative. But at a good dinner conversation, people take turns.
The repartee may still be quick, but it's easy to follow.
Non-speech sound is the same: have continuous sounds arrive and depart.
Bonus: doing this is a great way to avoid hard clipping.
A related principle: if a sound isn't changing, it should probably
fade into the background.
Another principle: you can only pay attention to so many sounds at
once. Out of all the sounds that might play at a given moment,
play only the 5 or 10 most important. This is what videogames do.
(You can learn a lot from the better videogames.)
- Use different tempos.
If you have several layers of continuous sound,
impose various tempos (amplitude envelopes or actual rhythms) on
different sounds with the
Sequence Actor
to make it easier to tune in to a particular sound.
[ up to main page ]
Back:
Parambient sound.