Oliver Hsu & Albert Bui CS91 Final Project


Visualizing MIDI


Description

This project was designed to visualize MIDI music in Maya. It is composed of two primary components, the MIDI/Maya back end and the Maya visualization elements that interacted with the back end.

Target Audience

The visualizations of this project were aimed at a broad audience. However, the core elements were geared mainly at the developer. The MIDI/Maya back end provides a comprehensive and easy to use interface to working with MIDI within MAYA.

MIDI/Maya Back End

The MIDI back end consists of two components: the MIDI parser, and the script interface.

The MIDI parser

A MIDI file is essentially a very long list of binary instructions to either a MIDI program or a MIDI device. The instructions define information such as time signature, tempo, and key signature, as well as 'note on' and 'note off' instructions for each instrument. The MIDI parser scans through the MIDI file, ignoring everything except for tempo and 'note on/off' instructions. Using the tempo information, the parser calculates at which frame the notes are being played and released, and outputs a 'tune script' (which is essentially a Mel script containing calls to the functions play() and release()). If the MIDI file contains more than one track (each track usually corresponds to one instruments), the parser will create a separate tune script for each instrument.

Script Interface

For each visualization, a script file needs to be created which interacts with the MIDI parser. This script file initializes the screen either by creating the elements directly or importing them from another maya database file. There are two main function in this script, a play and a release function. A play function is called every time a note is played in the MIDI file, and a release function is called every time a note is release in the MIDI file. For example in asimulation of a piano, the play function would visualize the pushing down of the keys, while the release function would visualize the release of the key. The creativity comes into play in the play and release functions. The developer can instruct Maya to do anything when a given note is played/releases at a certain time.

Visualizations

Keyboard Simulation (keyboard.mel)

This is a visualization of piano music using an animation of a piano. As each note is played, the respective key on the keyboard is pressed.

Musical Doughnut (torus.mel)

In this visualization, each musical note is represented as a segment of a torus. As each note is played, that section of the torus lights up and then slowly fades away as the music is released.

Flying Polygons (block.mel)

This is an adaptation of a visualization we saw in the SIDGRAPH video. It consists of a series of rectangles placed back to back at heights corresponding to the pitch of the notes. As each note is played the matching rectangle lights up.

Magical Bubbles (emitter.mel)

For this visualization, we have different bubbles appearing during the course of MIDI playback. Each bubble emitter corresponds to a certain note and appears when the note is played.

Cheesy Drummer (drummer.mel)

Jacquie makes a guess appearance as an ethnic drummer. As each note in the drum sequence is played, Jacquie hits the drum.

How to use the package

1. compiling the MIDI parser

To compile the parser, simply type make tune in the /Source/midiParser directory.

2. creating a 'tune script'

Copy a sample midi file (a collection can be found in the /Source/midi/ directory) into the midiparser directory (/Source/midiParser). Type makeMelTune [midi file] [output name]. The parser will create one or more files called [output name].trackXX.mel, depending on how many tracks are in the midi file.
NOTE: for some yet undiscovered reason, the parser currently fails on some midi files, either seemingly entering an infinite loop or aborting with a segmentation fault.

3. using a visualization

In the Maya Script Editor window, type source "[visualization file]", which will load the code for the visualization. Next, type init ([lowest note], [highest note], [name]). [lowest note] and [highest note] are the midi values for the lowest and highest note that will be played by the current visualization. The C++ program hilo.cpp (located in /Source/misc) will look through a tune script, outputting the lowest and highest notes in the script. [name] is a name for this particular instance of the visualization.
Next, type source "[tune script]". This step applies the instructions in the tune script to the current visualization and might take a while to complete.
Finally, type finalize ([time]) to finalize the visualization. This step performs some final steps such as grouping certain objects together etc. [Time] here is given in frames.

4. creating new visualizations

This is relatively easy. All visualization need four procedures defined:

1. init(int $low, int $hi, string $name)
This is the initialization procedure. Setting up materials, initial expressions, etc. is all done in this function.

2. play(int $note, float $time)
This procedure contains instructions that should be carried out when a note is played. $note is the note in MIDI value, $time is in terms of frames in Maya.

3. release(int $note, float $time)
This procedure contains instructions that should be carried out when a note is released. $note is the note in MIDI value, $time is in terms of frames in Maya.



4. finalize(int $frame)
This procedure contains finalizing steps such as grouping objects together, creating objects that were not created due to missing release commands etc. Variable $frame is the frame at which the animation is to end, which can be useful for filling in missing objects at the end of the animation.

Using the existing visualizations might be helpful in getting a feel for the format as well as gaining some idea about what sorts of techniques can be used.

Some animations created with this package

1. Brahms Hungarian dance (2 pianos, 4 hands)
2. Bach Brandenburg concerto no. 1
3. Brahms Requiem 1st movement
4. Drummer