mpigraph -- mpi communication diagnosis project

the more-or-less current sourcecode can be downloaded here, or can be browsed here
(note: this is a work in progress)

mpigraph is a project that aims to be used to graphically visualize aspects of an MPI system's topography. Primarily, inter-node and intra-node latency are the focus of this program, though it could easily be expanded to include other factors, such as bandwidth, with only a few modifications.

The Main Idea

The main idea behind the program is to create a NxN matrix (for a system with N nodes), where each location i,j represents the latency/bandwidth between nodes i and j. Next, these points are laid out on a x,y coordinate system with more or less random placement, and are put through a process of simulated annealing, where the lowest possible energy state is when their relative distances to one another are the same as the values stored in the NxN matrix.

How It Works (ideally ;p )

First, an MPI program is run in which the latencies and/or bandwidths are measured and recorded to a file. For the sake of simplicity, and due to my relative newness to MPI, I wrote a separate program to take over the task of analyzing/shaping the data after this point which did not use MPI. The process of simulated annealing follows more or less the following steps:

1. Points (nodes or processors) are placed randomly in an x,y grid
2. The "total energy state" is calculated
3. A small random change is made
1. if the change results in a lower energy state, take it
2. else, if a certain random probability is met, take it
3. else, don't take the change
4. Repeat Process Many, Many, Many Times

If the locations of the points are recorded before and after being sent through the previously described process (plotted with gnuplot), and the calculations are based on the latencies of both inter and intra node tasks in an MPI program, the following can be witnessed:

Various Simulated Annealing Test Runs
(click on an image to enlarge)

Process Topography	Before	After
4 nodes, 2 tasks per node
3 nodes, 6 tasks per node
4 nodes, 5 tasks per node

To Be Improved, Enhancements, et c...

There's still a lot that I'd add on if I had the time. A few ideas:

• Record more informative data -- a more robust sample would ideally include nodename, pid, time, latencies, bandwidths....
• Handle high task per node situations better. Perhaps the best way to do that is to run the annealing process on tasks within each node first, and come up with relative distances for tasks within each node. Then, instead of placing every task onto the x,y grid, place one task from each node, run the annealing process, and then place the tasks relative to the already placed task. And maybe then run the process again with everything placed...
• A fancier way of plotting the data. Sure GNUplot's nice and all, but it'd be nice to have the ability to add in lines, use images to represent nodes, et c.
• Integrating the annealing process into the same MPI program that checks the latencies/bandwidths