MPI Continued

Message Matching

Suppose process A executes the following send operation:


MPI_Send(send_buffer, send_count, send_type, destination, send_tag, send_comm);

And process B executes the following receive operation:


MPI_Recv(recv_buffer, recv_count, recv_type, source, recv_tag, recv_comm, &status);

The receive will only be matched with the send if all of the following are true:

The communicator groups are the same.
The tags are the same.
A = source.
B = destination.

For the matched send/receive pair to successfully communicate, the recv_buffer must also be at least as large as the send_buffer, and the types must match.

Receiving From Any Source

It might be the case that we want a process to receive data, and we do not care what process sends it.

For example, in the sum program, there is no reason to force the order of communications. We could allow the processes to send their partial sums in any order.

This can be done with specifying MPI_ANY_SOURCE as the source.

This will match send calls that are otherwise compatible regardless of the process which is sending the message.

If we need to know which process sent us data, we can read that from our status variable.

We can re-write our sum program to take advantage of this:


    /* do the calculation */
    int sum = 0, i;
    for (i = start; i <= end; i++) {
        sum += i;
    }

    /* MPI communication: process 0 receives everyone elses sum */
    if (rank == 0) {
        /* parent process: receive each partial sum and add it to ours -
         * now we no longer care about the order we receive them in */
        int partial, i;
        MPI_Status status;
        for (i = 1; i < size; i++) {
            MPI_Recv(&partial, 1, MPI_INT, MPI_ANY_SOURCE, 0, MPI_COMM_WORLD, &status);

            /* see which process we are reading from */
            printf("Process 0 got data from process %d.\n", status.MPI_SOURCE);

            sum += partial;
        }
    } else {
        /* worker process: send sum to process 0 */
        MPI_Send(&sum, 1, MPI_INT, 0, 0, MPI_COMM_WORLD);
    }

Receiving Any Tag

We can also specify that our receive should receive from any tag. This might be useful if we want to distinguish between multiple different messages, but we want to receive them all.

This is done similarly to reading from any source:

Specify MPI_ANY_TAG as the tag in the receive call.
Optionally check status.MPI_TAG to see which tag was read.

I/O

As we've seen performing output from multiple processes is tricky. Each process is allowed to write to stdout and stderr (on our implementation of MPI at least). However the order is non-deterministic because each process buffers printf calls independently.

In order to have predictable multi-process output, we should only output from one process.

Input is a different story. The following program attempts to read a number inside of each process:


#include <stdio.h>
#include <mpi.h>

int main(int argc, char** argv) {
    /* initialize MPI */
    MPI_Init(&argc, &argv);

    /* get the rank (process id) and size (number of processes) */
    int rank;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);

    /* try to read an integer */
    int number = 42;
    scanf("%d", &number);

    /* print a hello message */
    printf("Process %d read %d from stdin.\n", rank, number);

    /* quit MPI */
    MPI_Finalize();
    return 0;
}

What happens when we run this?

Non-Blocking Communication

The MPI_Send and MPI_Recv calls are blocking calls which means they do not return until the communication is finished. This is normally preferable since it is simpler.

There are times, however, when we would want to do other work while the communication is happening.

For example, in a producer/consumer program, we might want the producer process to work on producing more data as it is sending the data it has to a consumer.

Non-blocking communication is accomplished with the following 2 functions:


MPI_Isend(void* buffer, int count, MPI_Datatype type, int destination, int tag, MPI_Comm comm, MPI_Request* request);

MPI_Irecv(void* buffer, int count, MPI_Datatype type, int source, int tag, MPI_Comm comm, MPI_Request* request);

These functions have an extra parameter than the non-blocking versions, the request. They will return immediately, not waiting for the actual transfer to take place.

It is important to note that we should not write to the buffer after calling these, it may still be needed!

If we want to check if a message has been received, after calling a non-blocking communication function, we can check if it is finished with MPI_Iprobe:


MPI_Iprobe(int source, int tag, MPI_Comm comm, int* flag, MPI_Status *status)

This will set the integer passed in as flag to 1 if a message is ready, and 0 otherwise.

Or wait for it to finish with MPI_Wait:


MPI_Wait(MPI_Request *request, MPI_Status *status)

This will block until a message is received. Calling MPI_Irecv followed by MPI_Wait is equivalent to calling MPI_Recv - except we can do work in between.

Non-Blocking Example

As an example of non-blocking communication, consider a producer consumer relationship. One process is charged with producing data, which is consumed by another process.

With blocking communication, the producer must stop producing while the data is sent to the consumer, and the consumer must stop consuming while it waits for data:

Producer:
1. Produce some data.
2. Send it to the consumer.
3. Repeat.
Consumer:
1. Wait for some data.
2. Consume it.
3. Repeat.

A better approach is to use non-blocking communication so that we can overlap the communication with the actual work. This requires having additional memory as it's not safe to change a memory location after using a non-blocking send, or access data after a non-blocking receive.

Producer:
1. Set buffer to buffer1.
2. Produce some data into buffer.
3. Wait for the previous send (if any) to finish.
4. Start sending the buffer.
5. Swap buffers.
6. Go to step 2.
Consumer:
1. Set buffer to buffer1.
2. Start receiving data into buffer.
3. Wait for the previous recv to finish.
4. Start receiving into the other buffer.
5. Consume the buffer we waited for in step 3.
6. Swap buffers.
7. Go to step 3.

This idea is illustrated in the this program which uses the producer/consumer relationship with three processes: one to read data from the user, one to apply a transformation (capitalization) to it, and one to display it.

Warning, this program is rather long!