hpc · advanced

MPI: distributed-memory parallelism

MPI (Message Passing Interface) is the standard for distributed-memory parallelism across a cluster: instead of threads sharing memory, you run many independent processes (ranks), each with private memory on potentially different nodes, cooperating only by passing messages. It's SPMD — the same program runs as N ranks that branch on their rank id. Point-to-point MPI_Send/MPI_Recv move typed buffers but can deadlock if two ranks block sending to each other before receiving; the fix is MPI_Sendrecv or non-blocking MPI_Isend/Irecv. Real code relies on collectives — Bcast, Scatter/Gather, Reduce, Allreduce — which are implemented as O(log P) trees and far outperform hand-rolled message loops. Scaling is bounded by Amdahl's law (the serial fraction caps speedup) and by communication overhead, so you overlap communication with computation and distinguish strong scaling (fixed problem) from weak scaling (problem grows with the rank count). Production HPC commonly combines MPI across nodes with OpenMP within each node.

The code

#include <mpi.h>
int main(int argc, char** argv) {
    MPI_Init(&argc, &argv);
    int rank, size;
    MPI_Comm_rank(MPI_COMM_WORLD, &rank); // which process am I? (0..size-1)
    MPI_Comm_size(MPI_COMM_WORLD, &size); // how many processes total?

    double local = work(rank); // each rank computes its share
    double total = 0;
    MPI_Allreduce(&local, &total, 1, MPI_DOUBLE, MPI_SUM, MPI_COMM_WORLD);
    MPI_Finalize(); // run: mpirun -np 8 ./a.out
}

What this lesson walks through

1. 01Many processes, no shared memory
2. 02Point-to-point: Send / Recv
3. 03Collectives: the operations that actually scale
4. 04Scaling limits: Amdahl, strong vs weak

OpenMP scales to one machine's cores; MPI scales across a whole CLUSTER. The model is the opposite of threads: many independent PROCESSES, each with its own private memory, often on different nodes. They cooperate only by sending MESSAGES. The same program runs as N processes, each with a unique 'rank' (0..N-1).