mod_msg_layout Module

Laying an exchange out as one message per destination rank.

An exchange that moves many small chunks between the same pair of ranks does not need one message per chunk. If both ranks can put a peer's chunks in the same order without talking to each other, every chunk bound for that peer can occupy one contiguous run of a single buffer and travel in a single Isend. The message count is then the number of neighbouring ranks rather than the number of chunks, and the tag no longer has to encode a block index, so it stays trivially inside MPI_TAG_UB.

The ordering is supplied by a key that both ranks compute for the same chunk and that totally orders a peer's run - typically built from the receiving block's index, which the sender reads out of its neighbour tables and the receiver reads off its own igrid. layout_runs sorts by it; rank_keys does the sorting.

Two things to keep in mind when using this:

  • both ranks must enumerate from tree state that is already consistent, before any message is posted, and the chunk sizes must agree chunk for chunk;
  • aggregation gives up MPI's own mismatch detection, because a disagreement about which chunks a run holds is no longer a length mismatch on any single message. Check each arrival's MPI_GET_COUNT against the run length that was laid out. That catches a difference in the set of chunks the two ranks enumerated; a difference in order cannot arise, since both sort the same run by the same key.

Used by mod_fix_conserve (refluxing) and mod_coarsen_refine (coarsening).


Used by

  • module~~mod_msg_layout~~UsedByGraph module~mod_msg_layout mod_msg_layout proc~build_message_layout build_message_layout proc~build_message_layout->module~mod_msg_layout proc~exchange_coarsened_blocks exchange_coarsened_blocks proc~exchange_coarsened_blocks->module~mod_msg_layout proc~exchange_migrated_blocks exchange_migrated_blocks proc~exchange_migrated_blocks->module~mod_msg_layout

Subroutines

public subroutine layout_runs(n, pe_of, key_of, size_of, off_of, npeer, peer, peer_off, peer_len)

Give every peer one contiguous run of the exchange buffer, with the chunks inside a run ordered by the key. Both ranks compute the same key for the same chunk, so sorting by it makes the sender's run and the receiver's run agree element for element without any handshake.

Read more…

Arguments

Type IntentOptional Attributes Name
integer, intent(in) :: n
integer, intent(in) :: pe_of(:)
integer, intent(in) :: key_of(:)
integer, intent(in) :: size_of(:)
integer, intent(out) :: off_of(:)
integer, intent(out) :: npeer
integer, intent(out) :: peer(:)
integer, intent(out) :: peer_off(:)
integer, intent(out) :: peer_len(:)

public subroutine rank_keys(n, key, perm)

Rank key(1:n) ascending: perm(k) is the index of the k-th smallest key. Heapsort, so O(n log n) worst case and no scratch beyond perm itself. Local rather than the equivalent mrgrnk vendored with octree-mg, to keep the callers independent of the multigrid solver.

Arguments

Type IntentOptional Attributes Name
integer, intent(in) :: n
integer, intent(in) :: key(:)
integer, intent(out) :: perm(:)