mod_coarsen_refine Module

MPI recv send variables for AMR. One request per neighbouring rank now, not one per chunk, so these are sized by npe.

MPI recv send variables for AMR. One request per neighbouring rank now, not one per chunk, so these are sized by npe.

MPI recv send variables for AMR. One request per neighbouring rank now, not one per chunk, so these are sized by npe.

MPI recv send variables for staggered-variable AMR

maximum number of coarse blocks that can be sent after coarsening

Per outgoing / incoming chunk, appended by coarsen_grid_siblings during the walk and turned into per-peer runs by exchange_coarsened_blocks. cf_snd_igrid / cf_snd_ibuf and the rcv counterparts are read by the pack and unpack kernels and are therefore device-resident; cf_snd_dest, cf_snd_key, cf_rcv_src and cf_rcv_key are the host-side working set the layout is derived from.

The peers themselves, ascending, with the extent of each one's run. Host only: these drive the MPI calls and nothing else.

The peers themselves, ascending, with the extent of each one's run. Host only: these drive the MPI calls and nothing else.

Per outgoing / incoming chunk, appended by coarsen_grid_siblings during the walk and turned into per-peer runs by exchange_coarsened_blocks. cf_snd_igrid / cf_snd_ibuf and the rcv counterparts are read by the pack and unpack kernels and are therefore device-resident; cf_snd_dest, cf_snd_key, cf_rcv_src and cf_rcv_key are the host-side working set the layout is derived from.

MPI buffers to send non-local coarsened grids.

The exchange is aggregated per destination rank: every coarsened child bound for one peer occupies a contiguous run of snd_buff_cf and travels in a single Isend, so the message count is the number of neighbouring ranks rather than the number of (parent, child) pairs. That is also what lets the tag be a constant; it used to be itag=ipeFi+igridFi, which is unique only because the matching Irecv names ipeFi as its source, and was itself a workaround for MPI_TAG_UB.

Both ranks order a peer's run by the key

ikey = 8*(igridCo-1) + (ic1-1) + 2*(ic2-1) + 4*(ic3-1)

built from the receiving parent's block index: the sender has it as the igrid argument of coarsen_grid_siblings, the receiver as its own igrid. Both know it because amr_coarsen_refine walks the globally replicated coarsen() table in the same order on every rank and calls getnode() for every event, so the free-node bookkeeping is replicated too. Sorting by the key makes the two layouts agree element for element with no handshake.

Aggregation gives up MPI's own mismatch detection, so the arrival lengths are checked against the layout in exchange_coarsened_blocks.

MPI buffers to send non-local coarsened grids.

The exchange is aggregated per destination rank: every coarsened child bound for one peer occupies a contiguous run of snd_buff_cf and travels in a single Isend, so the message count is the number of neighbouring ranks rather than the number of (parent, child) pairs. That is also what lets the tag be a constant; it used to be itag=ipeFi+igridFi, which is unique only because the matching Irecv names ipeFi as its source, and was itself a workaround for MPI_TAG_UB.

Both ranks order a peer's run by the key

ikey = 8*(igridCo-1) + (ic1-1) + 2*(ic2-1) + 4*(ic3-1)

built from the receiving parent's block index: the sender has it as the igrid argument of coarsen_grid_siblings, the receiver as its own igrid. Both know it because amr_coarsen_refine walks the globally replicated coarsen() table in the same order on every rank and calls getnode() for every event, so the free-node bookkeeping is replicated too. Sorting by the key makes the two layouts agree element for element with no handshake.

Aggregation gives up MPI's own mismatch detection, so the arrival lengths are checked against the layout in exchange_coarsened_blocks.