mod_load_balance Module

MPI recv send variables for AMR. One request per neighbouring rank, so these are sized by npe.

MPI recv send variables for AMR. One request per neighbouring rank, so these are sized by npe.

MPI recv send variables for AMR. One request per neighbouring rank, so these are sized by npe.

high-water mark of the two buffers, in chunks

Per migrating block, appended during the walk and turned into per-peer runs by exchange_migrated_blocks. lb_snd_igrid / lb_snd_ibuf and the rcv counterparts are read by the pack and unpack kernels and are therefore device-resident; the rest is the host-side working set.

The peers themselves, ascending, with the extent of each one's run. Host only: these drive the MPI calls and nothing else.

The peers themselves, ascending, with the extent of each one's run. Host only: these drive the MPI calls and nothing else.

Per migrating block, appended during the walk and turned into per-peer runs by exchange_migrated_blocks. lb_snd_igrid / lb_snd_ibuf and the rcv counterparts are read by the pack and unpack kernels and are therefore device-resident; the rest is the host-side working set.

MPI buffers to send blocks.

The migration is aggregated per destination rank: every block bound for one peer occupies a contiguous run of snd_buff_lb and travels in a single Isend. Both ranks order a run by the key

ikey = recv_igrid

the block index the block will occupy on its new owner. That is unique among the blocks arriving at a given receiver, because each comes from its own getnode(), and the sender knows it because load_balance walks the globally replicated Morton list in the same order on every rank and every rank calls getnode() for every migration - so igrid_inuse, and with it the destination index, stays replicated.

Sizing matters more here than the message count. These buffers used to be block_nx^3 * nw * max_blocks doubles each, allocated on the first call whether or not a single block ever moved: 1.05 GB apiece at block_nx=16, nw=8, max_blocks=4000, and device-resident. They are now sized by what actually migrates, and only ever grow, so the steady state neither wastes the memory nor reallocates every regrid.

Deliberately NOT declare create: these are resized, and an OpenACC declare-create allocatable does not reliably get a fresh device copy when the host array is reallocated - an update device afterwards then writes through a stale mapping. They are managed explicitly with enter data create / exit data delete instead, the same way mod_fix_conserve handles its own resizable exchange buffers.

MPI recv send variables for staggered-variable AMR

MPI recv send variables for staggered-variable AMR

MPI buffers to send blocks.

The migration is aggregated per destination rank: every block bound for one peer occupies a contiguous run of snd_buff_lb and travels in a single Isend. Both ranks order a run by the key

ikey = recv_igrid

the block index the block will occupy on its new owner. That is unique among the blocks arriving at a given receiver, because each comes from its own getnode(), and the sender knows it because load_balance walks the globally replicated Morton list in the same order on every rank and every rank calls getnode() for every migration - so igrid_inuse, and with it the destination index, stays replicated.

Sizing matters more here than the message count. These buffers used to be block_nx^3 * nw * max_blocks doubles each, allocated on the first call whether or not a single block ever moved: 1.05 GB apiece at block_nx=16, nw=8, max_blocks=4000, and device-resident. They are now sized by what actually migrates, and only ever grow, so the steady state neither wastes the memory nor reallocates every regrid.

Deliberately NOT declare create: these are resized, and an OpenACC declare-create allocatable does not reliably get a fresh device copy when the host array is reallocated - an update device afterwards then writes through a stale mapping. They are managed explicitly with enter data create / exit data delete instead, the same way mod_fix_conserve handles its own resizable exchange buffers.