DPM Parallel Computing and Performance -- MPI Distribution, Load Balancing, and Speed Optimization

#+CATEGORY: cfd

#+TAGS: DPM parallel MPI load-balancing performance optimization GPU speedup scaling

#+MATURITY: growing

DPM Parallel Computing and Performance Optimization

As parcel counts grow beyond ~100,000, or when transient tracking requires thousands of time steps, computational cost becomes the primary constraint. This page covers parallel DPM distribution (MPI), GPU acceleration (Fluent 2024 R1), load balancing, and practical speed optimization.

How DPM Parallelizes

Domain decomposition (mesh-based)

Fluent partitions the mesh across MPI ranks. Each rank tracks the parcels that are currently in its subdomain. When a parcel crosses a partition boundary, Fluent transfers particle data between ranks.

Parcel distribution (load-balanced)

By default, parcels are assigned to the rank that owns the cell where the parcel was injected. As parcels move, they are transferred between ranks. This can cause load imbalance if one rank has significantly more parcels than others.

MPI Parallel Setup

Launching parallel Fluent

  # Linux/macOS
  fluent 3d -t8 -mpi=intel -ssh
  
  # Windows (from Fluent Launcher)
  # Set "Number of Compute Processes" = 8
  # Select MPI: Intel MPI (recommended) or Microsoft MPI

The -t8 flag launches 8 compute processes. Fluent also launches a host process (-tn-1 compute + 1 host).

DPM parallel settings

  parallel/set/dpm-distribution method
    > cell-based     (parcels follow cell ownership -- default)
    > load-balanced  (rebalance parcels across ranks)
  
  parallel/set/dpm-load-balance interval
    > 10             (rebalance every 10 DPM iterations)

Load balancing

Without load balancing, if most parcels are injected near one wall (owned by Rank 0), Rank 0 handles 90% of the DPM work while other ranks are idle.

With load balancing, Fluent periodically redistributes parcels so each rank has a similar count. The cost: inter-rank communication to transfer parcel data.

Practical Speedup Guidelines

Expected parallel scaling

DPM scales better than the flow solver because parcel tracking is embarrassingly parallel. Beyond 64 cores, communication overhead dominates and scaling degrades.

Amdahl's law estimate

If DPM takes 80% of the total run time and flow takes 20%:

• DPM speedup with 16 cores = 14x
• Flow speedup with 16 cores = 10x
• Total speedup = =1 / (0.2/10 + 0.8/14) = 1 / (0.02 + 0.057) = 13x=

GPU Acceleration (Fluent 2024 R1)

Fluent 2024 R1 supports GPU-accelerated DPM for NVIDIA GPUs (compute capability \geq 7.0: V100, A100, RTX 30/40 series).

Enabling GPU DPM

  # Launch Fluent with GPU
  fluent 3d -t8 -gpu -gpu_node=0
  
  # In Fluent:
  parallel/gpu/set yes
  solve/set/dpm-gpu yes

GPU acceleration scope

Limitations

• UDFs compiled for CPU cannot be used with GPU DPM (CUDA compilation required)
• Brownian force UDF must be rewritten in CUDA or run on CPU
• GPU memory limits the number of parcels that can be tracked simultaneously

Recommendation for shisha

If you have access to a GPU (A100 or better), GPU DPM can provide 5--20x speedup for the particle tracking portion. However, the Brownian UDF and DPM sample file operations still run on CPU, so the end-to-end speedup is typically 3--8x.

Mesh Partitioning for DPM

Partition methods

DPM-aware partitioning

  parallel/partition/mesh/method metis
  parallel/partition/mesh/quality    (check partition quality)
  parallel/partition/smooth          (smooth partition boundaries)

For surface injection on a specific wall: ensure the wall is entirely within one partition to avoid parcel-transfer overhead at injection time.

Memory Optimization

Parcel memory footprint

Each tracked parcel consumes approximately:

100,000 parcels \approx 12.8 MB. 10 million parcels \approx 1.28 GB.

Tips to reduce memory

1. Disable trajectory recording unless needed 2. Reduce DPM history depth 3. Use fewer parcels with larger parcel mass (statistical convergence must still hold) 4. Delete parcels at walls as soon as they are trapped (don't keep data)

Profiling and Bottleneck Diagnosis

Built-in timer

  solve/set/timer yes
  report/system/timer

Fluent reports wall-clock time per iteration broken down by component:

• Flow solve
• DPM tracking
• DPM source calculation
• Data I/O

Identifying the bottleneck

Performance Checklist for Shisha Transient

For the anticipated transient shisha simulation:

• [ ] Launch with 8--16 cores (or GPU if available)
• [ ] Partition mesh so injection surface is on a single rank
• [ ] Use load-balanced DPM distribution (injection is localized)
• [ ] Set =\Delta t = 0.001= s, 1000 time steps = 1 second of simulation
• [ ] Disable trajectory recording (use sample files instead)
• [ ] Run Brownian UDF on CPU; GPU only if rewritten in CUDA
• [ ] Monitor wall-clock time per step to estimate total runtime
• [ ] Budget: 1 second of puff simulation \approx 2--8 hours on 8 cores

See Also