application called MPI_Abort(compute, 59)

Question

Hello everyone,

I have been learning to use openCARP for simulations recently. When I try to use multi-core parallel acceleration (--np), I encounter some runtime errors, with the message: `application called MPI_Abort(compute, 59) - process 21`. Apart from this, I could not find any other useful information in the command line output.

The mesh I am using has an average edge length of 500um, containing 500,000 nodes and 20,000,000 tetrahedral elements. I used meshtool for resampling, smoothing, and reindexing operations. To reproduce the error and identify the cause, I tested different values of --np and found that the error occurs when np > 28. I also tested on another computer and encountered the same issue: when np exceeds a certain value, the `application called MPI_Abort` error appears.

Additionally, I tested with a smaller mesh (40,000 vertices, 90,000 elements) and found that the `application called MPI_Abort` error occurs when np > 83.

Does anyone know the reason of this error? Could it be related to mesh quality or size? Or is there a specific constraint on '--np' for the different mesh sizes?

I would appreciate it if you could give me some advice on how to solve the problem！

Answer 1

Hi!

You did not mention what kind of hardware you are using. As such, it is hard to say whether the issue is that you oversubscribe your CPU and/or run out of memory.

Generally speaking, the number of processes you request (--np option) should have something to do with the hardware you are running on. For example, using one process per physical CPU core makes sense. Thus, --np 83 is only a valid choice on a server with e.g. 96 or 128 physical cores.

It is hard to say why you are getting an MPI_Abort, it may be an issue in the MPI or in openCARP. One would have to debug it on the specific machine to find out.

Best, Aurel